NMS Monitor

Overview

The NMS Monitor runs on a WebLogic managed server at each site and periodically request the current status of the NMS Agents, WebLogic managed servers and databases. The monitor is configured to communicate with an Oracle Enterprise Manager instance to trigger a failover request via Site Guard in the event of loss to a critical component. A failover or switchover can be requested manually by an NMS Monitor user. Requests for failover or switchover are sent to the Site Guard instance via the Enterprise Manager Command Line Interface (emcli).

Each NMS Monitor in the system communicates with an Apache ZooKeeper cluster in order to:

• Manage and read the site configuration.

• Store the current site status.

• Coordinate with other NMS Monitor instances.

In this way, the NMS Monitor is able to determine the state of all sites in the system.

The NMS Monitor hosts a web page showing the current status at each site and provides configuration to allow the addition and maintenance of sites to be monitored.

Monitoring

The NMS Monitor monitors the following components using REST API calls and stores the status of each in the ZooKeeper cluster:

• From each NMS Agent the monitor retrieves the current and previous state of the services, timestamps and the NMS version.

• From the CESEJB service the monitor retrieves the status of the services and publisher.

• From the Mobile NMS service the monitor retrieves the status of the services.

• From the databases the monitor retrieves the active and staged environments, active and staged versions and the site name.

• From the WebLogic instance the monitor retrieves the status of each managed server and the applications that are deployed under it.

The NMS Monitor uses this information when determining:

• Whether the active NMS instance is fully running

• Which standby instances are viable candidates for failover.

Configuration

The NMS Monitor is configured via the nmmonitor.properties configuration file located in the $DOMAIN_HOME directory of the WebLogic instance. A configuration tool, nms‑monitor‑config is provided to help in the configuration of the monitor. See NMS Monitor for full details of the configuration of the monitor and the use of the nms-monitor-config tool.

Setting the Domain Environment

The WebLogic domain environment must be sourced prior to running the nms-monitor-config utility, for example:

cd $DOMAIN_HOME/bin

. ./setDomainEnv.sh

Configuration Tool Main Page

The main page displays the following options:

1: Display properties

2: Configure zookeeper

3: Configure datasource

4: Configure emcli

5: Keystore actions

6: Truststore actions

7: Credentials

0: Exit

Display properties: Selecting this option displays the current configuration properties for the NMS Monitor.

Configure ZooKeeper: This option configures the properties necessary to connect to the ZooKeeper cluster.

Configure datasource: This option is used to identify the database datasources configured in the WebLogic server that are used to connect to the NMS database.

Configure emcli: This option is used to configure the properties for the Enterprise Manager command line interface.

Keystore Actions: This options allows you to create an identity keystore for the monitor and export the monitor’s certificate.

Truststore Actions: This option allows you to create a trust keystore for the monitor and import certificates.

Credentials: This option allows the safe storage of the credentials required to communicate with the NMS cesejb, nms-ws, and emcli services.

In addition a list of outstanding issues that have still to be performed are listed. This is a subset of the options listed above. As options are completed they are removed from the outstanding issues.

As the NMS Monitor is configured, a property file (msmonitor.properties) is automatically created in the WebLogic $DOMAIN_HOME.

Configure ZooKeeper

Select the Configure ZooKeeper option to configure the ZooKeeper properties. The tool displays the current settings:

zookeeper.connectString: This is the connection string used to connect each node in the ZooKeeper cluster.

zookeeper.namespace: This is the ZooKeeper namespace. Default: nmsmonitor.

zookeeper.protocol: This is the protocol to use to communicate to the cluster. Valid values are http or https.

The following options are available:

1 : New connect string

2 : Add new node to connect string

3 : Set namespace

4 : Set protocol

New Connect String

Select the New connect string option to set the connection string for the cluster. For example to connect to the example ZooKeeper cluster with secure client ports configured on 2181:

server.1=<server1_hostname>:2888:3888;2181

server.2=<server2_hostname>:2888:3888;2181

server.3=<server3_hostname>:2888:3888;2181

The connection string would be:

<server1_hostname>:2181, <server2_hostname>:2181,

<server3_hostname>:2181

Add New Node to Connect String

Select this option to add an additional ZooKeeper node to the existing connection string.

Set Namespace

Select this option to configure the ZooKeeper namespace if used.

Set Protocol

Select this option to configure whether the connection to the ZooKeeper node is secure (https) or not (http). Secure is highly recommended.

Configure Datasource

Select the Configure datasource option to configure which database datasources to use to communicate with the NMS database. The list of existing datasources can be found in the WebLogic administration console -> Services -> Datasources. The datasources used should connect to the read/write schema of the corresponding NMS environment. The tool displays the current settings:

config.datasource_1: This is the datasource to use for the NMS instance in environment 1.

config.datasource_2: This is the datasource to use for the NMS instance in environment 2.

The following options are available:

1. Set datasource 1

2. Set datasource 2

Set datasource 1

Select this option to configure the datasource to use for the environment 1 NMS instance’s database schema.

Set datasource 2

Select this option to configure the datasource to use for the environment 2 NMS instance's database schema.

Configure emcli

Select the Configure emcli option to configure the location of the Enterprise Manager Command Line Interface. The tool displays the current settings:

emcli.home: The home directory where the emcli is installed.

emcli.state_dir: The emcli state directory if applicable. Sets the EMCLI_STATE_DIR environment variable when running emcli. This is typically not needed.

emcli.options: The emcli options if applicable. Sets the EMCLI_OPTS environment variable when running emcli. This is typically not needed. If emcli needs to connect to Enterprise Manager through a proxy server, set this option to:

-Dhttps.proxyHost=<proxy host> -Dhttps.proxyPort=<proxy port>

emcli.header.primary_system: The Primary System header in the output for the "emcli get_operation_plans" command. Default: Primary System. This is typically not needed; this option is available in case the emcli output from a future version of Enterprise Manager is different.

emcli.header.standby_system: The Standby System header in the output for the "emcli get_operation_plans" command. Default: Standby System. This is typically not needed; this option is available in case the emcli output from a future version of Enterprise Manager is different.

emcli.header.plan_name: The Plan Name header in the output for the "emcli get_operation_plans" command. Default: Plan Name. This is typically not needed; this option is available in case the emcli output from a future version of Enterprise Manager is different.

emcli.header.target_name: The Target Name header in the output for the "emcli get_targets" command. Default: Target Name. This is typically not needed; this option is available in case the emcli output from a future version of Enterprise Manager is different.

emcli.em_per_site: Specifies whether each site has its own Enterprise Manager instance. Default: false. If set to true, each NMS monitor instance must be configured to connect to a different Enterprise Manager instance. After a switchover of failover operation initiated from NMS Monitor completes successfully, NMS Monitor will update Site Guard at each other site to set the new primary site. If not set or set to false, all NMS Monitor instances must be configured to connect to the same Enterprise Manager instance.

The following options are available:

1. Set home: Select this option to set the emcli.home property.

2. Set state dir: Select this option to set the emcli.state_dir property.

3. Set options: Select this option to set the emcli.options property.

4. Set header primary system: Select this option to set the emcli. primary_system property.

5. Set header standby system: Select this option to set the emcli. standby_system property.

6. Set header plan name: Select this option to set the emcli. plan_name property.

7. Set header target name: Select this option to set the emcli. target_name property.

8. Set EM instance per site (true/false): Select this option to set the emcli. em_per_site property.

Keystore Actions

Select the Keystore actions option to configure the identity keystore. The tool displays the current settings for the keystore:

credentials.keystore: The location of the identity store. The default location is $DOMAIN_HOME/security/nmsmon_keystore.

credentials.keystore_password: The encrypted password used to access the identity store.

The following options are available:

• Set keystore location

• Create keystore

• Export certificate

Set keystore location Option

Select the Set keystore location option to configure the keystore location. You will be prompted for the location of the keystore. Enter the location or press Enter to select the default option.

If the file does not exist, the keystore actions page shall indicate that the file does not exist. The keystore is generated using option 2 Create keystore.

Create keystore Option

Select the Create keystore option to create the identity keystore and save its password in encrypted form. The tool will prompt for the following information:

Enter Password: Enter the desired password for accessing the new identity store.

Re-enter Password: Enter the password again to confirm it.

The tool will then continue to prompt for the Distinguished Name information for the NMS Agent certificate. An example is given below. Pressing Enter at each prompt will select the default value of Unknown.

• What is your first and last name? (Unknown) = nms@opal.com

• What is the name of your organizational unit? (Unknown) = GBU

• What is the name of your organization? (Unknown) = Oracle

• What is the name of your City or Locality? (Unknown) = Minneapolis

• What is the name of your State or Province? (Unknown) = Minnesota

• What is the two-letter country code for this unit? (Unknown) = US

You will be prompted to confirm the details.

• Is CN=nms@opal.com,OU=GBU,O=Oracle,L=Minneapolis,ST=Minnesota,C=US correct? (no) =

Enter yes to confirm the details or press Enter to select the default option no. Selecting no will cancel the action and the keystore shall not be created.

You are then prompted for the valid duration of the server certificate that will be created.

Enter validity period in days (365): Enter a period in days or press Enter to select 365 days.

The identity keystore is then created in the correct location and the encrypted password is set in the NMS Monitor property file.

Export certificate Option

The NMS Monitor certificate must be added to the truststore of each NMS Agents that communicates with the agent and each of the ZooKeeper nodes. In order to do this the certificate needs to be exported to file. Select the export certificate option to export the agent’s certificate.

You will be prompted for the keystore password.

After the password is correctly entered, the certificate is automatically exported to the file:

nmsmonitor.cert

Truststore Actions

Select the Truststore actions option to configure the truststore for the NMS Monitor. The tool displays the current settings for the truststore:

credentials.truststore

This is the location of the identity store. The default location is $DOMAIN_HOME/security/nmsmon_truststore.

credentials.truststore_password

This is the encrypted password used to access the truststore.

The following options are available:

1. Set truststore location

2. Create truststore

3. Import certificate

Set truststore location Option

Select the Set truststore location option to configure the truststore location. You will be prompted for the location of the truststore. Enter the location or press Enter to select the default option.

If the file does not exist, the truststore actions page shall indicate that the file does not exist. The truststore is generated using option 2, Create truststore.

Create truststore Option

Select the Create truststore option to create the truststore and save its password in encrypted form. The tool will prompt for the following information:

Enter Password: Enter the desired password for accessing the new identity store.

Re-enter Password: Enter the password again to confirm it.

The identity truststore is the created in the correct location and the encrypted password is set in the NMS Monitor property file.

Import Certificate Option

Select the Import certificate option to import a certificate into the monitor’s truststore. Each NMS Agents and ZooKeeper nodes that communicates with the agent will require its certificate to be imported into the monitor’s truststore.

You will be prompted for the truststore password.

You will then be prompted for the alias they wish to store the certificate under.

Enter alias =

Enter the name of the alias for the certificate being imported.

You will then be asked for the location of the certificate on the file system.

Enter certificate location (myalias.cert) =

Enter the location of the certificate or press Enter to use the default location of <alias>.cert

The certificate will then be loaded into the truststore.

Credentials

In order for the NMS Monitor to communicate to the cesejb & nms-ws instances, WebLogic admin and emcli services credentials are required to be configured. This option allows the credentials to be viewed and managed. The tool displays the current settings:

• A list of currently stored credentials and their encrypted value.

The name of each credential takes the form:

Where <hostname> is the server name where the service is installed and <port> is that service’s listen port.

The following credentials must be configured in the NMS Monitor:

emcli

Credentials for logging into the Enterprise Manager command line interface.

Hostname=emcli

Port=0

Username=<admin account username>

Password=<admin account password>

cesejb, nms-ws

Credentials for using the WebLogic cesejb and nms-ws deployments. This needs to be done for each of the WebLogic servers deploying these services.

Hostname=<hostname of the WebLogic server>

Port=<the secure port number for the WebLogic server>

Username=<valid NMS username>

Password=<valid NMS password>

Application=<valid mobile application id>

WebLogic Admin

Credentials for using the WebLogic Admin must be provided.

Hostname=<hostname of the WebLogic server>

Port=7001

Username=<the WebLogic admin username>

Password=<the WebLogic admin password>

The following options are available:

1: Create new credential

2: Delete credential

1. Create New Credential Option

Select this option to create a new credential. You will be prompted for the following information:

• Hostname: Enter the hostname of the service.

• Port: Enter the listen port of the service.

• Username: Enter the username to login with

• Enter password: Enter the password to login with.

• Re-enter password: Revalidate the password.

2. Delete Credential Option

Select this option to delete one of the listed credentials. You will be prompted for the credentials hostname and port.

Creating the NMS Monitor Server

The NMS Monitor needs to be deployed to a new managed server in the WebLogic instance. For full details on how to create a managed service, refer to the WebLogic Server Runtime Configuration section of the System Installation chapter in the Oracle Utilities Network Management System Installation Guide.

Create the Server

1. Open the WebLogic admin console using the following URL

http://<hostname>:<port>/console

Hostname represents the DNS name or IP address of the Administration Server, and port represents the number of the port on which the Administration Server is listening for requests (port 7001 by default).

2. Navigate to Environment, and then Servers

3. Select New to create a new server

4. Enter the server name and give it a unique listening port.

5. Create the server as Stand Alone.

6. Select the server in the server list to edit its details. This will invoke the server details page for the new server.

7. Enter the hostname of the server as the listen address.

8. Enable the SSL listen port and set the port to a unique port number.

9. Save the changes.

10. Edit the advanced settings and set and set the RMI JDBC Security to Secure.

11. Edit the keystore options and make sure the default Java truststore is configured.

12. Save the changes.

Configure SSL

In order to enable SSL for the NMS Monitor web application the keystores need to be configured.

1. Navigate to Environment, and then Servers.

2. Select the NMS Monitor server.

3. Select the Keystores tab.

4. Click the Change/New button

5. Select the Custom Identity and Java Standard Trust menu option

6. Save the selection.

In the Identity section:

1. In the Custom Identity Keystore field, enter the location of the NMS Monitor keystore. For example:

/scratch/gbuora/Oracle/Middleware/Oracle_Home/user_projects/domains/nms/security/nmsmon_keystore.p12

2. In the Customer Identity Keystore Type field leave as the default option JKS.

3. Enter and confirm the NMS Monitor keystore password.

4. Save the changes.

Select the SSL tab.

1. In the Private Key Alias field enter the alias of the NMS Monitor private key nmsmonitor.

2. Enter and confirm the NMS Monitor keystore password.

3. Save the changes.

Starting the Server

1. In the WebLogic admin console, select Environment, and then Servers.

2. Select the Control tab.

3. Select the NMS Monitor server, and then select Start.

Update the JDBC Datasources

The JDBC datasources for the NMS database need to be accessible to the new servers.

1. In the WebLogic admin console, select Services, and then Data Sources.

2. Select each of the NMS data sources in turn and perform the following actions:

• Select the Targets tab.

• In the targets list, select the newly created NMS Monitor server in addition to the targets already selected.

3. Save the datasources.

Deploying the NMS Monitor

The new NMS Monitor service needs to be deployed to the new NMS Monitor server in the WebLogic instance.

1. Copy $NMS_BASE/dist/install/nms-monitor.ear from the NMS installation to the WebLogic server.

2. In the WebLogic admin console using the following URL

http://<hostname>:<port>/console

Here hostname represents the DNS name or IP address of the Administration Server, and port represents the number of the port on which the Administration Server is listening for requests (port 7001 by default).

3. Navigate to Deployments, and then Configuration.

4. Click the Install button.

5. Navigate to the path where NMS Monitor is installed and select the nms‑monitor.ear.

6. Select Next.

7. Select Install this deployment as an application.

8. Select Next.

9. In the list of available targets, select the new NMS Monitor server.

10. Select Next, Next, and then Finish to complete the deployment.

NMS CESEJB Self-Signed Certificates

If the NMS CESEJB has been self-signed, the certificate must be imported into the default Java truststore in order for the NMS Monitor to trust it.

The certificate can be obtained from the NMS keystore or exported from a browser. Navigate to

Where <hostname> is the hostname of the WebLogic server and <port> is the secure port used for connecting to the NMS Applications.

View and export the certificate chain.

The certificate chain can then be imported into the default java truststore using the following command:

keytool -import -alias <alias> -file <certificate> -keystore $JAVA_HOME/lib/security/cacerts