Using Application Clustering

Unified Assurance provides clustering capabilities within most metric applications to support redundancy and enable distributed data processing. You can set up clustering with multiple instances of an application on a single server, or distribute clusters of applications across multiple servers. You can cluster all applications that poll and process data, such as the Generic SNMP Poller and Standard Thresholding Engine. You cannot cluster applications that receive data, such as event list aggregators. This section shows how to enable clustering for broker services. See Services and Configuring a Broker Service for more information.

Unified Assurance completes the following tasks before starting a cluster service:

The system applies an internal identifier to the applications.
The system divides devices to be polled evenly among the available applications and assigns them accordingly.
Before starting a poll cycle, the application will check for a broker message and perform any reconfiguration, if necessary (for example, if a broker is down).
Periodically the broker will trigger a configuration check and if there are changes, the cluster will reconfigure:
- The system removes devices that are no longer being polled from the applications to which they were assigned.
- The system distributes and assigns new devices to be polled among the applications, considering each application's current device count to achieve load balancing.
Note:
- Unchanged devices will continue to be polled by the same instance.
- If an individual service in a cluster is restarted, the entire cluster will completely re-balance the device pool regardless if any devices have been changed. When this occurs, individual applications may begin polling different devices.

Dependencies

The application must be available on all servers that will be running the cluster service.
The Cluster ID value must be the same in all service configurations that will be used by the applications in the cluster.

Best Practices

During initial deployment, make sure the application configuration has the LogLevel set to DEBUG. After functionality has been verified, this can be changed to reduce log data.
The health status of each application can be viewed under the Health tab in the Devices UI within the All Metrics Overview panel. If the application was configured to be run on different servers, each instance of the application has separate health status metrics saved on each server used in the cluster. In such cases, you can check the Host Name in the Failover Service States UI to find the server on which a particular instance of your application is hosted before checking its health in the Devices UI.

Creating a Cluster

To create a cluster:

Determine the number of application instances in the cluster you want to create, and the servers running each instance.
Create a service entry for each application added to the cluster on the Services UI. A separate entry is required for each instance of the application in the cluster.
Note:

While creating a service entry, consider the following configurations:
- Make sure to set the correct application configuration for each application.
- Set Failover Type to Cluster.
- Set Cluster ID to be the same value for all the individual service configurations.
- Be sure to enter the correct DeviceZoneID.
- Make sure to set the correct number of Threads. The number of total threads should not exceed eight times the total number of cores. The number of DB threads created will be one-third of the number Threads unless you manually set DBThreads to a different value.
Start each instance of the application across the cluster.
Verify application functionality and review the application logs to see if any errors occur. You can view the log files in from the command line in the $A1BASEDIR/logs directory, or from the Logs UI.

Changing the Cluster for Services

You can change the cluster of a service by updating the Cluster ID property. All instances belonging to the same cluster must be associated with the same cluster ID and each instance can only be associated with one cluster ID.

If you are updating the cluster IDs in bulk for several instances, stop the cluster services and update the cluster ID for all of them before restarting.

To change the cluster for services in a bulk:

Stop all related cluster services. For example, if you have multiple cluster Metric Generic SNMP Poller services:
1. From the Configuration menu, select Broker Control, and then Services.
2. Open the filter bar and enter Metric Generic SNMP Poller in the Name column.
3. Press CTRL or SHIFT and click to select the cluster services.
4. Click Stop.
For each cluster service, update the Cluster ID property and start the service. For example, if you have multiple cluster Metric Generic SNMP Poller services:
1. From the Configuration menu, select Broker Control, and then Services.
2. Open the filter bar and enter Metric Generic SNMP Poller in the Name column.
3. Repeat the following for each cluster service:
  1. Select the service.
  2. Change the value of the Cluster ID property.
    
    Note:
    
    You must use the same value for all instances of a service in the same cluster and one cluster can only contain instance of one application or service.
  3. Click Submit.
  4. Click Start.

If you only want to change the cluster for few handpicked instances of a service in a cluster, you can update the cluster ID for those service instances without stopping and restarting the service. In this case, you need to click Reload Config button after updating the cluster IDs for all the required instances.

To verify the service cluster information, including checking whether all instances of a single service are assigned to the required clusters, run the following command:

BrokerControl -b debug FailoverClusters

This command returns data for all cluster services.

Title and Copyright Information

Implementation Guide

G49441-01