Java Dynamic Management Kit 4.1 Tutorial

Chapter 7 Notification Forwarding

In this topic, we expand the notification mechanism to the manager side, looking at how remote applications receive notifications. Notifications are forwarded to a manager through existing structures of the Java Dynamic Management architecture: MBeans, the MBean server, and the connectors. Notably, this implies that the notification mechanism is designed to forward only notifications from registered MBeans on the agent side to proper listeners on the manager side.

As with the other management operations, listening for notifications is nearly as simple to do in a management application as it is to do locally in an agent. The interface of the connector client hides all communication issues, so that listeners may be registered through the connector or directly with existing proxy MBeans.

The code samples in this topic are taken from the files in the Notification example directory located in the main examplesDir (see "Directories and Classpath" in the preface).

Contents:

"Registering Manager-Side Listeners" shows how similar the agent side and manager side notification mechanisms are.
"Push Mode" demonstrates the simplest forwarding strategy whereby the agent sends notifications to the manager as they occur.
"Pull Mode" demonstrates an advanced forwarding strategy which buffers notifications in the agent until the manager requests them.
"Running the Notification Forwarding Example" shows how to compile and launch the manager application.

Registering Manager-Side Listeners

Like the other structures of the Java Dynamic Management Kit, the notification mechanism is designed to be homogeneous from the agent to the manager side. For this reason, notification objects and listener interfaces in manager applications are identical to those on the agent side.

This means that listeners in managers are similar to those in agents, and they could even be identical objects in some management solutions. However, chances are that manager-side listeners will want to receive different notifications and take different actions than their agent-side peers.

The Agent-Side Broadcaster

As in most management architectures, the notification broadcasters are agent-side entities. The broadcasters are MBeans registered in an agent's MBean server to which our management application will need to connect. Only notifications sent by registered MBeans can be forwarded to manager applications, and a manager-side listener can receive them only by registering through a connector object.

Since the notification support classes do not rely on the MBean server, they could be reused in the manager or any other Java-based application that requires a notification mechanism. In this case, listeners will need to register directly with local broadcasters and there is no longer any mechanism for forwarding notifications to a remote listener. Because the same classes are used everywhere, it would be possible to have a listener in a manager that is added to both a broadcaster MBean in a remote agent and directly to a local broadcaster object. However, none of these situations are covered in our example.

Here we give the code of the MBean that we will use to send notifications (the code for its MBean interface has been omitted). It extends the NotificationBroadcasterSupport class to reuse all of its listener registration facilities. It only contains one operation which can be called by our manager to trigger any number of notifications.

Example 7-1 The Agent-Side Broadcaster MBean

import javax.management.MBeanNotificationInfo;
import javax.management.NotificationBroadcasterSupport;
import javax.management.Notification;

public class NotificationEmitter
    extends NotificationBroadcasterSupport
    implements NotificationEmitterMBean {

    // Just to make the inheritance explicit
    public NotificationEmitter() {
        super();
    }

    // Provide details about the notification type and class that is sent
    public MBeanNotificationInfo[] getNotificationInfo() {
        
        MBeanNotificationInfo[] ntfInfoArray = new MBeanNotificationInfo[1];
        
        String[] ntfTypes = new String[1];
        ntfTypes[0] = myType;
        
        ntfInfoArray[0] = new MBeanNotificationInfo( ntfTypes,
            "javax.management.Notification", 
            "Notifications sent by the NotificationEmitter");
        return ntfInfoArray;
    }  

    // The only operation: sends nb number of notifications
    // whose sequence numbers go from 1 to nb
    public void sendNotifications(Integer nb) {

        for (int i=1; i<=nb.intValue(); i++) {
            sendNotification(new Notification(myType, this, i));
        }
    }
    private String myType = "notification.my_notification";
}

Our MBean invents a notification type string and exposes this information through the getNotificationInfo method. To demonstrate the forwarding mechanism, we are more interested in the sequence number: this will allow us to identify the notifications as they are received in the manager.

This MBean demonstrates that the broadcaster has total control over the contents of its notifications. Constructors for the Notification object allow you to specify all of the fields, even ones such as the time stamp. In this example, we control the sequence number, and our chosen policy is to reset the sequence number to 1 with every call to the operation. Of course, you are free to choose the notification contents, including the time-stamping and sequence-numbering policies that fit your management solution.

The Manager-Side Listener

In our simple example, the Client class itself is the listener object. Usually, a listener would be a separate instance of a special listener class and depending on the complexity of the manager, there might be several classes of listeners, each for a specialized category of notifications.

Example 7-2 The Manager-Side Listener

public class Client implements NotificationListener {

    [...] // Constructor omitted

    // Implementation of the NotificationListener interface  
    //
    public void handleNotification(Notification notif, Object handback) {

        System.out.println("Client: received a notification of type "
            + notif.getType() + "\nwith the sequence number "
            + notif.getSequenceNumber());
    }
    [...]  // main omitted
}

As explained in "Agent-Side Notifications", a listener on the agent side is typically an MBean which receives notifications about the status of other MBeans and then processes or exposes this information in some manner. On the agent side, notifications are often used to pass information around, allowing services to know the status of an agent. Only if a key value is observed, an overflow or a forbidden format, will this information be passed to a listening manager, probably by sending a different notification. This is how the Java Dynamic Management architecture can embed management intelligence into an agent.

In this manner, the notification model reduces the communication that is necessary between agents and managers. Your management solution determines how much decisional power resides in the agent and when situations are escalated. These parameters will affect your design of the notification flow between broadcasters, listeners, agents, and managers.

The usual role of a manager-side listener is to process the important information in a notification and take the appropriate action. As we shall see, our notification example is much simpler. Our goal is not to construct a real-world example, but to demonstrate the mechanisms that are built into the Java Dynamic Management Kit.

Adding the Listener Through the Connector

By extension of the ClientNotificationHandler interface, the RemoteMBeanServer interface exposes methods for adding and removing listeners. The signatures of these methods are identical to those of the agent-side MBeanServer interface. The only difference is that they are implemented in the connector client classes which make the communication protocol transparent.

Our manager application uses the RMI protocol. After creating the connector client object, we can use its RemoteMBeanServer interface to create our broadcaster MBean and then register as a listener to this MBean's notifications.

Example 7-3 Adding the Listener through the Connector

// Use RMI connector on port 8086 to communicate with the agent
System.out.println(">>> Create an RMI connector client");
RmiConnectorClient connectorClient = new RmiConnectorClient();

// agentHost was read from the command line or defaulted to localhost
RmiConnectorAddress rmiAddress = new RmiConnectorAddress(
    agentHost, 8086, com.sun.jdmk.ServiceName.RMI_CONNECTOR_SERVER);
connectorClient.connect(rmiAddress);

// Wait 1 second for connecting
Thread.sleep(1000);

// Create the MBean in the agent
ObjectName mbean = new ObjectName ("Default:name=NotificationEmitter");
connectorClient.createMBean("NotificationEmitter", mbean);

// Now add ourselves as the listener (no filter, no handback)
connectorClient.addNotificationListener(mbean, this, null, null);

You can see how similar this is to the agent code is by comparing it with Example 6-2 and Example 6-5.

If you have generated and instantiated proxy MBeans for your broadcaster MBeans, you can also register through the addNotificationListener method that they expose. When generating proxy classes with the proxygen tool, MBeans which implement the NotificationBroadcaster interface will have proxy classes which implement the NotificationBroadcasterProxy interface.

Again, the method signatures defined in a proxy MBean are identical to those of the MBeanServer or NotificationBroadcasterClient interfaces for adding or removing listeners. Listeners added through a proxy MBean will received the same notifications as listeners added to the same MBean through the interface of the connector client.

Note -

Following the Java programming model, the connector client limits its resource usage by only running one thread to notify all of its listeners. This thread calls all of the handler callback methods that have been added through this connector. Therefore, the callbacks should return quickly and use safe programming to avoid crashing.

Push Mode

Because the broadcaster and the listener are running on separate machines or in separate JVMs on the same host, their notifications must be forwarded from one to the other. The mechanism for doing this is completely transparent to the components of the Java Dynamic Management Kit.

Briefly, the connector client instructs the connector server to add its own agent-side listener to the designated broadcaster using the methods of the MBeans server. Then, the connector server implements a buffering cache mechanism to centralize notifications before serializing them to be forwarded to the connector client. By design, the agent doesn't control the buffering and forwarding mechanism; it is the manager where the listeners reside that has this privilege.

Neither the broadcaster nor the listener need to implement any extra methods, or even be aware that the other party is remote. Only the designer needs to be aware of communication issues such as delays: you can't expect the listener to be invoked immediately after a remote broadcaster sends a notification.

The forwarding mechanism allows you to configure how and when notifications are forwarded. This allows you to optimize the communication strategy between your agents and managers.

There are two basic modes for notification forwarding: push mode and pull mode. They are named according to the action performed on a notification that has been sent and is in the connector server's buffer. This notification is either pushed to the manager at the agent's initiative, or pulled by the manager at its own initiative.

The push mode for notification forwarding is the simplest because it implements the expected behavior of a notification. When a notification is sent from an MBean to its listener, it is immediately pushed to the manager-side where the listener's handler method is called. There is no delay in the caching, and if the communication layer is quick enough, the listener is invoked almost immediately after the notification is sent.

Push mode is the default forwarding policy of a newly instantiated connector client.

In our manager example, we explicitly set the connector client in push mode and then trigger the agent-side notifications.

Example 7-4

System.out.println("\n>>> Set notification forward mode to PUSH.");
connectorClient.setMode(ClientNotificationHandler.PUSH_MODE);

System.out.println(">>> Have our MBean broadcast 10 notifications...");
params[0] = new Integer(10);
signatures[0] = "java.lang.Integer";
connectorClient.invoke(mbean, "sendNotifications", params, signatures);

System.out.println(">>> Done.");
System.out.println(">>> Receiving notifications...\n");

// Nothing to do but wait while our handler receives the notifications
Thread.sleep(2000);

As we shall see when describing the pull mode, the connector client exposes the methods for controlling the agent's notification buffer. This caching buffer is not used in push mode, so these methods do not affect pushed notifications. The methods do however set internal variables that will be taken into account if and when pull mode is enabled. Future versions of the product may implement push-mode buffering to provide added functionality.

The advantage of push mode is that it works without any further intervention: notifications eventually reach their remote listeners. Push mode works when the communication layer and the listener's processing capacity are adapted to the notification emission rate, or more specifically to the potential emission rate. Since all notifications are immediately sent to the manager hosts, a burst of notifications will cause a burst of traffic that may or may not be adapted to the communication layer.

If your communication layer is likely to be saturated or must be preserved in all cases, either your design should control broadcasters to prevent bursts of notifications, or you should use the pull mode which has this control functionality built-in. The push mode is ideal if you have reliable and fast communication between your agents and your managers. You may also dynamically switch between modes, allowing a management application to fine-tune its communication policy depending on the number of notifications that must be handled.

Pull Mode

In pull mode, notifications are not immediately sent to their remote listeners. Rather, they are stored in the connector server's internal buffer until the connector client requests that they be forwarded. Pull mode has the following settings that let the manager define the notification forwarding policy:

A period for automatic pulling
The size of the agent-side notification buffer (also called the cache)
The policy for discarding notifications when this buffer is full

These settings are controlled through the methods exposed by the connector client. Therefore, the pull-mode policy is set for all manager-side listeners added through a given connector client, or through one of its dependent proxy MBeans.

Periodic Pulling

Pull mode forwarding is necessarily a compromise between receiving notifications in a timely manner and not saturating the communication layer. Notifications are stored temporarily in the agent-side buffer, but the manager-side listeners still need to receive them. Pull mode includes automatic pulling that retrieves all buffered notifications regularly.

The frequency of the pull forwarding is controlled by the pull period expressed in milliseconds. By default, this period is 1 second, meaning that when pull mode is enabled, the manager will automatically begin pulling notifications once per second. Whether or not there are any notifications to receive depends upon the agent.

Our manager application sets a half-second pull period and then triggers the notification broadcaster.

Example 7-5 Pulling Notifications Automatically

System.out.println(">>> Set notification forward mode to PULL.");
connectorClient.setMode(ClientNotificationHandler.PULL_MODE);

// Retrieve buffered notifications from the agent twice per second
System.out.println(">>> Set the forward period to 500 milliseconds.");
connectorClient.setPeriod(500);

System.out.println(">>> Have our MBean broadcast 20 notifications...");
params[0] = new Integer(20);
signatures[0] = "java.lang.Integer";
connectorClient.invoke(mbean, "sendNotifications", params, signatures);
System.out.println(">>> Done.");

// Wait for the handler to process all notifications
System.out.println(">>> Receiving notifications...\n");
Thread.sleep(2000);

When notifications are pulled, all notifications in the agent-side buffer are forwarded to the manager and the registered listeners. It is not possible to set a limit on the number of notifications which are forwarded, except by limiting the size of the buffer (see "Agent-Side Buffering"). Even in a controlled example such as ours, the number of notifications in the agent-side buffer at each pull period is completely dependent upon the agent's execution paths, and therefore unpredictable from the manager-side.

On-Demand Forwarding

We can also disable the automatic pulling by setting the pull period to zero. In this case, the connector client will not pull any notifications from the agent until instructed to do so. Use the getNotifications method of the connector client to pull all notifications when desired. This method will immediately forward all notifications in the agent-side buffer. Again, it is not possible to limit the number of notifications that are forwarded, except by limiting the buffer size.

In our example, we disable the automatic pulling and then trigger the notification broadcaster. The notifications will not be received until we request that the connector server pull them. Then, all of the notifications will be received at once.

Example 7-6 Pulling Notifications by Request

System.out.println(">>> Use pull mode with period set to zero.");
connectorClient.setMode(ClientNotificationHandler.PULL_MODE);
connectorClient.setPeriod(0);

System.out.println(">>> Have our MBean broadcast 30 notifications...");
params[0] = new Integer(30);
signatures[0] = "java.lang.Integer";
connectorClient.invoke(mbean, "sendNotifications", params, signatures);
System.out.println(">>> Done.");

// Call getNotifications to pull all buffered notifications from the agent
System.out.println("\n>>> Press <Enter> to pull the notifications.");
System.in.read();
connectorClient.getNotifications();

// Wait for the handler to process all notifications
Thread.sleep(100);

In the rest of our example, we use the pull-by-request mechanism to control how many notifications are buffered on the agent-side and thereby test the different caching policies.

Agent-Side Buffering

In pull mode, notifications are stored by the connector server in a buffer until they are pulled by the connector client. Any one of the pull operations, whether by-request or automatic, empties this buffer, and it fills up again as new notifications are sent.

By default, this buffer will grow to contain all notifications. In the worst case scenario of an overfull buffer, this can lead either to an "out of memory" error of the agent application, a saturation of the communication layer, or an overload of the manager's listeners when the notifications are finally pulled. The ClientNotificationHandler interface defines the static NO_CACHE_LIMIT field to represent an unlimited buffer size.

To change the size of the agent's cache, call the connector client's setCacheSize method. The size of the cache is expressed as the number of notifications which can be stored in its buffer. When a cache buffer of limited size is full, new notifications will cause an overflow. Therefore you should also choose an overflow mode when using a limited cache size. The two overflow modes are defined by static fields of the ClientNotificationHandler interface:

DISCARD_OLD - The oldest notifications will be lost and the buffer will always be renewed with the latest notifications which have been sent; this is the default value when a limit is first set for the cache size
DISCARD_NEW - Once the notification buffer is full, any new notifications will be lost until the buffer is emptied by forwarding the messages; the buffer will always contain the first notifications sent after the previous pull operation

We demonstrate each of these modes in our sample manager, by first setting the cache size and the overflow mode, then by triggering more notifications than the cache buffer can hold.

Example 7-7 Controlling the Agent-Side Buffer

System.out.println(">>> Use pull mode with period set to zero, " +
    "buffer size set to 10, and overflow mode set to DISCARD_OLD.");
connectorClient.setMode(ClientNotificationHandler.PULL_MODE);
connectorClient.setPeriod(0);
connectorClient.setCacheSize(10, true);
connectorClient.setOverflowMode(ClientNotificationHandler.DISCARD_OLD);

System.out.println(">>> Have our MBean broadcast 30 notifications...");
params[0] = new Integer(30);
signatures[0] = "java.lang.Integer";
connectorClient.invoke(mbean, "sendNotifications", params, signatures);
System.out.println(">>> Done.");

// Call getNotifications to pull all buffered notifications from the agent
System.out.println("\n>>> Press <Enter> to get notifications.");
System.in.read();
connectorClient.getNotifications();

// Wait for the handler to process the 10 notifications
// These should be the 10 most recent notifications
// (the greatest sequence numbers)
Thread.sleep(100);
System.out.println("\n>>> Press <Enter> to continue.");
System.in.read();

// We should see that the 20 other notifications overflowed the agent buffer
System.out.println(">>> Get overflow count = " +
    connectorClient.getOverflowCount());

The overflow count gives the total number of notifications that have been discarded because the buffer has overflowed. The number is cumulative from the first listener registration from the manager until all of the manager's listeners have been unregistered. The manager application can reset this value by calling the setOverflowCount method.

In our example application, we repeat the actions above, in order to cause the buffer to overflow again, but this time using the DISCARD_NEW policy. Again, the buffer size is ten, and there are 30 notifications. In this mode, the first 10 sequence numbers will remain in the cache to be forwarded when the manager pulls them from the agent, and 20 more will have overflowed.

Buffering Specifics

When the buffer is full and notifications need to be discarded, the time reference for applying the overflow mode is the order in which notifications arrived in the buffer. Neither the time stamps nor the sequence numbers of the notifications are considered, since neither of these are necessarily absolute. Even the sequence of notifications from the same broadcaster can be non-deterministic. And in any case, broadcasters are free to set both time stamps and sequence numbers as they see fit, or even to set them to null.

The second parameter of the setCacheSize method is a boolean which determines whether or not the potential overflow of the cache is discarded when reducing the cache size. If the currently buffered notifications do not fit into the new cache size and this parameter is true, excess notifications are discarded according to the current overflow mode. The overflow count is also updated accordingly.

In the same situation with the parameter set to false, the cache will not be resized. You need to check the return value of the method when you set this parameter to false. If the cache cannot be resized because it would lead to discarded notifications, you need to pull the waiting notifications with the getNotifications method and try resizing the cache again. When the existing notifications fit within the new cache size or when increasing the cache size, the second parameter of setCacheSize has no effect.

Because several managers may connect through the same protocol, the connector server object must handle the notifications for each separately. This implies that each connected manager has its own notification buffer and its own settings for controlling this cache. The overflow count is specific to each manager as well.

Buffering Generalities

Here we have demonstrated each setting independently by controlling the notification broadcaster. In practice, periodic pulling, agent-side buffering and buffer overflowing can all happen at once. And you can call getNotifications at any time to do an on-demand pull of any notifications in the agent-side buffer. You should adjust the settings to fit the known or predicted behavior of your management solution, depending upon communication constraints and your acceptable notification loss.

The caching policy is completely determined by the manager application. If notification loss in unacceptable, it is the manager's responsibility to configure the mechanism so that they are pulled as often as necessary. Also, the mechanism can be updated dynamically. For example, by checking the overflow count with every pull operation, the manager can know the number of lost packets, allowing it to compute a new notification emission rate. Using this rate, the manager can dynamically update any of the controls (buffer size, pull interval, and overflow mode) to keep up with the notification rate.

Running the Notification Forwarding Example

The examplesDir/Notification directory contains all of the files for the broadcaster MBean, the BaseAgent application, and our Client application which is itself the listener object.

Compile all files in this directory with the javac command. For example, on the Solaris platform with the Korn shell, you would type:

$ cd examplesDir/Notification/
$ javac -classpath classpath *.java

To run the notification forwarding example, we use the BaseAgent application which contains an RMI adaptor server. To run the example, launch the agent on another host or in another terminal window with the following command. Be sure that the classes for the NotificationEmitter MBean can be found in its classpath.

$ java -classpath classpath BaseAgent

Wait for the agent to be completely initialized, then launch the manager with the following command, where hostname is the name of the machine running the agent. If you launched the agent on the same machine, you can omit the hostname.

$ java -classpath classpath Client hostname

When launched, the manager application first creates the NotificationEmitter MBean and then registers itself as a listener. The manager then steps through the various notification forwarding situations that we have seen in this topic. Press <Enter> to step through the example when the application pauses.

You may also interact with the example through the HTML adaptor of the BaseAgent. Leave the agent application running and launch the manager again.

Interacting with the Notification Forwarding Mechanism

Load the following URL in your browser and go to the MBean view of the NotificationEmitter MBean:

http://hostname:8082/

If you get an error, you may have to switch off proxies in your preference settings. Any browser on your local network can also connect to this agent using this URL.

When the manager application pauses for the first time, invoke the sendNotifications method from your browser and enter a small integer as the parameter.

You should see the listener handle your notifications in the manager's terminal window. Since the manager is still in push mode, they were forwarded immediately.

Press <Enter> in the terminal window: the manager is now in pull mode with a pull period of 500 milliseconds. Through the MBean view, send 1000 notifications.

If your agent's host is slow enough, or your manager's host fast enough, you may be able to see the manager pause briefly after it has processed all notifications from one period and before the next ones are forwarded.

Press <Enter> in the terminal window: the manager will now forward notifications by request. Before pressing <Enter> again, have the MBean send 15 notifications.

You should see the manager pull all of the notifications: the 30 triggered by the manager and the 15 we just triggered. They were all kept in the buffer, waiting for the manager's request to forward them. Remember that the sendNotifications operation resets the sequence numbering every time it is invoked.

Press <Enter> in the terminal window: the manager will now set the cache size to 10 notifications and set the overflow mode to DISCARD_OLD. Before pressing <Enter> again, have the MBean send 15 more notifications.

Only the last ten of our notifications could fit into the cache buffer, all the rest, including those already triggered by the manager, overflowed and were discarded. Press <Enter> to see that they are tallied in the overflow count.

Press <Enter> in the terminal window: the cache size is still 10 notifications and the manager will now set the overflow mode to DISCARD_NEW. Before pressing <Enter> again, have the MBean send only 5 more notifications.

The first ten of the manager-triggered notifications are received: all of the more recent notifications, including ours, overflowed the cache buffer and were lost. Press <Enter> to see that they are tallied in the overflow count: the 35 from the last step plus 25 more from this step, for a total of 60.

Press <Enter> in the terminal window one last time to stop the Client application. Stop the agent application in the same manner when you are finished running the example.