test

Java Dynamic Management Kit 5.0 Tutorial

Chapter 12 Notification Forwarding

In this chapter, we expand the notification mechanism to the manager side, looking at how remote applications receive notifications. Notifications are forwarded to a manager through existing structures of the Java dynamic management architecture: MBeans, the MBean server, and the connectors. Notably, this implies that the notification mechanism is designed to forward only notifications from registered MBeans on the agent side to proper listeners on the manager side.

As with the other management operations, listening for notifications is nearly as simple to do in a management application as it is to do locally in an agent. The interface of the connector client hides all communication issues, so that listeners can be registered through the connector or directly with existing proxy MBeans.

The code samples in this topic are taken from the files in the Notification example directory located in the main examplesDir (see “Directories and Classpath” in the Preface).

This chapter covers the following topics:

Registering Manager-Side Listeners

Like the other structures of the Java DMK, the notification mechanism is designed to be homogeneous from the agent to the manager side. For this reason, notification objects and listener interfaces in manager applications are identical to those on the agent side.

The symmetry of the interfaces also means that code for listeners can easily be reused without modification in either agent or manager applications. Listeners in managers are similar to those in agents, and they could even be identical objects in some management solutions. However, in most cases, manager-side listeners will receive different notifications and take different actions from their agent-side peers.

Agent-Side Broadcaster

The notification broadcasters are MBeans registered in an agent's MBean server to which our management application needs to connect. Only notifications sent by registered MBeans can be forwarded to manager applications, and a manager-side listener can receive them only by registering through a connector client or a proxy object.

Other notification broadcasters can exist independently in the manager application, but listeners need to register directly with these local broadcasters. Nothing prevents a listener object from registering both with a connector client or proxy for remote notifications and with a local broadcaster.

The code example below shows how the sample NotificationEmitter MBean sends notifications (the code for its MBean interface has been omitted). It extends the NotificationBroadcasterSupport class to reuse all of its listener registration facilities. It only contains one operation that can be called by our manager to trigger any number of notifications.

Example 12–1 Agent-Side Broadcaster MBean

import javax.management.MBeanNotificationInfo;
import javax.management.NotificationBroadcasterSupport;
import javax.management.Notification;

public class NotificationEmitter
    extends NotificationBroadcasterSupport
    implements NotificationEmitterMBean {

    // Just to make the inheritance explicit
    public NotificationEmitter() {
        super();
    }

    // Provide details about the notification type and class that is sent
    public MBeanNotificationInfo[] getNotificationInfo() {
        
        MBeanNotificationInfo[] ntfInfoArray = new MBeanNotificationInfo[1];
        
        String[] ntfTypes = new String[1];
        ntfTypes[0] = myType;
        
        ntfInfoArray[0] = new MBeanNotificationInfo( ntfTypes,
            "javax.management.Notification", 
            "Notifications sent by the NotificationEmitter");
        return ntfInfoArray;
    }  

    // The only operation: sends any number of notifications
    // whose sequence numbers go from 1 to "nb"
    public void sendNotifications( Integer nb ) {

        for (int i=1; i<=nb.intValue(); i++) {
            sendNotification(new Notification(myType, this, i));
        }
    }
    private String myType = "notification.my_notification";
}

Our MBean invents a notification type string and exposes this information through the getNotificationInfo method. To demonstrate the forwarding mechanism, we are more interested in the sequence number, which enables us to identify the notifications as they are received in the manager.

This MBean demonstrates that the broadcaster has total control over the contents of its notifications. Constructors for the Notification object enable you to specify all of the fields, even ones such as the time stamp. In this example, we control the sequence number and our chosen policy is to reset the sequence number to 1 with every call to the operation. Of course, you are free to choose the notification contents, including the time-stamping and sequence-numbering policies that fit your management solution.

Note –

Due to possible loss in the communication layer and the inherent indeterminism of thread execution, the notification model does not guarantee that remote notifications will be received nor that their sequence will be preserved. If notification order is critical to your application, your broadcaster should set the sequence numbers appropriately, and your listeners should sort the received notifications.

Manager-Side Listener

In our simple example, the Client class itself is the listener object. Usually, a listener would be a separate instance of a special listener class and depending on the complexity of the manager, there might be several classes of listeners, each for a specialized category of notifications.

Example 12–2 The Manger-Side Listener

public class Client implements NotificationListener {

    [...] // Constructor omitted

    // Implementation of the NotificationListener interface  
    //
    public void handleNotification(Notification notif, Object handback) {

        System.out.println("Client: received a notification of type "
            + notif.getType() + "\nwith the sequence number "
            + notif.getSequenceNumber());
    }
    [...]  // main omitted
}

As explained in the notification mechanism Overview, a listener on the agent side is typically an MBean that receives notifications about the status of other MBeans and then processes or exposes this information in some manner. Only if a key value or some management event is observed will this information be passed to a listening manager, probably by sending a different notification.

In this manner, the notification model reduces the communication that is necessary between agents and managers. Your management solution determines how much decisional power resides in the agent and when situations are escalated. These parameters will affect your design of the notification flow between broadcasters, listeners, agents, and managers.

The usual role of a manager-side listener is to process the important information in a notification and take the appropriate action. As we shall see, our notification example is much simpler. Our goal is not to construct a real-world example, but to demonstrate the mechanisms that are built into the Java DMK.

Adding a Listener Through the Connector

By extension of the ClientNotificationHandler interface, the RemoteMBeanServer interface exposes methods for adding and removing listeners. The signatures of these methods are identical to those of the agent-side MBeanServer interface. The only difference is that they are implemented in the connector client classes that make the communication protocol transparent.

Our manager application uses the RMI protocol connector. After creating the connector client object, we use the methods of its RemoteMBeanServer interface to create our broadcaster MBean and then register as a listener to this MBean's notifications.

Example 12–3 Adding a Listener through the Connector

// Use RMI connector on port 8086 to communicate with the agent
System.out.println(">>> Create an RMI connector client");
RmiConnectorClient connectorClient = new RmiConnectorClient();

// agentHost was read from the command line or defaulted to localhost
RmiConnectorAddress rmiAddress = new RmiConnectorAddress(
    agentHost, 8086, com.sun.jdmk.ServiceName.RMI_CONNECTOR_SERVER);
connectorClient.connect(rmiAddress);

// Wait 1 second for connecting
Thread.sleep(1000);

// Create the MBean in the agent
ObjectName mbean = new ObjectName ("Default:name=NotificationEmitter");
connectorClient.createMBean("NotificationEmitter", mbean);

// Now add ourselves as the listener (no filter, no handback)
connectorClient.addNotificationListener(mbean, this, null, null);

You can see how similar this code is to the agent application by comparing it with the code example for Adding a Listener Through the MBean Server.

If you have generated and instantiated proxy MBeans for your broadcaster MBeans, you can also register through the addNotificationListener method that they expose. When generating proxy classes with the proxygen tool, MBeans that implement the NotificationBroadcaster interface will have proxy classes that implement the NotificationBroadcasterProxy interface.

Again, the method signatures defined in a proxy MBean are identical to those of the MBeanServer or NotificationBroadcasterClient interfaces for adding or removing listeners. See the code example for Adding a Listener Directly to an MBean. Listeners added through a proxy MBean receives the same notifications as listeners added to the same MBean through the interface of the connector client.

Note –

Following the Java programming model, the connector client limits its resource usage by only running one thread to notify all of its listeners. This thread calls all of the handler callback methods that have been added through this connector. Therefore, the callbacks should return quickly and use safe programming to avoid crashing the connector client.

Push Mode

Because the broadcaster and the listener are running on separate hosts or in separate virtual machines on the same host, their notifications must be forwarded from one to the other. The mechanism for doing this is completely transparent to the users of the Java DMK components.

Briefly, the connector client instructs the connector server to add its own agent-side listener to the designated broadcaster using the methods of the MBeans server. Then, the connector server implements a buffering cache mechanism to centralize notifications before serializing them to be forwarded to the connector client. By design, it is the connector client in the manager application that controls the buffering and forwarding mechanism for a connection.

Figure 12–1 summarizes the notification forwarding mechanism and its actors. In particular, it shows how the connector server uses internal listener instances to register locally for MBean notifications, even if this mechanism is completely hidden from the user. The path of listener registration through calls to the addNotificationListener method of the various interfaces is paralleled by the propagation of notifications through calls to the listeners' handleNotification method.

Figure 12–1 Notification Forwarding Internals

Diagram of the notification forwarding mechanism

Neither the broadcaster nor the listener need to implement any extra methods, or even be aware that the other party is remote. Only the designer needs to be aware of communication issues such as delays. You cannot expect the listener to be invoked instantaneously after a remote broadcaster sends a notification.

The forwarding mechanism allows you to configure how and when notifications are forwarded. This enables you to optimize the communication strategy between your agents and managers. There are two basic modes for notification forwarding: push mode and pull mode. A notification in the connector server's cache is either pushed to the manager at the agent's initiative, or pulled by the manager at its own initiative.

The push mode for notification forwarding is the simplest because it implements the expected behavior of a notification. When a notification is sent from an MBean to its listener, it is immediately pushed to the manager-side where the listener's handler method is called. There is no delay in the caching, and if the communication layer is quick enough, the listener is invoked almost immediately after the notification is sent.

Push mode is the default forwarding policy of a newly instantiated connector client.

In our manager example, we explicitly set the connector client in push mode and then trigger the agent-side notifications.

Example 12–4 Switching to the Notification Push Mode

System.out.println("\n>>> Set notification forward mode to PUSH.");
connectorClient.setMode(ClientNotificationHandler.PUSH_MODE);

System.out.println(">>> Have our MBean broadcast 10 notifications...");
params[0] = new Integer(10);
signatures[0] = "java.lang.Integer";
connectorClient.invoke(mbean, "sendNotifications", params, signatures);

System.out.println(">>> Done.");
System.out.println(">>> Receiving notifications...\n");

// Nothing to do but wait while our handler receives the notifications
Thread.sleep(2000);

The connector client exposes the methods for controlling the agent's notification buffer. This caching buffer is not used in push mode, so these methods do not affect pushed notifications. The methods do however set internal variables that will be taken into account if and when pull mode is enabled.

The advantage of push mode is that it works without any further intervention: notifications eventually reach their remote listeners. Push mode works when the communication layer and the listener's processing capacity are adapted to the notification emission rate, or more specifically to the potential emission rate. Because all notifications are immediately sent to the manager hosts, a burst of notifications will cause a burst of traffic that might or might not be adapted to the communication layer.

If your communication layer is likely to be saturated, either your design should control broadcasters to prevent bursts of notifications, or you should use the pull mode which has this control functionality built-in. The push mode is ideal if you have reliable and fast communication between your agents and your managers. You can also dynamically switch between modes, enabling a management application to fine-tune its communication policy depending on the number of notifications that must be handled.

Pull Mode

In pull mode, notifications are not immediately sent to their remote listeners. Rather, they are stored in the connector server's internal buffer until the connector client requests that they be forwarded. Instead of being sent individually, the notifications are grouped to reduce the load on the communication layer. Pull mode has the following settings that let the manager define the notification forwarding policy:

For a given connection, there is one cache for all listeners, not one cache per listener. This cache therefore has one buffering policy whose settings are controlled through the methods exposed by the connector client. The cache buffer contains an unpredictable mix of notifications in transit to all manager-side listeners added through a given connector client or through one of its bound proxy MBeans. The buffer operations such as pulling or overflowing apply to this mix of notifications, not to any single listener's notifications.

Periodic Forwarding

Pull mode forwarding is necessarily a compromise between receiving notifications in a timely manner, not saturating the communication layer, and not overflowing the buffer. Notifications are stored temporarily in the agent-side buffer, but the manager-side listeners still need to receive them. Pull mode includes automatic pulling that retrieves all buffered notifications regularly.

The frequency of the pull forwarding is controlled by the pull period expressed in milliseconds. By default, when pull mode is enabled, the manager will automatically begin pulling notifications once per second. Whether or not there are any notifications to receive depends upon events in the agent.

Our manager application sets a half-second pull period and then triggers the notification broadcaster.

Example 12–5 Pulling Notifications Automatically

System.out.println(">>> Set notification forward mode to PULL.");
connectorClient.setMode(ClientNotificationHandler.PULL_MODE);

// Retrieve buffered notifications from the agent twice per second
System.out.println(">>> Set the forward period to 500 milliseconds.");
connectorClient.setPeriod(500);

System.out.println(">>> Have our MBean broadcast 20 notifications...");
params[0] = new Integer(20);
signatures[0] = "java.lang.Integer";
connectorClient.invoke(mbean, "sendNotifications", params, signatures);
System.out.println(">>> Done.");

// Wait for the handler to process all notifications
System.out.println(">>> Receiving notifications...\n");
Thread.sleep(2000);

When notifications are pulled, all notifications in the agent-side buffer are forwarded to the manager and the registered listeners. It is not possible to set a limit on the number of notifications that are forwarded except by limiting the size of the buffer (see Agent-Side Buffering). Even in a controlled example such as ours, the number of notifications in the agent-side buffer at each pull period is completely dependent upon the agent's execution paths, and therefore unpredictable from the manager-side.

On-Demand Forwarding

You can disable automatic pulling by setting the pull period to zero. In this case, the connector client will not pull any notifications from the agent until instructed to do so. Use the getNotifications method of the connector client to pull all notifications when desired. This method will immediately forward all notifications in the agent-side buffer. Again, it is not possible to limit the number of notifications that are forwarded except by limiting the buffer size.

In this example, we disable the automatic pulling and then trigger the notification broadcaster. The notifications will not be received until we request that the connector server pull them. Then, all of the notifications will be received at once.

Example 12–6 Pulling Notifications by Request

System.out.println(">>> Use pull mode with period set to zero.");
connectorClient.setMode(ClientNotificationHandler.PULL_MODE);
connectorClient.setPeriod(0);

System.out.println(">>> Have our MBean broadcast 30 notifications...");
params[0] = new Integer(30);
signatures[0] = "java.lang.Integer";
connectorClient.invoke(mbean, "sendNotifications", params, signatures);
System.out.println(">>> Done.");

// Call getNotifications to pull all buffered notifications from the agent
System.out.println("\n>>> Press Enter to pull the notifications.");
System.in.read();
connectorClient.getNotifications();

// Wait for the handler to process all notifications
Thread.sleep(100);

In the rest of our example, we use the on–demand forwarding mechanism to control how many notifications are buffered on the agent-side and thereby test the different caching policies.

Agent-Side Buffering

In pull mode, notifications are stored by the connector server in a buffer until they are pulled by the connector client. Any one of the pull operations, whether on-demand or periodic, empties this buffer, and it fills up again as new notifications are triggered.

By default, this buffer will grow to contain all notifications. The ClientNotificationHandler interface defines the static NO_CACHE_LIMIT field to represent an unlimited buffer size. If the notifications are allowed to accumulate indefinitely in the cache, this can lead either to an “out of memory” error in the agent application, a saturation of the communication layer, or an overload of the manager's listeners when the notifications are finally pulled.

To change the size of the agent's cache, call the connector client's setCacheSize method. The size of the cache is expressed as the number of notifications that can be stored in its buffer. When a cache buffer of limited size is full, new notifications will overflow and be lost. Therefore, you should also choose an overflow mode when using a limited cache size. The two overflow modes are defined by static fields of the ClientNotificationHandler interface:

We demonstrate each of these modes in our sample manager, first by setting the cache size and the overflow mode, then by triggering more notifications than the cache buffer can hold.

Example 12–7 Controlling the Agent-Side Buffer

System.out.println(">>> Use pull mode with period set to zero, " +
    "buffer size set to 10, and overflow mode set to DISCARD_OLD.");
connectorClient.setMode(ClientNotificationHandler.PULL_MODE);
connectorClient.setPeriod(0);
connectorClient.setCacheSize(10, true); // see "Buffering Specifics"
connectorClient.setOverflowMode(ClientNotificationHandler.DISCARD_OLD);

System.out.println(">>> Have our MBean broadcast 30 notifications...");
params[0] = new Integer(30);
signatures[0] = "java.lang.Integer";
connectorClient.invoke(mbean, "sendNotifications", params, signatures);
System.out.println(">>> Done.");

// Call getNotifications to pull all buffered notifications from the agent
System.out.println("\n>>> Press Enter to get notifications.");
System.in.read();
connectorClient.getNotifications();

// Wait for the handler to process the 10 notifications
// These should be the 10 most recent notifications
// (the greatest sequence numbers)
Thread.sleep(100);
System.out.println("\n>>> Press Enter to continue.");
System.in.read();

// We should see that the 20 other notifications overflowed the agent buffer
System.out.println(">>> Get overflow count = " +
    connectorClient.getOverflowCount());

The overflow count gives the total number of notifications that have been discarded because the buffer has overflowed. The number is cumulative from the first manger-side listener registration until all of the manager's listeners have been unregistered. The manager application can modify or reset this value by calling the setOverflowCount method.

In our example application, we repeat the actions above, to cause the buffer to overflow again, but this time using the DISCARD_NEW policy. Again, the buffer size is 10 and there are 30 notifications. In this mode, the first 10 sequence numbers remain in the cache to be forwarded when the manager pulls them from the agent, and 20 more will overflow.

Buffering Specifics

When the buffer is full and notifications need to be discarded, the time reference for applying the overflow mode is the order in which notifications have arrived in the buffer. Neither the time stamps nor the sequence numbers of the notifications are considered, because neither of these are necessarily absolute; even the sequence of notifications from the same broadcaster can be non-deterministic. And in any case, broadcasters are free to set both time stamps and sequence numbers as they see fit, or even to make them null.

The second parameter of the setCacheSize method is a boolean that determines whether or not the potential overflow of the cache is discarded when reducing the cache size. If the currently buffered notifications do not fit into the new cache size and this parameter is true, excess notifications are discarded according to the current overflow mode. The overflow count is also updated accordingly.

In the same situation with the parameter set to false, the cache will not be resized. You need to check the return value of the method when you set this parameter to false. If the cache cannot be resized because it would lead to discarded notifications, you need to empty the cache and try resizing the cache size again. To empty the cache, you can either pull the buffered notifications with the getNotifications method or discard them all by calling the connector client's clearCache method.

When the existing notifications fit within the new cache size or when increasing the cache size, the second parameter of setCacheSize has no effect.

Because several managers can connect through the same connector server object, it must handle the notifications for each separately. This implies that each connected manager has its own notification buffer and its own settings for controlling this cache. The overflow count is specific to each manager as well.

Buffering Generalities

Here we have demonstrated each setting of the forwarding mechanism independently by controlling the notification broadcaster. In practice, periodic pulling, agent-side buffering and buffer overflow can all be happening at once. And you can call getNotifications at any time to do an on-demand pull of the notifications in the agent-side buffer. You should adjust the settings to fit the known or predicted behavior of your management architecture, depending upon communication constraints and your acceptable notification loss rate.

The caching policy is completely determined by the manager application. If notification loss is unacceptable, it is the manager's responsibility to configure the mechanism so that they are pulled as often as necessary. Also, the notification mechanism can be updated dynamically. For example, the manager can compute the notification emission rate and update any of the settings (buffer size, pull period, and overflow mode) to minimize the risk of a lost notification.

Running the Notification Forwarding Example

The examplesDir/Notification directory contains all of the files for the broadcaster MBean, the BaseAgent application, and our Client application that is the listener object. To run the notification forwarding example, we use the BaseAgent application that contains an RMI connector server.

To Run the Notification Forwarding Example

  1. Compile all files in this directory with the javac command.

    For example, on the Solaris platform with the Korn shell, type:


    $ cd examplesDir/Notification/
$ javac -classpath classpath *.java

  2. Start the agent on another host or in another terminal window with the following command.

    Be sure that the classes for the NotificationEmitter MBean can be found in its classpath:


    $ java -classpath classpath BaseAgent

  3. Wait for the agent to be completely initialized, then start the manager in another window with the following command, where hostname is the name of the host running the agent.

    If you started the agent on the same host, you can omit the hostname:


    $ java -classpath classpath Client hostname

    When started, the manager application first creates the NotificationEmitter MBean and then registers itself as a listener.

  4. Press Enter when the application pauses to step through the various notification forwarding situations that we have seen in this topic.

  5. Press Enter again in the manager window to exit the application.

    Leave the agent application running if you want to interact with the example through the HTML protocol adaptor of the BaseAgent.

To Interact With the Notification Forwarding Mechanism

  1. Start the manager in another window with the following command, where hostname is the name of the host running the agent.

    If you started the agent on the same host, you can omit the hostname:


    $ java -classpath classpath Client hostname

  2. Load the following URL in your browser and go to the MBean view of the NotificationEmitter MBean:

    If you get an error, you might have to switch off proxies in your browser preference settings. Any browser on your local network can also connect to this agent using this URL.

  3. When the manager application pauses for the first time, call the sendNotifications method from your browser with a small integer as the parameter.

    The listener handles your notifications in the manager's terminal window. Because the manager is still in push mode, they are forwarded immediately.

  4. Press Enter in the manager window.

    The manager is now in pull mode with a pull period of 500 milliseconds.

  5. Through the MBean view, send 1000 notifications.

    If your agent's host is slow enough, or if your manager's host is fast enough, you might be able to see the manager pause briefly after it has processed all notifications from one period and before the next ones are forwarded.

  6. Press Enter in the manager window.

    The agent now forwards notifications by request.

  7. Through the MBean view, send 15 notifications, then press Enter again.

    The manager pulls all of the notifications: the 30 triggered by the manager and the 15 we just triggered. They were all kept in the buffer, waiting for the manager's request to forward them. Remember that the sendNotifications operation resets the sequence numbering every time it is invoked.

  8. Press Enter in the manager's window.

    The cache size is set to 10 notifications and the overflow mode to DISCARD_OLD.

  9. Through the MBean view, send 15 more notifications, then press Enter again.

    Only the last 10 of our notifications fit into the cache buffer. All the rest, including those already triggered by the manager, overflow and are discarded.

  10. Press Enter to see that the discarded notifications are tallied in the overflow count.

  11. Press Enter in the manager's window.

    The cache size is still 10 notifications and the overflow mode is set to DISCARD_NEW.

  12. Through the MBean view, send only 5 more notifications, then press Enter again.

    The first 10 of the manager-triggered notifications are received. All of the more recent notifications, including ours, overflow the cache buffer and are lost.

  13. Press Enter to see that the lost notifications are tallied in the overflow count:

    35 from Step 12 plus 25 more from this step, for a total of 60.

  14. Press Enter in the manager's window again to stop the Client application.

  15. Press Enter in the other window to stop the agent application when you have finished running the example.