System Monitoring

C H A P T E R 1

System Monitoring

Content Delivery Server provides the Application Monitoring Agent for monitoring the state of Content Delivery Server. This agent supports Simple Network Management Protocol (SNMP) and can be integrated with your existing network monitoring system.

Content Delivery Server components also record information in log files that you can use to review the state of the system.

This chapter includes the following topics:

Application Monitoring Agent

Log Files

Synchronization

Search Indexes

1.1 Application Monitoring Agent

The Application Monitoring Agent provides information on the status of Content Delivery Server. This agent enables your existing network monitoring system to monitor the availability of the Content Delivery Server components. You can also monitor the connections between Content Delivery Server and external systems such as your billing system or subscriber database. The agent is activated when the Monitoring Service is started.

The Application Monitoring Agent provides a Management Information Base (MIB) that describes the statuses and alarms issued by Content Delivery Server. This information is contained in the CDSMibModule.mib file in the $CDS_HOME/deployment/deployment-name/conf directory. Configure your network monitoring system to use this MIB specification. See the product documentation for the network monitoring system that you are using for information on loading the MIB.

TABLE 1-1 shows the status information provided through the MIB. Your network monitoring system can query the Application Monitoring Agent for any of these statuses. A status of 1 indicates that the component is running. A status of 0 indicates that the component is down.

TABLE 1-1 Component Status
Status	Description
`cdsServerStatus`	Status of the Content Delivery Server core server
`eventServiceStatus`	Status of the Content Delivery Server Event Service
`messagingServiceStatus`	Status of the Content Delivery Server Messaging Service
`postPaidServiceStatus`	Status of the Content Delivery Server Postpaid Service

TABLE 1-2 shows the notifications sent by Content Delivery Server to report that an event occurred. In general, a notification indicates that a particular component is unavailable or that a component started. If a component is unavailable, determine the cause and restart the component to ensure that Content Delivery Server continues to operate correctly.

TABLE 1-2 Notifications Sent by Content Delivery Server
Alarm	Description
`catalogJMSDown`	Java Message Service (JMS) associated with the Catalog Manager is not running.
`catalogStarted`	Catalog Manager started.
`dbConnectionDown`	Connection to the database is down.
`downloadFailure`	Download failure occurred while attempting to download content to a subscriber. The subscriber might have cancelled the download, the network might be encountering problems, or there might be a problem with the download process.
`httpPushConnectionDown`	Connection to the HTTP-based SMSC is down.
`missingExternalContent`	Externally hosted content is not available from the remote location. The URL might be invalid, the developer might need to submit an updated location for the item, or a firewall might be preventing access to the item.
`newVendingAccountRegistered`	Catalog Manager administrator registered a new Vending Manager Server account.
`notificationProcessingFailure`	The Vending Manager failed to process a notification of change from the Catalog Manager.
`ppgConnectionDown`	Connection to the Push Proxy Gateway (PPG) is down.
`remoteVendingDown`	Catalog Manager noticed that the Vending Manager is not running.
`smscConnectionDown`	Connection to the Short Messaging Service Center (SMSC) is down.
`smtpConnectionDown`	Connection to the Simple Mail Transfer Protocol (SMTP) server is down.
`streamingConnectionDown`	Connection to the streaming server is down.
`subscriberDirectoryConnectionDown`	Connection to the external subscriber database is down.
`vendingAccountUnregistered`	Catalog Manager administrator removed the Vending Manager Server account.
`vendingJMSDown`	JMS associated with the Vending Manager is not running.
`vendingNetworkIdUpdated`	Catalog Manager administrator changed the network ID for a Vending Manager Server account.
`vendingStarted`	Vending Manager started.

1.2 Log Files

As Content Delivery Server executes, the components record information in the following log files, which are found in the $CDS_HOME/deployment/deployment-name/logs directory:

CDS.log.n. Contains messages generated by the Content Delivery Server components.

n is a value from 0 to the maximum number of archive files specified for the cds.FileHandler.count property in the $CDS_HOME/cfg/logging.properties file. CDS.log.0 is the current log. An archive file is created when Content Delivery Server is started or when the maximum number of records specified for the cds.FileHandler.limit property in the logging.properties file is reached.

cdsdb.log. Contains messages generated by the Database Encoding Check command.

Database Encoding Check is performed when Content Delivery Server is started using the cdse startserver command.

db.log. Contains messages generated when running the cdsi db command.

Review the log files for exceptions, lines flagged as SEVERE, or other indications of problems. See the Sun Java System Content Delivery Server Error Messages for descriptions of messages that you might receive and suggested actions that you can take in response.

Content Delivery Server uses the logging utility that is available with the Java Software Development Kit. You can manage the logging files according to the capabilities of the utility by setting properties in the $CDS_HOME/cfg/logging.properties file. This file controls the logging properties for all Content Delivery Server functions.

The messages that are written to the log are determined by the level of logging that is set. The following levels are valid in order from most restrictive to least restrictive.

TABLE 1-3 Valid Levels of Logging
Label	Description
`OFF`	No messages are logged.
`SEVERE`	Only messages indicating severe problems are logged.
`WARNING`	Warnings are logged.
`INFO`	Informational messages are logged.
`CONFIG`	Static configuration messages are logged.
`FINE`	Tracing information is logged.
`FINER`	Fairly detailed tracing information is logged.
`FINEST`	Highly detailed tracing information is logged.
`ALL`	All messages are logged.

The log includes messages at the specified level and messages at higher levels. For example, if the level is set to WARNING, the log contains messages marked as SEVERE and WARNING. Content Delivery Server uses only SEVERE, WARNING, INFO, and FINE for its messages.

The logging utility supports several handlers. Content Delivery Server uses only the file handler and the console handler. In addition, several components of Content Delivery Server log messages. The level of messages logged by these loggers can be set individually, if desired. The following table contains the names of the loggers used by Content Delivery Server.

TABLE 1-4 Loggers in Content Delivery Server
Admin	EventServiceLog	PricingModel	SubscriberV1
BillingManager	EventService_config	PricingPlan	SubscriberXmlRpc
Cache	Foundation	RAS	Synchronization
CapabilityMgr	Fulfillment	Reporting	SynchronizationAPI
Catalog	I18NConverter	RiMatcher	Task
ConfirmListener	I18n	Root	TaskMgr
ContentManager	Jobs	SearchService	UserManager
DBAccess	LDAPUserManager	Security	VmAPI
DBCommand	Mms	ServicePlanManager	VmAPIXmlRpc
DBPoolManager	MsgServer	SnmpLog	Watermarking
DRM	MultiPart	Stock	Web
DRMMobile	OMA	Streaming	Webapps
DeveloperPortal	OTA	SubscriberAPI	Webresource
EMF	Pricing	SubscriberPortal

To manage the log files, set the properties in the $CDS_HOME/cfg/logging.properties file as described in the following table.

TABLE 1-5 Logging Properties
Property	Description
`cds.level`	Default level of information written to the log. This level is used if no level is set for a specific logger. Valid values are described in TABLE 1-3. The default is `INFO`.
`cds.FileHandler.level`	Level of information written to the file.
`cds.FileHandler.pattern`	Default location of the log file. This location is used if the component that creates the log does not specify a location. The default is `/tmp/CDS.log`. Note - Content Delivery Server logs are in the `$CDS_HOME/deployment/`deployment-name`/logs` directory. Adapters that you write or other integration code might write logs to this directory if another location is not specifically stated.
`cds.FileHandler.limit`	Maximum number of records written to the log before an archive file is created. The default value is `10000000`.
`cds.FileHandler.count`	Maximum number of archive files. The default value is `50`.
`cds.FileHandler.formatter`	Fully qualified name of the class used to format the messages. The default is `com.sun.content.server.foundation.log.pattern.PatternFormatter`.
`cds.ConsoleHandler.level`	Level of information written to the console.
`cds.ConsoleHandler.formatter`	Fully qualified name of the class used to format the messages. The default is `com.sun.content.server.foundation.log.pattern.PatternFormatter`.
`com.sun.content.server.foundation.log.pattern.PatternFormatter.pattern`	Pattern used by the class specified for the File Handler and Console Handler formatters to format the messages. The following elements can be used: `%c` - Logger category name `%C` - Name of the class that generated the log event `%d` - Date the event was generated `%m` - Message provided by the event `%M` - Name of the method that generated the log event `%n` - New line `%p` - Logging level for the event `%t` - Thread ID The default pattern is `%d [%c,%t] %p %m%n`, which generates a message similar to the following sample: Tue May 01 16:58:47,608 PDT 2007 [cds.DRM,20] SEVERE <0052> Purchase authorization failed. Note - If you specify a different class for the formatters, a different property might be needed.
`cds.`logger`.level`	(Optional) Level of logging for a specific logger where logger is one of the values in TABLE 1-4.

1.3 Synchronization

The Catalog Manager and each Vending Manager has its own database. Information such as the status of content or the types of devices supported must be maintained across databases. As changes are made in the Catalog Manager, notifications are sent to the Vending Managers identified by the Vending Manager Server accounts.

If a Vending Manager is not receiving notification of changes or if synchronization fails, verify that the Vending Manager Server account contains the correct account name and password for the Vending Manager. If needed, correct the name or password in the Vending Manager Server account using the Catalog Manager administration console. You do not need to restart the Vending Manager if only the Vending Manager Server account is changed. If you change the vs.account.name or vs.account.password property in the $CDS_HOME/deployment/deployment-name/conf/CDS.properties file, you must restart the Vending Manager. You must also restart the Catalog Manager if it is deployed with the Vending Manager in a basic deployment.

If a Vending Manager Server account is deleted or inactivated, the Catalog Manager stops sending notifications of changes to that Vending Manager. If the Vending Manager Server account is later recreated or reactivated, the Vending Manager must be restarted to initiate synchronization. You must also restart the Catalog Manager if it is deployed with the Vending Manager in a basic deployment.

Additional information about the synchronization between the Catalog Manager and Vending Managers is available in Chapter 2 of the Sun Java System Content Delivery Server Reference Manual.

1.4 Search Indexes

The Catalog Search Service and the Vending Search Service components of Content Delivery Server use the Solr search server to index content and handle search queries. For each service, Solr creates an index for content in the Catalog Manager and the Vending Manager. The search indexes are managed independent of the content databases.

As content is added or changed, the search indexes are automatically updated. However, it is possible for the indexes to become out of sync with the content and require a manual rebuild. Some of the situations in which a manual rebuild is necessary are described in the following list:

If content is added between the time a custom field is defined and the time the field is added to the schema.xml file for the search service, the data for the new field is not included in the search index.

If a deployment is redirected to a different database, the search index contains data for the previous index.

If a database is reinitialized, the search index still contains data for content that was previously in the database.

Rebuild the search index when one of the situations described occurs or when you are not getting the results you expect from search queries. You can rebuild the entire index or you can add a specific item to an index. You can also remove an item from the index or remove the entire contents of an index in preparation for starting over if rebuilding intermittently does not solve your problem.

1.4.1 Rebuilding the Entire Index

Be aware that rebuilding the entire index can take awhile, depending on the number of content items and the load on the system. You might also want to avoid rebuilding the index while other resource-intensive operations are running, such as bulk content submission, bulk content management, or device capability matching.

To rebuild the search index for the Catalog Manager, log in to the Catalog Manager administration console and enter the following URL in the address field of your browser:

http://host:port/admin/main/search_update?action=rebuild

host:port is the host name and port number for the server on which the Catalog Manager is running. A message is shown when the rebuilding process completes.

To rebuild the search index for the Vending Manager, log in to the Vending Manager administration console and enter the following URL in the address field of your browser:

http://host:port/vsadmin/main/search_update?action=rebuild

host:port is the host name and port number for the server on which the Vending Manager is running. A message is shown when the rebuilding process completes.

1.4.2 Adding an Item to the Index

Adding an item to the index takes less time than rebuilding the entire index and is an option when you know the class ID of a specific item that is not appearing in query results.

To add a specific item to the index for the Catalog Manager, log in to the Catalog Manager administration console and enter the following URL in the address field of your browser:

http://host:port/admin/main/search_update?action=add&rcid=content-class-id

host:port is the host name and port number for the server on which the Catalog Manager is running and content-class-id is the class ID of the content item. A message is shown when the process completes.

To add a specific item to the index for the Vending Manager, log in to the Vending Manager administration console and enter the following URL in the address field of your browser:

http://host:port/vsadmin/main/search_update?action=add&rcid=class-id

host:port is the host name and port number for the server on which the Vending Manager is running and class-id is the class ID of the content item. A message is shown when the process completes.

1.4.3 Removing All Entries in the Index

If rebuilding the index does not resolve issues with incorrect search results, you can delete everything in the index and then rebuild the index as described in Section 1.4.1, Rebuilding the Entire Index.

To remove everything from the search index for the Catalog Manager, log in to the Catalog Manager administration console and enter the following URL in the address field of your browser:

http://host:port/admin/main/search_update?action=delete_all

host:port is the host name and port number for the server on which the Catalog Manager is running. A message is shown when the process completes.

To remove everything from the search index for the Vending Manager, log in to the Vending Manager administration console and enter the following URL in the address field of your browser:

http://host:port/vsadmin/main/search_update?action=delete_all

host:port is the host name and port number for the server on which the Vending Manager is running. A message is shown when the process completes.

1.4.4 Removing an Item From the Index

If an item appears in search results that you know should not be there and you know the class ID of the item you can manually remove that item from the search index.

To remove a specific item from the index for the Catalog Manager, log in to the Catalog Manager administration console and enter the following URL in the address field of your browser:

http://host:port/admin/main/search_update?action=delete&rcid=class-id

host:port is the host name and port number for the server on which the Catalog Manager is running and class-id is the class ID of the content item. A message is shown when the process completes.

To remove a specific item from the index for the Vending Manager, log in to the Vending Manager administration console and enter the following URL in the address field of your browser:

http://host:port/vsadmin/main/search_update?action=delete&rcid=class-id

host:port is the host name and port number for the server on which the Vending Manager is running and class-id is the class ID of the content item. A message is shown when the process completes.