Skip Headers
Oracle® Enterprise Manager Cloud Control Administrator's Guide
12c Release 1 (12.1.0.1)

Part Number E24473-01
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
PDF · Mobi · ePub

31 Installation to Support Real-time Configuration Change Monitoring

The Enterprise Manager Compliance features include the ability to monitor certain elements of your targets in real time to watch for configuration changes or actions that may result in configuration changes.

These features include Operating System level file change monitoring, process starts and stops, Operating System user logins and logouts, Oracle database changes and more.

The real-time monitoring for these features takes place from the Enterprise Manager agent. Some of these monitoring capabilities require specific setup steps depending on the type of monitoring you will do and what Operating System is being monitored.

This section outlines the specific requirements and pre-requisites that exist to use the Compliance Real-time Monitoring features.

Real-Time Monitoring

Real-time monitoring is configured through the Enterprise Manager Server. Users with the EM_COMPLIANCE_DESIGNER role create Compliance Standard Rules that are of type “Real-time Monitoring Rule.” These rules are then associated with Compliance Standards and these standards are subsequently associated with one or more targets.

After the Compliance Standard to target association is complete, the set of monitoring rules are sent to the agent to enable real-time monitoring.

Resource Consumption Considerations

The Real-time monitoring features are built into the Enterprise Manager agent. There are some specific resource considerations if you use the Real-time monitoring features. The following sections describe issues you should consider when using Real-time monitoring features.

OS File Monitoring Archiving

An optional setting when monitoring for file changes in real time is to make an archive copy of the file on the agent. When monitoring first begins, a copy of the file at that time is made and stored into a private directory in the ORACLE_HOME directory of the agent. Then, any subsequent changes to that file will result in additional copies of the file being archived in that same directory. This feature allows you to later perform a file diff from the user interface or to issue a job to roll back a file to a previous version.

This feature however will use disk space to make copies of the file. Care should be taken to ensure that this feature is only enabled for files that are critical.

OS File Read Monitoring

The Operating System File level monitoring can monitor many types of changes to files, but can also monitor reads to files. If you have a Rule to monitor a facet that has file patterns that are read frequently, this may result in a very large number of observations. You can reduce the number of observations by ensuring that your Rule includes a filter on either time or a user that you want to ensure does not read the file.

For instance, monitoring the /etc/passwd file for reads for All users will result in many observations being created. However, if you only monitor the /etc/passwd file by a specific user, then you will only receive an observation when that specific user attempts to read the file.

Creating Facets That Have Very Broad Coverage

It is possible to create a facet for any entity type (Operating System File for instance) that monitors every file on a host. This will result in increased overhead on the agent as well as significant numbers of observations coming into the Enterprise Manager Server. It is important to remember that facets are created to specify files that are very important to monitor for security/compliance purposes. For instance, monitoring all modifies to a log file that change every few seconds will add a great deal of resource usage to the agent and server. Instead, in this case, it may be appropriate to create a rule to monitor the log file for all changes, but filter only when the log change is made by a non-application user. This would only capture the log file change if a regular user attempted to change or tamper with the log rather than when the log is simply being updated by an application.

Enterprise Manager Repository Sizing

Database sizing considerations for Real-time monitoring depend on several factors. The most important factor is the number of observations expected in a month. The second factor is the number of months data will be retained in the repository. Repository retention rates are explained later in this chapter.

In general, each observation consumes roughly 1.5KB of space in the database. This is a guideline and this number can vary depending on many factors for each installation.

If a customer expected a total of 10 million Real-time observations per month across all targets and wanted to retain the data for 12 months, then the database size required for this would be roughly 180GB.

10,000,000 Observations x 12 Months x 1500 Bytes = 180,000,000,000 Bytes

This size represents Real-time monitoring data only and does not include database storage needs for other areas of Enterprise Manager.

The number of observations to expect per month can vary from environment to environment and can also depend on what types of monitoring are configured. You may be required to tune the expected size over time after Rules and Facets have been enabled for some time and configured to fit the organizations requirements. You can easily find your observations usage over a month by using the Real-time Observations > Browse By Systems screen to select your systems and see the related counts of observations for each system over a period of time.

Configuring Monitoring Credentials

Many of the real-time monitoring capabilities require monitoring credentials that maintain the ability to launch monitoring programs with root privileges. These processes begin with the prefix nmxc. Low-level monitoring uses operating system APIs that are not available to regular users.

Before starting to use the Real-time monitoring features on a target host for the first time, the following settings must be configured from the Enterprise Manager Console.

  1. Ensure that root.sh script is run after agent installation

    After installing the agent, the root.sh script must be run as the root user. This script must be run before configuring the rest of these credential steps.

  2. Configuring Privilege Delegation

    Privilege Delegation settings are found from the Setup Menu, Setup > Security > Privilege Delegation. On this page, you can either set privilege delegation for each host manually or you can create a Privilege Delegation Setting Template.

    Privilege delegation for each host that will have real-time monitoring must have SUDO setting enabled with the appropriate SUDO command filled in (for example, /usr/local/bin/sudo).

  3. Configuring Monitoring Credentials

    Monitoring Credential settings are found from the Setup Menu, Setup > Security > Monitoring Credentials. From this page, select the Host target type and click on the Manage Monitoring Credentials button.

    For each entry with the credential “Host Credentials For Real-time Configuration Change Monitoring“ set, select the entry and click on the Set Credentials button. You will be asked for a credential set to use. Ensure you also add “root” to the Run As entry. If “Run As” is not visible, then the privilege delegation was not set properly in the previous step.

    To set monitoring credentials in bulk on multiple hosts at once, you can use EMCLI. For more information on using EMCLI to set monitoring credentials, see the section, Managing Credentials Using EMCLI in the Security chapter of Oracle Enterprise Manager Administration. Likewise, for more information about configuring monitoring credentials in Oracle Enterprise Manager, the Security chapter of Oracle Enterprise Manager Administration.

Preparing To Monitor Linux Hosts

The following sections describe how to prepare Linux hosts for monitoring.

OS File Monitoring

Before using Real-time file monitoring for Linux, a loadable kernel module must be installed on the host. This loadable kernel module provides you with the most efficient way of monitoring the host. This loadable kernel module is referred to as the File Audit Module, or Audit Module for short.

Acquiring the Kernel Module

The kernel audit module is available from http://oss.oracle.com/projects/fileauditmodule. There are two ways to get the file audit kernel module:

  1. Prebuilt .ko files. for which Oracle has already prebuilt, you can use this in your environment. You can look for the Prebuilt kernel modules under the Downloads link. To find the matching prebuilt version, run the uname -r command on the host being monitored and compare that version to the version of the prebuilt modules. The complete version string must match perfectly.

  2. Build your own kernel module. To build your own kernel module, you can download the following RPM from the Downloads link:

    Fileauditmodule-version-noarch.rpm

    You should always retrieve the latest version available at the time you are installing this module.

    Install this RPM on the host you want to monitor as root. The installation of this RPM depends on the kernel-devel package matching your running kernel also existing on the host. This kernel-devel package comes with the same media as the Linux installers.

    In addition to installing this package, you must ensure that the version of gcc available on your host matches the version with which the kernel was built. To do this, view the /proc/version file to see what gcc version the kernel was built with and then run the command gcc –version to see what version of gcc is being used. These two versions should match.

    Also check that the file /boot/System.map-{version} exists where {version} must match the kernel version you see when you run the uname -r command. This file contains system symbols that are required to decode the kernel symbols we are monitoring for real-time changes. Without this file, real-time file monitoring will not function. This file is standard on all default Linux installations.

    After installing this package and checking prerequisites successfully, go to the directory where the package contents were installed (defaults to /opt/fileauditmodule) and run the compmod.sh script. This will build the kernel module file (.ko, .k64, or .o extension depending on the OS version) and place it in the /opt/fileauditmodule directory.

    If the audit module file is not created, check the make.log and build.log files for any errors in building the module.

If all of your hosts have the exact same kernel version as shown using the command uname –r, then you only need to compile the module on one machine. You can then copy the .ko, .k64, or .o file to the other servers without having to build on that specific host.

Deploying the Kernel Module

Once you have either the prebuilt .ko file or a .ko file that exists from building it from the source RPM, the .ko file must be located in the proper directory. The default location for this file is in the bin folder under the agent home directory. You can also place the file in any location on the host and change the nmxc.properties file under the AGENT_INST/sysman/config directory of the agent home. The property nmxcf.kernel_module_dir specifies the absolute path to the .ko directory.

Install Kernel Module Job

In addition to manually placing the .KO file on the agent, there is an Enterprise Manager job named CCC Kernel Module Installation. This job is configured with a list of Linux hosts on which you can install the kernel module. It will search in a directory locally on the Enterprise Manager server disk for prebuilt .ko files or the source RPM file. If it finds a matching prebuilt .ko file, it will send this to the matching agents; otherwise it will send the RPM to the agent and install it resulting in a new .KO file.

Prior to using this job, files from OSS.ORACLE.COM must be manually retrieved by the user and placed into the %ORACLE_HOME%/gccompliance/fileauditmodule/resources/linux directory. This directory already exists on the server with a README file indicating this is the location to place these files. The files that must be placed here are either prebuilt .KO files or the source RPM file. If you have built your own .KO files in your environment, you can also place those .KO files into this directory on the server and deploy it to other hosts in your environment.

Special Considerations for Enterprise Linux 5 and Greater

For Enterprise Linux 5 and greater, the kernel audit module is not required. The monitoring will use the built-in audit subsystem. However, the functionality of the audit subsystem is not as robust as the capability that the kernel audit module can provide.

You will lose the functionality that provides the granularity of what type of change there has been to a file, whether it was a create action or a modify action. Without the kernel module, all changes to a file will appear as a modify action.

It is recommended that you use the kernel audit module even with the newer versions of Linux, if possible.

Debugging Kernel Module Issues

You may detect a problem with the kernel module in a few different ways:

  1. You may have noticed that you do not receive real-time file changes on the Enterprise Manager console for file changes that you know should occur.

  2. In the Compliance Standard Target Associations page on the user interface, you may see an agent warning indicating a kernel module problem.

  3. When examining the nmxcf.log file under AGENT_INST/sysman/logs, you may see errors indicating that the kernel module could not be loaded or used for various reasons.

If you encounter any of these issues, most likely there was a problem with compiling or inserting the Linux kernel module at run time.

You can confirm whether the auditmodule was loaded properly by running the following command.

grep -i auditmodule /proc/modules

If you do not get any output, then the auditmodule is not loaded and the agent will not be able to do real time file monitoring.

If the audit module file was generated properly and it does not show up in the module list above, you can attempt to manually load the module to see if there are any errors. Use the following command where you replace {audit module file name} with the entire name of the .ko file that was created from compmod.sh:

insmod {audit module file name}

If you experience no errors during this command, you can check the module list again by using the grep command above. If the audit module now appears, then the file monitoring capability should work. An agent restart is not necessary; however there still may be a problem with the file monitoring process finding the .ko file which you will experience again next time your host is rebooted.

If the module still is not able to load and if you need to contact Oracle support about the issue, please be sure to include the following information with your support ticket:

  • Output of the command: uname -a

  • Output of the command: grep –i auditmodule /proc/modules

  • Output of the command: rpm –q –a |grep –i kernel-devel

  • The make.log and build.log files from the /opt/fileauditmodule directory where you ran compmod.sh if you built your own .ko file

  • The files AGENT_INST/sysman/logs/nmxc*.log

This information will help Oracle Support to determine if the real time file monitoring audit module of the agent can be built on your environment.

Preparing To Monitor Windows Hosts

The Real-time monitoring modules for Windows rely on various capabilities of the operating system to collect all of the information on actions. One part of this is to capture the user that made changes from the Windows Event Log. If you do not configure Windows to capture users that make changes, the agent will not capture this information. However it will still capture that a change occurred and when it occurred.

To configure the event log to work with real time monitoring, perform the following steps:

  1. From Windows Explorer, select the directory that is being monitored by a Real-time Monitoring Rule, right-click and select Properties

  2. Go to the Security tab

  3. Click the Advanced button

  4. Select the Auditing tab

  5. Click the Add button. (In Microsoft XP, double click the Auditing Entries window)

  6. Select the Name Everyone and click OK. You can also choose specific users if you are only monitoring for changes by specific users in Configuration Change Console rules. The rules filter the results by user as well, so even if you enable audit for everyone, only users that you want to monitor changes of in your rules will be captured

  7. Select the following options (Successful and/or Failed) from the Access window:

    • Create Files/Write Data

    • Create Folders/Append Data

    • Delete Files Subfolders and Files

    • Delete

  8. Click OK to exit out of the screen

  9. Repeat steps 1 through 7 for all other monitored directories and/or files

  10. Go Start --> Settings --> Control Panel --> Administrative Tools --> Local Security Policy --> Local Policies --> Audit Policy. Double-click, and turn on the following policies (Success and/or Failure):

    • Audit account logon events

    • Audit logon events

    • Audit object access

  11. Close the Local Security Settings screen

  12. Go to Start --> Settings --> Control Panel -->Administrative Tools --> Event Viewer

  13. Select System Log, and click on Action from the menu bar and select Properties

  14. From the System Log Properties panel, on the General tab, set the Maximum log size to at least 5120 KB (5 megabytes) and select Overwrite Events as Needed. Note that the log size depends on the number of events generated in the system during a two-minute reporting interval. The log size must be large enough to accommodate those events. If you extend the monitoring time for file events because you expect the change rate to be lower, you need to ensure that the audit log in Windows is large enough to capture the events.

  15. Click Apply and OK to exit.

If Windows auditing is not configured properly, you will see warnings on the Compliance Standard Target Association page on the Enterprise Manager user interface. This is the same page where you associated your Real-time Monitoring compliance Standards to your targets.

Verifying Auditing Is Configured Properly

To verify that the host records login and logout events, follow these steps:

  1. Log out of the host and then log back into the host.

  2. Go to Start --> Settings --> Control Panel --> Administrative Tools --> Event Viewer

  3. Select Security Log and go to View --> Filter. Select Security for the Event Source and Logon/Logoff for the Category fields

  4. Click OK

The Event Viewer should have the activity recorded as Event 528.

Subinacl External Requirements

As mentioned earlier, the agent will send warnings to the server when audit settings are not set properly. It, however, can only do this if the windows feature SUBINACL is installed. If this feature is not installed on the host, a warning will be sent to the server saying that the agent cannot detect whether audit settings are correct. This warning will be visible from the Compliance Standard Associate Targets page.

You can specify the absolute path to the directory that contain subinacl by setting the following property in the AGENT_INST/sysman/config/nmxc.properties file:

nmxcf.subinacl_dir=

SubInACL is available for download from Microsoft.com

Preparing To Monitor Solaris Hosts

Real-time monitoring on Solaris systems utilizes the Solaris audit system which is part of the Solaris Basic Security Model (BSM). BSM auditing allows system administrators to monitor events and to detect user account logins and logouts as well as file changes.

Verify that BSM auditing is enabled by running the following command:

/usr/sbin/auditconfig –getcond

You should see the following output:

audit condition = auditing

If you need to enable BSM auditing, you can use the following command:

/etc/security/bsmconv

See the Solaris BSM Auditing manuals for additional details on setting up BSM auditing.

If auditing is already enabled on the server, simply verify that the audit system configuration matches the configurations detailed below.

The audit file can be configured to include specific events. The /etc/security/audit_control file controls which events will be included in the audit file. This section summarizes the configuration; for further details, refer to the Sun Product Online Documentation site.

For monitoring entity types OS FILE (file changes) and OS USER (user logins/logouts), the flags line in the file /etc/security/audit_control should be set as follows:

flags: +fw,+fc,+fd,+fm,+fr,+lo

This configuration enables success/fail auditing for file writes (fw), file creates (fc), file deletes (fd), file attribute modifies (fm), file reads (fr) and login/logout events (lo); where '+' means to only log successful events.

If you are interested in logging the failed events as well, remove the "+" sign before each event in the flag.

Note:

Installing BSM on an existing host has the requirement that the host is rebooted.

Auditing Users

The audit_user file controls which users are being audited. The settings in this file are for specific users and override the settings in the audit_control file, which applies to all users.

Audit Logs and Disk Space

The audit_control file also has entries to control where the audit logs are stored, and the maximum amount of disk space used by the audit system. The minimum requirement for file monitoring is approximately 10 minutes worth of data stored on the hard drive or the configured reporting interval time.

Managing Audit Files

Enterprise Manager Real-time Monitoring only reads the audit logs; it does not delete the logs. This might flood the system with log files and prevent it from logging additional events. To manage and delete old audit events while maintaining minimum monitoring requirements, do the following:

  1. The auditing policy can be set to automatically drop new events (keeping only a count of the dropped events) rather than suspending all processes by running the following command:

    # auditconfig -setpolicy cnt

  2. Run the following command to force the audit daemon to close the current audit log file and use a new log file.

    /usr/sbin/audit -s

  3. Run the following command to merge all existing closed auditing log files into a single file with an extension of .trash and then delete the files.

    /usr/sbin/auditreduce -D trash

  4. Create a cron job to periodically run the commands in step 2 and 3 above. The frequency at which these two commands are run can be adjusted based on the anticipated event volume and the amount of disk space allocated to auditing. The only requirement is that the time between the audit -s command and the auditreduce - D trash command is at least 15 minutes or twice the reporting interval if that is changed.

Preparing To Monitor the Oracle Database

This section describes the steps involved in setting up auditing within an Oracle database. If you are going to monitor an Oracle database with any of the Oracle entity types, you will need to perform these steps before events will be captured.

Before configuring auditing it is suggested you review the Auditing Database Use section of the Oracle Database Administrator's Guide. This document provides an overview of Oracle's auditing functionality, as well as basic concepts and guidelines for auditing configurations. Note that this document does not cover all details of configuring and fine tuning the Oracle audit system. Instead, this document serves as an example of the basic steps involved to configure the Oracle audit system to enable Real-time monitoring through Real-time monitoring rules.

Setting Auditing User Privileges

When you create a Real-time Monitoring Compliance Standard Rule to monitor an Oracle instance target, the agent will read the audit trail to perform its monitoring.

Real-time monitoring for Oracle entity types requires the audit trail to be stored in the database as opposed to a file. To verify if a setting is correct, follow these steps:

  1. In Enterprise Manager, go to the target home page for the Oracle Database target for which you want to enable Real-time Monitoring.

  2. Click on the Administration menu and then choose Initialization Parameters.

  3. Log in to the database as a sys user, connecting as SYSDBA.

  4. Find the parameter audit_trail and ensure it is set to DB. If not, this parameter needs to be changed in the Oracle Database.

  5. This change will require a restart of the database.

Specifying Audit Options

Through SQL plus, an Oracle DBA can use audit and noaudit statements to configure audit options for the database. The audit statement allows you to set audit options at three levels:

Table 31-1 Audit Options Table

Level Effect

Statement

Audits specific SQL statements or groups of statements that affect a particular type of database object. For example, AUDIT TABLE audits the CREATE TABLE, TRUNCATE TABLE, COMMENT ON TABLE, and DELETE [FROM] TABLE statements.

Privilege

Audits SQL statements that are executed under the umbrella of a specified system privilege. For Example, AUDIT CREATE ANY TRIGGER audits statements issued using the CREATE ANY TRIGGER system privilege.

Object

Audits specific statements on specific objects, such as ALTER TABLE on the employee table


To use the audit statement to set statement and privilege auditing options a DBA must be assigned AUDIT SYSTEM privileges. To use the audit statement to set object audit options, the DBA must own the object to be audited or be assigned the AUDIT ANY privilege within Oracle. Privilege assignments are covered in the following section.

Audit statements that set statement and privilege audit options can also include a BY clause to supply a list of specific users or application proxies to audit, and thus limit the scope of the statement and privilege audit options.

Some examples of audit statements can be seen below. Feel free to use these as a basis for the audit settings you specify within your database. Once all audit settings are in place you can create application policies, using the Oracle (SQL Trace) agent module with which to monitor the Oracle database instance.

Statement Audit Options (User sessions)

The following statement audits user sessions of users Bill and Lori.

AUDIT SESSION BY scott, lori;

Privilege Audit Options

The following statement audits all successful and unsuccessful uses of the DELETE ANY TABLE system privilege:

AUDIT DELETE ANY TABLE BY ACCESS WHENEVER NOT SUCCESSFUL;

Object Audit Options

The following statement audits all successful SELECT, INSERT, and DELETE statements on the dept table owned by user jward:

AUDIT SELECT, INSERT, DELETE ON jward.dept BY ACCESS WHENEVER SUCCESSFUL;

Example Oracle Audit Monitor Configurations

The following command audits all basic statements. Extra statements are not audited.

Audit all by access;

The following statement audits all extra statements:

audit ALTER SEQUENCE, ALTER TABLE, DELETE TABLE, EXECUTE PROCEDURE, GRANT DIRECTORY, GRANT PROCEDURE, GRANT SEQUENCE, GRANT TABLE, GRANT TYPE, INSERT TABLE, LOCK TABLE, UPDATE TABLE by access;

The following command displays audit settings for statements:

SELECT * FROM DBA_STMT_AUDIT_OPTS;

Once you have specified your audit configuration you can then create real-time monitoring rules from the Enterprise Manager Server that uses the Oracle Database entity types.

Setting Up Change Request Management Integration

This section explains how to install and configure integration with a Change Management Server and to be able to determine whether changes that occur are authorized automatically.

BMC Remedy Action Request System 7.1 Integration

Remedy ARS 7.1 is a supported Change Management system for automatic reconciliation of observations. The following steps outline how to setup Remedy and also configure the integration with Enterprise Manager.

Remedy Installation and Customization

Follow these steps to install and customize Remedy ARS 7.1.

  1. Install Remedy ARS 7.1. Ensure the following components are all installed and properly licensed:

    ARS 7.1.00 Patch 011

    Midtier 7.1.00 Patch 011

    Flashboard Server 7.0.03

    Assignment Engine 7.1

    Asset Management 7.0.03*

    CMDB 2.1.00 Patch 4

    CMDB Extension Loader

    Approval Server 7.1

    Change Management Server 7.0.03 Patch 008*

    Problem Management Server 7.0.03*

    Incident Management Server 7.0.0*3

    User Client

    Administrator Client

    These packages all come with the IT Service Management Pack. Oracle provides example customizations for the Remedy under ITSM 7.0.03 Patch 008 environment. For different versions, the customizations may need to be adjusted to account for changes in the version of Remedy.

  2. Install the Enterprise Manager EMCLI_Client on the same host that Remedy is installed on. This will need to be able to communicate to your Enterprise Manager Server.

    1. Login to the Enterprise Manager console

    2. Navigate to Setup > My Preferences > Command Line Interface

    3. Click Download the EM CLI kit to your workstation link and download the jar to your Remedy server.

    4. Follow the steps given in the page to install the EMCLI client on the Remedy server.

  3. Get the latest version of the Change Request Management connector self-update package. Also acquire the latest version of the example Remedy ARS customizations for Enterprise Manager version 12c.

    These definition files provide a guideline of customizations that must be made in your environment for the integration. These customization files assume a fresh install of Remedy ARS. When integrating with a production instance of Remedy, care should be taken to make sure these customizations are compatible with any previous customizations that have been made to the Remedy instance.

    • ActiveLinks _Customization.def

    • Forms_Customization.def

    • Menus_Customization.def

    • Webservices_Customization.def

  4. Install the four definition files (.DEF) files in the running Remedy environment by completing these steps:

    1. Log into the Remedy Administrator tool

    2. Select the Remedy instance from the hierarchy on the left

    3. Click Tools menu > Import Definitions > From Definition File...

    4. Select the definition file to import from the list above

    5. Check the box labeled Replace Objects on the Destination Server

    6. Choose the drop down option Replace With New Type

    7. Click Import

    8. You should not encounter any errors during this process. At the end of import there should be an Import Complete message.

    9. When done, repeat for the rest of the customization files.

  5. Customize Web Services

    1. Log into Remedy Administrator tool

    2. Select Webservices and then select the webservice EMCCC_GetCR. Right click and select Open

    3. Select the WSDL tab

    4. In the input on top, modify the midtier_server and servername values in the WSDL Handler URL.

    5. If midtier is on localhost, you can enter localhost right after http://

    6. If the midtier uses port 80, you can omit the port, otherwise include the port after the server name

    7. For the servername after "public/", enter the name of the Remedy server.

    8. Click the View button

    9. You should see an XML representing the webservice WSDL file for the webservice

    10. If you see an error, check the midtier_server name, port, or servername. Also, you can try adding/removing the domain part of the servername.

    11. If you see the XML content after clicking View, then close this window and save the changes

    12. Repeat all above steps with the webservices EMCCC_PublishCSData and EMCCC_UpdateChangeRequest

  6. Customize Active Links

    1. Log into Remedy Administrator tool

    2. Select active links and then select the active link EMCCCC_ApprovedCR. Right click and select Open

    3. Click on the tab If Action

    4. Click on the Current Action Run Process at the end of the list of actions

    5. In the Command Line field, change the path to emcli.bat to match that of where you installed the emcli on the local host.

  7. Create a user in Remedy that will be used for creating requests that will be used for automatic observation reconciliation:

    1. Log into BMC Remedy User Client as an administrative user

    2. Click on Application Administration Console on the User Client Home Page

    3. Click on Create for each step 1 through 4 in this wizard

    4. When adding the person, add the support group under the Support Groups tab.

    5. Under the Support Groups Tab, select sub tab Support Group Functional Roles

    6. Add Support groups with functional role of Infrastructure Change Management. Without this, you will not be able to create change requests as the Infrastructure Change Manager fields support group will not have values.

    7. Go to AR System Administrator Console

    8. On the left side bar, click Application > Users/Groups/Roles > Select Users.

    9. This will load the user search page. Click the Search button at the top right

    10. Double click the newly created user above to bring up the user form

    11. Click the down arrow next to "Group List" field and select Infrastructure Change Master

    12. Repeat the previous step and add the following Groups to this user as well

      Infrastructure Change Submit

      Infrastructure Change User

      Infrastructure Change Viewer

    13. Save the changes to this user by clicking the Save button in the upper right hand corner of the window.

Adding the Connector to Enterprise Manager

Follow these steps to add the connector to Enterprise Manager.

  1. Add the Change Management Connector to Enterprise Manager

    1. Log into Enterprise Manager as an Administrative user that has privileges to create a connector.

    2. Go to Enterprise > Change Management > Software Library

    3. Click Actions > Administration

    4. Click Add

    5. Provide a name, such as "self update swlib"

    6. Provide a location where the swlib files will be located on the Enterprise Manager server. This can be anywhere, but must be a path that the Enterprise Manager user can access. You must put the full absolute path in this input.

    7. This process will take several minutes to complete

    8. Locate the connector self-update package file.

      The connectors jar can be downloaded from the Enterprise Manager store to EM@Customer using the Self Update console, and can be exported to any local directory using the export functionality of Self Update.

    9. Run: emcli import_update -file=<full path>/connector.zip –omslocal (where connector.zip is an example name of the self update package)

    10. If you have errors with the previous step, make sure the user you run emcli as has permissions to access this directory and file. Also, be sure you are using absolute path for the -file switch

    11. When successful, you will receive the following message:

      Operation completed successfully. Update has been uploaded to Enterprise Manager. Please use the Self Update Home to manage this update.

    12. Log into the Enterprise Manager console

    13. Navigate to Setup > Extensibility > Self Update

    14. Find the type "Management Connector" and click on the link "1" under "Downloaded Updates" for this entry.

    15. Select the Connector from the table and click the Apply button.

  2. Create a Change Management Connector instance

    1. Login into Enterprise Manager console

    2. Navigate to Setup > Extensibility > Management Connectors

    3. Select "Remedy Change Management Connector" from the dropdown after "Create Connector" and click Go

    4. Provide a name and description for the connector. This name is used to choose the connector when creating a Real-time Monitoring Compliance Standard Rule.

    5. After returning to the management connector listing page, select the newly added row and click Configure.

    6. Under the Web Service End Points label, change the [servername] and [port] to match that of your Remedy instance Web Services. The values you put here will be similar to what you configured in the Web Services step earlier in these instructions.

    7. Enter the Remedy username and password you are using for the connector integration

    8. Enter the locale ('en', for example)

    9. Enter the time zone offset of the remedy server from UTC, ('-08:00', for example)

    10. Enter the Change ID to use as a test. This should be a valid Change Request ID currently existing in Remedy that is used to test the connectivity between Enterprise Manager and Remedy.

Using Automatic Reconciliation Rules

Once Remedy is customized and the Enterprise Manager connector is configured, to utilize the automatic reconciliation features, you need to create Real-time Monitoring Rules that are configured to use automatic reconciliation. Use the following steps:

  1. Create a Real-time monitoring Rule:

    1. Follow normal steps to create a Real-time monitoring Rule

    2. On the Settings page, choose Authorized Observations Automatically using Change Request Management System

    3. Select the connector from the drop down

    4. Click to annotate change requests with authorized observations check box

    5. Continue to save the rule after this. The Real-time Monitoring Rule can be used like any other Real-time Monitoring rule. Create a Compliance Standard, add this rule to the Compliance Standard, and associate this compliance standard to one or more targets.

Creating Change Requests for Upcoming Changes

Now that integration is set up and Real-time monitoring rules have been created, Change Requests can be created by Remedy users in the Remedy interface. These Change Requests will be compared to observations that occur to automatically determine if these observations are from actions that were authorized by change requests or not.

To make this correlation, some new fields that have been added to the Change Request form must be filled out by the change request filer. Not all fields are required; correlation only occurs on the fields that are present in the Change Request.

For instance, the following fields have been added to the Change Request form under the Oracle Enterprise Manager Integration tab:

  • Connector: Choose the Enterprise Manager connector this Change Request will use to integrate with Enterprise Manager

  • Hostname: the hostname(s) this change request is for. These are the hosts that this change request is specifying someone needs to make changes to. An empty value in this field indicates that all hosts will be correlated to this change request.

  • Target User List: the user name(s) this change request is for based on target users. These are the target users you expect to log in to the target to make a change. An empty value in this field means that all users on the target will be correlated to this change request.

  • Target Type: the target type this change request is against. An empty value in this field means that any target type will be correlated to this change request.

  • Target: The target this change request is specifically for. An empty value in this field indicates that any target will be correlated to this change request.

  • Facet: The facet this change request is specifically for. An empty value in this field indicates that all facets on the above target type and target will be correlated for this change request.

When creating a change request that you want to use to authorize changes detected by Real-time monitoring rules, follow these steps in addition to whatever requirements your organization implements for creation of Change Requests:

  1. Under the Dates tab of the Change Request form, fill out the Scheduled Start date and Scheduled End Date. These are the date ranges the request is valid for reconciliation. If an action occurs outside this time, it is marked as unauthorized by the Real-time Monitoring feature.

  2. Click on the Oracle Enterprise Manager Integration tab

  3. Select the Enterprise Manager connector from the drop-down list

  4. Optionally select values for the five reconciliation criteria as described above: Hostname, Target User List, Target type, Target and Facet. The last three -- Target Type, Target, and Facet -- will be Choice lists based on content in Real-time Monitoring Rules that have been created in Enterprise Manager that belong to Compliance Standards which are associated to targets. You can add multiple values separated by commas.

    Note:

    This form can be customized in Remedy to look differently. The example form elements from the customizations loaded earlier are only examples.
  5. Change auditable status to True. This configures Remedy to use this change request by Enterprise Manager for reconciliation of Real-time Observations that are detected.

  6. Save the change request.

  7. A popup displays notifying you that active links will send the content to Enterprise Manager. You will see a DOS command window open and then close.

Overview of Reconciliation Functionality

After creating a change request that references a target and/or facet that is being monitored by Real-time Monitoring rules, any observations that happen against that rule will be correlated to all open and matching change requests.

When the observation arrives at the Enterprise Manager server, all open change requests that were active (based on Scheduled Start/Stop time) and have matching correlation criteria from the Enterprise Manager Integration tab will be evaluated. If any change request exists that matches the criteria of the observation, this observation will be marked with an “authorized” audit status. If the annotation check box was checked in the Rule configuration, details of these authorized observations will be put into a table in the Enterprise Manager Integration tab of the Remedy Change Request.

If no open change requests can be correlated to the observation and the rule was configured to use automatic reconciliation, then this observation is set to an Unauthorized audit status. The Observation bundle to which this observation belonged will be in violation and results in an Enterprise Manager event being created. This event can further be used through creation of an Enterprise Manager Event Rule.

An observations audit status can be seen whenever looking at observation details either in the Compliance > Real-time Observations > Observation Search or either of the Browse By screens. A user with the proper role can also override the audit status for individual observations from these pages.

Any bundles that are in violation because they contain unauthorized observations will be reflected as violations in the Compliance Results page. These violations cause the compliance score skew lower. If these violations are cleared, the score becomes higher; however, the history of these audit status changes will be retained for the given observation.

Repository Views Related to Real-time Monitoring Features

The following views exist to allow access to Real-time Monitoring data.

View: mgmt$ccc_all_observations

Description: This view returns all observations that have occurred. Any query against this view should ensure that filtering is done on appropriate fields with action_time being the first to take advantage of partitions.

Fields:

Field Description
OBSERVATION_ID Unique ID given to the observation when detected by the agent
BUNDLE_ID Bundle to which this observation belongs based on rule bundle settings
TARGET Target this observation was found against
TARGET_TYPE Type of the target
ENTITY_TYPE Entity type of the entity that had an action against it
ACTION Action that was observed
ACTION_TIME Time the action occurred
USER_TYPE Type of user that performed the action (for example, OS user versus DB user)
USER_PERFORMING_ACTION Name of the user that performed the action
ORIGINAL_USER_NAME Previous user name in the case of a SU/SUDO action (only applicable to some entity types)
AFFECTED_ENTITY_NAME Name of the entity that was affected by this action (file name, and so on)
SOURCE_HOST_IP Source IP of a connection when an action comes from another host (only applicable to some entity types)
ACTION_PROCESS_ID PID of the process that performed the action (only applicable to some entity types)
ACTION_PROCESS_NAME Name of the process that performed the action (only applicable to some entity types)
ACTION_PARENT_PROCESS_ID PID of the parent process of the process that performed the action (only applicable to some entity types)
ACTION_PARENT_PROCESS_NAME Name of the parent process of the process that performed the action (only applicable to some entity types)
ENTITY_PREVIOUS_VALUE Previous value of the entity (only applicable to some entity types)
ENTITY_NEW_VALUE New value of the entity (only applicable to some entity types)
FILE_ENTITY_PREVIOUS_MDS_HASH Previous MD5 hash value of the entity (only applicable to some entity types)
FILE_ENTITY_NEW_MDS_HASH New MD5 hash value of the entity (only applicable to some entity types
AUDIT_STATUS Current audit status of the observation (unaudited, authorized, unauthorized, and so on)
AUDIT_STATUS_SET_DATE Date the most recent audit status was set
AUDIT_STATUS_SET_BY_USER User who set the most recent audit status

View: mgmt$ccc_all_obs_bundles

Description: This view returns a summary of all observations bundles. Any query against this view should ensure that filtering is done on appropriate fields with bundle_start_time being the first to take advantage of partitions.

Fields:

Field Description
BUNDLE_ID Bundle to which this observation belongs based on rule bundle settings
TARGET Target this observation was found against
TARGET_TYPE Type of the target
RULE_NAME Name of the Real-time Monitoring Compliance Standard Rule
ENTITY_TYPE Entity type of the entity that had an action against it
USER_PERFORMING_ACTION Name of the user that performed the action
BUNDLE_IN_VIOLATION Boolean value if the bundle currently is in violation. This means at least one observation in the bundle is unauthorized. True indicates the bundle is in violation.
BUNDLE_START_TIME Date of the first observation in this bundle
BUNDLE_CLOSE_TIME Date when this bundle was closed
BUNDLE_CLOSE_REASON Explanation of why this bundle was closed
DISTINCT_OBS_COUNT Total number of observations in this bundle
AUTHORIZED_OBS_COUNT Number of observations in this bundle that are currently authorized
UNAUTHORIZED_OBS_COUNT Number of observations in this bundle that are currently unauthorized
UNAUTH_CLEARED_OBS_COUNT Number of observations in this bundle that are currently cleared (that were at one point unauthorized)
UNAUDITED_OBS_COUNT Number of observations in this bundle that are currently unaudited. They have not been evaluated manually or with Change Management integration to determine audit status.

Modifying Data Retention Periods

Real-time Monitoring features use partitioning and data retention configuration.

The following are the tables along with their default retention periods. When changing any retention periods, all tables related to Real-time monitoring must be changed to the same value to ensure that data is consistent across various features.

Note:

For more information about modifying data retention values, see the chapter "Maintaining and Troubleshooting the Management Repository" in the book Oracle Enterprise Manager Administration.
Table Name Default Retention Period
EM_CCC_WATCHDOG_ALERTS 366 Days
EM_CCC_HISTORY_JOBEXEC 366 Days
EM_CCC_OBSERVATION 366 Days
EM_CCC_OBSGROUP 366 Days
EM_CCC_OBS_GROUP_MAP 366 Days
EM_CCC_HISTORY_OBS_STATUS 366 Days
EM_CCC_HA_OBS 366 Days
BUNDLE_START_TIME 366 Days
BUNDLE_CLOSE_TIME 366 Days
BUNDLE_CLOSE_REASON 366 Days
EM_CCC_HA_OBSGROUP 366 Days
EM_CCC_FILEOBS_DIFF 366 Days
EM_CCC_AUTHOBS_CR_MAP 366 Days