17 Always-On Monitoring
The Enterprise Manager Always-On Monitoring application provides the ability to monitor critical target status and metric alerts when the Oracle Management Service is unavailable. The service continuously monitors critical targets through the Enterprise Manager Agent and can be easily configured to send email notifications for these events to administrators.
Always-On Monitoring can be configured to send notifications at any time, but is particularly useful when experiencing downtime of your central Enterprise Manager site for maintenance operations such as upgrade and patching.
The Always-On Monitoring is synchronized with Enterprise Manager to reuse the configuration of monitored targets as well as requisite notification data such as notification contacts and email gateway configuration. Once properly configured and synchronized, the service will receive alerts from Enterprise Manager Agents and send email notifications to the appropriate administrators.
This chapter covers the following topics:
Prerequisites
Before installing Always-On Monitoring, ensure that the following prerequisite tasks have been performed:
Environment Setup
-
Install or identify an Oracle Database instance to hold the Always-On Monitoring repository.
-
Enterprise Manager must be upgraded to 13.4. In addition, the Management Agent should also be upgraded to 13.4.
-
If you do not have database administrator access, you need to ask the DB administrator to add you to the Always-On Monitoring repository. You can use the script found in Granting Required Privileges to the Always-On Monitoring Schema Owner.
Save the EM key to the Enterprise Manager repository. See Saving the Em Key for more information.
Java Requirements
-
JDK8 must be installed in the environment where the user will run Always-On Monitoring (including emsca). The JAVA_HOME environment variable must point to the JDK8 location.
System Resource Requirements
-
The number of open file descriptors used by Always-On Monitoring is http.queueSize + http.maxThreads. Always-On Monitoring also uses the database connection pool. The number of open database connections ranges between the number of active threads up to a maximum of 100 connections.
-
Maximum memory requirement is estimated at 10 Megabytes per thread. Memory demand peaks when Always-On Monitoring is first started because the Agents must communicate the complete status of all their monitored targets.
Installing the Always-On Monitoring Repository Database
You must install and configure an Oracle database instance to house the Always-On Monitoring repository. Because you must create the schema for the repository separately, it is critical to have an idea of the space that needs to be allocated to the schema, and how the space is broken down.
Because the purpose of Always-On Monitoring is to continuously monitor and send notifications when Enterprise Manager is down, the Always-On Monitoring Repository database should be installed as a separate instance of Oracle with the Always-On Monitoring Schema on another machine so that Always-On Monitoring continues to run when the Enterprise Manager Repository is either being upgraded or unexpectedly goes down.
Like the Enterprise Manager Repository, the Always-On Monitoring Repository needs to be installed on an Oracle database. The database version supported depends on the Always-On Monitoring version as shown in the following table:
Always-On Monitoring Version | Oracle Database Version |
---|---|
All Versions | Oracle Database 12.1.0.2.0 with Bundle Patch 10 |
Always-On Monitoring 13.3.0.0.0 and Greater | Oracle Database 12.2.0.1.0, 18.0.0.0.0, 18.3.0.0.0, and 19.0.0.0.0. |
Database Sizing
The factors to consider when sizing and configuring the database are as follows:
-
Undo Tablespace: The Undo tablespace is utilized when running the SYNC processes, especially during the initial SYNC when potentially larger numbers of rows are pulled from the Enterprise Manager instance to Always-On Monitoring.
-
Temp Tablespace: The SYNC process will also utilize TEMP tablespace, especially when indexes are created or sorting occurs during the data movement process.
-
Always-On Monitoring Tablespace: Table data and indexes specific to Always-On Monitoring are stored in this tablespace.
-
Redo Logs: Redo logs should be large enough to minimize the number of checkpoints that occur during times when the SYNC is occurring. It is recommended to configure 3 x 1 GB REDO log files.
-
Special Oracle Parameter Settings: For Always-On Monitoring, it is desirable to maintain the parameters used in the Enterprise Manager repository, especially if the Always-On Monitoring schema is to be populated in the Enterprise Manager repository database. To ensure correctness of the Enterprise Manager parameter settings that are passed in as part of the Always-On Monitoring install, you can run the SYNC and other Always-On Monitoring functions that are part of the emsctl verbs. This is important to consider when configuring a separate database instance. As is the case with the Enterprise Manager Repository, the following parameter should be set for the Always-On Monitoring Repository to avoid unforeseen optimizer issues:
ALTER SYSTEM SET OPTIMIZER_ADAPTIVE_FEATURES=FALSE SCOPE=BOTH SID='*';
The following table represents an example of the sizing of the above components for an Always-On Monitoring schema that is being used for a large enterprise database. Note that these figures represent the sizes of the tablespaces after the initial full SYNC between Always-On Monitoring and Enterprise Manager. Also note that the Redo Log files are not included in these calculations:
Table 17-1 Always-On Monitoring Repository Tablespace Sizing
Tablespace Name Used Space (MB) Free Space (MB) Total Allocation (MB) Percentage Free (%) TEMP
0
30,720
30,720
100%
Users
6,430
44,797
51,200
87%
SYSAUX
1,385
85
1,470
6%
SYSTEM
896
4
900
0%
UNDOTBS1
84
30,636
30,720
100%
Totals
8.769
106,246
115,015
92%
For this particular configuration, the absolute minimum for disk space required was 9 GB based on the used Tablespace. To ensure room for growth with this schema, it is recommended to plan on allocating at least 120 GB of space for the Always-On Monitoring database if creating a new Oracle instance. The TEMP and UNDO tablespaces are configured to handle initial and subsequent SYNC operations ensuring there is no issue with TEMP or REDO saturation. In addition, the Always-On Monitoring tablespace is sized to ensure no interruption due to saturation of the tablespace.
In order to accurately size the Always-On Monitoring tablespace for current usage and future growth, it is important to understand the following:
-
What data from what tables is being transferred from the Enterprise Manager Repository to the Always-On Monitoring Schema.
-
Knowing the number of rows transferred for each table, what is the approximate number of Bytes per Row for the tables and indexes?
The following table shows a subset of the tables and indexes in the Always-On Monitoring schema, highlighting the number of rows, total space consumption, and the number of bytes per row:
Table 17-2 Always-On Monitoring Table and Index Space Allocation
Owner | Segment Name | Segment Type | Partition Name | Size (MB) | Number of Rows |
---|---|---|---|---|---|
Always-On Monitoring |
MGMT_TARGET_PROPERTIES |
TABLE |
Non-Partitioned |
1,472 |
15,408,952 |
Always-On Monitoring |
MGMT_TARGET_PROPERTIES_IDX_02 |
INDEX |
Non-Partitioned |
1,152 |
15,215,316 |
Always-On Monitoring |
MGMT_TARGET_PROPERTIES_PK |
INDEX |
Non-Partitioned |
856 |
15,462,104 |
Always-On Monitoring |
MGMT_TARGET_PROPERTIES_IDX_01 |
INDEX |
Non-Partitioned |
584 |
14,871,521 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES |
TABLE |
Non-Partitioned |
496 |
1,177,065 |
Always-On Monitoring |
MGMT_TARGET_PROPERTIES_IDX_03 |
INDEX |
Non-Partitioned |
472 |
16,105,162 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES_UK1 |
INDEX |
Non-Partitioned |
136 |
1,201,617 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES_UK2 |
INDEX |
Non-Partitioned |
120 |
1,133,804 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES_IDX01 |
INDEX |
Non-Partitioned |
112 |
1,184,414 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES_IDX07 |
INDEX |
Non-Partitioned |
112 |
1,166,536 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES_IDX03 |
INDEX |
Non-Partitioned |
96 |
1,182,817 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES_IDX04 |
INDEX |
Non-Partitioned |
80 |
1,218,977 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES_IDX05 |
INDEX |
Non-Partitioned |
72 |
1,104,428 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES_IDX06 |
INDEX |
Non-Partitioned |
59 |
673,838 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES_IDX08 |
INDEX |
Non-Partitioned |
44 |
1,215,411 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES_PK |
INDEX |
Non-Partitioned |
41 |
1,172,101 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES_IDX09 |
INDEX |
Non-Partitioned |
40 |
1,192,236 |
Always-On Monitoring |
EM_MANAGEABLE_ENTITIES_IDX02 |
INDEX |
Non-Partitioned |
37 |
1,178,410 |
Always-On Monitoring |
EM_VIOLATIONS |
TABLE |
Non-Partitioned |
28 |
112,106 |
Always-On Monitoring |
EM_METRIC_COLUMN_VER_PK |
INDEX |
Non-Partitioned |
17 |
611,561 |
Database Character Set Definition
To ensure that the data being transferred from the Enterprise Manager Repository to the Always-On Monitoring Repository is stored and represented properly when the Always-On Monitoring Repository is on a separate instance from Enterprise Manager, it is important to ensure that the character set matches between the two databases/instances. For Enterprise Manager, the NLS_CHARACTERSET is defined as AL32UTF8..
If using Oracle Installer to manually set up the database instance that Always-On Monitoring uses, the instance will default to WE8MSWIN1252. Ensure at this point that the option AL32UTF8 is selected from the drop down box so that the correct character set is installed from the start..
To determine the character set on the Enterprise Manager Repository, run the following query:
SELECT * from nls_database_parameters where parameter='NLS_CHARACTERSET';
Run this query as well on the Always-On Monitoring Repository to ensure the character sets match. If the Always-On Monitoring Repository character set is different, the following is recommended to change the character set on the Always-On Monitoring Repository to match that of the Enterprise Manager Repository.
--Shutdown and start the database in a restricted mode. SHUTDOWN IMMEDIATE; STARTUP RESTRICT; --Set JOB_QUEUE_PROCESSES and AQ_TM_PROCESSES to its default value i.e.0 ALTER SYSTEM SET JOB_QUEUE_PROCESSES = 0; ALTER SYSTEM SET AQ_TM_PROCESSES=0; -- Set the required character set. ALTER DATABASE CHARACTER SET AL32UTF8; -- if the above fails: ALTER DATABASE CHARACTER SET INTERNAL_USE AL32UTF8; --Shutdown and restart the database in normal mode. SHUTDOWN IMMEDIATE; STARTUP; /
Creating the Always-On Monitoring Repository User
Note:
The Always-On Monitoring Repository should not be housed within the Enterprise Manager Repository database. Housing the Always-On Monitoring Repository in a separate database allows Always-On Monitoring to function even if the Enterprise Manager Repository goes down.Note:
The Always-On Monitoring user should be created by a database administrator with SYSDBA privileges. Alternatively, you can supply SYSDBA credentials when running the emsca configuration utility. emsca will then create the Always-On Monitoring repository user automatically.If you have SYSDBA access, you can skip this step since it will be done automatically by the emsca configuration assistant. See Using the Always-On Monitoring Configuration Assistant (EMSCA) for more information.
Granting Required Privileges to the Always-On Monitoring Schema Owner
To ensure proper functionality of Always-On Monitoring, the Always-On Monitoring schema owner needs to have the correct privileges. The following sample script details all grants required for the Always-On Monitoring Schema:
create user ems identified by ems; grant CREATE SESSION, ALTER SESSION, CREATE DATABASE LINK, CREATE MATERIALIZED VIEW, CREATE PROCEDURE, CREATE PUBLIC SYNONYM, CREATE ROLE, CREATE SEQUENCE, CREATE SYNONYM, CREATE TABLE, CREATE TRIGGER, CREATE TYPE, CREATE VIEW, UNLIMITED TABLESPACE, SELECT ANY DICTIONARY to ems; grant EXECUTE ON SYS.DBMS_CRYPTO to ems; grant EXECUTE ON SYS.DBMS_AQADM to ems; grant EXECUTE ON SYS.DBMS_AQ to ems; grant EXECUTE ON SYS.DBMS_AQIN to ems; grant EXECUTE on SYS.DBMS_LOCK to ems; grant EXECUTE ON SYS.DBMS_SCHEDULER to ems; grant create job to ems;
Under certain circumstances you may need to grant privileges directly to an Always-On Monitoring user. However, directly granting privileges to any user may violate security policy and compliance rules. In this situation, you can can assign most of the privileges to a role and then assign that role to the user.
For multitenant environments, when creating a user in a multitenant container database (CDB) and not in one of its constituent pluggable databases (PDB), then the user has to be prefixed by "C##".
Best Practices
Oracle recommends keeping an Always-On Monitoring instance running at all times regardless of whether Enterprise Manager is up or down. By telling the Always-On Monitoring to turn off notifications, you can have it run in the background while Enterprise Manager is up. During Enterprise Manager downtime, whether planned or unplanned, you can issue an emsctl command to tell Always-On Monitoring to restart notifications. The primary advantage to having Always-On Monitoring running at all times is that it will stay up to date with any changes made to your Enterprise Manager installation. Always-On Monitoring keeps itself in sync with Enterprise Manager by periodically running a synchronization job. This job runs every 24 hours by default, however the job execution time can be changed to meet the needs of your monitored environment.
You may also run Always-On Monitoring only when needed instead of having it run constantly in the background. However, doing so will require Always-On Monitoring to synchronize with the Enterprise Manager OMS. Depending on the size of your Enterprise Manager deployment, synchronization may take 10-15 minutes in order to obtain current information on all Enterprise Manager targets and metrics.
Note:
Always-On Monitoring and Enterprise Manager must be in SYNC before Enterprise Manager goes down.
Installing Always-On Monitoring
Always-On Monitoring is a self-contained application that is supplied with the Enterprise Manager software distribution.
Steps to install Always-On Monitoring:
-
Uninstall any previous installations of Always-On Monitoring. For more information, see Uninstalling Always-On Monitoring.
-
Locate the latest Always-On Monitoring ZIP file. The ZIP file can be found in the
/sysman/ems
directory of the OMS. -
Copy the ZIP file to the Always-On Monitoring host and unzip the contents to the location where Always-On Monitoring is to be installed.
Installing Always-On Monitoring from an Enterprise Manager Software Distribution
The Always-On Monitoring installation zip file is shipped with Enterprise Manager. From the sysman/ems
directory, find the Always-On Monitoring installation ZIP file (e.g. ems.zip). Unzip the file to the location where you want to install Always-On Monitoring.
Installing Multiple Always-On Monitoring Instances
You can set up multiple Always-On Monitoring (AOM) instances to ensure Always-On Monitoring is available in the event of outages. See Setting Up Multiple Always-On Monitoring Instances for more information.
Configuring Always-On Monitoring
Once Always-On Monitoring has been installed, the Always-On Monitoring configuration assistant script can be used to configure the service to communicate with the Enterprise Manager repository.
Always-On Monitoring configuration involves the following steps:
Saving the Em Key
Always-On Monitoring requires the Em key in order to be able to synchronize with the Enterprise Manager repository and reuse the SMTP (email) gateway configuration information from Enterprise Manager.
Note:
The Em key needs to be copied to the Enterprise Manager repository BEFORE configuring Always-On Monitoring. Not doing so will result in errors when running emsca.
Once Always-On Monitoring configuration is complete (emsca has successfully run through completion), immediately remove the Em key from the Enterprise Manager repository. Note: Always-On Monitoring does not have to be restarted. To store the key in the repository, run the following emctl commands from the $MW_HOME/bin
directory, where $MW_HOME is the Enterprise Manager Middleware Home directory.
% emctl config emkey -copy_to_repos
Once Always-On Monitoring configuration is complete, execute the following command to remove the key.
% emctl config emkey -remove_from_repos
Using the Always-On Monitoring Configuration Assistant (EMSCA)
The Always-On Monitoring configuration assistant, emsca, is a script located under the Always-On Monitoring installation scripts directory. Running emsca requires the bash shell to be installed.
If you have database administrator credentials, you can pass them to the emsca script and it will create the Always-On Monitoring user for you.
If you do not have database administrator credentials, you need to:
-
Ask the database administrator to run the script. See Creating the Always-On Monitoring Repository User.
-
Run
emsca -createEmsDbUser=false
The following example usage scenarios assume Always-On Monitoring is installed in a location referred to using the environment variable AOM_HOME. Run the configuration assistant located in $AOM_HOME/scripts. The EMSCA may be invoked with no parameters and will prompt for the necessary information. Once the configuration has completed, record the Always-On Monitoring Upload URL as it will be used later to configure Enterprise Manager.
Example EMSCA Installation Scenarios
The user installing Always-On Monitoring has SYSDBA credentials.
Copyright (c) 2017, 2020 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Always-On Monitoring Repository Connection String : myserver.myco.com:25059:s307480 Always-On Monitoring Repository Username [ems] : Always-On Monitoring Repository Password [ems] : User "ems" cannot be found in the database. In order to create this user, SYSDBA credentials are required. If you do not want to continue, answer "n" to the question below. Create the Always-On Monitoring Repository user [y] : Always-On Monitoring Repository SYSDBA Username : sys Always-On Monitoring Repository SYSDBA Password : Enterprise Manager Repository Connection String : myserver.myco.com:25059:s307480 Enterprise Manager Repository Username : sysman Enterprise Manager Repository Password : Creating Always-On Monitoring repository user ems Agent Registration Password : Keystore for host myserver.myco.com created successfully. Connecting to Always-On Monitoring Repository. Creating Always-On Monitoring Repository schema Creating repository storage for Targets data. Creating repository storage for Alerts and Availability data. Creating repository storage for Notification Metadata data. Creating repository storage for Target Metric Metadata data. Registering Always-On Monitoring instance Always-On Monitoring Upload URL: https://myserver.myco.com:8081/upload
The user installing Always-On Monitoring does not have SYSDBA credentials.
Example 1: SYSDBA creates an Always-On Monitoring user and grants user privileges shown in the following example.
aomuser is the name of the Always-On Monitoring user in the database.
SYSDBA privileges are required to run the following script
create user aomuser identified by <password>; grant CREATE JOB, CREATE SESSION, ALTER SESSION, CREATE DATABASE LINK, CREATE MATERIALIZED VIEW, CREATE PROCEDURE, CREATE PUBLIC SYNONYM, CREATE ROLE, CREATE SEQUENCE, CREATE SYNONYM, CREATE TABLE, CREATE TRIGGER, CREATE TYPE, CREATE VIEW, UNLIMITED TABLESPACE, SELECT ANY DICTIONARY to aomuser; grant EXECUTE ON SYS.DBMS_CRYPTO to aomuser; grant EXECUTE ON SYS.DBMS_AQADM to aomuser; grant EXECUTE ON SYS.DBMS_AQ to aomuser; grant EXECUTE ON SYS.DBMS_AQIN to aomuser; grant EXECUTE ON SYS.DBMS_SCHEDULER to aomuser; grant EXECUTE ON SYS.DBMS_LOCK to aomuser;
Example 2: The Always-On Monitoring user invokes the EMSCA script..
Oracle Enterprise Manager Cloud Control 13c Release 4 Copyright (c) 2017, 2020 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Always-On Monitoring Repository Connection String : myserver.myco.com:25059:s307480 Always-On Monitoring Repository Username [ems] : aomuser Always-On Monitoring Repository Password [ems] : Always-On Monitoring Repository user "aomuser" has already been created Enterprise Manager Repository Connection String : myserver.myco.com:25059:s307480 Enterprise Manager Repository Username : sysman Enterprise Manager Repository Password : Agent Registration Password : Keystore for host myserver.myco.com created successfully. Connecting to Always-On Monitoring Repository. Creating Always-On Monitoring Repository schema Creating repository storage for Targets data. Creating repository storage for Alerts and Availability data. Creating repository storage for Notification Metadata data. Creating repository storage for Target Metric Metadata data. Registering Always-On Monitoring instance Always-On Monitoring Upload URL: https://myserver.myco.com:8081/upload
SYSDBA creates a role and assigns it to Always-On Monitoring user then emsca is executed.
Some sites prefer to create a role and assign that role to the Always-On Monitoring user. In this situation, you must run the scripts shown in the following examples.
create user <aom_user> identified by <aom_password>; create role ems_role; grant CREATE SESSION, ALTER SESSION, CREATE DATABASE LINK, CREATE MATERIALIZED VIEW, CREATE PROCEDURE, CREATE PUBLIC SYNONYM, CREATE ROLE, CREATE SEQUENCE, CREATE SYNONYM, CREATE TABLE, CREATE TRIGGER, CREATE TYPE, CREATE VIEW, SELECT ANY DICTIONARY to ems_role; grant ems_role to <aom_user>;
grant CREATE JOB, UNLIMITED TABLESPACE to <aom_user>; grant EXECUTE ON SYS.DBMS_CRYPTO to <aom_user>; grant EXECUTE ON SYS.DBMS_AQADM to <aom_user>; grant EXECUTE ON SYS.DBMS_AQ to <aom_user>; grant EXECUTE ON SYS.DBMS_AQIN to <aom_user>; grant EXECUTE ON SYS.DBMS_SCHEDULER to <aom_user>; grant EXECUTE ON SYS.DBMS_LOCK to <aom_user>;
Oracle Enterprise Manager Cloud Control 13c Release 4 Copyright (c) 2017, 2020 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Always-On Monitoring Repository Connection String : myserver.myco.com:25059:s307480 Always-On Monitoring Repository Username [ems] : Always-On Monitoring Repository Password [ems] : Always-On Monitoring Repository user "ems" has already been created Enterprise Manager Repository Connection String : myserver.myco.com:25059:s307480 Enterprise Manager Repository Username : sysman Enterprise Manager Repository Password : Agent Registration Password : Keystore for host myserver.myco.com created successfully. Connecting to Always-On Monitoring Repository. Creating Always-On Monitoring Repository schema Creating repository storage for Targets data. Creating repository storage for Alerts and Availability data. Creating repository storage for Notification Metadata data. Creating repository storage for Target Metric Metadata data. Registering Always-On Monitoring instance Always-On Monitoring Upload URL: https://myserver.myco.com:8081/upload
Running EMSCA with a Response File
$ emsca -responseFile=<response_filename>
emsRepConnectString=localhost:1521:xe emsRepUsername=ems emsRepPassword=ems emRepConnectString=mymachine.mycompany.com:15044:semgc3 emRepUsername=sysman emRepPassword=sysman # emsPort=8081 http.protocol=http #
The following table lists available EMSCA parameters.
Table 17-3 EMSCA Parameters
EMSCA Prompt | Description |
---|---|
Always-On Monitoring Repository Connection String |
The connect string used to locate the Always-On Monitoring repository. This may be any valid service descriptor or may be a |
Always-On Monitoring Repository Username |
The username associated with the Always-On Monitoring repository. See Creating the Always-On Monitoring Repository Userfor more details. |
Always-On Monitoring Repository Password |
The password associated with the Always-On Monitoring repository user. See Creating the Always-On Monitoring Repository User for more details. |
Enterprise Manager Repository Connection String |
The connect string used to locate the Enterprise Manager repository. This may be any valid service descriptor or may be a host:port:sid. |
Enterprise Manager Repository Username |
The username of the Enterprise Manager SYSMAN user. |
Enterprise Manager Repository Password |
The password of the Enterprise Manager SYSMAN user. |
Removing the Em Key
After Always-On Monitoring configuration is complete (after emsca has been run), execute the following command to remove the key.
% emctl config emkey -remove_from_repos
Always-On Monitoring does not have to be restarted.
Configuring Email Servers in Enterprise Manager
When Always-On Monitoring sends email notifications, it does so using email server configurations that it obtains from Enterprise Manager during the synchronization step. For Always-On Monitoring to function properly, email servers must be configured in Enterprise Manager before performing an Always-On Monitoring synchronization.
If changes to the email server configurations are made in Enterprise Manager, Always-On Monitoring must be synchronized again to retrieve the updated email server configuration. See Updating Always-On Monitoring “Running an Incremental Synchronization Manually” for additional details.
Configuring Downtime Contacts in Enterprise Manager
Once Always-On Monitoring is synchronized with Enterprise Manager, the monitoring service can send email notifications to the appropriate administrators when the OMS is down via configured downtime contacts. Prior to running Always-On Monitoring for the first time, downtime contacts should be configured in Enterprise Manager. The downtime contacts that Always-On Monitoring sends notification to may be any one of the following forms.
-
Global downtime contact
-
Per-target downtime contact based on target property
-
Per-target downtime contact based on event rules
If email addresses are specified for both Global and Per-target forms, then all email addresses will be used.
Global Downtime Contact
The global downtime contact is a single property set on the Enterprise Manager site. Once set, all target status events and metric alerts across all targets will be sent to the recipients specified in the global downtime contact property.
% emcli set_oms_property -property_name='oracle.sysman.core.events.ems.downtimeContact' -property_value='<email addresses>'
To delete the global downtime contact, run the following command:
emctl delete property -name 'oracle.sysman.core.events.ems.downtimeContact'
Per-Target Downtime Contact from Target Property
The email addresses for these downtime contacts should be specified in the Downtime Contact target property for each target. There are three ways to specify the Downtime Contact target property:
-
For each target, navigate to the Target Properties page, which can be accessed from the target's homepage. From the target menu on the target homepage, select Target Setup and then Properties. The Target Properties page displays.
Click on Edit and specify the email address in the Downtime Contact target property. You can specify multiple email addresses by separating them with commas.
-
Use the EM CLI set_target_property_value verb.
emcli set_target_property_value -property_records="target_name:target_type:property_name:property_value"
Example:
%./emcli set_target_property_value -property_records="myhost:host:Downtime Contact:John.Doe@mycompany.com"
For more information about the set_target_property_value verb, see the Oracle Enterprise Manager Command Line Interface Guide.
-
By leveraging the email recipients in the target down event rules in Enterprise Manager Cloud Control. See the following section for more details.
Per-Target Downtime Contact from Event Rules
Always-On Monitoring may also send email notifications to different users for each target. These contacts are generated in Enterprise Manager based on the event rules for that target. Therefore, as event rules are changed in Enterprise Manager, the contacts must be re-generated and an incremental synchronization performed
By leveraging the event rule setup, the downtime contact will be generated based on the email recipient for the event rule for a Target Availability event type where Down status has been selected.
Note:
Although downtime contacts are generating using only Target Availability event rules, Always-On Monitoring will send notifications for both target availability and metric threshold alerts.
You can review and update the recipients of your target availability (status down) event rules. Doing so allows you to generate a list of downtime contacts using EM CLI or by submitting the downtime contact generation job.
Generating Downtime Contact using EM CLI
From the EM CLI, use the generate_downtime_contact command to produce downtime contacts for a specific target. The command simulates ALL Target Availability rules that would affect that particular target and generates a list of email addresses that includes the output from all rules. The optional -set parameter will cause the contact to be saved so that Always-On Monitoring synchronization can retrieve it later.
% emcli generate_downtime_contact -target_name=<name> -target_type=<type> -set
Generating Downtime Contact Enterprise Manager Job
% emcli create_job -input_file=property_file:<job_prop_file> -name="GenerateDowntimeContacts"
Where the job_prop_file refers to a properties file with the following content:
# job type type=CoreSetDowntimeContacts # description description=You job description # target names, the list of target names variable.target_names=host1,database2 # target types, the list of target types corresponding to the target list above variable.target_types=host,oracle_database # set on all targets, true to update contacts, false to print output but not save variable.set_all_targets=true # frequency to run job, refer to Enterprise Manager documentation for options schedule.frequency=IMMEDIATE
Note:
The process of generating the downtime contacts thru either the EM CLI or the downtime contact generation job, does not immediately make the contacts available to the monitoring service. A separate Always-On Monitoring synchronization must be done each time these contacts are updated. This is done using the emsctl sync command.
Types of Alerts Received by Downtime Contacts
Once configured, Always-On Monitoring will send email notifications for:
-
All target status alerts and metric alerts (critical, warning, and clear).
-
Metric collection errors (both critical and clear).
Target status alerts for aggregate targets, such as groups and clusters, whose status is based on the status of its component members is not supported. Instead, alerts will be sent for the individual members of these aggregate targets..
Note:
Currently, email notification is the only type of notification supported.
The recipients of the email should be email addresses. The recipients are specified using the global property downtimeContact or the per-target Downtime Contact. If both are specified, emails will be sent to all recipients specified.
If the global property downtimeContact is specified, all alerts on all targets will be sent to the recipients specified in the global property downtimeContact. If the per-target property Downtime Contact is specified, all target status and metric alerts for that target will be sent to the recipients specified in that property.
Setting Downtime Contacts for Composite Targets
You can set the Downtime Contacts target property for a composite target so that the target property is propagated to its member targets. You can perform this update using the EMCLI set_target_property_value
verb. See set_target_property_value in the Oracle® Enterprise Manager Command Line Interface for more information about this verb.
Examples
Example 1: The following example sets the AOM Downtime Contact target property for a DB static group.
> ./emcli set_target_property_value -property_records="db_group:composite:Downtime Contact:aom@test.com" -propagate_to_members
Properties updated successfully
Example 2: The following example sets the AOM Downtime Contact target property for a DB system target.
> ./emcli set_target_property_value -property_records="eva4.us.oracle.com_sys:oracle_dbsys:Downtime Contact:aom@test.com" -propagate_to_members
Properties updated successfully
Synchronizing Always-On Monitoring with Enterprise Manager for the First Time
Before starting Always-On Monitoring for the first time, you must synchronize Always-On Monitoring with Enterprise Manager. Synchronization copies notification configuration and downtime contacts from Enterprise Manager targets, thus allowing Always-On Monitoring to monitor alerts and send email notifications.
To perform Always-On Monitoring-Enterprise Manager synchronization:
% $AOM_HOME/scripts/emsctl sync Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved. ------------------------------------------------------------------ Connecting to Always-On Monitoring repository. Starting synchronization with Enterprise Manager. Synchronizing with Enterprise Manager repository: sysman@myserver.myco.com:35074:semgc4 Synchronizing Targets data. Synchronizing Alerts and Availability data. Synchronizing Notification Metadata data. Synchronizing Target Metric Metadata data. Synchronization complete at : Thu Oct 22 06:50:56 PDT 2016
Note:
Synchronization should be run again whenever there are changes made to Enterprise Manager. For example, after adding new hosts or targets, changing the email server, or changing the downtime contacts, an incremental synchronization is required in order for Always-On Monitoring to pick up the latest configurations.. See "Running an Incremental Synchronization Manually" for additional details.
Configuring Enterprise Manager to Work with Always-On Monitoring
The final step in the configuration of Always-On Monitoring is to update Enterprise Manager to include the upload URL for the service. The upload URL is displayed by Always-On Monitoring Configuration Assistant upon successful completion of the configuration of the service. Always-On Monitoring configuration properties file also includes the protocol and port configured for the service so that the upload URL will be of the form:
https://yourhostname:8081/upload
Once configured, the URL will be sent to all Agents (version 13.1 and greater), both existing Agents and any Agents deployed in future. Hence, configuration only needs to be done once. Agents less than 13.1 will not receive the Always-On Monitoring upload URL.
HTTPS is the default protocol for Always-On Monitoring and the default port is 8081. Please refer to the emsca or Always-On Monitoring configuration file for the configured values.You can find the configuration file at the following location:
$AOM_HOME/conf/emsConfig.properties
To configure Enterprise Manager with Always-On Monitoring upload location, the following command should be executed from Enterprise Manager.
% emctl set property -name "oracle.sysman.core.events.ems.emsURL" -value "https://yourhostname:8081/upload" -sysman_pwd sysman
Starting Always-On Monitoring
Start Always-On Monitoring. This step is important to enable notifications. For details on starting AOM, see Starting Always-On Monitoring.
Enabling Notifications
% $AOM_HOME/scripts/emsctl enable_notification Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved. ------------------------------------------------------------------ Notifications have been enabled. There are downtime contacts configured. Connecting to Always-On Monitoring repository. Starting synchronization with Enterprise Manager. Synchronizing with Enterprise Manager repository: sysman@myserver.myco.com:35074:semgc4 Synchronizing Targets data. Synchronizing Alerts and Availability data. Synchronizing Notification Metadata data. Synchronizing Target Metric Metadata data. Synchronization complete at : Thu Oct 22 07:01:20 PDT 2020
$AOM_HOME/scripts/emsctl enable_notification -nosync Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved. ------------------------------------------------------------------ Notifications have been enabled. There are downtime contacts configured.
% $AOM_HOME/scripts/emsctl disable_notification Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Notifications have been disabled. Filters have been deleted.
Filtering Alert Notifications by Event Type and Severity
When only specific alert types are pertinent, you need to have a way of filtering out extraneous alerts so you don't have to scan through all alerts just to find the ones you're interested in. Always-On Monitoring lets you filter your alert notifications based on either event type or severity.
Alert Notification Filtering by event type lets you exclude alert notifications for certain event types. For example, you want to receive only target availability alerts. You can filter out alerts for the following event types:
- Metric Alert (
metricAlert
) - Availability (
targetAvail
) - Metric Evaluation Errors (
metricError
)
Default Settings
- Target availability & metric alert notifications are enabled
- Metric evaluation error alert notifications are disabled
$AOM_HOME/scripts/emsctl disable_notification
-alert_types=targetAvail,metricAlert,metricError
$AOM_HOME/scripts/emsctl enable_notification -alert_types=targetAvail
$AOM_HOME/scripts/emsctl enable_notification -alert_types=metricError
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021, Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Notification filters have been enabled. There are downtime contacts configured.
Notification filters
Target Availability (targetAvail) : true
Metric Alerts (metricAlert) : false
Metric Errors (metricError) : true
Alert Notification Filtering by severity lets you exclude alert notifications for certain severities. For example, you want to receive only Fatal, Critical, and Warning alerts. You can filter out alerts for the following event types:
- Fatal
- Critical
- Warning
- Advisory
- Informational
- Clear
Default Settings
- Fatal, Critical, and Warning alert notifications are enabled
- Advisory, Informational, and Clear alert notifications are disabled
$AOM_HOME/scripts/emsctl disable_notification –severities=fatal,critical,warning,advisory,informational,clear
$AOM_HOME/scripts/emsctl enable_notification - severities = fatal,critical
$AOM_HOME/scripts/emsctl enable_notification -severities=warning
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021, Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Notification filters have been enabled. There are downtime contacts configured.
Notification filters - Alert types
Target Availability (targetAvail) : true
Metric Alerts (metricAlert) : true
Metric Errors (metricError) : false
Notification filters - Severity
Fatal (fatal) : true
Critical (critical) : true
Warning (warning) : true
Advisory (advisory) : false
Informational (informational) : false
Clear (clear) : false
Verifying the Always-On Monitoring Upload URL on Enterprise Manager
You can determine the upload URL by issuing the following command:
$AOM_HOME/scripts/emsctl status
Example:
$AOM_HOME/scripts/emsctl statusOracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 2017, 2021, Oracle Corporation. All rights reserved. ------------------------------------------------------------------ Always-On Monitoring Version : 13.5.0.0.0 Always-On Monitoring Home : /mycomputer/ems Started At : December 23, 2019 2:24:25 PM PST Last Repository Sync : December 23, 2019 2:25:49 PM PST Upload URL : https://myserver:8081/upload Always-On Monitoring Process ID : 24924 Always-On Monitoring Repository : myserver.myco.com:15044:sarview Enterprise Manager Repository : myserver.myco.com:15044:sarview Notifications Enabled : true Notification filters - Alert types Target Availability (targetAvail) : true Metric Alerts (metricAlert) : true Metric Errors (metricError) : false Notification filters - Severity Fatal (fatal) : true Critical (critical) : true Warning (warning) : true Advisory (advisory) : false Informational (informational) : false Clear (clear) : false Total Downtime Contacts Configured : 1
You can verify the setting in the OMS by using the emctl get property
command.
You can verify the setting on a particular Agent by looking at $AGENT_HOME/sysman/config/emd.properties and searching for EMS_URL.
Controlling the Service
Starting Always-On Monitoring
% $AOM_HOME/scripts/emsctl start Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved. ------------------------------------------------------------------ Pinging Always-On Monitoring... Always-On Monitoring is not running. Starting Always-On Monitoring. Waiting for process to start Retrying... Notifications Enabled : false Total Downtime Contacts Configured : 1 Always-On Monitoring is up.
Getting Status & Logs
Use the emsctl
status
command to ascertain the operational status of Always-On Monitoring.
Example:
% $AOM_HOME/scripts/emsctl status Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved. ------------------------------------------------------------------ Pinging Always-On Monitoring... Always-On Monitoring is not running. Starting Always-On Monitoring. Waiting for process to start Retrying... Notifications Enabled : false Total Downtime Contacts Configured : 1 Always-On Monitoring is up.
Pinging Always-On Monitoring
In addition to running the emsctl status command, you can also issue the ping command if you just want to see that Always-On Monitoring service is up without listing all the operational details.
Example:
$AOM_HOME/scripts/emsctl ping Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved. ------------------------------------------------------------------ Always-On Monitoring is running
Log Files
Log files record Always-On Monitoring events that occur during operation and are generated as follows:
-
emsca logs: emsca.err (only errors), emsca.log.0 (rotating log file that contains all output including errors).
-
ems logs: ems.err (only errors), ems.log.0 (rotating log file that contains all output including errors).
These files are located in the $AOM_HOME/logs
directory.
Log levels determine the type and operational severity of information recorded in the log files. Levels can be set to any one of the following:
-
INFO: Always-On Monitoring operational events such and login, logout, or any other events tied to Always-On Monitoring lifecycle.
-
DEBUG: Information considered useful when tracing the flow of Always-On Monitoring events to isolate specific problems.
-
WARN (default setting): Unexpected operational Always-On Monitoring events. These are Always-On Monitoring events that should be tracked, but may not require immediate administrator intervention.
-
ERROR: Always-On Monitoring operational malfunction.
To change the log level, add the logLevel=... property to $AOM_HOME/conf/emsConfig.properties
. Note that this applies only to the current Always-On Monitoring instance.
Note:
You must bounce Always-On Monitoring in order for the log level changes to take affect.
Stopping the Service
Use the emsctl stop
command to stop Always-On Monitoring.
$AOM_HOME/scripts/emsctl stop Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved. ------------------------------------------------------------------ Shutting down Always-On Monitoring. Always-On Monitoring is stopped.
Always-On Monitoring Commands
Command | What it does |
---|---|
emsctl start | Starts Always-On Monitoring. |
emsctl stop | Shuts down Always-On Monitoring. |
emsctl enable_notification -nosync (optional) | Allow Always-On Monitoring to send email notifications. By default, an incremental SYNC is performed when enabling email notifications. To prevent synchronization with the Enterprise Manager repository, specify the -nosync option. |
emsctl disable_notification | Prevent Always-On Monitoring from sending email notifications. |
emsctl list_agents | Lists the URLs of Agents and a count of Agents that have communicated with the Always-On Monitoring Service in the past hour. |
emsctl ping | Ping the Always-On Monitoring service to determine whether it is up or down. |
emsctl sync | Synchronize Always-On Monitoring target and notification data with the Enterprise Manager repository. |
emsctl secure [-wallet=<absolute path to the wallet location>] | [-reg_pwd=<Agent registration password>] [-host=<Fully qualified Always-On Monitoring hostname>] [-cert_validity=<validity period in days - defaults to 10 years] [-key_strength=<certificate key strength - defaults to 1024] [sign_alg=<md5|sha1|sha256|sha384|sha512 - defaults to sha512> | Generates a wallet file for Always-On Monitoring service to run on a secured port. |
emsctl status | Display information related to Always-On Monitoring operational status. |
emsctl getstats | Display the current Always-On Monitoring performance statistics |
emsctl list_properties | Display Always-On Monitoring configuration properties. |
emsctl set_property -name=<property name> -value=<property value> | Set Always-On Monitoring configuration properties. For a complete list of configuration properties, see Modifiable Always-On Monitoring Properties |
emsctl unset_property -name=<property name> | Unset an existing Always-On Monitoring configuration property. For a complete list of configuration properties, see Modifiable Always-On Monitoring Properties. |
emsctl update_task -task_name=<SYNC|PURGE> -run_interval=<time in minutes between task executions> | Update the intervals for performing Always-On Monitoring tasks such as SYNC or PURGE. |
emsctl set_em_repos_conn -username=<em username> -password=<em password> -connect_string=<em connect descriptor> | Update the username, password and connect string used to connect to Enterprise Manager repository database. |
emsctl set_ems_repos_conn -username=<ems username> -password=<ems password> -connect_string=<ems connect descriptor> | Update username, password and connect string used to connect to the Always-On Monitoring repository database. |
Updating Always-On Monitoring
$AOM_HOME/scripts/emsctl sync
Incremental synchronization must be performed thereafter in order to keep Always-On Monitoring monitoring/notification configuration current.
The following changes to Enterprise Manager will require an incremental Always-On Monitoring synchronization:
-
New hosts (Agents) added
-
New targets added
-
Changes to downtime contacts
-
Changes to administrator email addresses
-
Changes to email server configuration
Always-On Monitoring stays in SYNC with Enterprise Manager automatically via an incremental synchronization job that runs every 24 hours. You can change the synchronization interval using the emsctl update_task command.
$AOM_HOME/scripts/emsctl update_task -task_name=<SYNC|PURGE> -run_interval=<time in minutes between task executions>
Incremental synchronization can be run manually via the emsctl sync command.
Running an Incremental Synchronization Manually
You can, at any time, perform an incremental Always-On Monitoring synchronization by running the sync command.
% $AOM_HOME/scripts/emsctl sync
When performing a manual incremental synchronization, you do not need to shut down Always-On Monitoring: You can run the SYNC command regardless of whether Always-On Monitoring is up or down.
Note:
An incremental synchronization is automatically performed whenever you run the enable_notification command. You can bypass the synchronization by specifying the -nosync option.
Data Maintenance
The Always-On Monitoring repository contains historical availability and metric violations data. This data is maintained for 6 months (default interval). Data older than 6 months is deleted (purged) from the repository.
Data maintenance is controlled by following settings:
-
Purge Interval—Determines how often the data is purged from Always-On Monitoring repository.
-
Purge Duration—Determines how much data is kept during the purge process.
You can change the purge interval via the update_task command:
$AOM_HOME/scripts/emsctl update_task -task_name=PURGE -run_interval=<the number of minutes between purge runs>
You can change the purge duration via the set_property command:
$AOM_HOME/scripts/emsctl set_property -name=purge.Duration -value=<the number of minutes of data to keep>
Controlling Always-On Monitoring Configuration Settings
Always-On Monitoring configuration settings define how your Always-On Monitoring deployment operates within your environment.
$AOM_HOME/conf/emsConfig.properties
Properties are added/updated in the emsConfig.properties
file using a text editor. Always-On Monitoring must be restarted in order for the changes to take effect. Global properties that are shared by all Always-On Monitoring instances are stored in the Always-On Monitoring repository and are manipulated using the emsctl
commands described below.
The following commands are used to view and set the global configuration properties:
To list all properties and their values:
$AOM_HOME/scripts/emsctl list_properties
To set a property value:
$AOM_HOME/scripts/emsctl set_property -name=<propertyname> -value=<propertyvalue>
To remove a property value:
$AOM_HOME/scripts/emsctl unset_property -name=<propertyname>
Getting Performance Information
You can see how well your Always-On Monitoring installation is operating by viewing Always-On Monitoring performance information such as load and throughput. Use the getstats command to display Always-On Monitoring performance statistics:
$AOM_HOME/scripts/emsctl getstats
Modifiable Always-On Monitoring Properties
Always-On Monitoring allows you to change various properties that define how your Always-On Monitoring deployment operates. The following table lists the global properties that are available. These properties can be set as global properties or locally in the emsConfig.properties
file..
Table 17-4 Always-On Monitoring Global Properties
Property Name | Description | Default Value | Range of Values |
---|---|---|---|
connPool.initialSize |
The initial number of Always-On Monitoring repository connections to be created in the repository connection pool. |
1 |
1-connPool.maxSize |
connPool.maxSize |
The maximum number of Always-On Monitoring repository connections to be created in the repository connection pool. |
100 |
connPool.minSize – 500 |
connPool.minSize |
The minimum number of Always-On Monitoring repository connections to be kept in the repository connection pool after growth. |
10 |
connPool.initialSize – connPool.maxSize |
failover.heartbeatInterval |
The interval in seconds between heartbeats sent from the running Always-On Monitoring to other Always-On Monitoring instances. |
60 |
30 – 3600 |
http.idleTimeout |
Time in milliseconds before idle web server connections in the Always-On Monitoring are released and terminated. |
60000 |
10000 – 120000 |
http.maxThreads |
The maximum number of Always-On Monitoring web server threads to be created in the web server connection pool. This value is of particular interest on large sites with 10000 hosts or more. |
50 |
http.minThreads – 100 |
http.minThreads |
The minimum number of Always-On Monitoring web server threads to be created in the web server connection pool. |
10 |
1 – http.maxThreads |
nls.useSystemLocale |
Reserved for future use. |
false |
|
notif.enabled |
If email notifications are enabled or not; typically sent using |
true |
true/false |
notif.senderAddress |
Override sender email address for emails sent from Always-On Monitoring. Otherwise the address is the same as the Enterprise Manager email address. |
null |
|
notif.senderName |
Override sender email username for e-mails sent from Always-On Monitoring. Otherwise name is the same as the Enterprise Manager email name. |
null |
|
purge.Duration |
Amount of data (in minutes) kept in all historical Always-On Monitoring tables. |
259200 |
10080 – 0 (infinite) |
sync.allowIncrSync |
Reserved for future use. |
false |
true/false |
tasks.poolSize |
Number of threads to use in the background task execution pool. |
20 |
1-50 |
tasks.purgeRunInterval |
Interval in minutes between instances of the data purge task. |
10080 |
3600 – 10080 |
tasks.syncRunInterval |
Interval in minutes between instances of the incremental sync task. |
1440 |
60 - 3600 |
emsConfig.properties
file.
Table 17-5 Always-On Monitoring Local Properties
Property Name | Description | Default Value | Range of Values |
---|---|---|---|
logLevel | The level of log messages to include in the Always-On Monitoring logs | INFO | DEBUG, INFO, WARN, ERROR |
emsPort | The port number to use when listening for requests from the agent. | 8081 | 1024-65535 |
oracle.sysman.core.notification.smtp.retry_enabled | Property that determines whether failed email notifications should be retried. | false | true/false |
oracle.sysman.core.notification.max_email_delivery_time | Property in seconds that determines how long failed email notifications should be retried. | 600 seconds | 60-86400 |
Creating an SSO Wallet and JKS for CA Certificates
Creating an SSO Wallet and JKS for CA Certificates
You can create a single sign-on (SSO) wallet and Java Keystore (JKS) for certificates issued by an external Certificate Authority for use with your Always-On Monitoring environment. To do so, perform the following:
- Copy the EM key from the Enterprise Manager repository by running:
emctl config emkey -copy_to_repos
-
Generate the certificate and the wallet for the host by running:
$AOM_HOME/bin/emsctl secure
$AOM_HOME/bin/emsctl secure -wallet=<absolute location of the wallet file>
Diagnosing Problems
Though robust and easy to install and configure, you may encounter problems while using Always-On Monitoring. In general, Always-On Monitoring problems can be classified into three types:
-
Problems starting Always-On Monitoring
-
Problems with Always-On Monitoring-Enterprise Manager repository SYNC.
-
Not receiving email
By performing diagnostic tasks listed below, you can resolve a majority of issues that may occur when using Always-On Monitoring.
-
Check the log files in the $AOM_HOME/log sub-directory
-
Verify that the URL is set in Enterprise Manager by running the following
emctl get property -name "oracle.sysman.core.events.ems.emsURL" -sysman_pwd <sysman-password>
-
Verify that the URL is set on the Management Agent by checking the $AGENT_HOME/sysman/config/emd.properties file and searching for EMS_URL
-
Verify that Always-On Monitoring is running and notifications are enabled by running emsctl status or emsctl ping
-
Verify that downtime contacts have been specified in Enterprise Manager and Always-On Monitoring by running emsctl status.
-
Check the version of the Agent by running the following command:
emctl status agent
Theemctl
command can be run form the following directory:$AGENT_HOME/agent_inst/bin
High Availability and Disaster Recovery
The purpose of Always-On Monitoring is to make sure your Enterprise Manager environment continues to be monitored while Enterprise Manager OMS is down. However, to protect against a single point of failure within the Always-On Monitoring environment itself, the Always-On Monitoring service can be made highly available by adhering to the Enterprise Manager Maximum Availability Architecture (MAA) guidelines. By setting up Always-On Monitoring as a highly available service, you can ensure that targets will continue to be monitored in the event of catastrophic system failure.
Running Multiple Always-On Monitoring Instances
Always-On Monitoring provides you with the ability to continue alert monitoring while Enterprise Manager is down. By itself, Always-On Monitoring can be made highly available by implementing multiple instances for load sharing/high availability. It is also important to note that the Always-On Monitoring repository should be configured as a cluster database since you need to implement basic High Availability (HA) for this component as well.
Adding multiple Always-On Monitoring instances provides the following HA advantages:
-
Shares the work of receiving and recording alerts from agents. If an instance goes down for some reason, a server load balancer (SLB) can redirect those requests to surviving instances
-
Shares the work of sending of notifications. For ordering reasons, all notifications for a target are directed to one Always-On Monitoring instance. If that Always-On Monitoring instance goes down, the responsibility for its "queues" are transferred to another instance.
emsca add_ems
The following diagram shows a typical Always-On Monitoring HA deployment.
Figure 17-1 Always-On Monitoring HA Deployment
Shared Configuration Storage for the Multiple Instances
Instead of maintaining Always-On Monitoring configuration in the file system on each node, all instances can share a common configuration. This allows a single configuration at one location to be modified and read from all the instances.
You use the following EMSCTL commands to modify properties:
-
emsctl set_property
-
emsctl unset_property
To obtain a list of current property values:
-
emsctl list_properties
When a property is modified on one instance, all other Always-On Monitoring instances receive a JMS AQ message that tells them to refresh the value from the repository.
Notification Queues for Tracking Incoming Alerts
Each Always-On Monitoring instance that receives an alert stores the alert and records an entry in one of 16 event notification queues. Ownership of these notifications queues is distributed among the running Always-On Monitoring instances.
A failover system notices when Always-On Monitoring instances are not up (and heartbeating) and switches ownership away from the instance that is down to other Always-On Monitoring instances that are up. The failover system then sends a JMS AQ message notifying all up instances of the change.
Task Scheduler System
A Task scheduler system remembers shared responsibilities among the Always-On Monitoring instances and is responsible for reminding the instances to do the work on a schedule. The task request is sent to an Always-On Monitoring instance that is known to be up. This reduces the possibility of multiple instances trying to do the same work and running into conflicts.
Configuring an SLB
When multiple Always-On Monitoring instances are present, we expect that traffic to these instances is directed through a SLB.
When HA is added to an Always-On Monitoring instance that was configured as a single instance, you need to repopulate the Always-On Monitoring instance's certificates with fresh copies that have the SLB host name in them. For Always-On Monitoring released with Enterprise Manager 13.3, you generate a certificate for the SLB and store that in a wallet by running the emsctl secure
command.
The URL displayed from emsca and/or emsctl status is the local URL for that Always-On Monitoring instance. In an HA configuration, this local URL is NOT used as the Always-On Monitoring URL, but is used to set up the Always-On Monitoring pool in the SLB. The host and port for each Always-On Monitoring instance is used in the pool with the /upload PUT URL as the target for that pool.
When configuring the SLB, the monitoring URL is the /upload GET URL. This URL returns the string "Always On Monitoring is active."
Note:
Always-on monitoring is supported on F5 from version 11.4.x onwards.
Always-On Monitoring Disaster Recovery
Disaster Recovery (DR) allows Always-On Monitoring functionality to transition to a different location in the case of a site-wide failure.
There are existing procedures that allow an Enterprise Manager site to have a standby created for DR reasons. These procedures involve having the primary repository database's contents mirrored at the standby site using DataGuard, and the ability to link to the file system contents from the secondary site.
A very similar implementation can be used for Always-On Monitoring instances as well. An Always-On Monitoring DR setup is active-passive: If the primary site goes down, the virtual IP fails over to the standby node and, once the standby Always-On Monitoring instance is brought up, it runs exactly as it did on the primary node (because of the file system sharing and the virtual IP). In summary, the Always-On Monitoring repository needs to be available at the standby site, and the Always-On Monitoring instances' file systems need to be replicated. The following diagram shows the typical DR deployment for Always-On Monitoring.
Figure 17-2 Always-On Monitoring Disaster Recovery Deployment
Setting Up Multiple Always-On Monitoring Instances
You can set up multiple Always-On Monitoring (AOM) instances to ensure Always-On Monitoring is available in the event of outages.
-
Uninstall any 13.1 Always-On Monitoring instances, if any.
Except for the last running AOM instance (in this example, there are three AOM instances AOM1, AOM2 and AOM3) you must perform the following for AOM1 and AOM2):
-
Shut down the instances (AOM1 and AOM2) by running
$AOM_HOME/scripts/emsctl stop
-
Remove all the files and sub-directories under $AOM_HOME
And on the last AOM instance (AOM3):
-
Shut down the instance (AOM3) by running
$AOM_HOME/scripts/emsctl stop
-
Uninstall AOM by running
$AOM_HOME/scripts/emsca uninstall
-
Remove all files and sub-directories under $AOM_HOME
-
To remove any trace of the previous AOM installation, you can drop the AOM repository user in the repository by executing drop user ems cascade.
-
-
Obtain the latest Always-On Monitoring release. The ZIP file can be found on the OMS'
/sysman/ems
directory. Copy the ems_13.5.x.x.x.zip file to a location where you want AOM to be installed and unzip the file.unzip ems.zip
Note:
For AOM 13.5 to run, both the Management Agent and Enterprise Manager installation MUST be upgraded to 13.5 as well. -
Install AOM.
For setting up the very first AOM instance, the user must run
$AOM_HOME/scripts/emsca
For installing subsequent AOM instances the user must run
$AOM_HOME/scripts/emsca add_ems
-
Finally, you must run the following steps on all instances before starting AOM.
-
To generate the certificate and the wallet for the SLB host, run
$AOM_HOME/scripts/emsctl secure
-
Configure the downtime contacts. For more information, see Configuring Downtime Contacts in Enterprise Manager.
-
To sync the AOM repository schema with the Enterprise Manager repository schema, execute
$AOM_HOME/scripts/emsctl sync
-
Set the upload URL, by running
emctl config emkey -copy_to_repos
on the Enterprise Manager host. -
Start the AOM instances by running,
$AOM_HOME/scripts/emsctl start
-
Uninstalling Always-On Monitoring
If the purpose of uninstalling Always-On Monitoring is to upgrade to a new release, you should back up the following files.
-
$AOM_HOME/conf/emsConfig.properties
-
Custom certificates (if any) located at
$AOM_HOME/conf
(cwallet.sso, cwallet.sso.lck, ewallet.p12, and ewallet.p12.lck)
Use emsca to uninstall Always-On Monitoring as shown in the following example:
$ ./emsca uninstall
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Loading configuration from: /homedirectory/ems/conf/emsConfig.properties
Connecting to Always-On Monitoring repository.
Execute the following:
emctl delete property -name "oracle.sysman.core.events.ems.emsURL" -sysman_pwd <pwd> to unset the value of the property.
Uninstall completed.
Remove all files and sub-directories under $AOM_HOME.
drop user ems cascade
Configuring the Always-On Monitoring Application for Secure Communication Using the TLSv1.2 Protocol
Transport Layer Security (TLS) is a cryptographic protocol used to increase security over computer networks by providing communication privacy and data integrity between applications. In the case of Always-On Monitoring (AOM), these secure communication channels are between the following components:
-
The AOM application and the AOM repository
-
The AOM application and the Enterprise Manager repository
The following instructions cover how to enable TLSv1.2 communication for Always-On Monitoring.
Storing CA Certificates
The server CA certificates of the Always-On Monitoring repository and Enterprise Manager repository can be stored either in an external Trust Store or in the Oracle Management Service’s JDK Trust Store (JAVA_HOME/jre/lib/security/cacerts
).
Storing the Certificates in an External Trust Store
If you choose to store the certificates in an external Trust Store, then you need to set the following environment variables:
-
AOM_DB_WALLET_LOC - Absolute path to the external Trust Store. For example:
/home/aom/externalTrustSTore.jks
-
AOM_DB_WALLET_TYPE - The type of Trust Store being used. JKS. PKCS12 (For Enterprise Manager 13.3, the SSO Trust Store is not supported)
-
AOM_DB_WALLET_PASSWORD - The Trust Store password (For JKS and PKCS12)
Note:
% setenv AOM_DB_WALLET_LOC /home/aom/externalTrustStore.jks
% export AOM_DB_WALLET_LOC=/home/aom/externalTrustStore.jks
Storing the Certificates in the Oracle Management Service’s JDK Trust Store
If you choose to store the certificates in the Oracle Management Service’s JDK Trust Store, then you can use the emsca or emsctl scripts without additional configuration.
Guidelines for Configuring Always-On Monitoring to use TLSv1.2
-
During initial Always-On Monitoring setup with emsca, when prompted for the DB connection string, you must specify the connection string using the long form that has the protocol type (TCPS) being used and not the short form as shown in the following examples.
Long Form
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=myserver.myco.com) (PORT=15044)) (CONNECT_DATA =(SID=dbview))
Short Form (will not work with TCPS):
myserver.myco.com:15044:dbview
-
If Always-On Monitoring was initially configured to use the TCP protocol as part of emsca configuration, and at a later point you want to switch over to TCPS, you can re-configure Always-On Monitoring by performing the following steps:
-
Change the Always-On Monitoring database connection string in the
emsConfig.properties
file ($AOM_HOME/conf/emsConfig.properties
). -
Change Enterprise Manager Repository connection string in Always-On Monitoring database which is stored in table - EMS_SYNC_CONNECT_PROPS.connect_string.
-
Ensure that either the JDK wallet (under
JAVA_HOME/jre/lib/security/cacerts
) or the external Trust Store has the root CA certificates for both the Always-On Monitoring and Enterprise Manager repositories.
-
-
If Always-On Monitoring was initially configured to use the TCPS protocol as part of emsca configuration, and at a later point you want to switch over to TCP, you can re-configure Always-On Monitoring by performing the following steps:
-
In the
AOM_HOME/conf/emsConfig.properties
file, replace the correct AOM database connection string with the property emsRepConnectString. If TCP protocol is used to connect to the database, you can use either short or long form for the connection string. If TCPS is used, you must provide the database connection string in the long form as discussed previously. -
To change the Enterprise Manager connection string, you must update the
EMS_SYNC_CONNECT_PROPS.connect_string
field in the Always-On Monitoring database.
-