Always-On Monitoring

17 Always-On Monitoring

The Enterprise Manager Always-On Monitoring application provides the ability to monitor critical target status and metric alerts when the Oracle Management Service is unavailable. The service continuously monitors critical targets through the Enterprise Manager Agent and can be easily configured to send email notifications for these events to administrators.

Always-On Monitoring can be configured to send notifications at any time, but is particularly useful when experiencing downtime of your central Enterprise Manager site for maintenance operations such as upgrade and patching.

The Always-On Monitoring is synchronized with Enterprise Manager to reuse the configuration of monitored targets as well as requisite notification data such as notification contacts and email gateway configuration. Once properly configured and synchronized, the service will receive alerts from Enterprise Manager Agents and send email notifications to the appropriate administrators.

This chapter covers the following topics:

Prerequisites

Before installing Always-On Monitoring, ensure that the following prerequisite tasks have been performed:

Environment Setup

Install or identify an Oracle Database instance to hold the Always-On Monitoring repository.
Enterprise Manager must be upgraded to 13.4. In addition, the Management Agent should also be upgraded to 13.4.
If you do not have database administrator access, you need to ask the DB administrator to add you to the Always-On Monitoring repository. You can use the script found in Granting Required Privileges to the Always-On Monitoring Schema Owner.

Save the EM key to the Enterprise Manager repository. See Saving the Em Key for more information.

Java Requirements

JDK8 must be installed in the environment where the user will run Always-On Monitoring (including emsca). The JAVA_HOME environment variable must point to the JDK8 location.

System Resource Requirements

The number of open file descriptors used by Always-On Monitoring is http.queueSize + http.maxThreads. Always-On Monitoring also uses the database connection pool. The number of open database connections ranges between the number of active threads up to a maximum of 100 connections.
Maximum memory requirement is estimated at 10 Megabytes per thread. Memory demand peaks when Always-On Monitoring is first started because the Agents must communicate the complete status of all their monitored targets.

Installing the Always-On Monitoring Repository Database

You must install and configure an Oracle database instance to house the Always-On Monitoring repository. Because you must create the schema for the repository separately, it is critical to have an idea of the space that needs to be allocated to the schema, and how the space is broken down.

Because the purpose of Always-On Monitoring is to continuously monitor and send notifications when Enterprise Manager is down, the Always-On Monitoring Repository database should be installed as a separate instance of Oracle with the Always-On Monitoring Schema on another machine so that Always-On Monitoring continues to run when the Enterprise Manager Repository is either being upgraded or unexpectedly goes down.

Like the Enterprise Manager Repository, the Always-On Monitoring Repository needs to be installed on an Oracle database. The database version supported depends on the Always-On Monitoring version as shown in the following table:

Always-On Monitoring Version	Oracle Database Version
All Versions	Oracle Database 12.1.0.2.0 with Bundle Patch 10
Always-On Monitoring 13.3.0.0.0 and Greater	Oracle Database 12.2.0.1.0, 18.0.0.0.0, 18.3.0.0.0, and 19.0.0.0.0.

Database Sizing

The factors to consider when sizing and configuring the database are as follows:

Undo Tablespace: The Undo tablespace is utilized when running the SYNC processes, especially during the initial SYNC when potentially larger numbers of rows are pulled from the Enterprise Manager instance to Always-On Monitoring.
Temp Tablespace: The SYNC process will also utilize TEMP tablespace, especially when indexes are created or sorting occurs during the data movement process.
Always-On Monitoring Tablespace: Table data and indexes specific to Always-On Monitoring are stored in this tablespace.
Redo Logs: Redo logs should be large enough to minimize the number of checkpoints that occur during times when the SYNC is occurring. It is recommended to configure 3 x 1 GB REDO log files.

Special Oracle Parameter Settings: For Always-On Monitoring, it is desirable to maintain the parameters used in the Enterprise Manager repository, especially if the Always-On Monitoring schema is to be populated in the Enterprise Manager repository database. To ensure correctness of the Enterprise Manager parameter settings that are passed in as part of the Always-On Monitoring install, you can run the SYNC and other Always-On Monitoring functions that are part of the emsctl verbs. This is important to consider when configuring a separate database instance. As is the case with the Enterprise Manager Repository, the following parameter should be set for the Always-On Monitoring Repository to avoid unforeseen optimizer issues:

ALTER SYSTEM SET OPTIMIZER_ADAPTIVE_FEATURES=FALSE SCOPE=BOTH SID='*';

The following table represents an example of the sizing of the above components for an Always-On Monitoring schema that is being used for a large enterprise database. Note that these figures represent the sizes of the tablespaces after the initial full SYNC between Always-On Monitoring and Enterprise Manager. Also note that the Redo Log files are not included in these calculations:

Table 17-1 Always-On Monitoring Repository Tablespace Sizing

Tablespace Name	Used Space (MB)	Free Space (MB)	Total Allocation (MB)	Percentage Free (%)
TEMP	0	30,720	30,720	100%
Users	6,430	44,797	51,200	87%
SYSAUX	1,385	85	1,470	6%
SYSTEM	896	4	900	0%
UNDOTBS1	84	30,636	30,720	100%
Totals	8.769	106,246	115,015	92%

For this particular configuration, the absolute minimum for disk space required was 9 GB based on the used Tablespace. To ensure room for growth with this schema, it is recommended to plan on allocating at least 120 GB of space for the Always-On Monitoring database if creating a new Oracle instance. The TEMP and UNDO tablespaces are configured to handle initial and subsequent SYNC operations ensuring there is no issue with TEMP or REDO saturation. In addition, the Always-On Monitoring tablespace is sized to ensure no interruption due to saturation of the tablespace.

In order to accurately size the Always-On Monitoring tablespace for current usage and future growth, it is important to understand the following:

What data from what tables is being transferred from the Enterprise Manager Repository to the Always-On Monitoring Schema.
Knowing the number of rows transferred for each table, what is the approximate number of Bytes per Row for the tables and indexes?

The following table shows a subset of the tables and indexes in the Always-On Monitoring schema, highlighting the number of rows, total space consumption, and the number of bytes per row:

Table 17-2 Always-On Monitoring Table and Index Space Allocation

Owner	Segment Name	Segment Type	Partition Name	Size (MB)	Number of Rows
Always-On Monitoring	MGMT_TARGET_PROPERTIES	TABLE	Non-Partitioned	1,472	15,408,952
Always-On Monitoring	MGMT_TARGET_PROPERTIES_IDX_02	INDEX	Non-Partitioned	1,152	15,215,316
Always-On Monitoring	MGMT_TARGET_PROPERTIES_PK	INDEX	Non-Partitioned	856	15,462,104
Always-On Monitoring	MGMT_TARGET_PROPERTIES_IDX_01	INDEX	Non-Partitioned	584	14,871,521
Always-On Monitoring	EM_MANAGEABLE_ENTITIES	TABLE	Non-Partitioned	496	1,177,065
Always-On Monitoring	MGMT_TARGET_PROPERTIES_IDX_03	INDEX	Non-Partitioned	472	16,105,162
Always-On Monitoring	EM_MANAGEABLE_ENTITIES_UK1	INDEX	Non-Partitioned	136	1,201,617
Always-On Monitoring	EM_MANAGEABLE_ENTITIES_UK2	INDEX	Non-Partitioned	120	1,133,804
Always-On Monitoring	EM_MANAGEABLE_ENTITIES_IDX01	INDEX	Non-Partitioned	112	1,184,414
Always-On Monitoring	EM_MANAGEABLE_ENTITIES_IDX07	INDEX	Non-Partitioned	112	1,166,536
Always-On Monitoring	EM_MANAGEABLE_ENTITIES_IDX03	INDEX	Non-Partitioned	96	1,182,817
Always-On Monitoring	EM_MANAGEABLE_ENTITIES_IDX04	INDEX	Non-Partitioned	80	1,218,977
Always-On Monitoring	EM_MANAGEABLE_ENTITIES_IDX05	INDEX	Non-Partitioned	72	1,104,428
Always-On Monitoring	EM_MANAGEABLE_ENTITIES_IDX06	INDEX	Non-Partitioned	59	673,838
Always-On Monitoring	EM_MANAGEABLE_ENTITIES_IDX08	INDEX	Non-Partitioned	44	1,215,411
Always-On Monitoring	EM_MANAGEABLE_ENTITIES_PK	INDEX	Non-Partitioned	41	1,172,101
Always-On Monitoring	EM_MANAGEABLE_ENTITIES_IDX09	INDEX	Non-Partitioned	40	1,192,236
Always-On Monitoring	EM_MANAGEABLE_ENTITIES_IDX02	INDEX	Non-Partitioned	37	1,178,410
Always-On Monitoring	EM_VIOLATIONS	TABLE	Non-Partitioned	28	112,106
Always-On Monitoring	EM_METRIC_COLUMN_VER_PK	INDEX	Non-Partitioned	17	611,561

Database Character Set Definition

To ensure that the data being transferred from the Enterprise Manager Repository to the Always-On Monitoring Repository is stored and represented properly when the Always-On Monitoring Repository is on a separate instance from Enterprise Manager, it is important to ensure that the character set matches between the two databases/instances. For Enterprise Manager, the NLS_CHARACTERSET is defined as AL32UTF8..

If using Oracle Installer to manually set up the database instance that Always-On Monitoring uses, the instance will default to WE8MSWIN1252. Ensure at this point that the option AL32UTF8 is selected from the drop down box so that the correct character set is installed from the start..

To determine the character set on the Enterprise Manager Repository, run the following query:

SELECT * from nls_database_parameters where parameter='NLS_CHARACTERSET';

Run this query as well on the Always-On Monitoring Repository to ensure the character sets match. If the Always-On Monitoring Repository character set is different, the following is recommended to change the character set on the Always-On Monitoring Repository to match that of the Enterprise Manager Repository.

--Shutdown and start the database in a restricted mode.  
SHUTDOWN IMMEDIATE;  
STARTUP RESTRICT;  
 
--Set JOB_QUEUE_PROCESSES and AQ_TM_PROCESSES to its default value i.e.0  
ALTER SYSTEM SET JOB_QUEUE_PROCESSES = 0;  
ALTER SYSTEM SET AQ_TM_PROCESSES=0;  
 
-- Set the required character set.  
ALTER DATABASE CHARACTER SET AL32UTF8;  
 
 -- if the above fails:  
ALTER DATABASE CHARACTER SET INTERNAL_USE AL32UTF8;  
 
--Shutdown and restart the database in normal mode.  
SHUTDOWN IMMEDIATE;  
STARTUP;  
/

Creating the Always-On Monitoring Repository User

The Always-On Monitoring user should be created by a database administrator with SYSDBA privileges. If you are running emsca as a user with SYSDBA privileges, you can skip this section as the emsca configuration assistant will automatically create the user for you.

Note:

The Always-On Monitoring Repository should not be housed within the Enterprise Manager Repository database. Housing the Always-On Monitoring Repository in a separate database allows Always-On Monitoring to function even if the Enterprise Manager Repository goes down.

Connect to the database to be used as the Always-On Monitoring Repository, create the Always-On Monitoring user and then grant it the required privileges.

Note:

The Always-On Monitoring user should be created by a database administrator with SYSDBA privileges. Alternatively, you can supply SYSDBA credentials when running the emsca configuration utility. emsca will then create the Always-On Monitoring repository user automatically.

If you have SYSDBA access, you can skip this step since it will be done automatically by the emsca configuration assistant. See Using the Always-On Monitoring Configuration Assistant (EMSCA) for more information.

Granting Required Privileges to the Always-On Monitoring Schema Owner

To ensure proper functionality of Always-On Monitoring, the Always-On Monitoring schema owner needs to have the correct privileges. The following sample script details all grants required for the Always-On Monitoring Schema:

create user ems identified by ems;
grant CREATE SESSION, 
      ALTER SESSION, 
      CREATE DATABASE LINK, 
      CREATE MATERIALIZED VIEW, 
      CREATE PROCEDURE, 
      CREATE PUBLIC SYNONYM, 
      CREATE ROLE, 
      CREATE SEQUENCE, 
      CREATE SYNONYM, 
      CREATE TABLE, 
      CREATE TRIGGER, 
      CREATE TYPE, 
      CREATE VIEW, 
      UNLIMITED TABLESPACE, 
      SELECT ANY DICTIONARY to ems;
grant EXECUTE ON SYS.DBMS_CRYPTO to ems;
grant EXECUTE ON SYS.DBMS_AQADM to ems;
grant EXECUTE ON SYS.DBMS_AQ to ems;
grant EXECUTE ON SYS.DBMS_AQIN to ems;
grant EXECUTE on SYS.DBMS_LOCK to ems;
grant EXECUTE ON SYS.DBMS_SCHEDULER to ems;
grant create job to ems;

Under certain circumstances you may need to grant privileges directly to an Always-On Monitoring user. However, directly granting privileges to any user may violate security policy and compliance rules. In this situation, you can can assign most of the privileges to a role and then assign that role to the user.

For multitenant environments, when creating a user in a multitenant container database (CDB) and not in one of its constituent pluggable databases (PDB), then the user has to be prefixed by "C##".

Best Practices

Oracle recommends keeping an Always-On Monitoring instance running at all times regardless of whether Enterprise Manager is up or down. By telling the Always-On Monitoring to turn off notifications, you can have it run in the background while Enterprise Manager is up. During Enterprise Manager downtime, whether planned or unplanned, you can issue an emsctl command to tell Always-On Monitoring to restart notifications. The primary advantage to having Always-On Monitoring running at all times is that it will stay up to date with any changes made to your Enterprise Manager installation. Always-On Monitoring keeps itself in sync with Enterprise Manager by periodically running a synchronization job. This job runs every 24 hours by default, however the job execution time can be changed to meet the needs of your monitored environment.

You may also run Always-On Monitoring only when needed instead of having it run constantly in the background. However, doing so will require Always-On Monitoring to synchronize with the Enterprise Manager OMS. Depending on the size of your Enterprise Manager deployment, synchronization may take 10-15 minutes in order to obtain current information on all Enterprise Manager targets and metrics.

Note:

Always-On Monitoring and Enterprise Manager must be in SYNC before Enterprise Manager goes down.

Installing Always-On Monitoring

Always-On Monitoring is a self-contained application that is supplied with the Enterprise Manager software distribution.

Steps to install Always-On Monitoring:

Uninstall any previous installations of Always-On Monitoring. For more information, see Uninstalling Always-On Monitoring.
Locate the latest Always-On Monitoring ZIP file. The ZIP file can be found in the /sysman/ems directory of the OMS.
Copy the ZIP file to the Always-On Monitoring host and unzip the contents to the location where Always-On Monitoring is to be installed.

Installing Always-On Monitoring from an Enterprise Manager Software Distribution

The Always-On Monitoring installation zip file is shipped with Enterprise Manager. From the sysman/ems directory, find the Always-On Monitoring installation ZIP file (e.g. ems.zip). Unzip the file to the location where you want to install Always-On Monitoring.

Installing Multiple Always-On Monitoring Instances

You can set up multiple Always-On Monitoring (AOM) instances to ensure Always-On Monitoring is available in the event of outages. See Setting Up Multiple Always-On Monitoring Instances for more information.

Configuring Always-On Monitoring

Once Always-On Monitoring has been installed, the Always-On Monitoring configuration assistant script can be used to configure the service to communicate with the Enterprise Manager repository.

Always-On Monitoring configuration involves the following steps:

Saving the Em Key

Always-On Monitoring requires the Em key in order to be able to synchronize with the Enterprise Manager repository and reuse the SMTP (email) gateway configuration information from Enterprise Manager.

Note:

The Em key needs to be copied to the Enterprise Manager repository BEFORE configuring Always-On Monitoring. Not doing so will result in errors when running emsca.

Once Always-On Monitoring configuration is complete (emsca has successfully run through completion), immediately remove the Em key from the Enterprise Manager repository. Note: Always-On Monitoring does not have to be restarted. To store the key in the repository, run the following emctl commands from the $MW_HOME/bin directory, where $MW_HOME is the Enterprise Manager Middleware Home directory.

% emctl config emkey -copy_to_repos

Once Always-On Monitoring configuration is complete, execute the following command to remove the key.

% emctl config emkey -remove_from_repos

Using the Always-On Monitoring Configuration Assistant (EMSCA)

The Always-On Monitoring configuration assistant, emsca, is a script located under the Always-On Monitoring installation scripts directory. Running emsca requires the bash shell to be installed.

If you have database administrator credentials, you can pass them to the emsca script and it will create the Always-On Monitoring user for you.

If you do not have database administrator credentials, you need to:

Ask the database administrator to run the script. See Creating the Always-On Monitoring Repository User.
Run emsca -createEmsDbUser=false

The following example usage scenarios assume Always-On Monitoring is installed in a location referred to using the environment variable AOM_HOME. Run the configuration assistant located in $AOM_HOME/scripts. The EMSCA may be invoked with no parameters and will prompt for the necessary information. Once the configuration has completed, record the Always-On Monitoring Upload URL as it will be used later to configure Enterprise Manager.

Example EMSCA Installation Scenarios

The user installing Always-On Monitoring has SYSDBA credentials.

Copyright (c) 2017, 2020 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Always-On Monitoring Repository Connection String : myserver.myco.com:25059:s307480
 Always-On Monitoring Repository Username [ems] :   
 Always-On Monitoring Repository Password [ems] : 
 
User "ems" cannot be found in the database.
In order to create this user, SYSDBA credentials are required. If you do not want to continue, answer "n" to the question below.
Create the Always-On Monitoring Repository user [y] : 
 Always-On Monitoring Repository SYSDBA Username : sys
 Always-On Monitoring Repository SYSDBA Password : 
 
 Enterprise Manager Repository Connection String : myserver.myco.com:25059:s307480
 Enterprise Manager Repository Username : sysman
 Enterprise Manager Repository Password : 
 
Creating Always-On Monitoring repository user ems
 Agent Registration Password : 
 
 Keystore for host myserver.myco.com created successfully.
 Connecting to Always-On Monitoring Repository.
 Creating Always-On Monitoring Repository schema
 Creating repository storage for Targets data.
 Creating repository storage for Alerts and Availability data.
 Creating repository storage for Notification Metadata data.
 Creating repository storage for Target Metric Metadata data.
 Registering Always-On Monitoring instance
 Always-On Monitoring Upload URL: https://myserver.myco.com:8081/upload

The user installing Always-On Monitoring does not have SYSDBA credentials.

Example 1: SYSDBA creates an Always-On Monitoring user and grants user privileges shown in the following example.

aomuser is the name of the Always-On Monitoring user in the database.

SYSDBA privileges are required to run the following script

create user aomuser identified by <password>;
grant CREATE JOB, 
         CREATE SESSION, 
         ALTER SESSION, 
         CREATE DATABASE LINK, 
         CREATE MATERIALIZED VIEW,
         CREATE PROCEDURE, 
         CREATE PUBLIC SYNONYM, 
         CREATE ROLE, 
         CREATE SEQUENCE, 
         CREATE SYNONYM,
         CREATE TABLE, 
         CREATE TRIGGER, 
         CREATE TYPE, 
         CREATE VIEW, 
         UNLIMITED TABLESPACE, 
         SELECT ANY DICTIONARY to aomuser;
 
grant EXECUTE ON SYS.DBMS_CRYPTO to aomuser;
grant EXECUTE ON SYS.DBMS_AQADM to aomuser;
grant EXECUTE ON SYS.DBMS_AQ to aomuser;
grant EXECUTE ON SYS.DBMS_AQIN to aomuser;
grant EXECUTE ON SYS.DBMS_SCHEDULER to aomuser;
grant EXECUTE ON SYS.DBMS_LOCK to aomuser;

Example 2: The Always-On Monitoring user invokes the EMSCA script..

Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 2017, 2020 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
 Always-On Monitoring Repository Connection String : myserver.myco.com:25059:s307480
 Always-On Monitoring Repository Username [ems] : aomuser
 Always-On Monitoring Repository Password [ems] : 
 
Always-On Monitoring Repository user "aomuser" has already been created
 Enterprise Manager Repository Connection String : myserver.myco.com:25059:s307480
 Enterprise Manager Repository Username : sysman
 Enterprise Manager Repository Password : 
 
 Agent Registration Password : 
 
 Keystore for host myserver.myco.com created successfully.
 Connecting to Always-On Monitoring Repository.
 Creating Always-On Monitoring Repository schema
 Creating repository storage for Targets data.
 Creating repository storage for Alerts and Availability data.
 Creating repository storage for Notification Metadata data.
 Creating repository storage for Target Metric Metadata data.
 Registering Always-On Monitoring instance
 Always-On Monitoring Upload URL: https://myserver.myco.com:8081/upload

SYSDBA creates a role and assigns it to Always-On Monitoring user then emsca is executed.

Some sites prefer to create a role and assign that role to the Always-On Monitoring user. In this situation, you must run the scripts shown in the following examples.

Example 1: SYSDBA creates a role and then assigns it to the Always-On Monitoring user.

create user <aom_user> identified by <aom_password>;
create role ems_role;
grant CREATE SESSION, 
         ALTER SESSION, 
         CREATE DATABASE LINK, 
         CREATE MATERIALIZED VIEW, 
         CREATE PROCEDURE, 
         CREATE PUBLIC SYNONYM, 
         CREATE ROLE, 
         CREATE SEQUENCE, 
         CREATE SYNONYM, 
         CREATE TABLE, 
         CREATE TRIGGER, 
         CREATE TYPE, 
         CREATE VIEW, 
         SELECT ANY DICTIONARY to ems_role;
grant ems_role to <aom_user>;

Example 2: SYSDBA grants the following privileges directly to the Always-On Monitoring user.

grant CREATE JOB, 
         UNLIMITED TABLESPACE to <aom_user>;
 
grant EXECUTE ON SYS.DBMS_CRYPTO to <aom_user>;
grant EXECUTE ON SYS.DBMS_AQADM to <aom_user>;
grant EXECUTE ON SYS.DBMS_AQ to <aom_user>;
grant EXECUTE ON SYS.DBMS_AQIN to <aom_user>;
grant EXECUTE ON SYS.DBMS_SCHEDULER to <aom_user>;
grant EXECUTE ON SYS.DBMS_LOCK to <aom_user>;

Example 3: The Always-On Monitoring user invokes the emsca script.

Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 2017, 2020 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
 Always-On Monitoring Repository Connection String : myserver.myco.com:25059:s307480
 Always-On Monitoring Repository Username [ems] : 
 Always-On Monitoring Repository Password [ems] : 
 
Always-On Monitoring Repository user "ems" has already been created
 Enterprise Manager Repository Connection String : myserver.myco.com:25059:s307480
 Enterprise Manager Repository Username : sysman
 Enterprise Manager Repository Password : 
 
 Agent Registration Password : 
 
 Keystore for host myserver.myco.com created successfully.
 Connecting to Always-On Monitoring Repository.
 Creating Always-On Monitoring Repository schema
 Creating repository storage for Targets data.
 Creating repository storage for Alerts and Availability data.
 Creating repository storage for Notification Metadata data.
 Creating repository storage for Target Metric Metadata data.
 Registering Always-On Monitoring instance
 Always-On Monitoring Upload URL: https://myserver.myco.com:8081/upload

Running EMSCA with a Response File

You can also use the -responseFile option to provide the parameters to the script using a response input file. For example

$ emsca -responseFile=<response_filename>

Where response_filename is the response file containing the following script parameters:

emsRepConnectString=localhost:1521:xe
emsRepUsername=ems
emsRepPassword=ems
emRepConnectString=mymachine.mycompany.com:15044:semgc3
emRepUsername=sysman
emRepPassword=sysman
#
emsPort=8081
http.protocol=http
#

The following table lists available EMSCA parameters.

Table 17-3 EMSCA Parameters

EMSCA Prompt	Description
Always-On Monitoring Repository Connection String	The connect string used to locate the Always-On Monitoring repository. This may be any valid service descriptor or may be a `host:port:sid`.
Always-On Monitoring Repository Username	The username associated with the Always-On Monitoring repository. See Creating the Always-On Monitoring Repository Userfor more details.
Always-On Monitoring Repository Password	The password associated with the Always-On Monitoring repository user. See Creating the Always-On Monitoring Repository User for more details.
Enterprise Manager Repository Connection String	The connect string used to locate the Enterprise Manager repository. This may be any valid service descriptor or may be a host:port:sid.
Enterprise Manager Repository Username	The username of the Enterprise Manager SYSMAN user.
Enterprise Manager Repository Password	The password of the Enterprise Manager SYSMAN user.

Removing the Em Key

After Always-On Monitoring configuration is complete (after emsca has been run), execute the following command to remove the key.

% emctl config emkey -remove_from_repos

Always-On Monitoring does not have to be restarted.

Configuring Email Servers in Enterprise Manager

When Always-On Monitoring sends email notifications, it does so using email server configurations that it obtains from Enterprise Manager during the synchronization step. For Always-On Monitoring to function properly, email servers must be configured in Enterprise Manager before performing an Always-On Monitoring synchronization.

If changes to the email server configurations are made in Enterprise Manager, Always-On Monitoring must be synchronized again to retrieve the updated email server configuration. See Updating Always-On Monitoring “Running an Incremental Synchronization Manually” for additional details.

Configuring Downtime Contacts in Enterprise Manager

Once Always-On Monitoring is synchronized with Enterprise Manager, the monitoring service can send email notifications to the appropriate administrators when the OMS is down via configured downtime contacts. Prior to running Always-On Monitoring for the first time, downtime contacts should be configured in Enterprise Manager. The downtime contacts that Always-On Monitoring sends notification to may be any one of the following forms.

Global downtime contact
Per-target downtime contact based on target property
Per-target downtime contact based on event rules

If email addresses are specified for both Global and Per-target forms, then all email addresses will be used.

Global Downtime Contact

The global downtime contact is a single property set on the Enterprise Manager site. Once set, all target status events and metric alerts across all targets will be sent to the recipients specified in the global downtime contact property.

% emcli set_oms_property -property_name='oracle.sysman.core.events.ems.downtimeContact' 
-property_value='<email addresses>'

To delete the global downtime contact, run the following command:

emctl delete property -name 'oracle.sysman.core.events.ems.downtimeContact'

Per-Target Downtime Contact from Target Property

The email addresses for these downtime contacts should be specified in the Downtime Contact target property for each target. There are three ways to specify the Downtime Contact target property:

For each target, navigate to the Target Properties page, which can be accessed from the target's homepage. From the target menu on the target homepage, select Target Setup and then Properties. The Target Properties page displays.

Description of the illustration target_downtime_contact.gif

Click on Edit and specify the email address in the Downtime Contact target property. You can specify multiple email addresses by separating them with commas.
Use the EM CLI set_target_property_value verb.

emcli set_target_property_value -property_records="target_name:target_type:property_name:property_value"

Example:

%./emcli set_target_property_value -property_records="myhost:host:Downtime Contact:John.Doe@mycompany.com"

For more information about the set_target_property_value verb, see the Oracle Enterprise Manager Command Line Interface Guide.
By leveraging the email recipients in the target down event rules in Enterprise Manager Cloud Control. See the following section for more details.

Per-Target Downtime Contact from Event Rules

Always-On Monitoring may also send email notifications to different users for each target. These contacts are generated in Enterprise Manager based on the event rules for that target. Therefore, as event rules are changed in Enterprise Manager, the contacts must be re-generated and an incremental synchronization performed

By leveraging the event rule setup, the downtime contact will be generated based on the email recipient for the event rule for a Target Availability event type where Down status has been selected.

Note:

Although downtime contacts are generating using only Target Availability event rules, Always-On Monitoring will send notifications for both target availability and metric threshold alerts.

You can review and update the recipients of your target availability (status down) event rules. Doing so allows you to generate a list of downtime contacts using EM CLI or by submitting the downtime contact generation job.

Generating Downtime Contact using EM CLI

From the EM CLI, use the generate_downtime_contact command to produce downtime contacts for a specific target. The command simulates ALL Target Availability rules that would affect that particular target and generates a list of email addresses that includes the output from all rules. The optional -set parameter will cause the contact to be saved so that Always-On Monitoring synchronization can retrieve it later.

% emcli generate_downtime_contact -target_name=<name> -target_type=<type> -set

Generating Downtime Contact Enterprise Manager Job

% emcli create_job -input_file=property_file:<job_prop_file> -name="GenerateDowntimeContacts"

Where the job_prop_file refers to a properties file with the following content:

# job type
type=CoreSetDowntimeContacts
# description
description=You job description
# target names, the list of target names
variable.target_names=host1,database2
# target types, the list of target types corresponding to the target list above
variable.target_types=host,oracle_database
# set on all targets, true to update contacts, false to print output but not save
variable.set_all_targets=true
# frequency to run job, refer to Enterprise Manager documentation for options
schedule.frequency=IMMEDIATE

Note:

The process of generating the downtime contacts thru either the EM CLI or the downtime contact generation job, does not immediately make the contacts available to the monitoring service. A separate Always-On Monitoring synchronization must be done each time these contacts are updated. This is done using the emsctl sync command.

Types of Alerts Received by Downtime Contacts

Once configured, Always-On Monitoring will send email notifications for:

All target status alerts and metric alerts (critical, warning, and clear).
Metric collection errors (both critical and clear).

Target status alerts for aggregate targets, such as groups and clusters, whose status is based on the status of its component members is not supported. Instead, alerts will be sent for the individual members of these aggregate targets..

Note:

Currently, email notification is the only type of notification supported.

The recipients of the email should be email addresses. The recipients are specified using the global property downtimeContact or the per-target Downtime Contact. If both are specified, emails will be sent to all recipients specified.

If the global property downtimeContact is specified, all alerts on all targets will be sent to the recipients specified in the global property downtimeContact. If the per-target property Downtime Contact is specified, all target status and metric alerts for that target will be sent to the recipients specified in that property.

Setting Downtime Contacts for Composite Targets

You can set the Downtime Contacts target property for a composite target so that the target property is propagated to its member targets. You can perform this update using the EMCLI set_target_property_value verb. See set_target_property_value in the Oracle® Enterprise Manager Command Line Interface for more information about this verb.

Examples

Example 1: The following example sets the AOM Downtime Contact target property for a DB static group.

>  ./emcli set_target_property_value -property_records="db_group:composite:Downtime Contact:aom@test.com" -propagate_to_members
 Properties updated successfully

Example 2: The following example sets the AOM Downtime Contact target property for a DB system target.

> ./emcli set_target_property_value -property_records="eva4.us.oracle.com_sys:oracle_dbsys:Downtime Contact:aom@test.com" -propagate_to_members
Properties updated successfully

Synchronizing Always-On Monitoring with Enterprise Manager for the First Time

Before starting Always-On Monitoring for the first time, you must synchronize Always-On Monitoring with Enterprise Manager. Synchronization copies notification configuration and downtime contacts from Enterprise Manager targets, thus allowing Always-On Monitoring to monitor alerts and send email notifications.

To perform Always-On Monitoring-Enterprise Manager synchronization:

% $AOM_HOME/scripts/emsctl sync

Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
 Connecting to Always-On Monitoring repository.
Starting synchronization with Enterprise Manager.
Synchronizing with Enterprise Manager repository: sysman@myserver.myco.com:35074:semgc4
Synchronizing Targets data.
Synchronizing Alerts and Availability data.
Synchronizing Notification Metadata data.
Synchronizing Target Metric Metadata data.
Synchronization complete at : Thu Oct 22 06:50:56 PDT 2016

Note:

Synchronization should be run again whenever there are changes made to Enterprise Manager. For example, after adding new hosts or targets, changing the email server, or changing the downtime contacts, an incremental synchronization is required in order for Always-On Monitoring to pick up the latest configurations.. See "Running an Incremental Synchronization Manually" for additional details.

Configuring Enterprise Manager to Work with Always-On Monitoring

The final step in the configuration of Always-On Monitoring is to update Enterprise Manager to include the upload URL for the service. The upload URL is displayed by Always-On Monitoring Configuration Assistant upon successful completion of the configuration of the service. Always-On Monitoring configuration properties file also includes the protocol and port configured for the service so that the upload URL will be of the form:

https://yourhostname:8081/upload

Once configured, the URL will be sent to all Agents (version 13.1 and greater), both existing Agents and any Agents deployed in future. Hence, configuration only needs to be done once. Agents less than 13.1 will not receive the Always-On Monitoring upload URL.

HTTPS is the default protocol for Always-On Monitoring and the default port is 8081. Please refer to the emsca or Always-On Monitoring configuration file for the configured values.You can find the configuration file at the following location:

$AOM_HOME/conf/emsConfig.properties

To configure Enterprise Manager with Always-On Monitoring upload location, the following command should be executed from Enterprise Manager.

% emctl set property -name "oracle.sysman.core.events.ems.emsURL" -value "https://yourhostname:8081/upload" -sysman_pwd sysman

Starting Always-On Monitoring

Start Always-On Monitoring. This step is important to enable notifications. For details on starting AOM, see Starting Always-On Monitoring.

Enabling Notifications

Always-On Monitoring initial configuration has notifications disabled—the service will not send emails as new alerts are received. You must first enable Always-On Monitoring to send email notifications.The Always-On Monitoring notification enable/disable setting is persistent across starts/stops of Always-On Monitoring. The enable_notification command will automatically perform an incremental sync.

% $AOM_HOME/scripts/emsctl enable_notification
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Notifications have been enabled. There are downtime contacts configured.
 Connecting to Always-On Monitoring repository.
Starting synchronization with Enterprise Manager.
Synchronizing with Enterprise Manager repository: sysman@myserver.myco.com:35074:semgc4
Synchronizing Targets data.
Synchronizing Alerts and Availability data.
Synchronizing Notification Metadata data.
Synchronizing Target Metric Metadata data.
Synchronization complete at : Thu Oct 22 07:01:20 PDT 2020

If you do not want to perform a SYNC then add the -nosync option.

$AOM_HOME/scripts/emsctl enable_notification -nosync
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Notifications have been enabled. There are downtime contacts configured.

To disable notifications but leave Always-On Monitoring running use the disable_notification command:

% $AOM_HOME/scripts/emsctl disable_notification
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Notifications have been disabled. Filters have been deleted.

Filtering Alert Notifications by Event Type and Severity

When only specific alert types are pertinent, you need to have a way of filtering out extraneous alerts so you don't have to scan through all alerts just to find the ones you're interested in. Always-On Monitoring lets you filter your alert notifications based on either event type or severity.

Filtering Alert Notifications by Event Type

Alert Notification Filtering by event type lets you exclude alert notifications for certain event types. For example, you want to receive only target availability alerts. You can filter out alerts for the following event types:

Metric Alert (metricAlert)
Availability (targetAvail)
Metric Evaluation Errors (metricError)

Default Settings

Target availability & metric alert notifications are enabled
Metric evaluation error alert notifications are disabled

Example 1: To receive only target availability alerts, run the following:

$AOM_HOME/scripts/emsctl disable_notification
          -alert_types=targetAvail,metricAlert,metricError 
$AOM_HOME/scripts/emsctl enable_notification -alert_types=targetAvail

Example 2: To add notification for metric error alerts in addition to target availability alerts:

$AOM_HOME/scripts/emsctl enable_notification -alert_types=metricError
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021, Oracle Corporation.  All rights reserved.
------------------------------------------------------------------
Notification filters have been enabled. There are downtime contacts configured.
Notification filters
    Target Availability (targetAvail) : true
    Metric Alerts (metricAlert) : false
    Metric Errors (metricError) : true

Filtering Alert Notifications by Severity

Alert Notification Filtering by severity lets you exclude alert notifications for certain severities. For example, you want to receive only Fatal, Critical, and Warning alerts. You can filter out alerts for the following event types:

Fatal
Critical
Warning
Advisory
Informational
Clear

Default Settings

Fatal, Critical, and Warning alert notifications are enabled
Advisory, Informational, and Clear alert notifications are disabled

Example 1: To receive only Fatal and Critical severity alerts, run the following:

$AOM_HOME/scripts/emsctl disable_notification –severities=fatal,critical,warning,advisory,informational,clear
$AOM_HOME/scripts/emsctl enable_notification - severities = fatal,critical

Example 2: To add notification for Warning alerts in addition to Fatal and Critical severity alerts:

$AOM_HOME/scripts/emsctl enable_notification -severities=warning

Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021, Oracle Corporation.  All rights reserved.
------------------------------------------------------------------
Notification filters have been enabled. There are downtime contacts configured.
Notification filters - Alert types
    Target Availability (targetAvail) : true
    Metric Alerts (metricAlert) : true
    Metric Errors (metricError) : false
Notification filters - Severity
    Fatal (fatal) : true
    Critical (critical) : true
    Warning (warning) : true
    Advisory (advisory) : false
    Informational (informational) : false
    Clear (clear) : false

Verifying the Always-On Monitoring Upload URL on Enterprise Manager

You can determine the upload URL by issuing the following command:

$AOM_HOME/scripts/emsctl status

Example:

$AOM_HOME/scripts/emsctl statusOracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021, Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Always-On Monitoring Version : 13.5.0.0.0
Always-On Monitoring Home : /mycomputer/ems
Started At : December 23, 2019 2:24:25 PM PST
Last Repository Sync : December 23, 2019 2:25:49 PM PST
Upload URL : https://myserver:8081/upload
Always-On Monitoring Process ID : 24924
Always-On Monitoring Repository : myserver.myco.com:15044:sarview
Enterprise Manager Repository : myserver.myco.com:15044:sarview
Notifications Enabled : true
Notification filters - Alert types
    Target Availability (targetAvail) : true
    Metric Alerts (metricAlert) : true
    Metric Errors (metricError) : false
Notification filters - Severity
    Fatal (fatal) : true
    Critical (critical) : true
    Warning (warning) : true
    Advisory (advisory) : false
    Informational (informational) : false
    Clear (clear) : false
Total Downtime Contacts Configured : 1

You can verify the setting in the OMS by using the emctl get property command.

You can verify the setting on a particular Agent by looking at $AGENT_HOME/sysman/config/emd.properties and searching for EMS_URL.

Controlling the Service

Starting Always-On Monitoring

% $AOM_HOME/scripts/emsctl start
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Pinging Always-On Monitoring...
Always-On Monitoring is not running.
Starting Always-On Monitoring.
Waiting for process to start
Retrying...
Notifications Enabled : false
Total Downtime Contacts Configured : 1
Always-On Monitoring is up.

Getting Status & Logs

Use the emsctl status command to ascertain the operational status of Always-On Monitoring.

Example:

% $AOM_HOME/scripts/emsctl status

Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Pinging Always-On Monitoring...
Always-On Monitoring is not running.
Starting Always-On Monitoring.
Waiting for process to start
Retrying...
Notifications Enabled : false
Total Downtime Contacts Configured : 1
Always-On Monitoring is up.

Pinging Always-On Monitoring

In addition to running the emsctl status command, you can also issue the ping command if you just want to see that Always-On Monitoring service is up without listing all the operational details.

Example:

$AOM_HOME/scripts/emsctl ping
 
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Always-On Monitoring is running

Log Files

Log files record Always-On Monitoring events that occur during operation and are generated as follows:

emsca logs: emsca.err (only errors), emsca.log.0 (rotating log file that contains all output including errors).
ems logs: ems.err (only errors), ems.log.0 (rotating log file that contains all output including errors).

These files are located in the $AOM_HOME/logs directory.

Log levels determine the type and operational severity of information recorded in the log files. Levels can be set to any one of the following:

INFO: Always-On Monitoring operational events such and login, logout, or any other events tied to Always-On Monitoring lifecycle.
DEBUG: Information considered useful when tracing the flow of Always-On Monitoring events to isolate specific problems.
WARN (default setting): Unexpected operational Always-On Monitoring events. These are Always-On Monitoring events that should be tracked, but may not require immediate administrator intervention.
ERROR: Always-On Monitoring operational malfunction.

To change the log level, add the logLevel=... property to $AOM_HOME/conf/emsConfig.properties. Note that this applies only to the current Always-On Monitoring instance.

Note:

You must bounce Always-On Monitoring in order for the log level changes to take affect.

Stopping the Service

Use the emsctl stop command to stop Always-On Monitoring.

$AOM_HOME/scripts/emsctl stop
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 2017, 2021 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Shutting down Always-On Monitoring.
Always-On Monitoring is stopped.

Always-On Monitoring Commands

The following EMSCTL commands are used to control Always-On Monitoring.

Command	What it does
emsctl start	Starts Always-On Monitoring.
emsctl stop	Shuts down Always-On Monitoring.
emsctl enable_notification -nosync (optional)	Allow Always-On Monitoring to send email notifications. By default, an incremental SYNC is performed when enabling email notifications. To prevent synchronization with the Enterprise Manager repository, specify the -nosync option.
emsctl disable_notification	Prevent Always-On Monitoring from sending email notifications.
emsctl list_agents	Lists the URLs of Agents and a count of Agents that have communicated with the Always-On Monitoring Service in the past hour.
emsctl ping	Ping the Always-On Monitoring service to determine whether it is up or down.
emsctl sync	Synchronize Always-On Monitoring target and notification data with the Enterprise Manager repository.
emsctl secure [-wallet=<absolute path to the wallet location>] \| [-reg_pwd=<Agent registration password>] [-host=<Fully qualified Always-On Monitoring hostname>] [-cert_validity=<validity period in days - defaults to 10 years] [-key_strength=<certificate key strength - defaults to 1024] [sign_alg=<md5\|sha1\|sha256\|sha384\|sha512 - defaults to sha512>	Generates a wallet file for Always-On Monitoring service to run on a secured port.
emsctl status	Display information related to Always-On Monitoring operational status.
emsctl getstats	Display the current Always-On Monitoring performance statistics
emsctl list_properties	Display Always-On Monitoring configuration properties.
emsctl set_property -name=<property name> -value=<property value>	Set Always-On Monitoring configuration properties. For a complete list of configuration properties, see Modifiable Always-On Monitoring Properties
emsctl unset_property -name=<property name>	Unset an existing Always-On Monitoring configuration property. For a complete list of configuration properties, see Modifiable Always-On Monitoring Properties.
emsctl update_task -task_name=<SYNC\|PURGE> -run_interval=<time in minutes between task executions>	Update the intervals for performing Always-On Monitoring tasks such as SYNC or PURGE.
emsctl set_em_repos_conn -username=<em username> -password=<em password> -connect_string=<em connect descriptor>	Update the username, password and connect string used to connect to Enterprise Manager repository database.
emsctl set_ems_repos_conn -username=<ems username> -password=<ems password> -connect_string=<ems connect descriptor>	Update username, password and connect string used to connect to the Always-On Monitoring repository database.

Updating Always-On Monitoring

In order to carry out the notification function of Enterprise Manager, Always-On Monitoring requires target monitoring and administrator notification configuration data to be synchronized with the Enterprise Manager repository. Full synchronization is performed when you install Always-On Monitoring and run the following command:

$AOM_HOME/scripts/emsctl sync

Incremental synchronization must be performed thereafter in order to keep Always-On Monitoring monitoring/notification configuration current.

The following changes to Enterprise Manager will require an incremental Always-On Monitoring synchronization:

New hosts (Agents) added
New targets added
Changes to downtime contacts
Changes to administrator email addresses
Changes to email server configuration

Always-On Monitoring stays in SYNC with Enterprise Manager automatically via an incremental synchronization job that runs every 24 hours. You can change the synchronization interval using the emsctl update_task command.

$AOM_HOME/scripts/emsctl update_task -task_name=<SYNC|PURGE> -run_interval=<time in minutes between task executions>

Incremental synchronization can be run manually via the emsctl sync command.

Running an Incremental Synchronization Manually

You can, at any time, perform an incremental Always-On Monitoring synchronization by running the sync command.

% $AOM_HOME/scripts/emsctl sync

When performing a manual incremental synchronization, you do not need to shut down Always-On Monitoring: You can run the SYNC command regardless of whether Always-On Monitoring is up or down.

Note:

An incremental synchronization is automatically performed whenever you run the enable_notification command. You can bypass the synchronization by specifying the -nosync option.

Data Maintenance

The Always-On Monitoring repository contains historical availability and metric violations data. This data is maintained for 6 months (default interval). Data older than 6 months is deleted (purged) from the repository.

Data maintenance is controlled by following settings:

Purge Interval—Determines how often the data is purged from Always-On Monitoring repository.
Purge Duration—Determines how much data is kept during the purge process.

You can change the purge interval via the update_task command:

$AOM_HOME/scripts/emsctl update_task -task_name=PURGE -run_interval=<the number of minutes between purge runs>

You can change the purge duration via the set_property command:

$AOM_HOME/scripts/emsctl set_property -name=purge.Duration -value=<the number of minutes of data to keep>

Controlling Always-On Monitoring Configuration Settings

Always-On Monitoring configuration settings define how your Always-On Monitoring deployment operates within your environment.

Properties that are specific to a particular instance of Always-On Monitoring can be found at the following location:

$AOM_HOME/conf/emsConfig.properties

Properties are added/updated in the emsConfig.properties file using a text editor. Always-On Monitoring must be restarted in order for the changes to take effect. Global properties that are shared by all Always-On Monitoring instances are stored in the Always-On Monitoring repository and are manipulated using the emsctl commands described below.

The following commands are used to view and set the global configuration properties:

To list all properties and their values:

$AOM_HOME/scripts/emsctl list_properties

To set a property value:

$AOM_HOME/scripts/emsctl set_property -name=<propertyname> -value=<propertyvalue>

To remove a property value:

$AOM_HOME/scripts/emsctl unset_property -name=<propertyname>

Getting Performance Information

You can see how well your Always-On Monitoring installation is operating by viewing Always-On Monitoring performance information such as load and throughput. Use the getstats command to display Always-On Monitoring performance statistics:

$AOM_HOME/scripts/emsctl getstats

Modifiable Always-On Monitoring Properties

Always-On Monitoring allows you to change various properties that define how your Always-On Monitoring deployment operates. The following table lists the global properties that are available. These properties can be set as global properties or locally in the emsConfig.properties file..

Table 17-4 Always-On Monitoring Global Properties

Property Name	Description	Default Value	Range of Values
connPool.initialSize	The initial number of Always-On Monitoring repository connections to be created in the repository connection pool.	1	1-connPool.maxSize
connPool.maxSize	The maximum number of Always-On Monitoring repository connections to be created in the repository connection pool.	100	connPool.minSize – 500
connPool.minSize	The minimum number of Always-On Monitoring repository connections to be kept in the repository connection pool after growth.	10	connPool.initialSize – connPool.maxSize
failover.heartbeatInterval	The interval in seconds between heartbeats sent from the running Always-On Monitoring to other Always-On Monitoring instances.	60	30 – 3600
http.idleTimeout	Time in milliseconds before idle web server connections in the Always-On Monitoring are released and terminated.	60000	10000 – 120000
http.maxThreads	The maximum number of Always-On Monitoring web server threads to be created in the web server connection pool. This value is of particular interest on large sites with 10000 hosts or more.	50	http.minThreads – 100
http.minThreads	The minimum number of Always-On Monitoring web server threads to be created in the web server connection pool.	10	1 – http.maxThreads
nls.useSystemLocale	Reserved for future use.	false
notif.enabled	If email notifications are enabled or not; typically sent using `emsctl enable_notification`	true	true/false
notif.senderAddress	Override sender email address for emails sent from Always-On Monitoring. Otherwise the address is the same as the Enterprise Manager email address.	null
notif.senderName	Override sender email username for e-mails sent from Always-On Monitoring. Otherwise name is the same as the Enterprise Manager email name.	null
purge.Duration	Amount of data (in minutes) kept in all historical Always-On Monitoring tables.	259200	10080 – 0 (infinite)
sync.allowIncrSync	Reserved for future use.	false	true/false
tasks.poolSize	Number of threads to use in the background task execution pool.	20	1-50
tasks.purgeRunInterval	Interval in minutes between instances of the data purge task.	10080	3600 – 10080
tasks.syncRunInterval	Interval in minutes between instances of the incremental sync task.	1440	60 - 3600

The following table lists the Always-On Monitoring local properties that can only be defined locally in the emsConfig.properties file.

Table 17-5 Always-On Monitoring Local Properties

Property Name	Description	Default Value	Range of Values
logLevel	The level of log messages to include in the Always-On Monitoring logs	INFO	DEBUG, INFO, WARN, ERROR
emsPort	The port number to use when listening for requests from the agent.	8081	1024-65535
oracle.sysman.core.notification.smtp.retry_enabled	Property that determines whether failed email notifications should be retried.	false	true/false
oracle.sysman.core.notification.max_email_delivery_time	Property in seconds that determines how long failed email notifications should be retried.	600 seconds	60-86400

Creating an SSO Wallet and JKS for CA Certificates

You can create a single sign-on (SSO) wallet and Java Keystore (JKS) for certificates issued by an external Certificate Authority for use with your Always-On Monitoring environment. To do so, perform the following:

Copy the EM key from the Enterprise Manager repository by running:
```
emctl config emkey -copy_to_repos
```
Generate the certificate and the wallet for the host by running:
```
$AOM_HOME/bin/emsctl secure
```

If a certificate is issued by a third-party Certificate Authority, then you need to store the certificate in the wallet. Once the wallet has the certificate, you must run the following:

$AOM_HOME/bin/emsctl secure -wallet=<absolute location of the wallet file>

Diagnosing Problems

Though robust and easy to install and configure, you may encounter problems while using Always-On Monitoring. In general, Always-On Monitoring problems can be classified into three types:

Problems starting Always-On Monitoring
Problems with Always-On Monitoring-Enterprise Manager repository SYNC.
Not receiving email

By performing diagnostic tasks listed below, you can resolve a majority of issues that may occur when using Always-On Monitoring.

Check the log files in the $AOM_HOME/log sub-directory

Verify that the URL is set in Enterprise Manager by running the following

emctl get property -name "oracle.sysman.core.events.ems.emsURL" -sysman_pwd <sysman-password>

Verify that the URL is set on the Management Agent by checking the $AGENT_HOME/sysman/config/emd.properties file and searching for EMS_URL
Verify that Always-On Monitoring is running and notifications are enabled by running emsctl status or emsctl ping
Verify that downtime contacts have been specified in Enterprise Manager and Always-On Monitoring by running emsctl status.
Check the version of the Agent by running the following command:
```
emctl status agent
```
The emctl command can be run form the following directory:
```
$AGENT_HOME/agent_inst/bin
```

High Availability and Disaster Recovery

The purpose of Always-On Monitoring is to make sure your Enterprise Manager environment continues to be monitored while Enterprise Manager OMS is down. However, to protect against a single point of failure within the Always-On Monitoring environment itself, the Always-On Monitoring service can be made highly available by adhering to the Enterprise Manager Maximum Availability Architecture (MAA) guidelines. By setting up Always-On Monitoring as a highly available service, you can ensure that targets will continue to be monitored in the event of catastrophic system failure.

Running Multiple Always-On Monitoring Instances

Always-On Monitoring provides you with the ability to continue alert monitoring while Enterprise Manager is down. By itself, Always-On Monitoring can be made highly available by implementing multiple instances for load sharing/high availability. It is also important to note that the Always-On Monitoring repository should be configured as a cluster database since you need to implement basic High Availability (HA) for this component as well.

Adding multiple Always-On Monitoring instances provides the following HA advantages:

Shares the work of receiving and recording alerts from agents. If an instance goes down for some reason, a server load balancer (SLB) can redirect those requests to surviving instances
Shares the work of sending of notifications. For ordering reasons, all notifications for a target are directed to one Always-On Monitoring instance. If that Always-On Monitoring instance goes down, the responsibility for its "queues" are transferred to another instance.

Once you have configured an Always-On Monitoring instance, you can add another instance by running the following emsca command:

emsca add_ems

The following diagram shows a typical Always-On Monitoring HA deployment.

Figure 17-1 Always-On Monitoring HA Deployment

Description of "Figure 17-1 Always-On Monitoring HA Deployment"

Shared Configuration Storage for the Multiple Instances

Instead of maintaining Always-On Monitoring configuration in the file system on each node, all instances can share a common configuration. This allows a single configuration at one location to be modified and read from all the instances.

You use the following EMSCTL commands to modify properties:

emsctl set_property
emsctl unset_property

To obtain a list of current property values:

emsctl list_properties

When a property is modified on one instance, all other Always-On Monitoring instances receive a JMS AQ message that tells them to refresh the value from the repository.

Notification Queues for Tracking Incoming Alerts

Each Always-On Monitoring instance that receives an alert stores the alert and records an entry in one of 16 event notification queues. Ownership of these notifications queues is distributed among the running Always-On Monitoring instances.

A failover system notices when Always-On Monitoring instances are not up (and heartbeating) and switches ownership away from the instance that is down to other Always-On Monitoring instances that are up. The failover system then sends a JMS AQ message notifying all up instances of the change.

Task Scheduler System

A Task scheduler system remembers shared responsibilities among the Always-On Monitoring instances and is responsible for reminding the instances to do the work on a schedule. The task request is sent to an Always-On Monitoring instance that is known to be up. This reduces the possibility of multiple instances trying to do the same work and running into conflicts.

Configuring an SLB

When multiple Always-On Monitoring instances are present, we expect that traffic to these instances is directed through a SLB.

When HA is added to an Always-On Monitoring instance that was configured as a single instance, you need to repopulate the Always-On Monitoring instance's certificates with fresh copies that have the SLB host name in them. For Always-On Monitoring released with Enterprise Manager 13.3, you generate a certificate for the SLB and store that in a wallet by running the emsctl secure command.

Copy the EM key from the Enterprise Manager repository by running the following command:

emctl config emkey -copy_to_repos
Run emsctl secure.
Restart the Always-On Monitoring instance.
Repeat for all Always-On Monitoring instances already configured.
Make sure to run emctl set property for the emsUrl property to use the newly configured Always-On Monitoring upload URL set up on the SLB. For example: ./emctl set property -name "oracle.sysman.core.events.ems.emsURL" -value "https://<slb hostname>:<slb aom port>/upload"

The URL displayed from emsca and/or emsctl status is the local URL for that Always-On Monitoring instance. In an HA configuration, this local URL is NOT used as the Always-On Monitoring URL, but is used to set up the Always-On Monitoring pool in the SLB. The host and port for each Always-On Monitoring instance is used in the pool with the /upload PUT URL as the target for that pool.

When configuring the SLB, the monitoring URL is the /upload GET URL. This URL returns the string "Always On Monitoring is active."

Note:

Always-on monitoring is supported on F5 from version 11.4.x onwards.

Always-On Monitoring Disaster Recovery

Disaster Recovery (DR) allows Always-On Monitoring functionality to transition to a different location in the case of a site-wide failure.

There are existing procedures that allow an Enterprise Manager site to have a standby created for DR reasons. These procedures involve having the primary repository database's contents mirrored at the standby site using DataGuard, and the ability to link to the file system contents from the secondary site.

A very similar implementation can be used for Always-On Monitoring instances as well. An Always-On Monitoring DR setup is active-passive: If the primary site goes down, the virtual IP fails over to the standby node and, once the standby Always-On Monitoring instance is brought up, it runs exactly as it did on the primary node (because of the file system sharing and the virtual IP). In summary, the Always-On Monitoring repository needs to be available at the standby site, and the Always-On Monitoring instances' file systems need to be replicated. The following diagram shows the typical DR deployment for Always-On Monitoring.

Figure 17-2 Always-On Monitoring Disaster Recovery Deployment

Description of "Figure 17-2 Always-On Monitoring Disaster Recovery Deployment"

Setting Up Multiple Always-On Monitoring Instances

You can set up multiple Always-On Monitoring (AOM) instances to ensure Always-On Monitoring is available in the event of outages.

Uninstall any 13.1 Always-On Monitoring instances, if any.

Except for the last running AOM instance (in this example, there are three AOM instances AOM1, AOM2 and AOM3) you must perform the following for AOM1 and AOM2):
- Shut down the instances (AOM1 and AOM2) by running $AOM_HOME/scripts/emsctl stop
- Remove all the files and sub-directories under $AOM_HOME
And on the last AOM instance (AOM3):
- Shut down the instance (AOM3) by running $AOM_HOME/scripts/emsctl stop
- Uninstall AOM by running $AOM_HOME/scripts/emsca uninstall
- Remove all files and sub-directories under $AOM_HOME
- To remove any trace of the previous AOM installation, you can drop the AOM repository user in the repository by executing drop user ems cascade.
Obtain the latest Always-On Monitoring release. The ZIP file can be found on the OMS' /sysman/ems directory. Copy the ems_13.5.x.x.x.zip file to a location where you want AOM to be installed and unzip the file.
```
unzip ems.zip
```
Note:
For AOM 13.5 to run, both the Management Agent and Enterprise Manager installation MUST be upgraded to 13.5 as well.
Install AOM.

For setting up the very first AOM instance, the user must run $AOM_HOME/scripts/emsca

For installing subsequent AOM instances the user must run $AOM_HOME/scripts/emsca add_ems
Finally, you must run the following steps on all instances before starting AOM.
- To generate the certificate and the wallet for the SLB host, run $AOM_HOME/scripts/emsctl secure
- Configure the downtime contacts. For more information, see Configuring Downtime Contacts in Enterprise Manager.
- To sync the AOM repository schema with the Enterprise Manager repository schema, execute $AOM_HOME/scripts/emsctl sync
- Set the upload URL, by running emctl config emkey -copy_to_repos on the Enterprise Manager host.
- Start the AOM instances by running, $AOM_HOME/scripts/emsctl start

Uninstalling Always-On Monitoring

If the purpose of uninstalling Always-On Monitoring is to upgrade to a new release, you should back up the following files.

$AOM_HOME/conf/emsConfig.properties
Custom certificates (if any) located at $AOM_HOME/conf (cwallet.sso, cwallet.sso.lck, ewallet.p12, and ewallet.p12.lck)

Use emsca to uninstall Always-On Monitoring as shown in the following example:

$ ./emsca uninstall
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Loading configuration from: /homedirectory/ems/conf/emsConfig.properties
 Connecting to Always-On Monitoring repository.
 Execute the following:
 emctl delete property -name "oracle.sysman.core.events.ems.emsURL" -sysman_pwd <pwd> to unset the value of the property.
 Uninstall completed.

Remove all files and sub-directories under $AOM_HOME.

To remove any trace of the previous AOM installation, you can drop the AOM repository user in the repository by running the following command:

drop user ems cascade

Configuring the Always-On Monitoring Application for Secure Communication Using the TLSv1.2 Protocol

Transport Layer Security (TLS) is a cryptographic protocol used to increase security over computer networks by providing communication privacy and data integrity between applications. In the case of Always-On Monitoring (AOM), these secure communication channels are between the following components:

The AOM application and the AOM repository
The AOM application and the Enterprise Manager repository

The following instructions cover how to enable TLSv1.2 communication for Always-On Monitoring.

Storing CA Certificates

The server CA certificates of the Always-On Monitoring repository and Enterprise Manager repository can be stored either in an external Trust Store or in the Oracle Management Service’s JDK Trust Store (JAVA_HOME/jre/lib/security/cacerts).

Storing the Certificates in an External Trust Store

If you choose to store the certificates in an external Trust Store, then you need to set the following environment variables:

AOM_DB_WALLET_LOC - Absolute path to the external Trust Store. For example: /home/aom/externalTrustSTore.jks
AOM_DB_WALLET_TYPE - The type of Trust Store being used. JKS. PKCS12 (For Enterprise Manager 13.3, the SSO Trust Store is not supported)
AOM_DB_WALLET_PASSWORD - The Trust Store password (For JKS and PKCS12)

Note:

In CSH, to set the environment variable, use the setenv command. For example, to set the AOM_DB_WALLET_LOC environment variable, run the following:

% setenv AOM_DB_WALLET_LOC /home/aom/externalTrustStore.jks

To set the same environment variable in a Bash environment, run the following:

% export AOM_DB_WALLET_LOC=/home/aom/externalTrustStore.jks

Storing the Certificates in the Oracle Management Service’s JDK Trust Store

If you choose to store the certificates in the Oracle Management Service’s JDK Trust Store, then you can use the emsca or emsctl scripts without additional configuration.

Guidelines for Configuring Always-On Monitoring to use TLSv1.2

During initial Always-On Monitoring setup with emsca, when prompted for the DB connection string, you must specify the connection string using the long form that has the protocol type (TCPS) being used and not the short form as shown in the following examples.

Long Form
```
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=myserver.myco.com) (PORT=15044)) (CONNECT_DATA =(SID=dbview))  
```
Short Form (will not work with TCPS):
```
myserver.myco.com:15044:dbview
```
If Always-On Monitoring was initially configured to use the TCP protocol as part of emsca configuration, and at a later point you want to switch over to TCPS, you can re-configure Always-On Monitoring by performing the following steps:
1. Change the Always-On Monitoring database connection string in the emsConfig.properties file ($AOM_HOME/conf/emsConfig.properties).
2. Change Enterprise Manager Repository connection string in Always-On Monitoring database which is stored in table - EMS_SYNC_CONNECT_PROPS.connect_string.
3. Ensure that either the JDK wallet (under JAVA_HOME/jre/lib/security/cacerts) or the external Trust Store has the root CA certificates for both the Always-On Monitoring and Enterprise Manager repositories.
If Always-On Monitoring was initially configured to use the TCPS protocol as part of emsca configuration, and at a later point you want to switch over to TCP, you can re-configure Always-On Monitoring by performing the following steps:
1. In the AOM_HOME/conf/emsConfig.properties file, replace the correct AOM database connection string with the property emsRepConnectString. If TCP protocol is used to connect to the database, you can use either short or long form for the connection string. If TCPS is used, you must provide the database connection string in the long form as discussed previously.
2. To change the Enterprise Manager connection string, you must update the EMS_SYNC_CONNECT_PROPS.connect_string field in the Always-On Monitoring database.