Using Blackouts

Blackouts and Notification Blackouts

Blackouts and Notification Blackouts (notification blackouts) help you maintain monitoring accuracy during target maintenance windows by providing you with the ability to suspend various Enterprise Manager monitoring functions for the duration of the maintenance period. For example, when bringing down targets for upgrade or patching, you may not want that downtime included as part of the collected metric data or have it affect a Service Level Agreement (SLA).

Blackout/Notification Blackout functionality is available from both the Enterprise Manager console as well as via the Enterprise Manager command-line interface (EMCLI).

About Blackouts

Blackouts allow you to suspend monitoring on one or more targets in order to perform maintenance operations. Blackouts, also known as patching blackouts, ensure that the target is not changed during the period of the blackout so that a maintenance operation on the actual target will not be affected. During this period, the Agent does not perform metric data collection on the target and no notifications will be raised for the target. Blackouts will allow Enterprise Manager jobs to run on the target during the blackout period by default. Optionally, job runs can be prevented during the blackout period.

A blackout can be defined for individual target(s), a group of multiple targets that reside on different hosts, or for all targets on a host. The blackout can be scheduled to run immediately or in the future, or stop after a specific duration. Blackouts can be created on an as-needed basis, or scheduled to run at regular intervals. If, during the maintenance period, the administrator discovers that he needs more (or less) time to complete his maintenance tasks, he can easily extend (or stop) the blackout that is currently in effect. Blackout functionality is available from both the Enterprise Manager console as well as via the Enterprise Manager command-line interface (EMCLI). EMCLI is often useful for administrators who would like to incorporate the blacking out of a target within their maintenance scripts.

Why use blackouts?

Blackouts allow you to collect accurate monitoring data. For example, you can stop data collections during periods where a managed target is undergoing routine maintenance, such as a database backup or hardware upgrade. If you continue monitoring during these periods, the collected data will show trends and other monitoring information that are not the result of normal day-to-day operations. To get a more accurate, long-term picture of a target's performance, you can use blackouts to exclude these special-case situations from data analysis.

Blackout Access

Enterprise Manager administrators that have at least Blackout Target privileges on all Selected Targets in a blackout will be able to create, edit, stop, or delete the blackout.

In case an administrator has at least Blackout Target privileges on all Selected Targets (targets directly added to the blackout), but does not have Blackout Target privileges on some or all of the Dependent Targets, then that administrator will be able to edit, stop, or delete the blackout. For more information on Blackout access, see "About Blackouts Best Effort".

About Notification Blackouts

Notification Blackouts are solely for suppressing the notifications on targets during the Notification Blackout duration. The Agent continues to monitor the target under Notification Blackout and the OMS will show the actual target status along with an indication that the target is currently under Notification Blackout. Events will be generated as usual during a Notification Blackout. Only the event notifications are suppressed.

The period of time under which the target is in Notification Blackout is not used to calculate the target's Service Level Agreement (SLA).

To place a target under Notification Blackout, you need to have at least Blackout Target privilege on the target.

There are two types of Notification Blackouts:

Maintenance Notification Blackout: The target is under a planned maintenance and administrators do not want to receive any notifications during this period. Since the target is brought down deliberately for maintenance purposes, the Notification Blackout duration should not be considered while calculating the availability percentage and SLA. In this scenario, an administrator should create a maintenance Notification Blackout.
Notification-only Notification Blackout: The target is experiencing an unexpected down time such as a server crash. While the administrator is fixing the server, they do not want to receive alerts as they are already aware of the issue and are currently working to resolve it. However, the availability percentage computation should consider the actual target status of the Notification Blackout duration and the SLA should be computed accordingly. In this scenario, the administrator should create a Notification-only Notification Blackout.

By default, when a Notification Blackout is created, it is a maintenance Notification Blackout (the Under Maintenance option will be selected by default and the administrator will need to select the Non-maintenance option in order to create a regular Notification-only Notification Blackout.

A Notification Blackout can be defined for individual target(s), a group of multiple targets that reside on different hosts, or for all targets on a host. The Notification Blackout can be scheduled to run immediately or in the future, or stop after a specific duration. Notification Blackouts can be created on an as-needed basis, or scheduled to run at regular intervals. If, during the maintenance period, the administrator discovers that he needs more (or less) time to complete his maintenance tasks, he can easily extend (or stop) the Notification Blackout that is currently in effect. Notification Blackout functionality is available from both the Enterprise Manager console as well as via the Enterprise Manager command-line interface (EMCLI). EMCLI is often useful for administrators who would like to incorporate the blacking out of a target within their maintenance scripts.

Notification Blackout Access

Enterprise Manager administrators that have at least Blackout Target privileges on all Selected Targets in a Notification Blackout will be able to create, edit, stop, or delete the Notification Blackout.

In case an administrator has at least Blackout Target privileges on all Selected Targets (targets directly added to the Notification Blackout), but does not have Blackout Target privileges on some or all of the Dependent Targets, then that administrator will be able to edit, stop, or delete the Notification Blackout.

Working with Blackouts/Notification Blackouts

Blackouts allow you to collect accurate monitoring data. For example, you can stop data collections during periods where a managed target is undergoing routine maintenance, such as a database backup or hardware upgrade. If you continue monitoring during these periods, the collected data will show trends and other monitoring information that are not the result of normal day-to-day operations. To get a more accurate, long-term picture of a target's performance, you can use blackouts to exclude these special-case situations from data analysis.

Creating Blackouts/Notification Blackouts

Blackouts/Notification Blackouts allow you to suspend monitoring on one or more managed targets.

To create a Blackout/Notification Blackout:

From the Enterprise menu, select Monitoring and then Blackouts.
From the table, click Create. Blackout and Notification Blackout selection dialog displays.
Choose either Blackout or Notification Blackout and click Create. The Create Blackout/Notification Blackout page displays.
When creating a Notification Blackout, you will also be able to specify the type of Notification Blackout via the Maintenance Window options.
- Under maintenance. Target downtime is excluded from Availability(%) calculations.
- Non-maintenance. Any target downtime will impact Availability(%) calculations.
Enter the requisite parameters for the new blackout/Notification Blackout and then click Submit.

Editing Blackouts/Notification Blackouts

Blackouts allow you to suspend monitoring on one or more managed targets.

To edit a Blackout:

From the Enterprise menu, select Monitoring and then Blackouts.
If necessary, use the Search and display options to show the blackouts you want to change in the blackouts table.
Select the desired Blackout/Notification Blackout. Details are displayed. click Edit. The Edit Blackout/Notification Blackout page displays.
Make the desired changes and click Submit.

Note: Enterprise Manager also allows you to edit blackouts after they have already started.

Viewing Blackouts/Notification Blackouts

To view information and current status of a blackout:

From the Enterprise menu, select Monitoring and then Blackouts.
If necessary, you can use the Search and display options to show the blackouts you want to view in the blackouts table.
Select the desired Blackout/Notification Blackout. Details are displayed.

Viewing Blackouts from Target Home Pages

For most target types, you can view a Blackout/Notification Blackout information from the target home page for any target currently under Blackout/Notification Blackout. The Blackout/Notification Blackout Summary region provides pertinent Blackout/Notification Blackout status information for that target.

Viewing Blackout/Notification Blackouts from Groups and Systems Target Administration Pages

For Groups and Systems, you can view Blackout/Notification Blackout information about the number of active/scheduled Blackouts/Notification Blackouts on a group/system and its member targets.

Purging Blackouts/Notification Blackouts that have Ended

When managing a large number of targets, the number of completed Blackouts/Notification Blackouts, or those Blackouts/Notification Blackouts that have been ended by an administrator can become quite large. Removing these ended Blackouts/Notification Blackouts facilitates better search and display for current Blackouts/Notification Blackouts.

To purge ended Blackouts/Notification Blackouts from Enterprise Manager:

From the Enterprise menu, select Monitoring and then Blackouts.
Use the search criteria to filter for the desired targets.
From the Blackout Timeframe drop-down menu, select History.
In the table, select the ended Blackouts/Notification Blackouts you want to remove and click Delete. The Delete Blackout/Brownout confirmation page appears.
Click Delete to complete the purge process.

Retroactive Blackouts and Outages

If a target is brought down for maintenance and a blackout is not scheduled for that target, the maintenance period would be reflected as target downtime, thus a negative impact on the target’s availability history. This would be a problem for say a target’s service-level agreement (SLA) where the collected metrics that define the level of expected service would be inaccurate.

Enterprise Manager lets you remedy this situation by allowing you to define blacklouts and outages retroactively.

Retroactive Blackouts

As mentioned previously, retroactive blackouts can be used to specify past maintenance periods where the administrator has forgotten to set a blackout. The retroactive period will be used to adjust the target's availability (%) period by excluding these periods from availability (%) calculations, thus increasing a target's availability (%). Retroactive blackouts can be created either from the console or using EM CLI.

The following sequence of events illustrates the typical scenario for a retroactive blackout.

The target is brought down for maintenance.
The target availability % goes down from 100% => 80%.
The target is brought back up after the maintenance work has been completed.
The administrator realizes no blackout was created for the maintenance period.
The administrator creates a retroactive blackout for the maintenance period.
The target availability goes back up from 80% ==> 100%.

Creating a Retroactive Blackout from the Enterprise Manager Console

From the Enterprise menu, select Monitoring and then Blackouts. The Blackouts page displays.
Click Settings. The Settings page displays.
At the bottom of the Settings page, check the Enable Retroactive Blackout Feature option box.
Click Create Retroactive Blackout. The Create Retroactive Blackout page displays.
Enter the requisite information and then click Submit.

Retroactive Outages

If a monitored target goes down (outage) and Enterprise Manager does not detect it, the target availability percentage will be inaccurate. In this situation, the availability percentage will be too high. To remedy this inaccuracy, Enterprise Manager lets you specify this outage retroactively. A retroactive outage is essentially a retroactive blackout that specifies target downtime should be included as part of the availability calculation.

Retroactive outage can be created either from the console or using EM CLI.

The following sequence of events illustrates the typical scenario for a retroactive outage.

The target goes down for an unknown reason.
Enterprise Manager does not detect the target's down state. Target availability remains at 100%.
The administrator brings the target back up.
The administrator creates a retroactive blackout with the Include target downtime in target availability (%) calculation option selected for the unplanned outage period.
Target availability goes down from 100% ==> 84%.

Creating a Retroactive Outage from the Enterprise Manager Console

From the Enterprise menu, select Monitoring and then Blackouts. The Bl
ackouts page displays.
Click Settings. The Settings page displays.
Click Create Retroactive Blackout. The Create Retroactive Blackout page displays.
Ensure that the Include target downtime in target availability (%) calculation option is checked.

Note:
The Include target downtime in target availability (%) calculation checkbox should be enabled during the creation of a retroactive blackout/outage, only if Enterprise Manager did not detect the outage. Checking this option will include target downtime in the target availability calculation and will result in a lower availability percentage.
Select a reason from the Reason drop-down menu and then click Add to add the target(s) for which you are creating the retroactive outage.
In the Schedule region, specify the time period in which the target was down.
Click Submit to create the retroactive outage. A success confirmation message will be displayed on the Blackouts page.

The following graphic shows target availability history for the repository database before the retroactive outage has been defined. Target availability percentage is 100% with no down time.
Graphic shows availability history before the retroactive outage has been defined.

The next graphic shows the target availability history after a retroactive outage of 56 minutes has been defined. Target availability percentage has been reduced to 84% with 56 minutes of target down time.

Creating a Retroactive Outage using the Command Line Interface (EM CLI)

Alternatively, you can create a retroactive blackout/outage using the EM CLI create_rbk verb. The verb syntax is as follows:

emcli create_rbk
          -add_targets="name1:type1;name2:type2;..."...
          -reason="reason"
          [-propagate_targets]
          -schedule=
              start_time:<yyyy-MM-dd HH:mm>;
              end_time:<yyyy-MM-dd HH:mm>;
              [tzregion:<...>]
          [-outage]

Table 4-1 create_rbk Options

Option	Description
add_targets	Targets to add to the blackout, each specified as `target_name:target_type` The `add_targets` option may be specified more than once.
reason	Reason for the retroactive blackout. This is used for storing in backup tables.
propagate_targets	When this option is specified, a blackout for a target of type host applies the blackout to all non-agent targets on that host. Regardless of whether this option is specified, a blackout for a target that is a composite or a group applies the blackout to all members of the composite or group.
schedule	Blackout schedule. Parameters start_time The start date/time of the blackout. The format of the value is `yyyy-MM-dd HH:mm:ss`. Example: `2017-09-20 12:12:12` end_time The end date/time of the blackout. The format of the value is `yyyy-MM-dd HH:mm:ss`. Example: `2017-09-20 12:15:00` tzregion The timezone region to use. If not provided tzregion is defaulted to UTC.
outage	Use this option with caution as it will lower the target availability (%). This option should be used only if Enterprise Manager did not detect the outage.

The following example shows the command output.
create_rbk verb output

General Usage Guidelines

For planned outages, where the administrator forgot to set a blackout, create a retroactive blackout without enabling the Include target downtime in target availability (%) calculation checkbox. This will increase the target's availability %. For example, 84% ==> 100%.
For unplanned outages, where Enterprise Manager did not detect the outage, create a retroactive blackout and enable the Include target downtime in target availability (%) calculation option. This will decrease the target's availability %. For example. 100% ==> 84%.
For unplanned outages, where Enterprise Manager did detect the outage, nothing needs to be done.

Controlling Blackouts Using the Command Line Utility

You can control blackouts from the Oracle Enterprise Manager 13c Cloud Control Console or from the Enterprise Manager command line utility (emctl). However, if you are controlling target blackouts from the command line, you should not attempt to control the same blackouts from the Cloud Control console. Similarly, if you are controlling target blackouts from the Cloud Control console, do not attempt to control those blackouts from the command line.

From the command line, you can perform the following blackout functions:

Starting Immediate Blackouts
Stopping Immediate Blackouts
Checking the Status of Immediate Blackouts

Note:

When you start a blackout from the command line, any Enterprise Manager jobs scheduled to run against the blacked out targets will still run. If you use the Cloud Control Console to control blackouts, you can optionally prevent jobs from running against blacked out targets.

To use the Enterprise Manager command-line utility to control blackouts:

Change directory to the AGENT_HOME/bin directory (UNIX) or the AGENT_INSTANCE_HOME\bin directory (Windows).
Enter the appropriate command as described in Table 4-2.

Note:

When you start a blackout, you must identify the target or targets affected by the blackout. To obtain the correct target name and target type for a target, see Administering Enterprise Manager Using EMCTL Commands.

Table 4-2 Summary of Blackout Commands

Blackout Action	Command
Set an immediate blackout on a particular target or list of targets	`emctl start blackout <Blackoutname> [<Target_name>[:<Target_Type>]].... [-d <Duration>]` Be sure to use a unique name for the blackout so you can refer to it later when you want to stop or check the status of the blackout. The `-d` option is used to specify the duration of the blackout. Duration is specified in `[days] hh:mm where:` days indicates number of days, which is optional hh indicates number of hours mm indicates number of minutes If you do not specify a target or list of targets, Enterprise Manager will blackout the local host target. All monitored targets on the host are not blacked out unless a list is specified or you use the `-nodelevel` argument. If two targets of different target types share the same name, you must identify the target with its target type.
Stop an immediate blackout	`emctl stop blackout <Blackoutname>`
Set an immediate blackout for all targets on a host	`emctl start blackout <Blackoutname> [-nodeLevel] [-d <Duration>]` The `-nodeLevel` option is used to specify a blackout for all the targets on the host; in other words, all the targets that the Management Agent is monitoring, including the Management Agent host itself. The `-nodeLevel` option must follow the blackout name. If you specify any targets after the `-nodeLevel` option, the list is ignored.
Check the status of a blackout	emctl status blackout `[<Target_name>[:<Target_Type>]]....`

Use the following examples to learn more about controlling blackouts from the Enterprise Manager command line:

To start a blackout called "bk1" for databases "db1" and "db2," and for Oracle Listener "ldb2," enter the following command:
```
$PROMPT> emctl start blackout bk1 db1 db2 ldb2:oracle_listener -d 5 02:30
```
The blackout starts immediately and will last for 5 days 2 hours and 30 minutes.
To check the status of all the blackouts on a managed host:
```
$PROMPT> emctl status blackout
```
To stop blackout "bk2" immediately:
```
$PROMPT> emctl stop blackout bk2
```
To start an immediate blackout called "bk3" for all targets on the host:
```
$PROMPT> emctl start blackout bk3 -nodeLevel
```
To start an immediate blackout called "bk3" for database "db1" for 30 minutes:
```
$PROMPT> emctl start blackout bk3 db1 -d 30
```
To start an immediate blackout called "bk3" for database "db2" for five hours:
```
$PROMPT> emctl start blackout bk db2 -d 5:00
```

About Blackouts Best Effort

The Blackouts Best Effort feature allows you to create blackouts on aggregate targets, such as groups or systems, for which you do not have Blackout Target (or Higher) privileges on all members of the aggregate target.

Here, an Enterprise Manager administrator has Blackout Target privilege on an aggregate target but do not have OPERATOR privilege on its member/associated targets. You should ideally create a Full Blackout on this aggregate target. When defining the blackout, you are allowed to select any member target, even those member targets for which you have no Blackout Target privileges.

When the blackout actually starts, Enterprise Manager checks privileges on each member target and only blackout those on which you have Blackout Target( or Higher) privileges. This automated privilege check and target blackout selection is Enterprise Manager's "best effort" at blacking out the aggregate target.

When to Use Blackout Best Effort

The Blackout Best Effort functionality is targeted towards the creation of blackouts on targets of any aggregate type, such as Group, Hosts, Application Servers, Web Applications, Redundancy Groups, or Systems.

All targets the blackout creator has Blackout Target (or higher) privilege on will be displayed in the first step of Create/Edit Blackout Wizard. Once the blackout creator selects an aggregate type of target to be included in the Blackout Definition, this Blackout is "Full Blackout" by default.

The creator has the option of choosing the Blackout to run on “All Current" or “Selected" Targets, by selecting the appropriate values from the List box. Only when the "Full Blackout" option is chosen, will Blackout Best Effort affect targets for which the creator does not have Blackout Target (or higher) privileges.

Example Use Case

Consider 3 targets T1,T2 and T3 (all databases). A Group G1 contains all these 3 targets.

User U1 has OPERATOR privilege on T1,T2 and G1. User U1 has VIEW privilege on T3.

User U1 creates a scheduled full blackout on target G1. Scheduled implies that the blackout will start at a later point in time.

At the time of blackout creation, the tip text Needs Blackout Target privilege, see Tip below the table would be shown beside target T3.

When this blackout starts, if by that time User U1 has been granted OPERATOR privileges on target T3, then target T3 would also be under blackout. Otherwise only targets T1, T2 and G1 will be under blackout.