This chapter covers the following topics:
Blackouts and Notification Blackouts (notification blackouts) help you maintain monitoring accuracy during target maintenance windows by providing you with the ability to suspend various Enterprise Manager monitoring functions for the duration of the maintenance period. For example, when bringing down targets for upgrade or patching, you may not want that downtime included as part of the collected metric data or have it affect a Service Level Agreement (SLA).
Blackout/Notification Blackout functionality is available from both the Enterprise Manager console as well as via the Enterprise Manager command-line interface (EMCLI).
Blackouts allow you to suspend monitoring on one or more targets in order to perform maintenance operations. Blackouts, also known as patching blackouts, ensure that the target is not changed during the period of the blackout so that a maintenance operation on the actual target will not be affected. During this period, the Agent does not perform metric data collection on the target and no notifications will be raised for the target. Blackouts will allow Enterprise Manager jobs to run on the target during the blackout period by default. Optionally, job runs can be prevented during the blackout period.
A blackout can be defined for individual target(s), a group of multiple targets that reside on different hosts, or for all targets on a host. The blackout can be scheduled to run immediately or in the future, or stop after a specific duration. Blackouts can be created on an as-needed basis, or scheduled to run at regular intervals. If, during the maintenance period, the administrator discovers that he needs more (or less) time to complete his maintenance tasks, he can easily extend (or stop) the blackout that is currently in effect. Blackout functionality is available from both the Enterprise Manager console as well as via the Enterprise Manager command-line interface (EMCLI). EMCLI is often useful for administrators who would like to incorporate the blacking out of a target within their maintenance scripts.
Why use blackouts?
Blackouts allow you to collect accurate monitoring data. For example, you can stop data collections during periods where a managed target is undergoing routine maintenance, such as a database backup or hardware upgrade. If you continue monitoring during these periods, the collected data will show trends and other monitoring information that are not the result of normal day-to-day operations. To get a more accurate, long-term picture of a target's performance, you can use blackouts to exclude these special-case situations from data analysis.
Blackout Access
Enterprise Manager administrators that have at least Blackout Target privileges on all Selected Targets in a blackout will be able to create, edit, stop, or delete the blackout.
In case an administrator has at least Blackout Target privileges on all Selected Targets (targets directly added to the blackout), but does not have Blackout Target privileges on some or all of the Dependent Targets, then that administrator will be able to edit, stop, or delete the blackout. For more information on Blackout access, see "About Blackouts Best Effort".
Notification Blackouts are solely for suppressing the notifications on targets during the Notification Blackout duration. The Agent continues to monitor the target under Notification Blackout and the OMS will show the actual target status along with an indication that the target is currently under Notification Blackout. Events will be generated as usual during a Notification Blackout. Only the event notifications are suppressed.
The period of time under which the target is in Notification Blackout is not used to calculate the target's Service Level Agreement (SLA).
To place a target under Notification Blackout, you need to have at least Blackout Target privilege on the target.
There are two types of Notification Blackouts:
Maintenance Notification Blackout: The target is under a planned maintenance and administrators do not want to receive any notifications during this period. Since the target is brought down deliberately for maintenance purposes, the Notification Blackout duration should not be considered while calculating the availability percentage and SLA. In this scenario, an administrator should create a maintenance Notification Blackout.
Notification-only Notification Blackout: The target is experiencing an unexpected down time such as a server crash. While the administrator is fixing the server, they do not want to receive alerts as they are already aware of the issue and are currently working to resolve it. However, the availability percentage computation should consider the actual target status of the Notification Blackout duration and the SLA should be computed accordingly. In this scenario, the administrator should create a Notification-only Notification Blackout.
By default, when a Notification Blackout is created, it is a maintenance Notification Blackout (the Under Maintenance option will be selected by default and the administrator will need to select the Non-maintenance option in order to create a regular Notification-only Notification Blackout.
A Notification Blackout can be defined for individual target(s), a group of multiple targets that reside on different hosts, or for all targets on a host. The Notification Blackout can be scheduled to run immediately or in the future, or stop after a specific duration. Notification Blackouts can be created on an as-needed basis, or scheduled to run at regular intervals. If, during the maintenance period, the administrator discovers that he needs more (or less) time to complete his maintenance tasks, he can easily extend (or stop) the Notification Blackout that is currently in effect. Notification Blackout functionality is available from both the Enterprise Manager console as well as via the Enterprise Manager command-line interface (EMCLI). EMCLI is often useful for administrators who would like to incorporate the blacking out of a target within their maintenance scripts.
Notification Blackout Access
Enterprise Manager administrators that have at least Blackout Target privileges on all Selected Targets in a Notification Blackout will be able to create, edit, stop, or delete the Notification Blackout.
In case an administrator has at least Blackout Target privileges on all Selected Targets (targets directly added to the Notification Blackout), but does not have Blackout Target privileges on some or all of the Dependent Targets, then that administrator will be able to edit, stop, or delete the Notification Blackout.
Blackouts allow you to collect accurate monitoring data. For example, you can stop data collections during periods where a managed target is undergoing routine maintenance, such as a database backup or hardware upgrade. If you continue monitoring during these periods, the collected data will show trends and other monitoring information that are not the result of normal day-to-day operations. To get a more accurate, long-term picture of a target's performance, you can use blackouts to exclude these special-case situations from data analysis.
Blackouts/Notification Blackouts allow you to suspend monitoring on one or more managed targets.
To create a Blackout/Notification Blackout:
Blackouts allow you to suspend monitoring on one or more managed targets.
To edit a Blackout:
Note: Enterprise Manager also allows you to edit blackouts after they have already started.
To view information and current status of a blackout:
Viewing Blackouts from Target Home Pages
For most target types, you can view a Blackout/Notification Blackout information from the target home page for any target currently under Blackout/Notification Blackout. The Blackout/Notification Blackout Summary region provides pertinent Blackout/Notification Blackout status information for that target.
Viewing Blackout/Notification Blackouts from Groups and Systems Target Administration Pages
For Groups and Systems, you can view Blackout/Notification Blackout information about the number of active/scheduled Blackouts/Notification Blackouts on a group/system and its member targets.
When managing a large number of targets, the number of completed Blackouts/Notification Blackouts, or those Blackouts/Notification Blackouts that have been ended by an administrator can become quite large. Removing these ended Blackouts/Notification Blackouts facilitates better search and display for current Blackouts/Notification Blackouts.
To purge ended Blackouts/Notification Blackouts from Enterprise Manager:
If a target is brought down for maintenance and a blackout is not scheduled for that target, the maintenance period would be reflected as target downtime, thus a negative impact on the target’s availability history. This would be a problem for say a target’s service-level agreement (SLA) where the collected metrics that define the level of expected service would be inaccurate.
Enterprise Manager lets you remedy this situation by allowing you to define blacklouts and outages retroactively.
Retroactive Blackouts
As mentioned previously, retroactive blackouts can be used to specify past maintenance periods where the administrator has forgotten to set a blackout. The retroactive period will be used to adjust the target's availability (%) period by excluding these periods from availability (%) calculations, thus increasing a target's availability (%). Retroactive blackouts can be created either from the console or using EM CLI.
The following sequence of events illustrates the typical scenario for a retroactive blackout.
The target is brought down for maintenance.
The target availability % goes down from 100% => 80%.
The target is brought back up after the maintenance work has been completed.
The administrator realizes no blackout was created for the maintenance period.
The administrator creates a retroactive blackout for the maintenance period.
The target availability goes back up from 80% ==> 100%.
From the Enterprise menu, select Monitoring and then Blackouts. The Blackouts page displays.
Click Settings. The Settings page displays.
At the bottom of the Settings page, check the Enable Retroactive Blackout Feature option box.
Click Create Retroactive Blackout. The Create Retroactive Blackout page displays.
Enter the requisite information and then click Submit.
Retroactive Outages
If a monitored target goes down (outage) and Enterprise Manager does not detect it, the target availability percentage will be inaccurate. In this situation, the availability percentage will be too high. To remedy this inaccuracy, Enterprise Manager lets you specify this outage retroactively. A retroactive outage is essentially a retroactive blackout that specifies target downtime should be included as part of the availability calculation.
Retroactive outage can be created either from the console or using EM CLI.
The following sequence of events illustrates the typical scenario for a retroactive outage.
The target goes down for an unknown reason.
Enterprise Manager does not detect the target's down state. Target availability remains at 100%.
The administrator brings the target back up.
The administrator creates a retroactive blackout with the Include target downtime in target availability (%) calculation option selected for the unplanned outage period.
Target availability goes down from 100% ==> 84%.
From the Enterprise menu, select Monitoring and then Blackouts. The Bl
ackouts page displays.Note:
The Include target downtime in target availability (%) calculation checkbox should be enabled during the creation of a retroactive blackout/outage, only if Enterprise Manager did not detect the outage. Checking this option will include target downtime in the target availability calculation and will result in a lower availability percentage.Select a reason from the Reason drop-down menu and then click Add to add the target(s) for which you are creating the retroactive outage.
In the Schedule region, specify the time period in which the target was down.
Click Submit to create the retroactive outage. A success confirmation message will be displayed on the Blackouts page.
The following graphic shows target availability history for the repository database before the retroactive outage has been defined. Target availability percentage is 100% with no down time.
The next graphic shows the target availability history after a retroactive outage of 56 minutes has been defined. Target availability percentage has been reduced to 84% with 56 minutes of target down time.
Creating a Retroactive Outage using the Command Line Interface (EM CLI)
create_rbk
verb. The verb syntax is as follows:
emcli create_rbk -add_targets="name1:type1;name2:type2;..."... -reason="reason" [-propagate_targets] -schedule= start_time:<yyyy-MM-dd HH:mm>; end_time:<yyyy-MM-dd HH:mm>; [tzregion:<...>] [-outage]
Table 4-1 create_rbk Options
Option | Description |
---|---|
add_targets | Targets to add to the blackout, each specified as target_name:target_type
The |
reason | Reason for the retroactive blackout. This is used for storing in backup tables. |
propagate_targets | When this option is specified, a blackout for a target of type host applies the blackout to all non-agent targets on that host. Regardless of whether this option is specified, a blackout for a target that is a composite or a group applies the blackout to all members of the composite or group. |
schedule | Blackout schedule.
Parameters
|
outage | Use this option with caution as it will lower the target availability (%). This option should be used only if Enterprise Manager did not detect the outage. |
The following example shows the command output.
General Usage Guidelines
For planned outages, where the administrator forgot to set a blackout, create a retroactive blackout without enabling the Include target downtime in target availability (%) calculation checkbox. This will increase the target's availability %. For example, 84% ==> 100%.
For unplanned outages, where Enterprise Manager did not detect the outage, create a retroactive blackout and enable the Include target downtime in target availability (%) calculation option. This will decrease the target's availability %. For example. 100% ==> 84%.
For unplanned outages, where Enterprise Manager did detect the outage, nothing needs to be done.
You can control blackouts from the Oracle Enterprise Manager 13c Cloud Control Console or from the Enterprise Manager command line utility (emctl
). However, if you are controlling target blackouts from the command line, you should not attempt to control the same blackouts from the Cloud Control console. Similarly, if you are controlling target blackouts from the Cloud Control console, do not attempt to control those blackouts from the command line.
From the command line, you can perform the following blackout functions:
Starting Immediate Blackouts
Stopping Immediate Blackouts
Checking the Status of Immediate Blackouts
Note:
When you start a blackout from the command line, any Enterprise Manager jobs scheduled to run against the blacked out targets will still run. If you use the Cloud Control Console to control blackouts, you can optionally prevent jobs from running against blacked out targets.
To use the Enterprise Manager command-line utility to control blackouts:
Table 4-2 Summary of Blackout Commands
Blackout Action | Command |
---|---|
Set an immediate blackout on a particular target or list of targets |
Be sure to use a unique name for the blackout so you can refer to it later when you want to stop or check the status of the blackout. The
If you do not specify a target or list of targets, Enterprise Manager will blackout the local host target. All monitored targets on the host are not blacked out unless a list is specified or you use the If two targets of different target types share the same name, you must identify the target with its target type. |
Stop an immediate blackout |
|
Set an immediate blackout for all targets on a host |
The |
Check the status of a blackout |
Use the following examples to learn more about controlling blackouts from the Enterprise Manager command line:
To start a blackout called "bk1" for databases "db1" and "db2," and for Oracle Listener "ldb2," enter the following command:
$PROMPT> emctl start blackout bk1 db1 db2 ldb2:oracle_listener -d 5 02:30
The blackout starts immediately and will last for 5 days 2 hours and 30 minutes.
To check the status of all the blackouts on a managed host:
$PROMPT> emctl status blackout
To stop blackout "bk2" immediately:
$PROMPT> emctl stop blackout bk2
To start an immediate blackout called "bk3" for all targets on the host:
$PROMPT> emctl start blackout bk3 -nodeLevel
To start an immediate blackout called "bk3" for database "db1" for 30 minutes:
$PROMPT> emctl start blackout bk3 db1 -d 30
To start an immediate blackout called "bk3" for database "db2" for five hours:
$PROMPT> emctl start blackout bk db2 -d 5:00
The Blackouts Best Effort feature allows you to create blackouts on aggregate targets, such as groups or systems, for which you do not have Blackout Target (or Higher) privileges on all members of the aggregate target.
Here, an Enterprise Manager administrator has Blackout Target privilege on an aggregate target but do not have OPERATOR privilege on its member/associated targets. You should ideally create a Full Blackout on this aggregate target. When defining the blackout, you are allowed to select any member target, even those member targets for which you have no Blackout Target privileges.
When the blackout actually starts, Enterprise Manager checks privileges on each member target and only blackout those on which you have Blackout Target( or Higher) privileges. This automated privilege check and target blackout selection is Enterprise Manager's "best effort" at blacking out the aggregate target.
The Blackout Best Effort functionality is targeted towards the creation of blackouts on targets of any aggregate type, such as Group, Hosts, Application Servers, Web Applications, Redundancy Groups, or Systems.
All targets the blackout creator has Blackout Target (or higher) privilege on will be displayed in the first step of Create/Edit Blackout Wizard. Once the blackout creator selects an aggregate type of target to be included in the Blackout Definition, this Blackout is "Full Blackout" by default.
The creator has the option of choosing the Blackout to run on “All Current" or “Selected" Targets, by selecting the appropriate values from the List box. Only when the "Full Blackout" option is chosen, will Blackout Best Effort affect targets for which the creator does not have Blackout Target (or higher) privileges.
Example Use Case
Consider 3 targets T1,T2 and T3 (all databases). A Group G1 contains all these 3 targets.
User U1 has OPERATOR privilege on T1,T2 and G1. User U1 has VIEW privilege on T3.
User U1 creates a scheduled full blackout on target G1. Scheduled implies that the blackout will start at a later point in time.
At the time of blackout creation, the tip text Needs Blackout Target privilege, see Tip below the table would be shown beside target T3.
When this blackout starts, if by that time User U1 has been granted OPERATOR privileges on target T3, then target T3 would also be under blackout. Otherwise only targets T1, T2 and G1 will be under blackout.