1. Cloud Control Monitoring Guide
  2. Monitoring and Managing Targets
  3. Using Dynamic Runbooks

7 Using Dynamic Runbooks

Dynamic Runbooks allow you to capture procedural knowledge and expertise in diagnosing and resolving issues from your subject matter experts and store them in Enterprise Manager for ready access and execution by other Enterprise Manager users. Dynamic Runbooks are supported for Incidents and Metrics.

This chapter covers the following topics:

Feature Updates

Feature/Update Release
Metric Steps Support for Related Targets Enterprise Manager 13c Release 5 Update 21 (13.5.0.21)
Launch Runbooks from the All Metrics Page Enterprise Manager 13c Release 5 Update 21 (13.5.0.21)
Support Links to EM Pages in Step Instructions and Note Steps Enterprise Manager 13c Release 5 Update 21 (13.5.0.21)
Save Step Output into Runbook Variables Enterprise Manager 13c Release 5 Update 15 (13.5.0.18)
Comments in the Runbook Session Enterprise Manager 13c Release 5 Update 15 (13.5.0.17)
Runbook Variables as bind parameters on SQL Enterprise Manager 13c Release 5 Update 15 (13.5.0.17)
Warning and Critical Thresholds in Metric Step chart Enterprise Manager 13c Release 5 Update 15 (13.5.0.17)
Oracle Provided Runbooks and OS Command Step Enterprise Manager 13c Release 5 Update 15 (13.5.0.15)
Import/Export and Note Step External Links Enterprise Manager 13c Release 5 Update 11 (13.5.0.11)
Relevant Runbooks Enterprise Manager 13c Release 5 Update 10 (13.5.0.10)
Create Like Enterprise Manager 13c Release 5 Update 8 (13.5.0.8)
Dynamic Runbooks Initial Release Enterprise Manager 13c Release 5 Update 7 (13.5.0.7)

About Dynamic Runbooks

Background and Concepts

Dynamic Runbooks are documented procedures that IT Operations staff follow to resolve an issue. The steps in the Dynamic Runbook are often based on the operational experience of subject matter experts who are responsible for providing a consistent way to respond to and resolve issues in a timely manner.

Dynamic Runbooks typically consist of a set of ordered instructions (steps) that include looking at metric data, correlating data across targets, and executing SQL against the Enterprise Manager repository or target databases to resolve the issue. Many, if not all such tasks, can be performed entirely within Enterprise Manager. With Dynamic Runbooks, users can execute these steps inside Enterprise Manager.

Terminology

  • Runbook

    Set of ordered instructions, data, and SQL designed to provide a standard way to diagnose and respond to an incident.

  • Runbook Session

    Execution of a Runbook by a user

  • Runbook Variables

    Context values applied to a Runbook. These could be: metric name, target name, target type, etc. These can be populated from an incident or metric in which the Runbook session is executed, or they could be an output of a previous Runbook step, i.e. the value for a Runbook variable can be manually filled in by the session user based on the output from one of the previous steps. Instructions for how this should be manually done from a previous step can be covered via the description in the step.

Prerequisites

You need to set the following OMS properties in order for the Repository SQL and Target SQL Runbook steps to work:

  • oracle.sysman.db.restfulapi.executesql.repository.query.enable
  • oracle.sysman.db.restfulapi.executesql.target.query.enable
  • oracle.sysman.db.restfulapi.executesql.target.update.enable
  • oracle.sysman.db.restfulapi.executesql.throttle.max.req.per.user.interval.sec
  • oracle.sysman.db.restfulapi.executesql.throttle.max.req.per.user
  • oracle.sysman.db.restfulapi.executesql.throttle.max.concurrent.request

Examples: 

emctl set property -name oracle.sysman.db.restfulapi.executesql.repository.query.enable -value true -sysman_pwd "<your password>"
emctl set property -name oracle.sysman.db.restfulapi.executesql.target.query.enable -value true -sysman_pwd "<your password>"
emctl set property -name oracle.sysman.db.restfulapi.executesql.target.update.enable -value true -sysman_pwd  "<your password>"
emctl set property -name oracle.sysman.db.restfulapi.executesql.throttle.max.req.per.user.interval.sec -value 60 -sysman_pwd "<your password>"
emctl set property -name oracle.sysman.db.restfulapi.executesql.throttle.max.req.per.user -value 30 -sysman_pwd "<your password>"
emctl set property -name oracle.sysman.db.restfulapi.executesql.target.update.enable -value true -sysman_pwd "<your password>"
emctl set property -name oracle.sysman.db.restfulapi.executesql.throttle.max.concurrent.request -value 20 -sysman_pwd "<your password>"

Run any SQL on Database and Connect Target privilege is required on the database in order to run a Target SQL type step.

Working with Dynamic Runbooks

At a high level, working with Dynamic Runbooks involves the following workflows:

Create a New Dynamic Runbook

You can create a Runbook in one of three ways:

You can go to Incident Manager, choose the incident for which you want to create a Runbook, and then click on the Create Runbook link.


Graphic displays the Runbook Sessions region of the Incident Manager page.

Go to the All Metrics page of a target homepage and select the metric for which you would like to create a runbook. In the Runbook Sessions section, click on Create Runbook link.


Create runbook from All metrics page

Alternatively, you create a Runbook by creating a Runbook Draft and then specify the context and the context ID for which you would like to create a Runbook. The context ID can either be an Incident context or Metric context. For Incidents, get the Incident ID from Incident Manager. For Metrics, get the metric context from the Runbook Sessions section associated with the metric in the target's All Metrics page.


create Runbook workflow

In all these ways, choosing the incident or metric will provide the context that will be used to develop and test the Runbook steps. Once the Runbook is complete, you can publish the Runbook.

Create a New Version of a Dynamic Runbook

You create a new version of a published Dynamic Runbook by creating a Runbook draft from the Runbook, editing the steps and re-publishing the Runbook. When you publish the new version of the Runbook, the new published version replaces the prior version.


create a new version of a Runbook

Use a Published Dynamic Runbook

By executing a Dynamic Runbook from the context of an incident or metric, a Runbook session is created.


using a Runbook workflow

Types of Dynamic Runbook Users

There are different types of Dynamic Runbook users that play specific roles in the creation, running, and administrative maintenance of Runbooks.

  • Runbook Author

    This user can:

    • Create and publish Runbooks
    • Show/hide published Runbooks

    Create Runbooks privilege is required.

  • Runbook User

    This user can:

    • Execute published Runbooks (start Runbook session)
    • If a user can see an incident, they can execute the Runbook session

    All Enterprise Manager Users are Runbook users.

  • Runbook Administrator

    This user can:

    • Delete and show/hide on any Runbook
    • Show/hide published Runbooks
    • List and delete Runbook sessions marked done that have been created by other users
    • Extend expiry of any runbook session marked done
    • View Runbook session data without access to incidents/jobs

    Manage Runbooks privilege is required.

Note:

The Enterprise Manager Super Administrator has complete access to all facets of the Runbook lifecycle (creation, administration, editing, publication)

Accessing Dynamic Runbooks

From the Enterprise Manager console, you can access Runbooks from the following locations:

Runbooks List Page

From the Enterprise menu, select Monitoring and then Runbooks. The Runbooks page displays all Published Runbooks. If you have the Create Runbooks privilege, you will also see available Runbook drafts. You can initiate Runbook draft creation from this page. However, the recommended way to create an new Runbook is from Incident Manager, in context of the incident for which you want to create a Runbook.

Runbook Sessions Page

From the Enterprise menu, select Monitoring and then Runbook Sessions. Here, you'll see a list of Runbook sessions you own (both active and done). To start a new Runbook session, go to Incident Manager, locate the incident for which you would like to start a Runbook session, then click on the option to Start Runbook Session.

Target's All Metrics Page

From any target homepage menu, select Monitoring then All Metrics. When you choose a metric from the list of metrics, a Runbook Sessions section will be available next to the metric chart. From here you can start a Runbook session from any available runbooks for that metric or create a new Runbook in context of that metric.

Incident Manager

From the Enterprise menu, select Monitoring and then Incident Manager. When you click on a particular incident, you'll see any Runbook sessions created for that incident. You'll also have the option to view the top (up to three) Runbooks that may be relevant to the incident (see Relevant Runbooks for more information). From here, you'll be able to start a Runbook session or create a new Runbook within the context of the incident.

All Metrics

In the target instance homepage, from the target menu, select Monitoring and then All Metrics. When you click on a particular metric in a metric group, you will see a Runbooks section similar to the one you would see in Incident Manager with all of the same options for starting a Runbook session or creating a new Runbook draft.

Creating Dynamic Runbooks

Any Runbook Author can create a Dynamic Runbook

  1. To create a Dynamic Runbook for an incident, from the Enterprise menu, select Monitoring and then Incident Manager. View incident details by clicking its row and then go to the Runbook Sessions region. If there is a Runbook session that is currently active, it will be listed.

    Alternatively, go to the All Metrics page to access Runbooks. To create a Dynamic Runbook for a metric, locate the metric in the All Metrics page of the appropriate target. You should see a Runbook Sessions region next to the metric's performance charts.

  2. In the Runbook Sessions region, click Create Runbook. The New Runbook authoring UI displays in the context of the incident or metric, thus allowing you to create Runbook steps.
  3. By default, the authoring UI opens with the Overview & Prerequisites as the first step. Here, you enter a description of what the new Runbook will accomplish and any prerequisites that are required to complete all Runbook steps. To aid readability, the Overview & Prerequisites step supports simple formatting via markdown notation. Click on the information icon (“I”) in the text entry area to view a list of markown elements supported by the text editor. The markdown formatting will be applied when you save the step.
  4. Click Save Step.
  5. From the Add a Step drop-down menu, select the desired step type you want to define. There are four step types:
    • Note: Use the Note step to provide instructions to the user. You can use markdown formatting with this step and also include Runbook variables. Refer to the Runbooks UI for details on the Runbook variables
    • Metric Data: Show a graphical view of a metric over time
    • Repository SQL: Allows you to run a SQL statement against documented Enterprise Manager repository schema views (MGMT$)

      Note:

      Accessing other elements from the Enterprise Manager repository schema is not supported.

    • Target SQL: Allows you to execute SQL against any database target in Enterprise Manager

    • OS Commands and Scripts: Allows you to execute a single operation, as you would do on the command line or to specify an interpreter to be used to execute a script which you enter as a part of the step definition.

    For more information about step types and how to use them, see Step Types.

  6. When you select a step type, the step authoring UI displays. Enter a step title and description of what the step will do, instructions, and any other information regarding interpretation of results.

  7. By default, the step will be prepopulated with settings from the context of the incident or metric. Depending on the step type, you can select other step parameters such as Metric Group Name by clicking on the current selection to display the pop-up selector and choose a new value from the drop-down list.

    Note:

    At the right, you can choose from a list of variables pulled from the incident context. If none of the variables meet your requirement, you can create your own variable by clicking Add a Runbook Variable.

    When defining a step, you can assign data from the output of the step to populate values of existing author-defined Runbook variables.

    Starting with Enterprise Manager 13c Release 5 Update 21 (13.5.0.21) details, such as target name and internal target type, for a different target instance, can be set directly or through the use of Runbook Variables.

  8. Save to Variable (Optional)

    When defining a step, you can dynamically assign data from the output of the step to populate values of author-defined Runbook variables, following the steps below:

    Note:

    A Runbook Variable can be assigned a value from only one cell of one single step's output.
    1. Click Save to Variable to open the tab.
    2. From the output of the step, click the cell containing the value you want to save to a variable. Once you do so, you will see a Save to Runbook Variable dialog appear.
    3. In the Save to Runbook Variable dialog box, click to open the dropdown list providing the available variables, and select the one you want to contain the data, then click Save. If you are saving a value to a Runbook Variable from another step, your author-defined Runbook Variable will not show up in the dropdown list. If the list is empty or you want to create a new variable, click Create New Runbook Variable at the top of the list, and use the pop-up window to create a new variable.

    The Saved Variables list, on the right side of the Save to Variables tab, shows the variables that take data from the step output.

  9. Links to EM Pages in Step Instructions (Optional)

    Use the MarkDown language to specify a link to an EM page. To construct the link, navigate to an EM page and then copy the part of the URL which begins with " / " after the host and port name. For example, if the url is http://my.emsite.com:1234/em/faces/si-host-home?type=oracle_database&target=Oemrep_Database then you could use it as a link as [Target Home Page](/em/faces/si-host-home?)

    Other examples:

    [Runbook Page](/em/faces/runbookPage)
    [Target Page](/em/faces/core-uifwk-targetSearch-home)

    You can use also use variables in your URLs. For example, if your target Oemrep_Database is type=oracle_database, they can be put into variables target_type and target_name respectively such that the link would be [Target Home Page](/em/faces/si-host-home?type=$target_type&target=$target_name)

  10. Click Run to execute the step and view any rendered visualizations such as charts or tables defined by the current parameter settings. You can run multiple tests to ensure that your settings provide the desired results. Once the step is run, you will see that any values selected to be saved to Runbook Variables will now be assigned to those Runbook Variables.
  11. Click Save Step. The step is now ready to use and will be added to the list of steps.
  12. Once you've finished adding steps, you've created a Runbook draft as it's still editable. It will appear in the Runbooks list page. (From the Enterprise menu, select Monitoring and then Runbooks). In draft mode, only the author can view/edit the Runbook.

Step Types

You can create the following types of steps:

Note:

The following step limitations apply to all step types:
  • 1MB of data (configurable via the oracle.sysman.core.runbooks.maxStepDataSizeKB is the OMS property)
  • 20 steps per Runbook (configurable via the oracle.sysman.core.runbooks.maxSteps OMS property)
  • Note Step

    Any type of text meant to provide information or instruction pertaining to the Runbook. The Note can be plain text, but can accommodate simple formatting via markdown language. A markdown reference table showing supported formatting is provided when creating a Note step.

    Input:

    1. Any type of text
    2. Basic formatting of text (markdown)
      1. bulleted lists (1 level)
      2. numbered lists (1 level)
      3. bold text
      4. italic text
      5. bold, italic text
      6. Header (h3, h4, h5)
      7. horizontal rule
      8. blockquote
    3. Runbook Variables
    4. External Links

      You can specify two types of external links within a Note step using the following markdown notation:

      Links to a specific URL: [Link Text](http//www.anothercompany.com)

      Links to My Oracle Support notes (only the MOS ID is required): [MOS Link Text](MOS:MOS ID). For example [Enterprise Manager configuration issues](MOS:1234567.1)

      Links to EM pages: [Target Page](/em/faces/core-uifwk-targetSearch-home)

      Note:

      External linking is available starting with Enterprise Manager 13c Release 5 Update 11 (13.5.0.11).

    Output:

    Formatted text with Runbook Variables substituted out for values.

    Example:

    Remember to have a named credential for the target database **$ora_target_name** on which the FRA incident has occurred. The Note step will show:Remember to have a named credential for the target database orcl_database on which the FRA incident has occurred.

  • Metric

    This step shows the time series chart for the specified metric. Defaults to the metric from passed in context, but it can be changed. When the Runbook is launched in context of an incident/event, the chart also shows when the event occurred.

    With Oracle Enterprise Manager 13c Release 5 Update 17 (13.5.0.17), the chart shows lines representing the current warning and critical thresholds for the metric. This allows you to evaluate the metric trend against warning and critical thresholds.

    Starting with Enterprise Manager 13c Release 5 Update 21 (13.5.0.21) details, such as target name and internal target type, for a different target instance, are available to be typed directly in the target selector input box or set through the use of Runbook Variables.

    Note:

    Configuration metrics are not supported.

    Input

    1. Target name/specification
    2. Metric Group
    3. Metric
    4. Metric Key Values
    5. Time range notation:

      You can also specify a metric time span by providing start and end dates. You can make those times relative to now or evt_time.

      • now corresponds to current time.
      • evt_time corresponds to the date and time when the event occurred.

      Well known Runbook variable named ora_event_time from the incoming context corresponds to the evt_time. Use the keyword now or evt_time followed by + or - and finally an integer and m for minutes, h for hours, or d for days.

      Examples:

      now : current time when the step is run

      now - 6h : 6 hours prior to the current time

      evt_time + 3d : 3 days after the time the event occurred

    Output:

    1. Time series line chart display
    2. Legend

    Note:

    If the Runbook was launched in context of an incident/event, the chart will show when the incident occurred so long as the specified time range includes when the incident happened.

  • Repository SQL

    You provide a SQL statement to be run against MGMT$ views in the repository schema. You are limited to running the SQL against the repository view tables (MGMT$ or GC$). You can use BIND parameters. There is a limit on step size data: The maximum limit for a SQL query is 20 columns and 1000 rows, or 1 MB of data (whichever limit is reached first).

    Using Bind Parameters

    You can bind values from Runbook Variables into your SQL statement using a colon followed by the Runbook Variable reference. For instance:

    select target_name from MGMT$TARGETS where target_type = :$ora_target_type_name

    This will create a bind parameter in the SQL statement and use whatever the value of $ora_target_type_name is as the value for that bind parameter when the SQL is executed.

    Starting with Enterprise Manager 13c Release 5 Update 21 (13.5.0.21) you can use a repository SQL step to find the details of another target instance (target name, and internal target type). These values can be assigned to Runbook variables, that can be used in the metric step edit screen to drive the values of the step.

    SQL Query Resultset formatting

    1. If the query column list contains a clob, wrap the CLOB column_name in TO_CHAR() function.

      Example:

      SELECT em_varchar_col, to_char(clob_column) from MGMT$CLOB_TABLE
    2. If the query column list contains a timestamp, wrap the date column name in a TO_CHAR() function and provide the proper date format mask.

      Example:

      SELECT em_varchar_col, to_char(em_timestamp_col,'DD-MON-YYYY HH24:MI:SS') from MGMT$DATE_TABLE
    3. If a user defined variable is a string that represents a date and the variable is used in the query WHERE clause, wrap the variable name in a TO_TIMESTAMP() or TO_DATE() function and provide the corresponding date format mask.

      Example:

      SELECT em_varchar_col,to_char(em_timestamp_col,'DD-MON-YYYY HH24:MI:SS') FROM MGMT$DATE_TABLE where em_timestamp_col
< to_date(:$l_user_defined_date_variable,'DD-MON-YYYY')
  • Target SQL

    Ability to run predefined SQL (single statement only) on any database target. In addition to executing select statements, you can run Data Manipulation Language (DML) and Data Definition Language (DDL). The named credential is required whether it is a select statement or DDL/DML.

    Pre-requisites:

    In order to run the Target SQL step on a database target, you will need to have these privileges on your database targets:

    1. Run any sql on Database
    2. Connect Target
    3. Access to the named credential referenced in the step

    Input

    1. Target Name
    2. Target Type
    3. SQL
    4. Named Credential

    Using Runbook Variables

    You can use Runbook Variables in your Target SQL queries, DML or DDL statements.

    1. SQL queries (SELECT statements):

      Using Bind Parameters you can bind values from Runbook Variables into your SQL statement using a colon followed by the Runbook Variable reference.

      Example:

      select target_name from MGMT$TARGETS where target_type =:$ora_target_type_name

      This will create a bind parameter in the SQL statement and use whatever the value of $ora_target_type_name is as the value for that bind parameter when the SQL is executed.

    2. DML or DDL statements

      You can use Runbook Variables in your DML or DDL statements by using a colon followed by the Runbook Variable, such as :$ora_target_name or :$fra_size.

      Example:

      alter system set db_recovery_file_dest_size=:$fra_size  scope=both

      It will substitute the value of the variable before executing the DML or DDL statement.

    Using ISO 8601 formatted Runbook variables in SQL Query Where Clause

    If a Runbook variable's value is in ISO 8601 date format (such as 2022-03-01T19:21:00.000Z) and you need to use it in a where clause on a database table/view with date column:

    1. Use to_utc_timestamp_tz function to convert the value into a database timestamp with time zone value.
    2. Value from #1 can be converted to another time zone (such as target time zone) as needed.
    3. cast the output from #1 or #2 into date type, and use it as part of the where clause to join with tables having date column.

    4. Following examples assume the utc_time1 Runbook variable is set to a value in ISO 8601 date format (such as '2022-03-01T19:21:00.000Z').

      Examples:

      Example #1: Shows how to cast the variable value to date without taking timezone conversion into account.

      
select to_char(min(collection_timestamp),'YYYY-MM-DD HH24:MI:SS') as min_time, to_char(max(collection_timestamp), 'YYYY-MM-DD HH24:MI:SS') as max_time, count(1) as total_count
from mgmt$metric_current
where collection_timestamp >= cast(TO_UTC_TIMESTAMP_TZ(:$ora_event_time_utc) as date)

      Example #2: Shows how to convert the variable value to target timezone first, then cast it to date;

      
select to_char(min(collection_timestamp),'YYYY-MM-DD HH24:MI:SS') as min_time, to_char(max(collection_timestamp), 'YYYY-MM-DD HH24:MI:SS') as max_time, count(1) as total_count
from mgmt$metric_current
where collection_timestamp >= cast((TO_UTC_TIMESTAMP_TZ(:$ora_event_time_utc) at time zone TIMEZONE_REGION) as date)

    Output

    1. Tabular display of return data
  • OS Commands and Scripts

    Allows you to execute a single operation, as you would do on the command line on the host of the specified target. or to execute a script which you enter as a part of the step definition using the specified interpreter.

    Pre-requisites:

    In order to run the OS Command and Scripts step, you will need to have these privileges:

    1. View privilege on the target, target's host and the agent that monitors the target's host
    2. Access to the named credential referenced in the step

    Input

    1. Target Name
    2. Target Type
    3. Named Credential

    4. Command Execution Type:

      Single Operation

      1. Command

        Note:

        The interpreter will be defaulted for whatever the OS is on the host.

      Script

      Note:

      For Windows platform, Script as a Command Execution Type is currently not supported.

      1. Script

      2. Interpreter

        Note:

        If left blank, the interpreter is defaulted according to the OS on the target host where the script is executed.

    Output:

    Returns the output from running the command or script on the target host.

Creating Runbook Variables

Runbook Variables are populated with values from the incident details when a session or draft is created. These values can be used by steps for display in the instruction text or as a part of the step definition which gets data or performs some task. This allows a Runbook session to provide information and perform tasks based on context specific to that session. And for a draft, it allows an author to define a Runbook using the Runbook Variables rather than hardcoded values.

For values which are not included as context when the draft is created, the author can create their own Runbook Variables. This Runbook Variable can be populated in these ways:

  • The runbook Variable can have a static value which remains the same across the sessions that will eventually be created on that Runbook

  • The Runbook Variable can require the session user to provide a value each time a new session is created for the Runbook. For instance, for target DB SQL steps, a named credential is required. Creating a Runbook Variable for this named credential and requiring the value to be set by the session user ensures that the SQL as a part of that step will be executed with a named credential the session user has access to for that session’s target DB.

  • The Runbook Variable can also by dynamically populated with data from the output of a step. This feature makes the use of Runbook Variables more powerful and is available starting with Oracle Enterprise Manager 13c Release 5 Update 18. As an example, you may have a step that determines the filesystem used by the database. The author can save the output of that step into a Runbook Variable, and then use that Runbook Variable in the next step that determines the available free space of the filesystem. Note that a Runbook Variable can save data from a specific value cell of a step's output, hence if you need to save multiple data values, you'll need to save these in multiple Runbook Variables.

There are some restrictions around the naming of Runbook Variables an author can create:

  1. It cannot start with ora_.
  2. It can contain only letters, numbers, and underscores.
  3. It cannot be longer than 64 characters.
  4. Additionally, for credentials, _cred should be included in the name (this allows us to filter for the specific case of named credentials used in the target DB step)

When creating a Runbook Variable, the author will provide:

  1. A name, which is how the Runbook Variable will be referenced in the step (for instance, user_target_db_cred)
  2. A display name, which is a short, user readable name (for instance, Target Database Named Credential)
  3. A description (optionally), which will be displayed in the session UI when the session user is asked to fill in a value (for instance, Provide a named credential for this target database)
  4. Value to be used in the draft, which allows you to create the Runbook Draft with a value the author can use to execute the step
  5. Options for Value used in a Runbook Session:

    There are options for how the value will be populated in a Runbook Session. The table below has a summary of these options and usage guidelines.

Value used in a Runbook Session When to use this option How the runbook variable value is populated Can runbook session user change it? Usage Notes
Specified by session user or by running a step Use this option when you want the runbook session user to specify the value of the runbook variable. Typically, this is done by running a runbook step to populate the runbook variable. For advanced users, the session user can manually enter the runbook value if needed. In a runbook session, the user runs a step to populate the runbook variable or he types in a value for the variable. Yes - by typing in a value or by running a step The initial value of the variable is empty (null).
Value provided by author or by running a step Use this option if you don't want the session user the ability to manually specify the runbook variable value. Instead, you want the session user to run a step to populate the value or you want to define the value as part of the runbook definition. The runbook author defines the value as part of runbook definition or he has the user run a step to populate it. Yes - only by running a step, if the runbook author creates a step that populates the runbook variable If the author expects the value of the variable to be populated by running a step, the initial value of the variable is null.

Graphic shows the Add a Runbook Variable dialog.

Publishing Runbooks

Once you are finished defining your Runbook draft, you're ready to publish it. Once published it will become available to all Runbook users. Only the author of the Runbook draft can publish it.

Navigate to the Runbooks list page (Enterprise->Monitoring->Runbooks) and go to the Drafts tab. From the Actions menu for your Runbook, select Publish. Click on the Published tab to view your Runbook in the published Runbook list.

Creating New Versions of Runbooks

  • Create a draft from a published Runbook. Navigate to the Runbooks page. From the Actions menu for the desired Runbook, select Create New Version.
  • While the draft is being edited, the published version of the Runbook remains available for users to continue to create sessions.
  • The Runbook author will be required to provide a context ID to do so.
  • When the new draft is published, it replaces the previous published Runbook after user confirmation.
  • This new published Runbook will be used to create all future Runbook sessions. It has no impact on prior Runbook sessions.

Creating New Runbooks from Existing Runbooks

To save time and effort, you can create a new Dynamic Runbook from an existing one by using the Create Like action. Create Like can be used on either published or draft Runbooks.

Using Published Dynamic Runbooks

Note:

You do not have to be the owner of the published Runbook to use the Create Like action.
  1. From the Enterprise menu, select Monitoring and then Runbooks.
  2. Click on the Published tab to view the table of published Runbooks.
  3. From the Actions drop-down menu for the chosen Runbook, select Create Like. The Create Like dialog is displayed.
  4. Enter a context ID to use to create the new draft Runbook.
  5. Click OK. The Runbook editor screen is displayed with a copy of the published runbook in draft mode. As the owner of this draft, you can edit this runbook in the context of the specified context ID.

Using Draft Dynamic Runbooks

Since you own the draft version of the Runbook, a more complete copy can be made. You do not need to provide an context ID since the selected draft Runbook will already have that information.

  1. From the Enterprise menu, select Monitoring and then Runbooks.
  2. Click on the Drafts tab to view the table of draft Runbooks that you own.
  3. From the Actions drop-down menu for the chosen Runbook, select Create Like. The Create Like confirmation dialog is displayed.
  4. Click OK. The Runbook editor page displays the new draft Runbook.

Other Operations

After you've published the Runbook, you can perform other operations on it from the Actions menu.

You can:
  • Hide/Unhide the Runbook to control availability of published Runbooks for public use.
  • Delete the published Runbook.

Note:

These operations can only be performed by the Runbook author or Runbook administrator.

Adding External Links

You can add external links to information that may assist a future Runbook user with additional diagnostic or contextual information.

Creating a Rule to designate a Dynamic Runbook for an Incident

After creating and publishing a Runbook, you may want to designate specific Runbooks to be used for an incident. The designation of the specific Runbook for an incident can be done as part of Event Rule actions:

  1. Start creating an event rule.

  2. In the Add Actions page, select Create Incident.


    Add actions page

  3. Click the search bar in the Associate Runbook section.
  4. Search and select the Runbook you want to designate for the incident, and click Select.

When the incident rule is applied to create an incident, the Runbook associated with it in the step above will be a part of the incident and appear in the section below under Start Runbook Session. It will be the only entry with text Recommended By Incident Rule. All other relevant Runbooks are still accessible through the All Relevant Runbooks... link.


Runbooks display

Runbooks associated through the incident rule will always appear at the top of the list and have the text [Recommended by Incident Rule] next to the Runbook name. The relevance score will be automatically 100% (full bar is shown) and the text will show Recommended.


runbooks selector page

Using Dynamic Runbooks

As a Runbook user, you can create a Runbook session from Incident Manager or from the All Metrics page.

  1. Navigate to the Runbook Sessions region, and click Start Runbook Session.

    The Start a Runbook Session page appears and displays a table where you can select a Runbook with which to start a session.

    Note:

    From the Incident Manager Runbook Sessions region, you can also view a list of Runbooks that may be relevant to the specific incident (see Relevant Runbooks for more information). To access one of the Relevant Runbooks, click All Relevant Runbooks… to display the Start a Runbook Session page listing the Relevant Runbooks.

  2. Click Start Session. The Runbook session page opens for that incident or metric. Incident or metric context details are listed at the top of the page.
  3. The Overview & Prerequisites step is intended to show what the Runbook is used for and any prerequisites required of the user before executing subsequent steps in the Runbook. Ensure all prerequisites are met before beginning the Runbook session. Once you've completed this step, you can click on the checkbox to indicate the step has been completed.
  4. For each step, click the play icon to execute the operation and review live diagnostic information produced by the step. If you see a gear icon next to a Runbook step, that means user input is required before the step can be executed. An example of user input is a named credential for the target database. Once the input is specified, the gear icon will change to the play icon, which will indicate the step can be executed.
    Graphic shows examples of the gear and play runbook icons.

    At the right is a list of Runbook Variables that pertain to the incident context.

    Starting with Enterprise Manager 13c Release 5 Update 21 (13.5.0.21) metric steps for other target instances will be available.

    Note:

    The final step should be a Success Criteria step to allow the session user to verify that the Runbook steps performed by the user resolved the issue.
  5. If you don’t complete the Runbook session, you can leave it open and come back to it. Only you, as the session owner will be able to return to continue running steps and complete the Runbook session. Note that there is a 14 day purge policy, which means you have 14 days to complete the Runbook session before it is purged from the system.

    Note:

    The default purge policy can be increased by setting the oracle.sysman.core.runbooks.runbookSessionMinLifeDays OMS property.
  6. Once you have finished with the Runbook session, Click Mark as Done.
    • This places the Runbook session into read-only mode. No more edits or running steps allowed.
    • Once marked done, the session can be saved for a longer period of time by extending the expiration date in the Runbooks Sessions list page.

Extend the Expiry Date

  • Sessions are kept for a default amount of time and automatically purged
  • The time can be extended if the session is marked as done.
  • The default time a session is saved is 14 days, default extended time is 45 days from when session is started.

    Note:

    The default purge policy can be increased by setting the oracle.sysman.core.runbooks.runbookSessionMaxLifeDays OMS property.

Exporting and Importing Dynamic Runbooks

Beginning with Oracle Enterprise Manager 13c Release 5 Update 11 (13.5.0.11), you can export and import published Runbook definitions, thus significantly enhancing Runbook reuse and sparing administrators from having to develop Runbooks from scratch.

Note:

You can only export/import published Runbooks. Runbook drafts and Runbook sessions cannot be used as they have contextual properties that are specific to the Enterprise Manager system in which they are authored. They will not be portable across Enterprise Manager installations.

Usage Guidelines

Exporting/importing published Runbooks across Enterprise Manager versions:

  • Export from an older Enterprise Manager version and import to a newer Enterprise Manager version will always work.
  • Export from a newer Enterprise Manager version and import to an older Enterprise Manager version will only work if step types are compatible (supported in both versions).

The following table lists the currently available step types and the corresponding minimum compatible Enterprise Manager release.

Step Type Applies to EM release
Note Enterprise Manager 13c Release 5 Update 7 and later
Target SQL Enterprise Manager 13c Release 5 Update 7 and later
Repository SQL Enterprise Manager 13c Release 5 Update 7 and later
Metric Enterprise Manager 13c Release 5 Update 7 and later

Republish Runbooks authored before Enterprise Manager 13c Release 5 Update 11

For published Runbooks created using Enterprise Manager 13c Release 5 Update 7 thru 10, you must republish the Runbooks you plan on exporting/importing. This ensures that Runbook properties are up-to-date and compatible across Enterprise Manager deployments.

Note:

To republish a Runbook, simply create a new draft from an existing published Runbook, optionally edit the draft, and then publish Runbook.

Required User Privileges

In order to export or import a published Runbook, you must have at least one of the following privileges:

  • CREATE_RUNBOOK—Allows you to create and publish Runbooks.
  • MANAGE_RUNBOOK—Allows you to perform administrative operations, such as deleting/showing/hiding Runbooks, or viewing Runbook session data.

For more information, see Types of Dynamic Runbook Users.

Export a Published Runbook

  1. Navigate to the Published page. From the Enterprise menu, select Monitoring and then Runbooks.
  2. Select Export from the Actions menu for the Runbook you want to export. An Export dialog displays with the default Export Filename. By default, the name of the currently published Runbook is used, however, you can enter a new filename.
  3. Click OK. A Runbook definition JSON file is saved to your local file system.

Import a Published Runbook

  1. Navigate to the Published page. From the Enterprise menu, select Monitoring and then Runbooks.
  2. Click Import. The Import a Published Runbook dialog displays
  3. Either drag and drop a Runbook definition file directly into the dialog, or click the Drag and Drop region to open a file browser to select a file. The imported definition file will immediately appear in the Published Runbook table.
  4. Click Close.

Note:

The user importing a Runbook definition becomes the new owner of that Runbook.

Oracle Provided Runbooks

Beginning with Oracle Enterprise Manager 13c Release 5 Update 15 (13.5.0.15), you can use these Runbooks to help triage and resolve issues in your Enterprise Manager environment.

Dynamic Runbook Description

Loader issues causing Agent backoff requests

This Runbook helps you diagnose and resolve incidents that the EM administrator receives when many agents have been asked to back off their uploading of data to the Oracle Management Server (OMS).

DB Tablespace Used Metric Time Out Error

This Runbook helps diagnose metric collection errors for the Oracle database Tablespace Space Used (%) metric.
Job Suspended Agent Not Ready This Runbook helps troubleshoot job executions that are in status Suspended Agent Not Ready.

Advanced Operations and Configuration

The following topics cover areas that may be of interest to Runbook administrators and authors.

Dropping a User and the Impact on Dynamic Runbooks

Deleting an Enterprise Manager user will affect Dynamic Runbooks in the following ways:

  1. Published Runbooks
    1. When a user is deleted, published Runbooks owned by the deleted user can be reassigned to any other Enterprise Manager User
      1. The reassigned user does not need Create Runbooks privilege. It can be granted later
    2. When the deleted user owns published Runbooks and no reassigned user is specified, then Runbooks are reassigned to the repository owner
  2. Draft Runbooks
    1. Draft (unpublished) Runbooks owned by the deleted user are always deleted
    2. Any data saved as a part of the Draft Runbook will be deleted as well
  3. Runbook Sessions
    1. Runbook sessions are not reassigned to any user, even if a reassign user is specified
    2. Open Runbook sessions owned by the deleted user are automatically marked as DONE
    3. DONE Runbook sessions owned by the deleted user will be dropped by the automated purge policy when they expire, as are any other sessions.

Configurable Properties

Runbook OMS Properties

The following OMS properties are used to configure Dynamic Runbook. See EMCTL Commands for OMS for information on setting OMS properties.

  • Max Step Data Size Limit

    Name: oracle.sysman.core.Runbooks.maxStepDataSizeKB

    Description: The maximum data size allowed for a Runbook step in kilobytes.

    Default: 1024 (1 MB)

    Min: 100 (100KB)

    Max: 10240 (10MB)

  • Max Steps Allowed

    Name: oracle.sysman.core.Runbooks.maxSteps

    Description: The maximum number of steps allowed for a Runbook.

    Default: 20

    Min: 10

    Max: 50

  • Minimum days of life for session

    Name: oracle.sysman.core.Runbooks.RunbookSessionMinLifeDays

    Description: Initial number of days a session will be kept before being purged.

    Default: 14

    Min: 14

    Max: 30

  • Maximum days of life for session

    Name: oracle.sysman.core.Runbooks.RunbookSessionMaxLifeDays

    Description: Maximum number of days allowed for a Runbook session before it is purged.

    Default: 45

    Min: 45

    Max: 90

  • Max Output Size Returned By OS Command Step

    Name: oracle.sysman.core.runbooks.oscmd.maxRunStepOutputSize.KB

    Description: The maximum data size that os command run step returns in kilobytes.

    Default: 1024 (1 MB)

    Min: 10 (100KB)

    Max: 10240 (10MB)

  • Max Time To Run OS Command Step

    Name: oracle.sysman.core.runbooks.oscmd.maxRunStepTime.seconds

    Description: The maximum time allowed for os command run step REST API in seconds.

    Default: 120 ( 2 minutes)

    Min: 15 (15 seconds)

    Max: 600 (10 minutes)

Repository SQL Throttling

To minimize impact on Enterprise Manager performance, you can enable SQL throttling. See Repository Session (SQL) Throttling for more information.

Relevant Runbooks

A Runbook session can be started directly from the Incident Details page for a relevant Runbook, or you can click on All Relevant Runbooks... and a list of Runbooks ranked by Relevance Score is available from which to choose. Any Runbook that applies for the incident will appear in the Start a Runbook Session page table, paginated if necessary.


Image shows the Start a Runbook Session page for Relevant Runbooks

From this page, you can start a Runbook session that best matches the resolution requirements of the incident using the Relevance Score as a guide. As shown above, Relevance Scores are conveniently sorted in ascending order.

About Relevance Scores

A Relevance Score is a measure of how well a Runbook matches the current incident. The greater level of matching, the higher the Relevance Score. A Runbook draft is created in context of a specific incident and carries forward some of those values when it is published. The Runbook carries parts of this incident’s context even when it is published and available to use for any incident. That initial context tells the published Runbook which properties it provides are required and which are optional.

When a Relevance Score is created, the context from the incident you are currently viewing is compared to the published Runbook. The following is used to generate the score:

  • All required properties must be present. This is required to create a Runbook Session for the current incident.
  • Optional properties are evaluated. The Relevance Score is increased if an optional property exists. The score is further increased if the property value from the published Runbook matches the current incident context.
  • Runbook variables used in steps are evaluated. Similar to optional properties, the Relevance Score is increased if a variable used in a step appears in the information provided by the incident. The score is further increased if the incident and variable values match.

Note:

By default, Runbooks published prior to Enterprise Manager 13c Release 5 Update 10 (13.5.0.10) will have lower Relevance Scores. To bring them up to parity with newer Runbooks, you need to republish these older Runbooks. Republishing will cause the system to recompute the Relevance Score.

Follow these steps to quickly republish your older Runbooks.

As the Runbook owner:

  1. Locate the published version of the Runbook and create a new version. It will create a Runbook draft. There is no need to change anything. The Runbook draft will be saved automatically.
  2. Next, immediately publish this new draft. A Publish Runbook Draft dialog will appear, asking you if you want to replace the existing published Runbook with this new version. Click OK to accept this option. Your Runbook will then be published.