1. Using Oracle Application Performance Monitoring
  2. Administer Oracle Application Performance Monitoring

4 Administer Oracle Application Performance Monitoring

An Oracle Application Performance Monitoring administrator can deploy and administer the service in your environment.

Topics

Typical Tasks for Administering Oracle Application Performance Monitoring

Here are the administrative tasks for Oracle Application Performance Monitoring.

Task Description More Information
Deploy Oracle Application Performance Monitoring Download and install Oracle Application Performance Monitoring

Deploy Oracle Application Performance Monitoring
Specify servlet configuration options Specify how your web application / servlet is being monitored by Oracle Application Performance Monitoring.

Define Servlet Configuration Options
Set rules for alerts Set rules for email alerts

Set Alert Rules
Set up Synthetic Monitoring Set up synthetic monitoring, and monitor application performance through synthetic tests Typical Workflow for Using Synthetic Monitoring
Set up End User Monitoring Set up End User Monitoring with manual browser injection, enable and configure settings Set Up End User Monitoring
Set up Synthetic Monitoring Set up Synthetic Monitoring by simulating user paths to detect possible errors Set Up Synthetic Monitoring
Set up Custom Instrumentation Set up Custom Instrumentation and start monitoring technologies not supported out-of-the-box. Set Up Custom Instrumentation
Enable Privacy Settings Enable Privacy Settings to stop APM from storing personal identifiable information. Enable Privacy Settings

Customize APDEX Settings

The Application Performance Index, APDEX is a measurement of overall performance of your application.

APDEX is a summary of various measurements, with appropriate weightage on certain important results. Outliers which are otherwise ignored are given due importance in this index. The formula used to calculate APDEX = (Number of Satisfactory samples + (Number of Tolerating samples) / 2 ) / Total samples.

Session Health uses APDEX results for page and AJAX requests, along with the impact of error executions of java scripts and AJAX requests. With the Session health settings, you can influence the weightage given to the various components while calculating session health.

You can customize settings for the Application Performance Index. To customize settings for APDEX:
  1. In the left navigation pane, click Administration and select Metric Settings.
  2. In the APDEX Settings pane, provide Satisfactory and Tolerable values for Page Load Time and Ajax Call Response Time.
  3. In the Session Health Settings pane, provide a weightage for each element of the formula used to calculate session health:

    1. Page Performance Weight

    2. Ajax Performance Weight

    3. Javascript Errors Weight

    4. Ajax Errors Weight

  4. Click Save on the top right corner of the screen to save your settings.
The new settings will be used to determine APDEX value and to rate the session health. Note that elements for which there is no data (Example: No page views) will not be considered in the formula.

Associate Entities Using Tags

You can associate Application Servers to Database using tags.

To associate an application server to a database:
  1. From the Oracle Management Cloud left navigation menu, click Administration and select Entities Configuration.
  2. Select Tags to view a list of tags that have already been discovered.
  3. In the Tags screen, click New Key to create a new association with tags.
  4. In the Create New Key screen, provide a name and value for the key.
Once these associations are established, you can view information about the database from the App Server screen, and from the Application Definition.

Deleting Tags

  1. From the Oracle Management Cloud left navigation menu, click Administration and select Entities Configuration.
  2. Select Tags to view a list of tags that have already been discovered.
  3. In the Tags screen, search for the tag using the search box.
  4. Select the tag, and click the Delete Tag button.
  5. Click OK to confirm the operation.
    The selected tag is deleted.

Associate Application Servers to a Database Automatically

Application Performance Monitoring automatically creates an association between the App Server and the Database. When the association is not automatic, you can dynamically make the association through matching association tags.

Associate Application Servers to a Database through tags for a new Target

After installing and deploying Application Performance Monitoring, you can associate application servers to a database through tags. The below procedure is for a fresh installation of Application Performance Monitoring.

To associate an APM-Invisible Database with Application Server using tags:

  1. Download the APM Agent Software Installer and provision APM Agent. See Types of APM Agents to install the correct APM Agent.

  2. Shut down the application server to which you are associating the database.

  3. Add the below property in the Properties file to generate an association from the application server to the database:

    For APM Java agent:

    1. From the apmagent/config folder, edit the AgentStartup.properties file and add the below property: 

      <KeyName>=<Tag Value>
oracle.apmaas.agent.appServer.uses = Tag_Weblogic_To_Oracle

    2. Generic property tag: In the AgentStartup.properties file, you can add a new list of tags in the format tag-name [ = tag-value ] to the below property: 

      oracle.apmaas.agent.appServer.tags
      Example:
      oracle.apmaas.agent.appServer.tags=credToUse=exampletag1,exampletag2,tag3=exampletag3
      The above property will create 3 tags:

      • credToUse=exampletag1

      • exampletag2

      • tag3=exampletag3

    For APM .Net agent:

    • From the <Path where APM Agent Software is extracted>\Oracle APM .NET Agent folder, edit the AgentConfig.ini file and add the below property: 

      <KeyName>=<Tag Value>
oracle.apmaas.agent.appServer.uses = Tag_DotNet_To_Oracle

    In the above configuration file, the following is the Source Entity Marker tag:

     assoc_source:<AssociationHint>=<AssociationTag>
    Where:

    • assoc_source:<AssociationHint> is the tag key.

      <AssociationHint> can be any free form text used to identify, and therefore associate the set of destination entities. The <AssociationHint> used for the source entity, should match the one used for the destination entity(entities).

    • <AssociationTag> is the tag value.

      <AssociationTag> is the intended association tag. It is optional, and if a value is not provided, it defaults to omc_uses association tag.

    An entity can be tagged with one or more source marker tags if the source entity can have different associations with different sets of destination entities. Ensure that the source marker tag is unique.

  4. Start the application server.

  5. Verify that the tags are added to the application.

    1. From the Oracle Management Cloud left navigation menu, click Administration and select Entities Configuration.

    2. Select Tags to view a list of tags that have already been discovered.

  6. Verify that the application server is associated with the database in any of these ways:

    • From the Application Definition screen, view the topology.

    • From the App Server screen, click the Topology button to see the association.

You can also associate an App Server to a database from the APM UI. See Associating Entities Using Tags.

Discovering App Servers and Database entity using tag from Monitoring Service for a New Installation

  1. Install the Oracle Cloud Agent.

  2. To discover the database, modify the db.json and add the tag element.

    Example:

    { "entities":[
      {
         "name":"Oracle_DB",
         "type":"omc_oracle_db",
         "displayName":"Oracle_DB",
         "timezoneRegion":"PDT",
         "credentialRefs":["SQLCreds"],
         "properties":{
                        "host_name":{"displayName":"MachineName","value":"sample.test.com"},
                        "port":{"displayName":"OraclePort","value":"1000"},
         "protocol":{"displayName":"Protocol","value":"TCP"},
                        "sid":{"displayName":"SID","value":"sample"},
         "capability": {"displayName": "capability","value": "monitoring"}
                      },
"tags": {
       "assoc_dest:<assoc_name>" : ""
   }
        }
]}

    Where <assoc_name> can be any string based on user preference. Discover the database entity using cloud agent ( omcli add_entity agent db.json ).

  3. To discover the application server, modify the app server json and add the tag element.

    Example:
    {
"entities":[
       {
         "name":"Tomcat_Sample_Demo8",
        "tags": {
                "assoc_source:<assoc_name>" : ""
                },
         "type":"omc_tomcat",
         "displayName":"Tomcat_Sample_Demo8",
         "timezoneRegion":"PDT",
         "properties":{
        "host_name":{
            "displayName":"Host",
                        "value":"abcd.test.com"
                           },
                "jmx_port":{
            "displayName":"JMX Port Number",
            "value":"9999"},
        "jmx_user_name":{
            "displayName":"JMX User Name",
            "value":""},
        "jmx_password":{
            "displayName":"JMX Password",
            "value":""},
        "jmx_protocol":{
            "displayName":"Communication Protocol",
            "value":"rmi"},
        "jmx_service":{
            "displayName":"Service Name",
            "value":"jmxrmi"},
        "ssl_trust_store":{
            "displayName":"SSL Trust Store (required when SSL is enabled)",
            "value":""},
        "ssl_trust_store_password":{
            "displayName":"SSL Trust Store JMXPassword (required when SSL is enabled)",
            "value":""},
        "ssl_trust_store_password":{
            "displayName":"SSL Trust Store JMXPassword (required when SSL is enabled)",
            "value":""},"version":{
            "displayName":"Apache Tomcat Version",
            "value":"8"},
        "catalina_base_directory_path":{
            "displayName":"Catalina Base Directory Path",
            "value":"/scratch/opt/apache-tomcat-8.0.29"},
        "LoggingConfigurationFilePath":{
            "displayName":"Logging Configuration File Path",
            "value":""},
        "LogLocationCatalina":{
            "displayName":"Log Location Catalina",
            "value":""},
        "LogLocationLocalhost":{
            "displayName":"Log Location Local Host",
            "value":""},
        "LogLocationHostManager":{
            "displayName":"Log Location Manager",
            "value":""},
        "LogLocationHostManager":{
            "displayName":"Log Location Host Manager",
            "value":""},
        "capability": {
          "displayName": "capability",
          "value": "monitoring"}

          }
        }
]
}
    where <assoc_name> string should be same as in the database json in step 2. Discover the Tomcat entity using cloud agent (omcli add_entity agent tomcat.json.)

  4. "omc_uses" association should be automatically created between Tomcat and the database entity.

Update the App Servers and Database entity using tag from monitoring Service

You can add new tags to existing entities, but cannot update existing tags from the cloud agent omcli cmd. Here is a sample json:
{ "entities":[
      {
         "name":"Sample_DB",
         "type":"omc_sample_db",
         "displayName":"Sample_DB",
         "timezoneRegion":"PDT",
         "credentialRefs":["SQLCreds"],
         "properties":{
                        "host_name":{"displayName":"MachineName","value":"sample.test.com"},
                        "port":{"displayName":"OraclePort","value":"1111"},
         "protocol":{"displayName":"Protocol","value":"TCP"},
                        "sid":{"displayName":"SID","value":"orcl12c"},
         "capability": {"displayName": "capability","value": "monitoring"}
                      },
"tags": {
       "assoc_dest:<assoc_name>" : "",
       "assoc_dest:<assoc_name2>" : "",

       "key1" : "value1"

   }
        }
]}
Use the omcli update_entity agent db.json command to upload the new tags from the entity json to Oracle Management Cloud.

Define Locations for Synthetic Tests

An administrator defines locations from where synthetic tests will be run.

Prerequisites before you can define locations for synthetic tests:
  • Install Firefox browser on the machine where the cloud agent is installed:
    • Oracle Linux 6: Firefox version 45
    • Oracle Linux 7: Firefox version 61-66

    Note:

    Firefox is the only supported browser. Other Firefox versions including Beta versions are unsupported.

    You can check if Firefox is present on the cloud agent machine by running the command firefox --version.

    Install the relevant Firefox version on the host, and specify the correct path of the Firefox executable while defining a location. To migrate existing tests from Firefox 45 to Firefox 61, see Migrating Synthetic Tests to Firefox 61.

  • Firefox should be part of the PATH shell variable of the user used to run the cloud agent. For example, if Firefox executable is available at /scratch/firefox, then /scratch/firefox should be part of the PATH variable.

  • Install a Cloud Agent on the machine where the tests will be run. See Deploying Cloud Agents in Deploying and Managing Oracle Management Cloud Agents for instructions.

  • Ensure Perl is installed on the machine running the Cloud Agent.

  • Create an X Server pool, a set of X Servers which will be used to execute the Selenium test.

    You can check if an X Server pool exists by running the command ps —ef. For example, to check if there are any Xvfb X servers running, run following command: 
    ps -ef | grep Xvfb

  • Any X Server can be used, but the recommended X Server is Xvfb. To create an X Server pool using Xvfb, run the following commands:

    Xvfb :1
Xvfb :2
Xvfb :3
Xvfb :4
Xvfb :5
    Every time the above commands are run, an X Server with the specified display port is created. These display ports can be used to specify the X Server pools, while creating a location.

    Note:

    In case of an error when you run the above command, run this command instead: 
    Xvfb :1 -nolisten inet6
    The following is an example on how to setup Xvfb server and auto-start at system boot:

    1. $ sudo mkdir -p /usr/local/sbin # (if not already present)

    2. $ sudo cp xvfb_start.sh xvfb_stop.sh /usr/local/sbin

    3. $ sudo chmod 744 /usr/local/sbin/xvfb_start.sh /usr/local/sbin/xvfb_stop.sh

    4. $ sudo cp etc_systemd_system_oracle.service /etc/systemd/system/xvfb.service

    5. $ sudo systemctl daemon-reload

    6. $ sudo systemctl enable xvfb

    7. $ sudo systemctl start xvfb

    8. $ sudo systemctl status xvfb

  • In the file /etc/hosts, replace

    127.0.0.1  localhost.localdomain  loghost localhost

    with

    127.0.0.1  localhost   localhost.localdomain  loghost.

  • The cloud agent should not be installed in a symbolic link directory, when installing the cloud agent the value provided for AGENT_BASE_DIRECTORY parameter should not be a symbolic link.
To define locations:
  1. In the left navigation pane, click Administration.
  2. Select Locations to view a list of locations that are available.
  3. On the Locations screen, click Add Location.
  4. In the Add Location screen, specify information about the new location:

    • Name: Specify a name for the location

    • Available Agents: Select a cloud agent for the location.

    • Specify other optional details for the location — Longitude, Latitude, City and Country.

    • Optionally specify Proxy details for the location — Host, Port, Username and Password.

    • Capacity: Specify the maximum number of machines for this location.

    • X Server Pool: Specify the X Server pool to be used while executing the Selenium test. If you have X servers running on ports, you can specify the port number.

      Example: If X servers are running on display ports 1, 2 and 3, specify 1-3 as value for this property.

    • Firefox Path: Specify the path of Firefox executable. Example: /usr/bin/firefox.

  5. Click Save.
The new location will be displayed in the list of Locations, and will be available for selection while creating a synthetic test.

Migrating Synthetic Tests to Firefox 61

  1. Install version 61 (or higher) of Firefox on the machine where you will install the Cloud Agent.
  2. On the same machine, install Cloud Agent version 1.33 or higher.
  3. Define a Location using this Cloud Agent.
  4. Once this Location is active in the system, edit your synthetic test/s and add this new location to the test/s.
  5. Wait for Synthetic Tests to run on this new Location. You can verify this in the Instances tab where you will see data coming from two Locations. You can then edit the test and delete the location which was created using an older version of Firefox.

Configure Errors and Error Messages

APM Java Agent can automatically detect operation invocation errors while monitoring application flows. An invocation error could be defined as an unexpected event happening as an operation is invoked. To detect such events and classify them as errors, APM agents rely on a set of default rules:

  • HTTP response status code that ranges from 400 to 599

  • Exceptions thrown during operations execution

  • SOAP message response with a Fault block

  • OSB error codes

In some cases, however, you may want to ignore an error and have the invocation classified as success. For example, based on your Application behavior, you might want to treat HTTP 403 response codes as success. With Error configuration, you can modify the default error classification rules, customizing them using an opt-out / opt-in mechanism, excluding or including your own defined rules. 
To configure errors:
  1. Open the file Error.json in an editor from the following path:
    • To configure errors on all the APM Agents in a domain, edit this file - <DOMAIN_HOME>/apmagent/config/Error.json.
    • To configure errors on the APM Agent on a specific server, edit this file - <DOMAIN_HOME>/apmagent/config/<server-name>/Error.json.
  2. Delete the content of the Error.json file. Replace it with the following lines, modifying to specify your error rules: 
    {
  "type" : "error",
  "configurationsPerComponentType" :
  [
    {
      "componentTypes" : [],
      "excludeReturnCodes" : {},
      "excludeReturnCodesRegex" : {},
      "excludeErrors" : {},
      "excludeErrorMessages" : {},
      "includeReturnCodes" : {},
      "includeReturnCodesRegex" : {},
      "includeErrors" : {},
      "includeErrorMessages" : {},
      "includeFirst" : false
    }
  ]
}
  3. Save and close the Error.json file. You do not have to restart the application server for these rules to take effect.
    Example: Here's an example where a HTTP error 403 is converted from an error to an acceptable event.
    {
  "type" : "error",
  "configurationsPerComponentType" :
  [
    {
      "componentTypes" : ["SERVLET"],
      "excludeReturnCodes" : {"HTTP 403" : []},
      ...
      "includeFirst" : false
    }
  ]
}
You can use configurationsPerComponentType to define the error rules. Each rule (exclude/include) is expressed as a key-value pair, where the key defines the error code selection and value defines the list of operations the rule applies to. The value is optional and if not specified, the error code selection applies to all operations of the specified componentTypes. Here are the rules you can use to configure the errors:
Rule Type Property Type Description
Component Type componentTypes Array List of component types the configuration should be applied for. The same configuration can be applied to multiple types.
Opt-out (Exclude Rules) excludeReturnCodes Map

Key - an error code in String

Value - an array of operation names in String
excludeReturnCodesRegex Map

Key - Regex pattern for the error code

Value - a list of regex patterns for operation names
excludeErrors Map

Key - a string of error type. Error type key has to be exactly the same as the error type shown in Error Detail page.

Value - a list of regex patterns for operation names
excludeErrorMessages Map

Key - a regex pattern of an error message

Value - a list of regex patterns for operation names
Opt-in (Include Rules) includeReturnCodes Map

Key - error code in String

Value - an array of operation names in String
includeReturnCodesRegex Map

Key - Regex pattern for the error code

Value - a list of regex patterns for operation names
includeErrors Map

Key - a string of error type. Error type key has to be exactly the same as the error type shown in Error Detail page.

Value - a list of regex patterns for operation names
includeErrorMessages Map

Key - a regex pattern of an error message

Value - a list of regex patterns for operation names
Include First includeFirst boolean Determines which set of properties the configuration apply first to exclude or include server request instances from being marked as fault.

Notes on how these error rules work:

  • includeFirst

    • If includeFirst is false, opt-out rules are applied first in the following order:

      excludeReturnCodes → excludeReturnCodesRegex → excludeThrowableClasses → excludeThrowableMessages→includeReturnCodes → includeHttpReturnCodesRegex → includeThrowableClasses → includeThrowableMessages

    • If includeFirst is true, opt-in rules are applied first::

      includeReturnCodes → includeHttpReturnCodesRegex → includeThrowableClasses → includeThrowableMessages → excludeReturnCodes → excludeReturnCodesRegex → excludeThrowableClasses → excludeThrowableMessages

  • Opt-in and Opt-out properties are independent of each other. A server request needs to satisfy only one of those properties in the configuration.
  • All operations:
    For opt-in and opt-out properties, specify an empty list value or a list with an empty string to exclude or include any operation name with a key. For example, if you want agent to not report any error with HTTP 500 return code regardless of its operation name, you would define the following:
    excludeReturnCodes : {"HTTP 500" : []}
or,
excludeReturnCodes : {"HTTP 500" : [""]}
  • Look for these values in these APM views:
  • Using Opt-in Properties: Use Opt-in properties if you use regex to opt-out many errors from the default rules but want to allow some cases to be still treated as errors. For example, if you want to exclude HTTP 5XX from the rules except HTTP 500, set the value of includeFirst to true so that opt-in properties can be applied first in the configuration.
    {
  "type" : "error",
  "configurationsPerComponentType" :
  [
    {
      "componentTypes" : ["SERVLET"],
      "excludeReturnCodes" : {},
      "excludeReturnCodesRegex" : {"HTTP 5[0-9][0-9]" : []},
      "excludeErrors" : {},
      "excludeErrorMessages" : {},
      "includeReturnCodes" : {"HTTP 500" : ["/hello/errorConfigTest/throwException (GET)", "/hello/errorConfigTest/throwException (POST)"]},
      "includeReturnCodesRegex" : {},
      "includeErrors" : {},
      "includeErrorMessages" : {},
      "includeFirst" : true
    }
  ]
}
  • Two Types with Overlapping Properties
    Two or more component types can share the same customized rules if you specify all the types: 
    {
  "type" : "error",
  "configurationsPerComponentType" :
  [
    {
      "componentTypes" : ["SERVLET", "JAXRS", "SOA"],
      "excludeReturnCodes" : {},
      "excludeReturnCodesRegex" : {},
      "excludeErrors" : {"java.lang.RuntimeException" : ["/hello/errorConfigTest/throwException .*"]},
      "excludeErrorMessages" : {},
      "includeReturnCodes" : {},
      "includeReturnCodesRegex" : {},
      "includeErrors" : {},
      "includeErrorMessages" : {},
      "includeFirst" : false
    }
  ]
}
  • One Type with Multiple Configurations
    One type can belong to more than one configurationsPerComponentType if properties in configurationsPerComponentType do not overlap. This is useful if one type share the same properties as other component types but have one or two property that don't. Make sure there is no duplicate property in two different configurationsPerComponentType to avoid redundancy.
    {
  "type" : "error",
  "configurationsPerComponentType" :
  [
    {
      "componentTypes" : ["SERVLET", "JAXRS", "SOA"],
      "excludeReturnCodes" : {},
      "excludeReturnCodesRegex" : {},
      "excludeErrors" : {"java.lang.RuntimeException" : ["/hello/errorConfigTest/throwException .*"]},
      "excludeErrorMessages" : {},
      "includeReturnCodes" : {},
      "includeReturnCodesRegex" : {},
      "includeErrors" : {},
      "includeErrorMessages" : {},
      "includeFirst" : false
    },
    {
      "componentTypes" : ["SERVLET"],
      "excludeReturnCodes" : {"HTTP 500" : ["/hello/errorConfigTest/throwException (GET)"]},
      "excludeReturnCodesRegex" : {},
      "excludeErrors" : {},
      "excludeErrorMessages" : {},
      "includeReturnCodes" : {},
      "includeReturnCodesRegex" : {},
      "includeErrors" : {},
      "includeErrorMessages" : {},
      "includeFirst" : false
    }
  ]
}

Enable Privacy Settings

Enable Privacy Settings to stop APM from storing personal identifiable information.

Storage of certain data can be toggled in these settings like:
  • Storing full URLs, not only the domains
  • Storing page titles and click names

To enable or disable privacy settings:

  1. Log into Oracle Management Cloud and click APM.
  2. In the APM Admin page, click Privacy Settings.

Note that Web Application data will not be affected. Follow instructions on the configuration page to configure IP address reporting and URL related privacy settings based on your reporting requirements. It is strongly recommended to avoid collecting any personally identifiable information.