1. Configure Oracle Cloud Observability and Management on Apache Tomcat on LINUX ARM by using APM and Stack Monitoring
  2. Configure and Deploy Monitoring Assets

Configure and Deploy Monitoring Assets

To onboard the Linux ARM hosts with Oracle Database 19c and Apache Tomcat, you need to complete the following tasks:
  1. Deploy an APM Java Agent on Apache Tomcat.
  2. Configure synthetic monitoring to monitor application endpoints.
  3. Deploy the management agents.
  4. Discover infrastructure on Stack Monitoring.

Configure Application Performance Monitoring

First, you configure application performance monitoring.

From your web browser, login to your OCI tenancy. You'll enter most of the following commands from the Cloud shell. If you are not familiar with it, see related documentation. If you don’t have a compartment for O&M resources (such as APM domains), create it now:
  1. Select Identity & Security then Compartments page and click Create Compartment.
  2. Provide a meaningful name and description, for example:
    • Name = om-demo
    • Description = Compartment for Observability & Management resources
    Wait a minute for the compartment to be provisioned. Once that's complete, note of the compartment’s OCID.

Now, do the following:

  1. Create a new APM Free Tier domain by opening the Cloud shell and doing the following:
    1. Choose a compartment for O&M resources by entering: 
      export COMPARTMENT_OCID=YOUR_COMPARTMENT_OCID
    2. Choose APM domain name by entering: 
      export APM_DISPLAY_NAME=demoapm01 

oci apm-control-plane apm-domain create --compartment-id $COMPARTMENT_OCID --display-name $APM_DISPLAY_NAME --is-free-tier TRUE --wait-for-state SUCCEEDED --wait-interval-seconds 15
  2. Select Observability & Management then Application Performance Monitoring and then Administration, to display the creation status (be sure to select the correct compartment).
    Wait for the domain to be created, which takes about five minutes. You can also use the Cloud shell to see the progress by using the following commands: 
    export COMPARTMENT_OCID=YOUR_COMPARTMENT_OCID 

oci apm-control-plane apm-domain list --compartment-id $COMPARTMENT_OCID --all --query "data[].{\"display-name\":\"display-name\",\"lifecycle-state\":\"lifecycle-state\",\"id\":\"id\"}" --output table
    You should see a response like this:
    +--------------+-----------------------------+-----------------+
| display-name | id                          | lifecycle-state |
+--------------+-----------------------------+-----------------+
| demoapm01    | ocid1.apmdomain.oc1.***.*** | ACTIVE          |
+--------------+-----------------------------+-----------------+
  3. Note of APM domain OCID and enter the following commands to generate the data-upload-endpoint and the datakeys (which you'll use later to install the Java APM agent):
    export APM_OCID=YOUR_APM_DOMAIN_OCID

oci apm-control-plane apm-domain get --apm-domain-id $APM_OCID --query "data.{\"data-upload-endpoint\":\"data-upload-endpoint\"}" --output table

oci apm-control-plane data-key list --apm-domain-id $APM_OCID --all --query "data[].{\"name\":\"name\",\"type\":\"type\",\"value\":\"value\"}" --output table
    You should see a response like this:
    +-----------------------------------------------------------------------------+
| data-upload-endpoint                                                        |
+-----------------------------------------------------------------------------+
| https://***.apm-agt.***.oci.oraclecloud.com                                 |
+-----------------------------------------------------------------------------+ 

+--------------------------------+---------+----------------------------------+
| name                           | type    | value                            |
+--------------------------------+---------+----------------------------------+
| auto_generated_private_datakey | PRIVATE | ******************************** |
| auto_generated_public_datakey  | PUBLIC  | ******************************** |
+--------------------------------+---------+----------------------------------+
  4. Select Observability & Management then Application Performance Monitoring and then Administration, download the Java Agent from the Download APM Agents page.
  5. Copy the JAR file to each Apache Tomcat host:
    [tomcat@stvociapp021 ~]$ pwd
/home/tomcat
[tomcat@stvociapp021 ~]$ ls -l apm-java-agent-installer-1.10.1.3.jar
-rwxrwxr-x. 1 tomcat tomcat 11741769 Jul 26 07:39 apm-java-agent-installer-1.10.1.3.jar
[tomcat@stvociapp021 ~]$
  6. Install Java Agent with the Tomcat user by using the following commands (configure the VARIABLES based on your own environment):
    1. Set the Apache Tomcat installation home path: 
      export APM_INSTALL_DESTINATION=/opt/tomcat
    2. Set the APM installation JAR full path: 
      export APM_INSTALLER=/home/tomcat/apm-java-agent-installer-1.10.1.3.jar
    3. Choose the service name to be used on APM domain export: 
      APM_SERVICE_NAME=apex_tomcat_service
    4. Add APM domain information:
      export APM_PRIVATE_KEY=YOUR_PRIVATE_KEY 
export APM_ENDPOINT=YOUR_APM_ENDPOINT 
export APM_PUBLIC_KEY=YOUR_PUBLIC_KEY
    5. If required, add additional APM options; for example, thread snapshots or Enable APM Browser Agent Injection: 
      export APM_ADDITIONAL_OPTIONS="-property=com.oracle.apm.agent.deepdive.trace.snapshots.collection.enable=true"
    6. Set JAVA_HOME and PATH variable: 
      export JAVA_HOME=PATH_TO_JAVA_HOME 
export PATH=$JAVA_HOME/bin:$PATH
    7. Install the APM Java agent: 
      java -jar ${APM_INSTALLER} provision-agent -service-name=${APM_SERVICE_NAME} -destination=${APM_INSTALL_DESTINATION} -private-data-key=${APM_PRIVATE_KEY} -data-upload-endpoint=${APM_ENDPOINT} ${APM_ADDITIONAL_OPTIONS} -public-data-key=${APM_PUBLIC_KEY}
  7. Modify Apache Tomcat JAVA_OPTS by adding the javaagent property; for example:
    export JAVA_OPTS="-Dconfig.url=${ORDS_CONFIG} -Xms1024M -Xmx4096M -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -javaagent:${TOMCAT_HOME}/oracle-apm-agent/bootstrap/ApmAgent.jar "
  8. Restart Apache Tomcat.
  9. Verify the Java Agent log file (change TOMCAT_HOME to the Tomcat installation home path): 
    ls -l ${TOMCAT_HOME}/oracle-apm-agent/log/*/Agent*
  10. Finally, select Observability & Management then Application Performance Monitoring and then Trace Explorer and verify that traces are collected correctly.

Configure Synthetic Monitoring

After configuring the APM Java agent on Apache Tomcat, you can create monitors to simulate user activity in the application. Use the OCI Console or OCI Cloud Shell to programmatically create the OCI resources.

Navigate to the Synthetic Monitoring page (Observability & Management then Application Performance Monitoring then Synthetic Monitoring) to create the monitors that will continuously test your application endpoints.

Open the cloud shell to rapidly create a BROWSER monitor for your application. Then do the following:

  1. Get the list of available vantage points: 
    export APM_OCID=YOUR_APM_DOMAIN_OCID 

oci apm-synthetics public-vantage-point-collection list-public-vantage-points --apm-domain-id $APM_OCID
  2. Test the application endpoints (this example uses the Ashburn region to test these endpoints):
    echo "[\"OraclePublic-us-ashburn-1\"]" > $HOME/vantage_points.json
  3. Create the JSON file for Availability Configuration:
    echo {\"maxAllowedFailuresPerInterval\": 0, \"minAllowedRunsPerInterval\": 1} >  availability_configuration.json
  4. Create the JSON file for Network Measurements:
    echo {\"probeMode\": \"SACK\",  \"probePerHop\": 3,  \"protocol\": \"TCP\" } > network_measurements.json
  5. Finally, create the monitor:
    1. Identify your APM domain:
      export APM_OCID=YOUR_APM_DOMAIN_OCID
    2. Provide the monitor name: 
      export MONITOR_DISPLAY_NAME=demo-monitor-01
    3. Provide your application endpoint URL: 
      export MONITOR_TARGET=http://LOAD_BALANCER_PUBLIC_IP/APPLICATION_TEST_URL
    4. Provide the monitoring interval, in seconds:
       export MONITOR_REPEAT_INTERVAL=360 oci apm-synthetics monitor create-browser-monitor --apm-domain-id $APM_OCID --display-name $MONITOR_DISPLAY_NAME --monitor-type BROWSER --repeat-interval-in-seconds $MONITOR_REPEAT_INTERVAL --vantage-points file://$HOME/vantage_points.json --is-certificate-validation-enabled FALSE --is-failure-retried true --target $MONITOR_TARGET --availability-configuration file://$HOME/availability_configuration.json --network-configuration file://$HOME/network_measurements.json
From the Synthetic Monitoring page, you can see the status of the monitor you just created. In this example, the provided URL is monitored every six minutes (to view the details of test results, click the monitor name and select History from the Resources menu).

Deploy the Management Agent

Before configuring Stack Monitoring, you need to install the Oracle Management Agent in each host you want to monitor. This demo uses RPM installation for the Management Agent (you can use this method if your Linux ARM host is running on-premises or on a different cloud provider).

Note:

If you don't know how to install the Oracle Management Agent, see "Install Management Agents" in the OCI documentation, accessible from the "Explore More" topic at the end of this playbook.
Before you begin, download the Management Agent software from the Downloads and Keys page:
  1. Select Management Agent then Downloads and Keys).
  2. Select Agent for LINUX (AARCH64) with package type RPM.
  3. From the same page, create a new installation key to configure the agent. Provide the required parameters:
    • Key name = demo-key-01
    • Compartment = om-demo
    • Maximum installations = default

Note:

If you don't know how to create an installation key, see "Create Install Key" in the OCI documentation, accessible from the "Explore More" topic at the end of this playbook.
Now, deploy the management agent:
  1. Connect to Linux instances by using SSH and, if it's not already there, install JDK8: 
    # dnf install -y java-1.8.0-openjdk
  2. Install the Management Agent: 
    # rpm -ivh oracle.mgmt_agent.230622.2155.Linux-Aarch64.rpm
  3. Configure the Management Agent, as shown in the example below:
    1. Open a vi editor:
      # vi /usr/share/mgmt_agent/input.rsp
    2. Add the following content: 
      ManagementAgentInstallKey = <COPY_YOUR_KEY_HERE>
CredentialWalletPassword = <CHOOSE_YOUR_WALLET_PASSWORD>
    3. Change the owner of management agent:
      # chown mgmt_agent. /usr/share/mgmt_agent/input.rsp
  4. Run the following command to finalize management agent configuration:
    # sudo /opt/oracle/mgmt_agent/agent_inst/bin/setup.sh opts=/usr/share/mgmt_agent/input.rsp
After few minutes, you will see the agent from “Management Agent” then “Agents” page. You do not need to enable the Stack Monitoring plug-in; Stack Monitoring automatically enables it for you.

Enable Full Host Monitoring

With the Stack Monitoring service enabled, you next want to enable full monitoring of the hosts the application relies on. Stack Monitoring will automatically prepare a task to enable monitoring of an OCI compute instance. To enable full monitoring, select Promote to full monitoring. Identify the Tomcat and Oracle Database hosts from the list of available hosts to promote and select Promote. This will invoke a slide-out; from this slide-out verify the resource name matches that of the host's fully-qualified-domain name. Next, click Promote. Within minutes, Stack Monitoring will create a new homepage and begin collecting the new metrics.

Configure Stack Monitoring

Finally, you need to configure Stack Monitoring.

Before actually configuring the service, you need to meet these prerequisites:
  1. Enable the Stack Monitoring service.
  2. Enable JMX monitoring.
  3. Create the Oracle Database monitoring user.
Then complete these steps to configure Stack Monitoring:
  1. Enable full monitoring of the hosts.
  2. Discover Oracle Database.
  3. Discover Apache Tomcat.
These processes are described in the following steps.
  1. Enable the Stack Monitoring service:
    Navigate to Observability & Management then Application Performance Monitoring, then Stack Monitoring and click Enable Stack Monitoring.
  2. Enable JMX on Apache Tomcat:
    Stack Monitoring requires JMX to be enabled prior to discovery. To enable JMX on Apache Tomcat, do the following:
    1. Add the following lines to the setenv.bat: 
      set CATALINA_OPTS=-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=9012 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false
    2. Create the configuration folder to store the jmxremote.access and jmxremote.password files: 
      mkdir -p /opt/tomcat/conf
    3. Create jmxremote.access as tomcat user: 
      cd /opt/tomcat/conf 
echo "monitorRole readonly 
controlRole readwrite" > jmxremote.access
    4. Add the following to the jmxremote.password file. Note that the password file should be read-only and only accessible by the operating system user Tomcat is running as. As tomcat user: 
      cd /opt/tomcat/conf 
echo "monitorRole tomcat 
controlRole tomcat" > jmxremote.password
    5. Restart Apache Tomcat.
  3. Create the database monitoring user:
    Prior to discovering a database within Stack Monitoring, ensure you have access to the monitoring user. You can either use DBSNMP user that is built-in with the Oracle Database and has the privileges required to monitor the database OR create a custom user with only the necessary privileges. Steps to create a database monitoring user can be found in MOS Note: 2857604.1, which you can access from the "Explore More" at the end of this playbook.
  4. Enable full host monitoring:
    With the Stack Monitoring service enabled, the first step is to enable full monitoring of the hosts the application relies on. Stack Monitoring will automatically prepare a task to enable monitoring of an OCI Compute Instance. To enable full monitoring:
    1. Click Promote to full monitoring.
    2. Identify the Tomcat and Oracle Database hosts from the list of available hosts to promote and click Promote. This will invoke a slide-out.
    3. From this slide-out, verify that the Resource Name matches that of the hosts fully-qualified-domain name and then click Promote.

    Within minutes, Stack Monitoring will create a new homepage and begin collecting the new metrics.

  5. Discover the Oracle database that the application relies on:
    Discovering resources are easy with Stack Monitoring’s easy fill-in-the-blank discovery UI. To get started:
    1. Navigate to the “Resource discovery” page and click Discover New Resource. Once selected, a slide-out window will appear.
    2. From the drop-down menu select Oracle Database.
    3. Complete the fields such as Resource Name, DNS hostname, Service Name and click Discover New Resource.

    When the job completes Stack Monitoring will immediately begin collecting metrics and create a homepage.

  6. Discover the Apache Tomcat servers
    To discover Apache Tomcat:
    1. Navigate to “Resource discovery” and click Discover New Resource.
    2. From the drop-down list select Apache Tomcat.
    3. Complete the fill-in-the-blank fields, such as JMX port, and monitoring username and password and click Discover New Resource.

    Once the job completes, navigate to Stack Monitoring’s “Enterprise Summary”. The dashboard will display the performance and status of the Oracle Database, Apache Tomcat and the hosts on which Tomcat relies.

    Next, relate the resources, which will allow for easy navigation up and down the application stack. For more information on relating the resources see "Updating Application Topology" (which you can access from the "Explore More" topic at the end of this playbook).

    Finally, configure alarms for the resources. Examples of alarms:
    • Host
      • Filesystem Utilization
      • CPU and Memory Utilization
      • Monitoring Status
    • Oracle Database
      • Monitoring Status
      • CPU and Memory
      • Wait time
      • Storage Utilization by Tablespace
    • Apache Tomcat
      • Monitoring Status
      • JVM Memory Utilization (Heap and Non-heap)
      • CPU Utilization