14 Troubleshoot the Oracle Fusion Applications Tier Performance

Administrators will find it useful to study these suggestions on further analysis to undertake when a metric value exceeds threshold. The commands provided are for the Unix OS operating system.

When logical free memory/swap activity or paging activity is beyond threshold

This usually happens when memory is not sufficient to handle demands from all the running processes.

• Linux: Check cat/proc/meminfo and confirm total RAM is expected.

  AIX: Check /usr/sbin/prtconf | grep "Memory Size" and confirm total RAM is expected.

  Solaris: Check /usr/sbin/prtconf | grep "Memory size" and confirm total RAM is expected.

• Check if there are unallocated huge pages. If there are and the WebLogic Server/Oracle instances are not expected to use them, reduce the huge page pool size.

• Linux: Run top and sort by resident memory (type OQ). Look for processes using the most resident memory and investigate those processes.

  Solaris: Run top -o res. Look for processes using the most resident memory and investigate those processes.

When page activity is beyond threshold

Follow the steps in When logical free memory/swap activity or paging activity is beyond threshold to view and analyze memory usage.

When Network Interface Error Rates Is Beyond Threshold

The normal cause is misconfiguration between the host and the network switch. A bad network card or cabling also can cause this error. Run /sbin/ifconfig to identify which interface is having packet errors. Contact network administrator to ensure the host and the switch are using same data rate and duplex mode.

Otherwise, check if cabling or the network card is faulty and replace as appropriate.

When Packet Loss Rate Is Beyond Threshold

The normal cause of this error is network saturation of bad network hardware.

• On a UNIX system:

  • Run lsof -Pni | grep ESTAM to determine which network paths are generating the problem.

  • Linux: Then run mtr <target host> or ping <target host> and look for packet lost on that segment.

      20 packets transmitted, 20 received, 0% packet loss, time 18997ms
      rtt min/avg/max/mdev = 0.168/0.177/0.200/0.010 ms
    

    The packet loss should be 0% and rtt should be less than .5 ms.

  • Solaris: Then run /usr/sbin/ping -s <target host> and look for packet loss on that segment.

    2 packets transmitted, 2 packets received, 0% packet loss
    round-trip (ms) min/avg/max/stddev = 0.453/0.465/0.477/0.017
    

    The packet loss should be 0% and round-trip should be less than .5 ms.

• Ask the network monitoring staff to look for saturation or network packet loss from their side.

When Network Utilization Is Beyond Threshold

The normal cause is very heavy application load.

On a UNIX system:

• Linux/Solaris: Run top or lsof to determine which processes are moving a lot of data.

• Solaris: Use /usr/sbin/snoop to capture and inspect network packets.

• Linux: Use atop, iftop, ntop or pkstat to see which processes are moving data.

  Solaris: Use top to see which processes are moving data.

When CPU Usage or Run Queue Length Is Beyond Threshold

The normal cause is runaway demand, a poorly performing application, or poor capacity planning.

• Linux/Solaris: Run top to identify which application/process is using time.

• If top processes are WebLogic Server JVM processes, conduct a basic WebLogic Server health check. That is, review logs to see if there are configuration errors causing excessive exceptions, and review metrics to see if the load has increased. Use JVMD for a more detailed analysis.

• If top processes are Oracle processes, use Enterprise Manager to look for high load SQL.

When System CPU Usage Is Beyond Threshold

• High system CPU use could be due to kernel processes looking for pages to swap out during a memory shortage. Follow the steps listed in the When logical free memory/swap activity or paging activity is beyond threshold section to further diagnose the problem.

• High system CPU use is also frequently related to various device failures. Run {{dmesg | less}} and look for repeated messages about errors on some particular device, and also have hardware support personnel check the hardware console to see if there are any errors reported.

When Filesystem Usage Is Beyond Threshold

The normal cause is an application that is logging excessively or leaving behind temporary files.

On a UNIX system:

• Linux/Solaris: Run lsof -d 1-99999 | grep REG | sort -nrk 7 | less to see currently open files sorted by size from largest to smallest. Investigate the large files.

• Run du -k /mount_point_running_out_of_space > /tmp/sizes to get space used for directories under the mount point. This may take a long time. While it is running, run sort -nr /tmp/sizes and find the directories using most space and investigate those first.

When Total Processes Is Beyond Threshold

The normal cause is runaway code or a stuck NFS filesystem.

• Solaris: Run /usr/ucb/ps aux. If many processes are in status D, run df to check for stuck mounts.

  If there are hundreds or thousands of processes of a particular program, determine why.

• Linux: Run ps -o pid,nlwp,cmd | sort -nrk 2 | head to look for processes with many threads.

  Solaris: Run ps -o pid,nlwp,comm,user | sort -nrk 2 | head to look for processes with many threads.

When Disk Device Busy Is Beyond Threshold

• Check for disk drive failure.

  UNIX: As root, check /var/log/messages* and /var/log/mcelog for Linux, and /var/adm/messages* for Solaris, to see if there are any error messages indicating disk failure. For a RAID array, the disk controller needs to be checked. The commands will be specific to the controller manufacturer.

• Look for processes that are using the disk. From a shell window, execute ps aux | grep ' D. ' several consecutive times to look for processes with "stat" D.

Poor performance is a major indicator of network connectivity problems.

• Check for cumulative dropped packets drops for each host.

  UNIX

  netstat -s | grep 'TCP data loss'
    4007 segments retransmited
    3302 TCP data loss events
  

  The counts should be 0 or growing very slowly over time.

• Check for realtime dropped packets on specific network paths.

  Run the ping command.

  ping -c 20 other_host
    20 packets transmitted, 20 received, 0% packet loss, time 18997ms
    rtt min/avg/max/mdev = 0.168/0.177/0.200/0.010 ms
  

  Packet loss should be 0%.

  rtt should be less than .5 ms, except that it can be higher between the browser and load balancer.

  Solaris: /usr/sbin/ping -s other_host <data_size> <npackets>

  2 packets transmitted, 2 packets received, 0% packet loss
  

  Packet loss should be 0%.

• Check for network interface errors.

  Linux

  /sbin/ifconfig eth0 | grep errors
     RX packets:842803463 errors:0 dropped:0 overruns:0 frame:0
     TX packets:667946307 errors:0 dropped:0 overruns:0 carrier:0
  

  Solaris

  /sbin/ifconfig  -a // to list all the adapter names
  kstat <adapter name> |grep -i error
  

These metrics provide an indication of whether the WebLogic Server is in a healthy state. Performance may degrade if any of the metrics is exceeding its threshold.

Table 14-2 describes the WebLogic Server metrics that should be monitored in Cloud Control.

When CPU Usage on Host is Beyond Threshold and WebLogic Server Process is Identified as Top CPU Consumer

• Examine the % Time spent in the GC metric to see if JVM is doing excessive GC (>20 percent). If so, follow the process for diagnosing WebLogic Server heap pressure as indicated in section When Percent Time Spent in Garbage Collector is Beyond Threshold.

• Look for incident creation rate and error logs and see if something is triggering a massive amount of logging/errors.

• In JVMD, select the CPU state filter and look at top methods. Look for threads that are consistently in a CPU state.

When There is a Spike in Active Web Sessions

• Check access logs to see if there is a spike in the number of users.

• Check if there are stuck threads, which could cause users to log in again.

• Check session distribution across WebLogic Server managed servers and see if there is a problem with the load balancer.

• Check session timeout in web.xml, and see if it is too high or too low.

When There are Stuck Threads on the System

Analyzing live stuck thread can be done using the Live Thread Analysis (LTA). For more information see sections Viewing the JVM Live Thread Analysis Page and Using Java Workload Explorer for investigating both live and past stuck thread; in the Oracle Enterprise Manager Cloud Control Oracle Fusion Middleware Management Guide .

When There are Deadlocks Detected on the System

• In JVMD, inspect the threads that are in a blocked state.

• Deadlock threads normally also will be reported as a stuck thread in the WebLogic Server log. Use the Request Monitor to search for the ECID and expand down into JVMD to show the blocking thread.

When Request Processing Time is Beyond Threshold

• Examine the % Time spent in GC metric to see if JVM is doing excessive garbage collection

• Look for incident create rate and error logs and see if something is triggering a massive amount of logging/errors.

• In JVMD, look at the thread states and see where most processing time is going.

• Check the metric Garbage Collection - Invocation Time (ms) under the JVM Garbage Collectors metric category.

  Sometimes if many managed server instances are run on the same host, it is possible to reduce time spent in garbage collection by reducing the number of garbage collector threads in each JVM. The default is based on the number of CPUs and could be too high if there are multiple active JVMs running on the same machine. In those cases, if JRockit is being used, add the -XXgcThreads=4 option when starting the JVM. To add the option, edit the DOMAIN_HOME/bin/fusionapps_start_params.properties file, look for -Xgc:genpar and add the -XXgcThreads=4 option after it (for example -Xgc:genpar -XXgcThreads=4). The value 4 directs the JVM to use four threads to perform garbage collection. Try different values from 4 to the number of CPU cores and observe if the % Time spent in GC metric improves.

When Percent Time Spent in Garbage Collector is Beyond Threshold

• Check the session count. If there is a sudden surge of sessions due to user load, the JVM could be short on heap. Increase heap if possible, or add additional managed server instances.

• Look at the stuck threads count. Stuck threads could increase the number of active session, as users could be launching new sessions hoping for a faster response.

• Look at the incident creation rate and error logs and see if something is triggering a massive amount of logging/errors. The incident creation/logging operations could be causing a high amount of object creation and garbage collection stress.

• Generate a heap dump using JVMD and analyze the top retainer of memory.

• Use JRMC to connect and extract a JFR recording. Examine the Memory panel and allocation details to see what is doing a lot of allocations.

When Percent Connection Requests Waiting is Beyond Threshold

• Examine the number of sessions and request rate, and see if there is a spike in the load that would account for an increased demand for connections.

• In JVMD, see where time is spent. For example, requests could be running longer due to slow SQLs (and retain the connection longer). In that case, identify and tune slow SQLs.

• Consider increasing the initial capacity setting of the corresponding data source.

When Busy Threads % is Beyond Threshold

When Request Processing Time for a Virtual Host Exceeds Threshold

These metrics provide an indication of whether the Oracle Business Intelligence Server is in a healthy state.

To start monitoring:

Fusion Applications Control can monitor various BI components, including:

• Weblogic Analytics Application

• Oracle BI Presentation Services

• Oracle BI Server

• Oracle Weblogic Server (administration and managed servers)

Oracle Access Manager and Oracle Identity Manager are both WebLogic Server instances. See Analyze WebLogic Server Metrics to monitor their health. See also Tune and Troubleshoot Oracle Identity Management.

Use Cloud Control to monitor the Oracle Internet Directory and Oracle Identity Manager databases. See Table 14-4 for Oracle Identity Manager metrics.

See Table 14-5 for Oracle Internet Directory metrics.

To enable the collection of user LDAP Operation Statistics, edit the configuration using Fusion Applications Control:

The metrics shown in Table 14-6 provide an indication of whether the Enterprise Scheduler instance is performing well.

When the Value of Average Elapsed Time for the Completed Jobs is Higher than Expected

• Check the key host and WebLogic Server metrics and see if any component that could be involved in process batch jobs is in an unhealthy state.

• Follow the steps listed in section Using Java Workload Explorer in the Oracle Enterprise Manager Cloud Control Oracle Fusion Middleware Management Guide.

When the Value of Elapsed Time Under the Long Running Job Category is Higher than Expected

• Open the Enterprise Scheduler home page in Oracle Enterprise Manager Fusion Applications Control and examine the Top 10 Long Running Jobs.

• Follow the steps listed in section Using Java Workload Explorer in the Oracle Enterprise Manager Cloud Control Oracle Fusion Middleware Management Guide.

When Average Wait Time for Requests in Ready State (seconds) is Higher than Expected

Follow the steps listed in section Using Java Workload Explorer in the Oracle Enterprise Manager Cloud Control Oracle Fusion Middleware Management Guide.

Monitoring SOA involves monitoring SOA Infrastructure, SOA Composite and SOA servers.

Use Table 14-7 to locate the key performance metrics for SOA Composite.

Use Table 14-8 to locate the key performance metrics for SOA Infrastructure.