This chapter describes how to monitor and debug partitions in WebLogic Server Multitenant (MT). You can use either Fusion Middleware Control (FMWC) or WLST to monitor partitions, as described in this chapter. The chapter refers to the Fusion Middleware, Coherence, and Oracle Traffic Director documentation sets and online help for additional information as appropriate.
This chapter includes the following sections:
The WebLogic Diagnostics Framework (WLDF) provides the following partition scope diagnostic capabilities:
Logging
The logs for several WebLogic Server components, such as partition scope JMS, SAF, and servlet resources, are available in the partition file system directory. In a multitenant environment, partition users can configure partition scope loggers and control the levels of these loggers without affecting other partitions. See Configuring Partition Scope Logging.
Note:
The logs for server and domain scope resources, such as the server scope HTTP access log, the Harvester component, the Instrumentation component, and also the server and domain logs, are only available from the WLDF data accessor. For information, see Accessing Diagnostic Data.Debugging
In coordination with the domain system administrator, a partition user can set debug flags on WebLogic Server components so that debug messages can be emitted on behalf of work performed for that partition, without affecting other partitions. See Configuring Partition Scope Debugging.
Monitoring of partition resources
Diagnostic system modules can be configured in a resource group, or in a resource group template, to enable harvesting of partition-specific metrics and to configure policies and actions on partition scope resources. See Configuring Diagnostic System Modules.
WLDF also provides the ability to gather partition scope resource consumption metrics, which are surfaced as attributes on the PartitionResourceMetricsRuntimeMBean. See Monitoring Resource Consumption Management
Viewing log data
Partition administrators do not have file-level access to log files, but they can use the WLDF data accessor and supported WLST functions. See Accessing Diagnostic Data.
Instrumenting Application instrumentation
Applications deployed within a partition may be instrumented using WLDF instrumentation. See Instrumenting Partition Scope Applications.
Diagnostic image capture
Image capture can be initiated either manually by a partition scope user, or by a policy configured in a partition scope diagnostic system module. Only the content specific to the partition is included in the generated diagnostic image. See Configuring Partition Scope Diagnostic Image Capture.
WebLogic Server maintains the following log and diagnostics data files to track events and activity performed on behalf of partitions. Partition-specific log files are located in the partitions/<partition>/system/servers/<server-name> directory.
| Log File | Contents |
|---|---|
| ServerLog | Log records for Weblogic Server events generated when doing work within the scope of a partition. The server log is not available within the partition file system — it is available only through the WLDF data accessor. |
| DomainLog | Log records domain-scope events generated on behalf of a partition. The domain log is not available within the partition file system — it is available only through the WLDF data accessor. |
| HTTPAccessLog (partition scope virtual target access log) | The access.log file for each virtual target's web server is maintained in separately for each partition and can be accessed by the partition user. |
| HarvestedDataArchive | MBean metrics applicable to a partition are collected by the WLDF Harvester component. This file is available only through the WLDF data accessor. |
| EventsDataArchive | Events generated by the WLDF Instrumentation component. This file is available only through the WLDF data accessor. |
| JMSMessageLog (partition scope) | Log records for partition scope JMS resources. An individual log file is maintained for each partition and may be accessed by the partition user. |
| JMSSAFMessageLog (partition scope) | Log records for partition scope SAF agent resources. An individual log file is maintained for each partition and may be accessed by the partition user. |
| DataSourceLog | Data source profile records generated when doing work within the scope of a partition. |
| WebAppLog (partition scope) | Servlet context logs from partition scope applications. An individual log file is maintained for each partition and may be accessed by the partition user. |
| ConnectorLog (partition scope) | Java Connector Architecture resource adapter logs from partition scope resource adapters. An individual log file is maintained for each partition and may be accessed by the partition user. |
WebLogic Server MT allows configuration of levels for java.util.logging loggers that are used by applications deployed to a partition. Within a WebLogic domain, different partitions may contain copies of the same application. In such cases, different instances of the application may use java.util.logging loggers with same name. The application may have dependencies on libraries that use java.util.logging loggers and that are on the system classpath. In a multitenant environment, partition users can control the levels of these loggers within the scope of their partition without affecting other partitions.
The ability to configure java.util.logging logger levels for partition scope resources is enabled by the following:
A custom log manager, WLLogManager, which the domain system administrator must configure as the global log manager for the domain
The PartitionLogMBean.PartitionLoggerLevels attribute, which partition users can configure by specifying key-value pairs consisting of logger names and the corresponding java.util.logging.Level names
To enable the ability for partition users to configure partition scope logger levels:
The domain system administrator must configure WLLogManager for the domain by including the following weblogic.Server option in the WebLogic Server start command:
-Djava.util.logging.manager=weblogic.logging.WLLogManager
The partition user configures key-value pairs in the PartitionLogMBean.PartitionLoggerLevels attribute.
Example 27-1 shows using WLST to configure partition scope logging and debugging on partition p1.
Example 27-1 Configuring Partition Scope Logging and Debugging
startEdit()
cd('/')
p1 = cmo.createPartition('p1')
plog = p1.getPartitionLog()
plog.addEnabledServerDebugAttribute('DebugJNDI')
props = java.util.Properties()
props.put('foo.bar.logging','WARNING')
plog.setPlatformLoggerLevels(props)
save()
activate()
For information about debugging, see Configuring Partition Scope Debugging.
WebLogic Server MT supports debugging of partition scope resources by the following means:
The ServerDebugMBean includes the PartitionDebugLoggingEnabled attribute. This attribute controls whether partition scope debugging is enabled, and may be accessed only by the WebLogic domain administrator.
This attribute is disabled by default. When enabled, partition scope debugging is available for all partitions in the domain.
The PartitionLogMBean includes the EnabledServerDebugAttributes attribute, for which a partition user can define, as an array of Strings, any of the debug flags available for the ServerDebugMBean.
When debugging is enabled in the domain and configured in a partition, debug messages that are emitted by a system resource doing work on behalf of the partition are sent to that partition.
Note:
Oracle strongly recommends that when troubleshooting a server problem, the partition user and the WebLogic domain administrator should consult one another prior to enabling server-level debugging for a partition. Furthermore, partition scope debugging should be enabled only for a short period of time.A typical use case for partition scope debugging involves consultation with Oracle Support, who identifies the specific debug flags that the partition user should configure on the PartitionLogMBean.EnabledServerDebugAttributes attribute.
See Example 27-1 for an example WLST script that shows configuring partition scope debugging.
You can include WLDF diagnostic system modules in resource groups and resource group templates to allow partition specific monitoring. Note that not all functionality provided by domain scope diagnostic system modules is enabled when scoped to a partition. In particular, the server-level instrumentation, which affects the server as a whole, is not available at the partition level.
The following sections explain how to configure partition scope diagnostic system modules:
The WLDF Harvester component periodically samples JMX run-time MBean attributes consistent with its configuration in a diagnostic system module and stores the data in the Archive. This capability is available within the scope of a partition, but the diagnostic system module:
May access the run-time MBeans for resources within the same partition only. Partition users have read-only access to certain monitoring information from MBeans at the global scope. These MBean attributes can be harvested from a partition scope WLDF diagnostic system module.
May not access run-time MBeans for resources in other partitions
May configure the Harvester in a partition-neutral manner. In other words, the partition-id or partition-name need not be hard-wired in the configuration. This allows WLDF configurations to be portable regardless of the partitions to which they are assigned.
Partition scope data that is saved in the Archive is tagged with partition-specific identifiers so that the data is accessible to domain system administrators and users of that partition, but not to others.
For a list of the WLST commands that partition administrators can use for diagnostic system modules, see WLST Diagnostic Commands for Partition Administrators.
You can use the WLDF Policies and Actions component within a diagnostic system module to create policies that evaluate common JMX run-time MBean metrics, log records, and instrumentation events. This functionality is available at the partition level within the following constraints:
Policies may access only those MBean attributes that are visible to partitions.
Log policies are evaluated only for server and domain log records that are specific to the partition.
SNMP actions are not supported.
SMTP actions require that the mail session that is used must be visible to the partition in JNDI.
All policy expressions in partition scope diagnostic system modules must use Java Expression Language (EL). For more information, see "Policy Expression" in Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.
Scaling actions (scale up and scale down) are not supported within the scope of a partition.
As a partition administrator, you do not have file level access to all log files in the domain, particularly those that are server or domain scope. However, you can selectively access log content pertaining to your partition by using the WLDF data accessor and supported WLST functions.
The domain system administrator, who has full access to all log files in the domain, is responsible for configuring the appropriate security policies for log files to ensure the following:
You are authenticated when you attempt to access a log record.
You are able to access log records that are specific to your partition.
You are denied access to log records not pertaining to your partition.
You can access log records for your partition by using the following MBeans that are created under your partition's WLDFPartitionRuntimeMBean, which is a child of the PartitionRuntimeMBean:
| MBean | Description |
|---|---|
WLDFPartitionAccessRuntimeMBean |
Discovers log files that are available to your partition. Log files that are not available to your partition are not exposed. |
WLDFDataAccessRuntimeMBean |
Accesses log file records specific to your partition. Log records that do not pertain to your partition are not exposed. |
The following logs and diagnostic data are shared by all partitions in the domain, and the partition administrator can access records within these logs that pertain to their partition:
| Log Type | Content Description |
|---|---|
| Server | Log events from server and application components pertaining to the partition and that are recorded in the server log file. |
| Domain | Log events collected centrally from all server instances in the domain pertaining to the partition and that are recorded in the domain log file. |
| Data source | Data source log events pertaining to the partition and that are recorded in the data source log file. |
| Harvested data archive | Metric data gathered by the WLDF Harvester from MBeans pertaining to the partition and that are recorded in the harvested data archive. |
| Events data archive | WLDF Instrumentation events that are generated by applications deployed in the partition and that are recorded in the events data archive. |
When partitions are configured in the domain, each record created in these shared log files includes the following two attributes:
partition-name — a human-readable partition name
partition-id — a unique, automatically generated identifier of the partition
The harvested data archive and events data archive include two additional table columns, PARTITION_ID and PARTITION_NAME, in which these attributes are listed.
Note:
When partitions are configured in the domain, log records generated on behalf of server or domain scope work are assigned thepartition-id value 0, and the partition-name value DOMAIN.The following example shows a log record entry in a server log file. The partition-id and partition-name attributes are highlighted in bold.
####<Oct 2, 2015 1:20:28 PM EDT> <Notice> <Partition Lifecycle>
<tiger-mac.local> <partitionAdmin> <[ACTIVE] ExecuteThread: '2' for queue:
'weblogic.kernel.Default (self-tuning)'> <system> <>
<cfd4d584-ee45-49a6-ae91-9c12551330d8-000000af> <1443806428565> <[severity-value: 32] [rid: 0]
[partition-id: d75a4899-ca61-4ed8-b519-317b0ec15061] [partition-name: p1] > <BEA-2192303> <The partition lifecycle operation "START" for partition "p1" is initiated.>
When the data accessor obtains log records pertaining to a specific partition, it filters them based on the partition-id value.
Note:
The JFR flight recording, which includes events from the JVM and from any other event producer, such as WebLogic Server and Oracle Dynamic Monitoring System (DMS), can includepartition-id and partition-name attributes to distinguish data among partitions. However, only the domain system administrator may have access to the JFR data that contains the partition information that can be used to diagnose multi tenancy issues.The following log files are specific to each partition and are available to the partition administrator. These log files are stored in the partitions/<partition>/system/servers/<server-name> directory.
| Log Type | Content Description |
|---|---|
| HTPP access log | Log events of all HTTP transactions from the partition's virtual target web server, which are stored in the file access.log. |
| JMS server | JMS server message life cycle events for JMS server resources that are defined within a resource group, or resource group template, and that are scoped to the partition. |
| SAF agent | SAF agent message life cycle events for SAF agent resources that are defined within a resource group, or resource group template, and that are scoped to the partition. |
| Connector | Log data generated by Java Connector Architecture resource adapter modules that are deployed to a resource group, or resource group template, within the partition. |
| Servlet context | Servlet context log data generated by Java EE web application modules that are deployed to a resource group, or resource group template, within the partition. |
Using the WLDF data accessor, you can perform data lookups by type, component, and attribute. You can filter by severity, source, and content. You can also access diagnostic data in tabular form. For complete details about the data accessor, see "Accessing Diagnostic Data With the Data Accessor" in Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.
The following table lists the logical names for each of the log and diagnostic data files you can view using the data accessor. You use the logical name as a key to refer to a log type when using the WLDFDataAccessRuntimeMBean or WLST.
| Log Type | Logical Name |
|---|---|
| Server log |
ServerLog |
| Domain log |
DomainLog |
| JDBC log |
DataSourceLog |
| Harvested data archive |
HarvestedDataArchive |
| Events data archive |
EventsDataArchive |
| HTTP access log |
HTTPAccessLog/<WebServer-Name>
|
| JMS server |
JMSMessageLog/<JMSServer-Name>
|
| SAF agent |
JMSSAFMessageLog/<SAFAgent-Name>
|
| Servlet context |
WebAppLog/<WebServer-Name>/context-path |
| Connector |
ConnectorLog/connection-Factory-jndiName$partition-name |
As a partition administrator, you can use the following tools to access log and diagnostic data for your partition:
Fusion Middleware Control
You can use Fusion Middleware Control to view records from both shared and partition-specific log files. For information, see "Monitor domain partitions" in Administering Oracle WebLogic Server with Fusion Middleware Control.
WebLogic Scripting Tool
For a list of the WLST commands that partition administrators can use for using the data accessor, see WLST Diagnostic Commands for Partition Administrators. For the syntax and examples of these commands, see "Diagnostics Commands" in WLST Command Reference for WebLogic Server.
The WebLogic Server Administration Console does not support viewing partition log and diagnostic data.
Resource Consumption Management (RCM) provides a flexible, dynamic mechanism for WebLogic Server system administrators to manage shared resources and provide consistent performance of domain partitions in MT environments. WebLogic Server supports the ability to monitor resource consumption management (RCM) within a partition scope.
For more information about RCM, see Chapter 27, "Monitoring and Debugging Partitions".
For information about configuring RCM, see Configuring Resource Consumption Management: Main Steps.
Table 27-1 lists and summarizes the attributes on the PartitionResourceMetricsRuntimeMBean that can provide partition scope RCM metrics:
Table 27-1 PartitionResourceMetricsRuntimeMBean Attributes for RCM
| Attribute | Description |
|---|---|
RCMMetricsDataAvailable |
Boolean value indicating whether RCM metrics data is available for this partition. |
CpuTimeNanos |
Total CPU time spent in the context of a partition in the time since server start or partition creation, whichever is later. |
AllocatedMemory |
Total allocated memory in bytes in the context of a partition in the time since server or partition creation, whichever is later. |
RetainedHeapHistoricalData |
Snapshot of the historical data for retained heap memory usage for the partition. Data is returned as a two-dimensional array for the usage of retained heap scoped to the partition over time. Each item in the array contains a tuple of |
CpuUtilizationHistoricalData |
Snapshot of the historical data for CPU usage for the partition. CPU utilization percentage indicates the percentage of CPU utilized by a partition with respect to available CPU to Weblogic Server. Data is returned as a two-dimensional array for the CPU usage scoped to the partition over time. Each item in the array contains a tuple of |
ThreadCount |
Number of threads currently assigned to the partition. |
TotalOpenedSocketCount |
Total number of sockets opened in the context of a partition in the time since server start or partition creation, whichever is later. |
CurrentOpenSocketCount |
Number of sockets currently open in the context of a partition. |
NetworkBytesRead |
Total number of bytes read from sockets for a partition in the time since server start or partition creation, whichever is later. |
NetworkBytesWritten |
Total number of bytes written to sockets for a partition in the time since server start or partition creation, whichever is later. |
TotalOpenedFileCount |
Total number of files opened in the context of a partition in the time since server start or partition creation, whichever is later. |
CurrentOpenFileCount |
Number of files currently open in the context of a partition. |
FileBytesRead |
Total number of bytes read in the context of a partition in the time since server start or partition creation, whichever is later. |
FileBytesWritten |
Total number of bytes written in the context of a partition in the time since server start or partition creation, whichever is later. |
TotalOpenedFileDescriptorCount |
Total number of file descriptors opened in the context of a partition in the time since server start or partition creation, whichever is later. |
CurrentOpenFileDescriptorCount |
Number of file descriptors currently open in the context of a partition. |
For general RCM configuration requirements, see Configuring Resource Consumption Management.
The Instrumentation component of the WebLogic Diagnostics Framework (WLDF) provides a mechanism for adding diagnostic code to WebLogic Server instances and the applications running on them. For complete details about this feature, see "Configuring Instrumentation" in Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.
WebLogic Server MT supports the ability to instrument an application that is deployed within a partition by using the deployment descriptor (META-INF/weblogic-diagnostics.xml) or a deployment plan. This capability is supported with the following conditions:
The domain system administrator must configure the diagnostic system module at the system level and target it to the appropriate Managed Server instance at the global level (that is, not partition scoped), and its instrumentation component must be enabled.
The instrumented application must contain the META-INF/weblogic-diagnostics.xml deployment descriptor and its instrumentation component must be enabled.
The application must not be configured in such a way that it shares class loaders with application instances in other partitions.
Diagnostic image contains relevant data from various subsystems of WebLogic Server as it exists when a diagnostic image capture is initiated. You can initiate diagnostic image capture manually initiated with any of the following tools:
WebLogic Server Administration Console
Fusion Middleware Control
WLST
JMX
WLDF Policies and Actions component
When a partition user initiates a diagnostic image capture, the contents of the image is limited to the partition scope resources only. When a diagnostic image capture is initiated by invoking an operation on the WLDFPartitionImageRuntimeMBean that corresponds to a partition, the generated image contain data pertaining to that partition only. For partition scoped diagnostic images, the partition-id and partition-name is encoded into the file names.
Note that only the domain system administrator may have access to the JFR data containing the information corresponding to a partition.
When scoped to a partition, the following components can generate partition-specific records into the diagnostic image capture:
Connector
Instrumentation
JDBC
JNDI
JVM
Logging
RCM
Work Manager
JTA
For a list of the WLST commands that partition administrators can use for diagnostic image capture, see WLST Diagnostic Commands for Partition Administrators.
The following table lists the WLST diagnostics commands that are available to partition administrators for accessing partition scope logging and diagnostics data. These commands are presented in three categories:
Data accessor — commands for using the WLDF data accessor to access partition scope diagnostic data from various sources, including log records, data events, and harvested metrics
Diagnostic image capture — commands for capturing, saving, and accessing diagnostic image capture
Diagnostic modules — commands for exporting metric data collected by a diagnostic system module within a specific time interval
Each of these commands includes a partition argument for specifying the partition name.
| Command | Category | Summary |
|---|---|---|
exportDiagnosticDataFromServer |
Data accessor | Executes a query on the server side and retrieves the exported WLDF data. |
getAvailableDiagnosticDataAccessorNames |
Data accessor | Gets the logical names of diagnostic data accessor instances currently available on a server or partition, and returns them as an array of string values. |
captureAndSaveDiagnosticImage |
Diagnostic image | Captures a diagnostic image and download it to the client. |
getAvailableCapturedImages |
Diagnostic image | Returns, as an array of strings, a list of the previously captured diagnostic images that are stored in the image destination directory configured on the server. |
saveDiagnosticImageCaptureEntryFile |
Diagnostic image | Downloads a specific entry from the diagnostic image capture that is located on the server to which WLST is currently connected. |
saveDiagnosticImageCaptureFile |
Diagnostic image | Downloads the specified diagnostic image capture from the server to which WLST is currently connected. |
purgeCapturedImages |
Diagnostic image | Purges the diagnostic image files on the server as per the age criteria specified from the server's configured image destination directory. |
exportHarvestedTimeSeriesData |
Diagnostic module | Exports harvested metric data from the diagnostic archive for a particular server-scoped or partition-scoped diagnostic system module. |
For more information about the WLST commands available for diagnostics, see "Diagnostics Commands" in WLST Command Reference for WebLogic Server.