This chapter describes the auditing framework, performance profiling, and logging capabilities provided with the Oracle Data Service Integrator. It contains the following sections:
The auditing framework is used to collect auxiliary runtime data using a normal XQuery operation in an Oracle Data Service Integrator dataspace. This information may be used for security auditing, performance profiling, and other purposes.
Audit Data Structure
The data structure comprises a sequence of audit records containing an unordered collection of audit properties. Each audit record contains properties of a specific type, usually identified using a hierarchal name. Each audit record corresponds to an operation performed by Oracle Data Service Integrator. For example, access to a relational data source may generate a record of "evaluation/wrappers/relational" type that includes the following audit properties: sql, datasource, returnedRows, evaluationTime, parameters, message, and exception.
Any individual property may be configured to be collected. Each property has an individual intrinsic severity level that can be used to configure an overall threshold of what properties to collect. In certain cases, like when an exception occurs, some properties may be added to the record even if they are not configured to be collected. Typically, this information would be identifiers for a failed data source or update operation.
On the other hand, a property configured for collection may not be collected. This might be attributed to one of the following reasons:
Data might be unavailable due to internal implementation logic.
A property is collected by an audit based on the need to record internal conditions, for external analysis.
If an exception is encountered. This will result in an alternate execution path and impact the information being collected.
Collected elements of the data structure can be individually configured to be:
Submitted to the Oracle WebLogic Server auditing framework and processed by an auditing provider.
Written to an application server or system logging stream.
Transferred to a client application.
Note:
Auditing occurs whenever the engine is invoked and the Auditing option is enabled. Timestamps and other collected data enable you to match auditing information with particular query operations.
Use the System Administration category in Oracle Data Service Integrator Administration Console to configure audits such as setting the global audit severity level and overriding audit settings for individual properties that you may need to monitor.
Setting Global Audit Properties
There are some global auditing options that inherently apply to every aspect of the auditing process. To set these properties:
Acquire the lock.
Select the System Administration category and then the Audit tab shown in Figure 9-1, which allows you to configure these options.
By default, the audit report generation utility is turned off. Before you start generating audit reports, you need to enable auditing.
Note:
With auditing enabled, performance may be affected, depending on the audit levels and the number of properties being audited.
Figure 9-1 Audit Options
Table 9-1 describes available global auditing options. Select the respective check box in the Oracle Data Service Integrator Console to implement the required audit options.
Table 9-1 Oracle Data Service Integrator Global Auditing Options
Options
Description
Enable Auditing
Determines whether the auditing is activated or not.
Note:
When auditing is enabled, performance can be affected to a degree, depending on the audit level and the number of items being tracked.
Audit Queries
Determines whether the auditing is activated or not, during a query evaluation.
Audit Administrative Actions
Collects audit data during dataspace deployment and configuration.
Audit Updates
Determines whether auditing is activated or not during update operations.
Severity Level
Determines the level of information to be captured by the auditing process. See Auditing Severity Levels section for more information.
Send Audit Events Asynchronously
Determines whether the events are processed synchronously or asynchronously.
Enable Logging of Audit Events
Determines whether the auditing information is to be included in the application server log file.
Note:
If you enable this option (logging), ensure that the Log Level value in the General tab is set to either Info or Debug. Any other value will result in the log file not accepting any information.
Incremental Audit Dispatch
Determines if the audit records are to be dispatched partially or completely every time there is a new record.
Auditing Severity Levels
You can set the severity levels using the Severity Level drop down list in the Audit tab (Figure 9-1). Severity levels are similar to those provided with Oracle WebLogic Server security. For WebLogic Server details, see Message Severity.
Table 9-2 Oracle Data Service Integrator Audit Severity Levels
Level
Description
Debug
This setting is often referred to as “verbose”. Any audit property that can be added to the audit report is collected.
Information
Properties with information or higher conditions are collected for the audit report.
Warning
Properties with warning or higher conditions are collected for the audit report.
Failure
Properties with error or more higher conditions are collected for the audit report.
Setting Individual Auditing Properties
This section describes the individual auditing properties that you can audit and to what level. To configure these auditing properties:
Acquire the lock.
Select the System Administration category
Click the Audit Properties tab as shown in Figure 9-2.
Figure 9-2 Audit Properties Tab
After configuring audit properties, click Save > Activate Changes to implement the audit settings.
Note:
Click Default Settings to rearrange the auditing properties to the default values.
Audit properties can be configured at different levels and you can select the level using the Is Audited drop-down list. Table 9-3 lists the audit levels that you can set for each property. All levels listed in the table are not applicable to all the properties. Typically, each property has only three levels to choose from.
Note:
If you want the property-specific audit information to be returned to a client, then select the Available to Client check box.
Table 9-3 Setting Individual Audit Properties
Level
Description
Always
In this setting, the audit information of the property is always collected.
Never
In this setting, the audit information of the property is always ignored.
At Default Level
This setting configures the property at the default level.
Note:
This option is available only for the Select All Properties audit property.
At Info Level
In this setting, the audit information is collected if the global threshold level is Information or lower.
At Warning Level
In this setting, the audit information is collected if the global threshold level is Warning or lower.
At Failure Level
In this setting, the audit information is collected if the global threshold level is Failure or lower.
At Debug Level
In this setting, the audit information is collected if the global threshold level is Debug.
Note:
After you set and apply individual auditing property settings, any changes you make on the individual properties will override the initial settings for that property only.
Oracle Data Service Integrator Administration Console provides you with the option to select all audit properties to be audited using the Select All Properties node. You can set this property at the following levels:
Always
At Default Level
Never
All other individual properties are categorized into the following overall types depending on the corresponding operation that generates the audit data:
The audit information in this section pertains to the information exchanged while performing administration tasks such as configuration and application deployment. Only changes to the application made in the Oracle Data Service Integrator Administration Console are collected during audit.
Table 9-4 Administrator Properties
Property
Description
Configuration
notification
Records notification of deployed access control resource. For example:
Records any instance of the property that was changed in the Oracle Data Service Integrator Console. For example:
notification: jmx.attribute.change
value
Records a new value instance, for example:
value: 101
Dataspace
Note:
This information is displayed in the audit log by default. You cannot change the audit level for this property.
name
Records the name of the dataspace
operation
Records create, modify, delete operations for a dataspace
updatediff
Records changes from the last configuration update.
Common Audit Properties
The common audit information provides the generic transaction related information. It includes generic information on the event, such as event type, application name, user id, user access rights, date, and time.
Table 9-5 Common Properties
Property
Description
Application
eventkind
Records the type of event or operation, it could be a query or an update and so on. For example:
eventkind: evaluation
exception
Records the exception message, if one occurred. For example:
exception: ld:DataServices/ApparelDB/CUSTOMER_ORDER_LINE_ITEM.ds, line 77, column 7: {err}FORG0005: expected exactly one item, got 0 items
name
Records the deployed application name. For example:
name: RTLApp
principals
Records the groups to which the user belongs. For example:
Records the type of resource used, such as dataservice, application, submit and so on. For example:
resourcetype: function
Session
query
invocation
blocksize
Records the size of the returned serialized data block, in bytes
duration
Records the duration or the time required to compute the next block of the result, in milliseconds.
time
Records the time of call for the next data block.
Session
SQL
Invocation
time
Records the date and time of the call to the next () method on the server side of the JDBC driver.
duration
Records the duration or the time required to compute the next block of the result, in milliseconds.
blocksize
Records the size of the returned serialized data block, in bytes.
Time
duration
Records the time used to complete the audit event, in milliseconds. Calculates the time difference from initiation of the audit to its completion. For example:
duration: 2834
timestamp
Records the time when the audit event was initiated, for example:
timestamp: Tue Feb 14 09:21:02 IST 2006
Query Audit Properties
The audit information in this section pertains to all the information collected during query evaluation. The information includes the query itself, its result, the execution time, and details on the data source queried.
Note:
When using the streaming APIs, or when using the RequestConfig.OUTPUT_FILENAME feature, the results of the query are not audited because they are presumed to be very large. This means the AuditEvent dispatched to the audit provider, as well as the DataServiceAudit returned to the client, will not contain a value for the audit property Query/Service/results.
Table 9-6 Query Properties
Property
Description
Adhoc
query
Records the query that was executed.
result
Records the results obtained after execution of the query.
variablenames
Records names of the variables passed to the query.
variables
Records the external parameters or variables passed to the query.
Cache
Data
forcedrefresh
Boolean value where TRUE indicates the data is from a current data source or FALSE if it is from a cache.
functionid
Records the name of the function.
remainttl
Indicates the time remaining, in seconds, before the query cache is refreshed.
retrieved
Indicates whether the data was obtained from the query cache or not.
time
Indicates the duration of the cache retrieval operation.
Queryplan
Note:
Queryplan audit properties are not collected when a function is executed from the Test view. This is because the function cache is not utilized for functions executed in the Test view.
flushed
True and set when the query plan was flushed
found
Indicates whether the query plan cache has been located or not.
inserted
Indicates whether the query plan cache has been inserted or not.
type
Indicates the type of the query plan such as XQUERY_PLAN_CACHE, SQL_PLAN_CACHE, or STORED_PROC_CACHE.
Failover
exception
In the event of a failover, this records the exception that caused it.
function
Records the function name which can be either fn:bea:timeout or fn:bea:fail-over. For example:
Records the user-defined label, if any. For example:
label: lab
sourcecolumn
Records the source column of the function call. For example:
sourcecolumn: 2
sourcefile
Records the source file of the function call. For example:
sourcefile: [ad-hoc]
sourceline
Records the source line of the function call. For example:
sourceline: 4
timeout
Records the time-out that was exceeded, if applicable. For example:
timeout: 0
Function
Note:
Function audit properties are collected only when the individual functions of a data service are selected for auditing. See Function-level Auditing for more information.
name
Records the name of the audited function. For example:
Records the parameters passed through the audited function. For example:
parameters: CUSTOMER1
result
Records the result after executing the audited function. For example:
result: <ns0:CUSTOMER
Performance
compiletime
Records the query compilation time, in milliseconds. For example:
compiletime: 19
Note:
The query/performance/compiletime audit property does not include compilation time of any XQuery function or XQuery inline declaration. Rather it simple reports the cost of XQSEStatement.prepare.
evaltime
Records the query evaluation time, in milliseconds. For example:
evaltime: 90
Service
arity
Records the number of arguments for the invoked function.
dataservice
Records the name of the data service, for example:
Records the results of the executed query, for example:
ORDER_10_0 CUSTOMER0 2001-10-01 GROUND
SQL
Procedure
name
Records the name of the SQL procedure.
parameters
Records the parameters associated with the SQL procedure.
parametertypes
Records the types of the parameters.
Statement
parameters
Records the parameters of the query.
parametertypes
Records the parameter types of the query.
query
Records the text of the query.
Wrappers
File
exception
Records an exception, if any, when a function invoked belongs to a data service created over a File data source. For example:
exception: com.bea.ld.wrappers.df.exceptions.DFException: {bea-err}DF0004: [ld:DataServices/Demo/Valuation.csv]: Expected end of line at (row:2, column:3).
name
Records the unique function name. For example:
name: ld:DataServices/Demo/Valuation.csv
time
Records the time taken to query, in milliseconds. For example:
time: 20000
Java
exception
Records an exception, if any, when a function invoked belongs to a data service created over a Java class. For example:
exception: {ld:DataServices/Demo/Java/Physical/PRODUCTS}getFirstProduct:0, line 4, column 5: {bea-err}JFW0401: Class or Method not found exception : {ld:DataServices/Demo/Java/Physical/PRODUCTS}getFirstProduct
name
Records the name of the service. It is always recorded if an exception property was added. For example:
name: public static int Demo.Java.JavaSource4West.echoInt(int)
parameters
Records the external parameters passed to the service. For example:
parameters: 11
result
Records the results of the executed query. For example:
result: 11
time
Records the time taken to execute the query, in milliseconds. For example:
time: 20000
Procedure
datasource
Records the name of the data source, for example:
datasource: newDS
exception
Records an exception, if any, when a function invoked belongs to a data service created over a stored procedure. For example:
exception: weblogic.xml.query.exceptions.XQueryDynException: {err}XP0021: "-ss": can not cast to {http://www.w3.org/2001/XMLSchema}decimal}
name
Records the procedure identifier. It is always recorded if an exception property was added. For example:
name: WIRELESS.SIDEEFFECT_REG_PACKAGE.READ2
parameters
Records the external parameters passed to the data service method. For example:
parameters: s 2.2 22.0 ss
rows
Records the number of rows returned after execution of the procedure, for example:
rows: 0
time
Records the time taken to execute the procedure, in milliseconds. For example:
time: 170
Relational
basesql
Records the base SQL statement text.
exception
Records the relational database query exception, if any. For example:
The audit information in this section pertains to all the information related to performing an update function. It includes information on the time taken to update the source, when it was started, the unique transaction id and so on.
Table 9-7 Update Properties
Property
Description
Error
Fault
exception
Records the value of the tostring() of the update exception.
exceptionobject
Records the exception object for dataspace audit update error.
status
Records the status of the update.
updateid
Records the globally-unique update identifier.
Error
Procedure
arity
The arity of the update procedure.
dataservice
The data service of the update procedure.
id
The index of the update procedure invocation.
name
The name of the update procedure.
parameters
The parameters of the update procedure invocation.
result
The result of the update procedure invocation.
status
Status of the procedure executed by this update.
xid
The xid of the update procedure invocation.
Extension
id
Records the id of the source being updated.
time
Records the time spent, in milliseconds, for the update.
Procedure
name
Records the name of the audit procedure.
parameters
Records the parameters passed to the audited procedure.
result
Records the results of update procedure execution.
Relational
exception
Records the update exception, if any.
parameters
Records the parameters passed during the update of the relational database.
rowsmodified
Records the number of rows updated in the relational database, on successful completion.
source
Records the data source name. It is always recorded if an exception property was added.
sql
Records the SQL statement used during the update of the relational database.
time
Records the time spend, in milliseconds, in updating the relational database.
Service
arity
Records the number of arguments associated with the invoked function.
dataservice
Records the data service used for the update.
parameters
Records the parameters passed to the update procedure.
procedure
Records the data service fully qualified procedure name.
result
Records the results of the update.
script
Records the complete text of the executed script.
sdocount
Records the number of top level SDOs that were submitted for the update.
time
Records the total execution time, in milliseconds, for the update.
Function-level Auditing
By default, auditing for all directly invoked functions can be enabled through the /query/service record for the dataspace using the Audit tab. However, to limit auditing to specific functions, set all properties of the /query/service record to NEVER and then enable audit for individual functions. To do so:
Acquire the lock and select the System Administration category.
Figure 9-3 Enabling Auditing for Individual Functions
If auditing for a function is enabled, all external calls to this function are audited. If the Enable Audit of Indirect Calls check box is selected, all calls originating from other data services are also audited.
Note:
Enabling audit of indirect calls may disable query optimization for that function, and decrease performance.
Retrieving Audit Information
You can record the audit information collected in the following ways.
Values of the audit properties are represented as Java objects of types: String, Integer, java.util.Date, Boolean, or String [ ].
WebLogic Server Security Framework
Each audit event is sent to the WebLogic Server Security Framework as an instance of the weblogic.security.spi.AuditEvent interface. Table 9-8 describes each event.
Table 9-8 WebLogic Server Audit Events
Event
Description
getEventType()
Returns the event type, in this case DSPaudit.
getFailureException()
Returns the exception type, if one is encountered.
getSeverity()
Returns the event severity level.
toString()
Returns the audit event details in an XML formatted representation.
Depending on the configuration, each event can be sent to the WebLogic Server audit API asynchronously and buffered by the Oracle Data Service Integrator application.
The weblogic.security.spi.AuditEvent interface is implemented in the ld.server.audit.DSPAuditEvent interface, which collects all the information in the form of a list, where each entry is an instance of com.bea.dsp.DSPAuditEvent.
DSPAuditEvent adds the interface described in Table 9-9.
Table 9-9 Oracle Data Service Integrator AuditEvent API
AuditEvent API
Description
getAllRecords()
Returns all records as a list of com.bea.ld.DSPAuditRecord.
getRecords(String recordType)
Returns all records of a particular type as a list of com.bea.ld.DSPAuditRecord.
getProperty(String propertyId)
Returns all values for a particular property, across multiple records.
getApplication()
Returns the Oracle Data Service Integrator application identifier.
getUser()
Returns the user name of the application server user.
getTimeStamp()
Returns the time when the event was created.
getEventKind()
Returns the event type, which can be EVALUATION_EVENT, CONFIGURATION_EVENT or UPDATE_EVENT.
getVersion()
Returns the event version, for example 10.3 for the Oracle Data Service Integrator 10gR3 release.
com.bea.ld.DSPAuditRecord has the interface shown in Table 9-10.
Table 9-10 Oracle Data Service Integrator AuditRecord API
AuditRecord API
Description
getRecordType()
Returns the type of record, for example common/time/duration.
getAuditProperties()
Returns all properties in the record. Maps from String identifier to Object value.
A sample security services audit provider is included that demonstrates use of this API.
Oracle Data Service Integrator Client API
You can use the com.bea.ld.DataServiceAudit client side instance as part of the com.bea.dsp.RequestConfig class, to collect the audit information from the client API. This class collects the audit information and returns it when the operation is successful. If the operation fails for any reason, the com.bea.ld.QueryException class can be used to collect the information as part of the exception thrown.
Note:
When using Streaming APIs, auditing will not be complete until the returned XMLInputStream has its close( ) method called. This means that the AuditEvent will not be dispatched to the audit provider by the server, and the RequestConfig.getDataServiceAudit() method will return null, until close( ) is called.
Following are the four steps, with code examples, that need to be performed in order to retrieve audit information.
Initializing the RequestConfig Class
You need to initialize the RequestConfig class as shown in the following code example:
RequestConfig requestCfg = new RequestConfig(); requestCfg.enableFeature(RequestConfig.RETURN_DATA_SERVICE_AUDIT); requestCfg.enableFeature(RequestConfig.RETURN_AUDIT_PROPERTIES); requestCfg.setStringArrayAttribute(RequestConfig.RETURN_AUDIT_PROPERTIES, new String[] {"query/service/dataservice"});
Passing the RequestConfig Object
You need to pass the RequestConfig object to the invoked operation. The code example below uses getCustomer as the invoked operation.
You need to filter the data and ensure there is no unsecured access to it. Only those audit properties that are configured in the Oracle Data Service Integrator Administration Console to be allowed to return to the client, will be returned to the client application.
Retrieving Data Service Audit
You need to retrieve the data service audit from the RequestConfig object, as shown in the code example below:
RequestConfig.RETURN_AUDIT_PROPERTIES is an array of string identifiers for audit properties. If you set this request attribute those specified properties will be collected for this particular evaluation even if they are not configured to be collected through the administration console. They will be returned only if it is allowed. If the RETURN_DATA_SERVICE_AUDIT request attribute is not enabled, only those properties will be returned.
RequestConfig.RETURN_DATA_SERVICE_AUDIT configures all collected audit information (that is allowed to be returned to the client application) to be returned.
Oracle Data Service Integrator Performance Profiling
Performance profiling allows you to store select audit information in a relational database. Relational databases supported by the Oracle Data Service Integrator audit provider are: Oracle, DB2, PointBase, Sybase, and MS SQL.
Information about audit events are stored as records in a table. A table can be used to record audit events for Oracle Data Service Integrator dataspaces running on a server, or for dataspaces running on shared servers in a cluster.
You can deploy the Oracle Data Service Integrator audit provider for performance profiling using the WebLogic Administration Console and configure it using the Oracle Data Service Integrator Profiler MBean. Configuration parameters you need to set at the time of deployment are described in Table 9-11.
Table 9-11 Configuration Parameters for Performance Profiling
Parameter
Description
Data Source
Name of the JDBC data source.
Table
Name of the table in the JDBC data source that logs query execution information.
Source Table
Name of the table in the JDBC data source that logs source access information.
Summary Table
Name of the table in the JDBC data source that logs aggregated information (summary).
Event Buffer
Size of the internal event buffer. Determines the number of events a buffer stores before the profiler starts processing events.
Collect Execution Aggregate
Stores aggregates (by function) of individual query executions in memory; eventually writes the aggregate to the database.
Aggregate Group Size
Number of events processed by the profiler before the aggregates are written to the database. Default value is 10.
Collect Execution Detail
Writes a row to the database for every query execution, including aggregate of source access within the query. Useful in application development environment.
Collect Source Detail
Writes a row to the database for every source access in a query. Collect Execution Detail needs to be configured for this parameter to take effect.
Creating a Performance Profiler
This section lists the steps needed to create a performance profiler.
Create a table to store the following audit properties:
common/time/timestamp
query/service/function
query/performance/evaltime
common/application/user
common/application/name
common/application/server
In addition to the above mentioned properties, you will also need to store:
information about the audit event exception, if any.
audit event severity level, which can be of types I (Information), W (Warning), S (Success), E (Error), F (Failure).
Modify the CLASSPATH to include a pointer to the JAR file.
Start WebLogic Server.
In the Audit page, configure the database tables as required.
Run the data service application and use the applicable database visualizer to view the results.
Using the Sample Performance Profiler
An Oracle Data Service Integrator audit provider sample file profiler.zip, is available in the Oracle Data Service Integrator root installation directory. The zip file contains the following files:
README.txt lists steps to use the sample audit provider).
Notice. Possible errors that do not affect runtime operation, as well as error level events.
Information. Start/stop events, unsuccessful access attempts, query execute times, and so on, as well as error and notice level events.
Debug logging occurs by default for any server in development mode. Client applications can contribute to the server log through the WebLogic Logger facility. For more information, see Using WebLogic Logging Services.
Query strings are echoed in the server log as a debug-level log message when the log level is set to Information in the Oracle Data Service Integrator Console and the WebLogic Administration Console is set to log debug messages to stdout.
Monitoring a WebLogic Domain
You can use the WebLogic Server Administration Console to monitor the health and performance of the domain in which WebLogic is deployed, including resources such as servers, JDBC connection pools, JCA, HTTP, the JTA subsystem, JNDI, and Enterprise Java Beans (EJB).
The domain log is located in the following directory:
You can use performance monitoring tools, such as the OptimizeIt and JProbe profilers, to identify Oracle Data Service Integrator application “hot spots” that result in either high CPU utilization or high contention for shared resources.
