19 Monitoring Integration Processes
This chapter includes the following sections:
Introduction to Monitoring
Monitoring your development executions consists of viewing the execution results and managing the development executions when the executions are successful or in error. This section provides an introduction to the monitoring features in Oracle Data Integrator. How to work with your execution results is covered in Monitoring Executions Results. How to manage your development executions is covered in Managing your Executions.
Introduction to Operator Navigator
Through Operator Navigator, you can view your execution results and manage your development executions in the sessions, as well as the scenarios and Load Plans in production.
Operator Navigator stores this information in a work repository, while using the topology defined in the master repository.
Operator Navigator displays the objects available to the current user in six navigation trees:
-
Session List displays all sessions organized per date, physical agent, status, keywords, and so forth
-
Hierarchical Sessions displays the execution sessions also organized in a hierarchy with their child sessions
-
Load Plan Executions displays the Load Plan Runs of the Load Plan instances
-
Scheduling displays the list of logical agents and schedules
-
Load Plans and Scenarios displays the list of scenarios and Load Plans available
-
Solutions displays the list of solutions
The Operator Navigator Toolbar Menu
You can perform the main monitoring tasks via the Operator Navigator Toolbar menu. The Operator Navigator toolbar menu provides access to the features detailed in Table 19-1.
Table 19-1 Operator Navigator Toolbar Menu Items
Icon | Menu Item | Description |
---|---|---|
|
Refresh |
Click Refresh to refresh the trees in the Operator Navigator. |
|
Filter Filter activated |
Click Filter to define the filters for the sessions to display in Operator Navigator. |
|
Auto Refresh |
Click Auto Refresh to refresh automatically the trees in the Operator Navigator. |
|
Connect Navigator |
Click Connect Navigator to access the Operator Navigator toolbar menu. Through the Operator Navigator toolbar menu you can:
|
Scenarios
A scenario is designed to put a source component (mapping, package, procedure, variable) into production. A scenario results from the generation of code (SQL, shell, etc.) for this component.
When a scenario is executed, it creates a Session.
Scenarios are imported into production environment and can be organized into Load Plan and Scenario folders. See Managing Scenarios and Load Plansfor more details.
Sessions
In Oracle Data Integrator, an execution results in a Session. Sessions are viewed and managed in Operator Navigator.
A session is an execution (of a scenario, a mapping, a package or a procedure, and so forth) undertaken by an execution agent. A session is made up of steps which are themselves made up of tasks.
A step is the unit of execution found between a task and a session. It corresponds to a step in a package or in a scenario. When executing a mapping or a single variable, for example, the resulting session has only one step.
Two special steps called Command On Connect and Command On Disconnect are created if you have set up On Connect and Disconnect commands on data servers used in the session. See Setting Up On Connect/Disconnect Commands for more information.
The task is the smallest execution unit. It corresponds to a command in a KM, a procedure, and so forth.
Sessions can be grouped into Session folders. Session folders automatically group sessions that were launched with certain keywords. Refer to Organizing the Log with Session Folders for more information.
Note that certain privileges are required for you to stop a session or clean stale sessions. These privileges are listed below:
You can stop the session if you have one of the following Sessions privileges:
-
You are a Supervisor user.
-
You have started the session and you are the owner of the session.
-
You have been explicitly assigned the Stop Immediate or the Stop Normal privileges. Note that you can view these method objects in the Security Navigator by expanding the Objects tree and expanding the Session Object node of a given profile.
You can clean stale sessions if you have one of the following Sessions privileges:
-
You are a Supervisor user.
-
You have been explicitly assigned the Clean Stale Sessions privileges. Note that you can view this method object in the Security Navigator by expanding the Objects tree and expanding the Agent Object node of a given profile.
You can also stop a session or clean stale sessions if the OPERATOR or NG OPERATOR profile is assigned to you.
Load Plans
A Load Plan is the most course-grained type of executable object in Oracle Data Integrator, used to organize and run finer-grained objects. It uses Scenarios in its steps. A Load Plan is an organized hierarchy of child steps. This hierarchy allows conditional processing of steps in parallel or in series.
Load Plans are imported into production environments and can be organized into Load Plan and Scenario folders. See Managing Scenarios and Load Plansfor more details.
Load Plan Executions
Executing a Load Plan creates a Load Plan instance and the first Load Plan run for the instance. This Load Plan instance is separated from the original Load Plan and can be modified independently. Every time a Load Plan instance is restarted, a Load Plan run is created for this Load Plan instance. A Load Plan run corresponds to an attempt to execute the instance. See the Load Plan Execution Lifecycle section in Developing Integration Projects with Oracle Data Integrator for more information.
When running, a Load Plan Run starts sessions corresponding to the scenarios sequenced in the Load Plan.
Note that in the list of Load Plan executions, only the Load Plan runs appear. Each run is identified by a Load Plan Instance ID and an Attempt (or Run) Number.
Schedules
You can schedule the executions of your scenarios and Load Plans using Oracle Data Integrator's built-in scheduler or an external scheduler. Both methods are detailed in Scheduling Scenarios and Load Plans.
The schedules appear in Designer and Operator Navigator under the Scheduling node of the scenario or Load Plan. Each schedule allows a start date and a repetition cycle to be specified.
Log
The Oracle Data Integrator log corresponds to all the Sessions and Load Plan instances/runs stored in a repository. This log can be exported, purged or filtered for monitoring. See Managing the Logfor more information.
Status
A session, step, task or Load Plan run always has a status. Table 19-2 lists the six possible status values:
Table 19-2 Status Values
Status Name | Status Icon for Sessions | Status Icon for Load Plans | Status Description |
---|---|---|---|
Done |
The Load Plan, session, step or task was executed successfully. |
||
Done in previous run |
The Load Plan step has been executed in a previous Load Plan run. This icon is displayed after a restart. |
||
Error |
The Load Plan, session, step or task has terminated due to an error. |
||
Running |
The Load Plan, session, step or task is being executed. |
||
Waiting |
The Load Plan, session, step or task is waiting to be executed. |
||
Warning (Sessions and tasks only) |
|
|
|
Queued (Sessions only) |
The session is waiting for an agent to be available for its execution |
When finished, a session takes the status of the last executed step (Done or Error). When finished, the step, takes the status of the last executed task (Except if the task returned a Warning. In this case, the step takes the status Done).
A Load Plan is successful (status Done) when all its child steps have been executed successfully. It is in Error status when at least one of its child steps is in error and has raised its exception to the root step.
Task Types
Tasks are the nodes inside of steps in a session. The types of tasks are shown in Table 19-3.
Table 19-3 Task Types
Task Type | Task Icon for Sessions | Task Description |
---|---|---|
Normal Task |
This task is executed according to its sequential position in the session. It is marked to DONE status when completed successfully. If it completes in error status, it is failed and marked with the error status. |
|
Serial Task |
The children tasks are executed in a sequential order. The serial task is completed and marked to DONE status when all its children tasks have completed successfully. It is considered failed and marked with error status when the first child task completes in error status. |
|
Parallel Task |
The children tasks are executed concurrently. The parallel task is considered completed and marked with DONE status when all its children tasks have completed successfully. It is considered failed and marked with ERROR status if any of its child tasks has failed. |
Monitoring Executions Results
In Oracle Data Integrator, an execution results in a session or in a Load Plan run if a Load Plan is executed. A session is made up of steps which are made up of tasks. Sessions are viewed and managed in Operator Navigator.
Load Plan runs appear in the Operator Navigator. To review the steps of a Load Plan run, you open the editor for this run. The sessions attached to a Load Plan appear with the rest of the sessions in the Operator Navigator.
Monitoring Sessions
To monitor your sessions:
- In the Operator Navigator, expand the Session List navigation tree.
- Expand the All Executions node and click Refresh in the Navigator toolbar.
- Optionally, activate a Filter to reduce the number of visible sessions. For more information, see Filtering Sessions.
- Review in the list of sessions the status of your session(s).
Handling Failed Sessions
When your session ends in error or with a warning, you can analyze the error in Operator Navigator.
To analyze an error:
You can fix the code of the command in the Code tab and apply your changes. Restarting a session (see Restarting a Session) is possible after performing this action. The session will restart from the task in error.
Note:
Fixing the code in the session's task does not fix the source object that was executed (mapping, procedure, package or scenario). This source object must be fixed in Designer Navigator and the scenario (if any) must be regenerated. Modifying the code within the session is useful for debugging issues.
WARNING:
When a session fails, all connections and transactions to the source and target systems are rolled back. As a consequence, uncommitted statements on transactions are not applied.
Reviewing Successful Sessions
When your session ends successfully, you can view the changes performed in Operator Navigator. These changes include record statistics such as the number of inserts, updates, deletes, errors, and the total number of rows as well as execution statistics indicating start and end time of the execution, the duration in seconds, the return code, and the message (if any).
Session level statistics aggregate the statistics of all the steps of this session, and each step's statistics aggregate the statistics of all the tasks within this step.
To review the execution statistics:
-
In the Operator Navigator, identify the session, the step, or the task to review.
-
Double click the session, the step, or the task. The corresponding editor opens.
-
The record and execution statistics are displayed on the Definition tab.
Record Statistics
Properties | Description |
---|---|
No. of Inserts |
Number of rows inserted during the session/step/task. |
No. of Updates |
Number of rows updated during the session/step/task. |
No. of Deletes |
Number of rows deleted during the session/step/task. |
No. of Errors |
Number of rows in error in the session/step/task. |
No. of Rows |
Total number of rows handled during this session/step/task. |
Execution Statistics
Properties | Description |
---|---|
Start |
Start date and time of execution of the session/step/task. |
End |
End date and time of execution of the session/step/task. |
Duration (seconds) |
The time taken for execution of the session/step/task. |
Return code |
Return code for the session/step/task. |
Execution Details |
The metrics about the performance of the loading task and any configuration data that have an impact on performance. |
For session steps in which a mapping has been executed or a datastore check has been performed, the target table details are displayed. Also, for session steps in which a mapping has been executed, the Knowledge Modules used in the mapping are displayed in the Knowledge Module Details section of the Session Step Editor.
If tracking is enabled for a session, information regarding the variable or sequence is displayed in the Variable and Sequence Values section of the Session Step Editor and Session Task Editor.
You can view the options selected to limit concurrent executions of a session in the Concurrent Execution Controller section of the Session Editor. See the Controlling Concurrent Execution of Scenarios and Load Plans section in Developing Integration Projects with Oracle Data Integrator for more information.
Handling Failed Load Plans
When a Load Plan ends in error, review the sessions that have failed and caused the Load Plan to fail. Fix the source of the session failure.
You can restart the Load Plan instance. See Restarting a Load Plan Runfor more information.
Note that it will restart depending on the Restart Type defined on its steps. See the Handling Load Plan Exceptions and Restartability section in Developing Integration Projects with Oracle Data Integrator for more information.
You can also change the execution status of a failed Load Plan step from Error to Done on the Steps tab of the Load Plan run Editor to ignore this particular Load Plan step the next time the Load Pan run is restarted. This might be useful, for example, when the error causing this Load Plan step to fail is not possible to fix at the moment and you want to execute the rest of the Load Plan regardless of this Load Plan step.
Reviewing Successful Load Plans
When your Load Plan ends successfully, you can review the execution statistics from the Load Plan run editor.
You can also review the statistics for each session started for this Load Plan in the Session Editor.
To review the Load Plan run execution statistics:
- In the Operator Navigator, identify the Load Plan run to review.
- Double click the Load Plan run. The corresponding editor opens.
- The record and execution statistics are displayed on the Steps tab.
Managing your Executions
Managing your development executions takes place in Operator Navigator. You can manage your executions during the execution process itself or once the execution has finished depending on the action that you wish to perform. The actions that you can perform are:
Managing Sessions
Managing sessions involves the following tasks
-
New sessions can be created by executing run-time objects or scenarios. See Running Integration Processesfor more information on starting sessions.
-
Sessions in progress can be aborted. How to stop sessions is covered in Stopping a Session.
-
Sessions failed, or stopped by user action can be restarted. Restarting sessions is covered in Restarting a Session.
In addition to these tasks, it may be necessary in production to deal with stale sessions.
Managing Load Plan Executions
Managing Load Plan Executions involves the following tasks:
-
New Load Plan Instances and Runs can be created by executing Load Plans. See Executing a Load Planfor more information on starting Load Plans.
-
Load Plan Runs in progress can be aborted. How to stop Load Plan runs is covered in Stopping a Load Plan Run.
-
Load Plan Runs failed, or stopped by user action can be restarted. Restarting Load Plan Runs is covered in Restarting a Load Plan Run.
Managing the Log
Oracle Data Integrator provides several solutions for managing your log data:
-
Filtering Sessions to display only certain execution sessions in Operator Navigator
-
Purging the Log to remove the information of past sessions
-
Exporting and Importing Log Data for archiving purposes
-
Runtime Logging for ODI components (ODI Studio, ODI Java EE agent, ODI Standalone agent, and ODI Standalone Colocated agent)
Filtering Sessions
Filtering log sessions allows you to display only certain sessions in Operator Navigator, by filtering on parameters such as the user, status or duration of sessions. Sessions that do not meet the current filter are hidden from view, but they are not removed from the log.
To filter out sessions:
Sessions that do not match these criteria are hidden in the Session List navigation tree. The Filter button on the toolbar is activated.
To deactivate the filter click Filter in the Operator toolbar menu. The current filter is deactivated, and all sessions appear in the list.
Purging the Log
Purging the log allows you to remove past sessions and Load Plan runs from the log. This procedure is used to keeping a reasonable volume of sessions and Load Plans archived in the work repository. It is advised to perform a purge regularly. This purge can be automated using the OdiPurgeLog tool (see: the OdiPurgeLog section in Oracle Data Integrator Tools Reference) in a scenario.
To purge the log:
Oracle Data Integrator removes the sessions and/or Load Plan runs from the log.
Note:
It is also possible to delete sessions or Load Plan runs by selecting one or more sessions or Load Plan runs in Operator Navigator and pressing the Delete key. Deleting a Load Plan run in this way, deletes the corresponding sessions.
Organizing the Log with Session Folders
You can use session folders to organize the log. Session folders automatically group sessions and Load Plan Runs that were launched with certain keywords. Session folders are created under the Keywords node on the Session List or Load Plan Executions navigation trees.
Each session folder has one or more keywords associated with it. Any session launched with all the keywords of a session folder is automatically categorized beneath it.
To create a new session folder:
- In Operator Navigator, go to the Session List or Load Plan Executions navigation tree.
- Right-click the Keywords node and select New Session Folder.
- Specify a Folder Name.
- Click Add to add a keyword to the list. Repeat this step for every keyword you wish to add.
Note:
Only sessions or load plans with all the keywords of a given session folder will be shown below that session folder. Keyword matching is case sensitive.
Table 19-5 lists examples of how session folder keywords are matched.
Table 19-5 Matching of Session Folder Keywords
Session folder keywords | Session keywords | Matches? |
---|---|---|
DWH, Test, Batch |
Batch |
No - all keywords must be matched. |
Batch |
DWH, Batch |
Yes - extra keywords on the session are ignored. |
DWH, Test |
Test, dwh |
No - matching is case-sensitive. |
To launch a session with keywords, you can for example start a scenario from a command line with the -KEYWORDS
parameter. Refer to Running Integration Processesfor more information.
Note:
Session folder keyword matching is dynamic. If the keywords for a session folder are changed or if a new folder is created, existing sessions are immediately re-categorized.
Exporting and Importing Log Data
Export and import log data for archiving purposes.
Exporting Log Data
Exporting log data allows you to export log files for archiving purposes.
To export the log:
-
Select Export... from the Designer, Topology, Security or Operator Navigator toolbar menu.
-
In the Export Selection dialog, select Export the Log.
-
Click OK.
-
In the Export the log dialog, set the log export parameters as described in Table 19-6.
Table 19-6 Log Export Parameters
Properties Description Export to directory
Directory in which the export file will be created.
Export to zip file
If this option is selected, a unique compressed file containing all log export files will be created. Otherwise, a set of log export files is created.
Zip File Name
Name given to the compressed export file.
Filters
This set of options allow to filter the log files to export according to the specified parameters.
Log Type
From the list, select for which objects you want to retrieve the log. Possible values are:
All|Load Plan runs and attached sessions|Sessions
From / To
Date of execution: specify either a date From or a date To, or both.
Agent
Agent used to execute the session. Leave the default All Agents value, if you do not want to filter based on a given agent.
Context
Session's execution Context. Leave the default All Contexts value, if you do not want to filter based on a context.
Status
The possible states are
Done
,Error
,Queued
,Running
,Waiting
,Warning
andAll States
. Leave the default All States value, if you do not want to filter based on a given session state.User
User who launched the session. Leave the default All Users value, if you do not want to filter based on a given user.
Session Name
Use
%
as a wildcard. For exampleDWH%
matches any session whose name begins withDWH
.Encryption
These fields allow you to provide an Export Key, used to encrypt any sensitive data that is contained in the exported object. See the Export Keys section in Developing Integration Projects with Oracle Data Integrator for details.
Export Key
Specifies the AES KEY for any sensitive data encryption needed during the export.
The export key string is minimum 8 characters long and maximum 100 characters long. It should have at least one special character (@#$%+/=) or digit, and at least one alphabetic lower or upper case character.
Confirm Export Key
Enter your Export Key again.
Save Export Key
If checked, your Export Key is saved for all future exports.
Advanced options
This set of options allow to parameterize the output file format.
Character Set
Encoding specified in the export file. Parameter encoding in the XML file header.
<?xml version="1.0"
encoding="ISO-8859-1"
?>
Java Character Set
Java character set used to generate the file.
-
Click OK.
The log data is exported into the specified location.
Note that you can also automate the log data export using the OdiExportLog tool (see: the OdiExportLog section in Oracle Data Integrator Tools Reference).
Importing Log Data
Importing log data allows you to import into your work repository log files that have been exported for archiving purposes.
To import the log:
-
Select Import... from the Designer, Topology, Security or Operator Navigator toolbar menu.
-
In the Import Selection dialog, select Import the Log.
-
Click OK.
-
In the Import of the log dialog:
-
-
Select the Import Mode. Note that sessions can only be imported in Synonym Mode INSERT mode. Refer to the Import Modes section in Developing Integration Projects with Oracle Data Integrator for more information.
-
Select whether you want to import the files From a Folder or From a ZIP file.
-
Enter the file import folder or zip file.
-
Click OK.
-
If prompted, enter the Export Key used when the log data was exported. If you do not enter an Export Key, any encrypted sensitive (cipher) data will be stripped from the imported log data. For more information about the Export Key, see the Export Keys section in Developing Integration Projects with Oracle Data Integrator.
-
The specified folder or ZIP file is imported into the work repository.
Runtime Logging for ODI components
You can set up runtime logging to trace ODI components or set a verbose level to investigate different issues or to monitor the system. The following ODI components can be traced: ODI Studio, ODI Java EE agents, ODI Standalone agents, and ODI Standalone Colocated agents.
Note:
A verbose logging will slow down the ODI performance.
Log Level
The log level can be set against a log_handler and/or logger elements. Note the following when setting the log level in the log configuration file for the ODI component:
-
If it is set against a log_handler, then it applies to all usages of the log_handler.
-
If it is set against a logger, then it applies to all of its handlers and any descendent loggers that do not have an explicit level setting.
-
A message is logged if its log level is:
-
Greater or equal than (>=) the level of its logger AND
-
Greater or equal than (>=) the level of its log_handler
-
ODI components use the Oracle Java Debugging Levels (OJDL). Table 19-7 shows the mapping between the Java log levels and the Oracle Java Debugging Levels.
Table 19-7 Mapping between Java Log Levels and Oracle Java Debugging Levels
Java Log Levels | Oracle Java Debugging Levels |
---|---|
SEVERE intValue()+100 |
INCIDENT_ERROR:1 |
SEVERE |
ERROR:1 |
WARNING |
WARNING:1 |
INFO |
NOTIFICATION:1 |
CONFIG |
NOTIFICATION:16 |
FINE |
TRACE:1 |
FINER |
TRACE:16 |
FINEST |
TRACE:32 |
Setting Up Runtime Logging
To set up runtime logging you have to enable different ODI loggers and log handlers. For ODI Java EE agents and ODI Standalone Colocated agents, this can be done through the logging configuration mechanism within Oracle Enterprise Manager Console. For ODI Studio and Standalone agents, set the related log level in the ODI logging system configuration file, as shown below:
-
Open the ODI logging system configuration file of the ODI component.
Each component has its own configuration file:
-
ODI Studio:
$ODI_HOME/odi/studio/bin/ODI-logging-config.xml
-
ODI Standalone agent:
<DOMAIN_HOME>/config/fmwconfig/components/ODI/<INSTANCE_NAME>/ODI-logging-config.xml
-
-
Make sure that the path to your log files is a valid and existing path. For example:
<log_handler name="ODI-file-handler" class="oracle.core.ojdl.logging.ODLHandlerFactory" level="ALL"> <property name="format" value="ODL-Text"/> <property name="path" value="/u01/oracle/odi11g/oracledi/agent/log/${LOG_FILE}"/> <property name="maxFileSize" value="1000000"/> <!-- in bytes --> <property name="maxLogSize" value="50000000"/> <!-- in bytes --> <property name="encoding" value="UTF-8"/> </log_handler>
Note the following concerning the log files path:
-
If you are on Windows, the path could be for example:
%ODI_HOME%\oracledi\agent\log\${LOG_FILE}
-
You can use a relative path on Windows and Unix.
-
-
Enable the logger and set the log level. For example, for logger oracle.odi.agent you can enable the most verbose logging setting:
<logger name="oracle.odi.agent" level="TRACE:32" useParentHandlers="false"> <handler name="ODI-file-handler"/> <handler name="ODI-console-handler"/> </logger>
-
Save the configuration file and restart ODI Studio and the ODI Standalone agent for the changes to take effect.
Example INFO (NOTIFICATION:1) Message
The INFO (NOTIFICATION:1)
Message is logged if:
-
The logger level (possibly inherited) is
<= NOTIFICATION:1
-
The log_handler level is
<= NOTIFICATION:1
(for example:TRACE:1
)
The INFO (NOTIFICATION:1)
Message is not logged if:
-
The logger level (possibly inherited) is
> NOTIFICATION:1
-
The log_handler level is
> NOTIFICATION:1
(for example:WARNING:1
)
The Runtime logger is called oracle.odi.agent. Its message levels cover the following content:
NOTIFICATION:1 (or INFO) Agent Level NOTIFICATION:16 (or CONFIG) the above + Session Level TRACE:1 (or FINE) the above + Step Level TRACE:16 (or FINER) the above + Task + SQL TRACE:32 (or FINEST) the above + more detail
It is recommended that the console level for oracle.odi.agent is set so that agent startup messages are displayed (NOTIFICATION:1
).
Managing Scenarios and Load Plans
You can also manage your executions in Operator Navigator by using scenarios or Load Plans.
Before running a scenario, you need to generate it in Designer Navigator or import from a file. See the Using Scenarios chapter in Developing Integration Projects with Oracle Data Integrator. Load Plans are also created using Designer Navigator, but can also be modified using Operator Navigator. See the Using Load Plans chapter in Developing Integration Projects with Oracle Data Integrator for more information.
Launching a scenario from Operator Navigator is covered in Executing a Scenario from ODI Studio, and how to run a Load Plan is described in Executing a Load Plan.
Managing Schedules
A schedule is always attached to one scenario or one Load Plan. Schedules can be created in Operator Navigator. See Scheduling Scenarios and Load Plansfor more information.
You can also import an already existing schedule along with a scenario or Load Plan import. See the Importing Scenarios in Production and Exporting, Importing and Versioning Load Plans sections in Developing Integration Projects with Oracle Data Integrator for more information.
You can view the scheduled tasks of all your agents or you can view the scheduled tasks of one particular agent. See Displaying the Schedulefor more information.