20 Monitoring Integration Processes

This chapter describes how to manage your development executions in Operator Navigator. An overview of the Operator Navigator's user interface is provided.

This chapter includes the following sections:

Section 20.1, "Introduction to Monitoring"
Section 20.2, "Reviewing your Execution Results"
Section 20.3, "Managing your Executions"

20.1 Introduction to Monitoring

Monitoring your development executions consists of viewing the execution results and managing the development executions when the executions are successful or in error. This section provides an introduction to the monitoring features in Oracle Data Integrator. How to work with your execution results is covered in Section 20.2, "Reviewing your Execution Results". How to manage your development executions is covered in Section 20.3, "Managing your Executions".

20.1.1 Introduction to Operator Navigator

Through Operator Navigator, you can view your execution results and manage your development executions in the sessions, as well as the scenarios in production.

Operator Navigator stores this information in a work repository, while using the topology defined in the master repository.

Operator Navigator displays the objects available to the current user in five accordions:

Session List displays all sessions organized per date, physical agent, status, keywords, etc
Hierarchical Sessions displays the execution sessions also organized in a hierarchy with their child sessions
Scheduling displays the list of physical agents and schedules
Scenarios displays the list of scenarios available
Solutions displays the list of solutions

The Operator Navigator Toolbar Menu

You can perform the main monitoring tasks via the Operator Navigator Toolbar menu. The Operator Navigator toolbar menu provides access to the features detailed in Table 20-1.

Table 20-1 Operator Navigator Toolbar Menu Items

Icon	Menu Item	Description
	Refresh	Click Refresh to refresh the trees in the Operator Navigator accordions.
	Filter Filter activated	Click Filter to define the filters for the sessions to display in Operator Navigator.
	Auto Refresh	Click Auto Refresh to refresh automatically the trees in the Operator Navigator accordions.
	Connect Navigator	Click Connect Navigator to access the Operator Navigator toolbar menu. Through the Operator Navigator toolbar menu you can: Import a scenario Import and export the log Perform multiple exports Purge the log Display the scheduling information Clean stale sessions

20.1.2 Sessions

In Oracle Data Integrator, an execution results in a session. Sessions are viewed and managed in Operator Navigator.

A session is an execution (of a scenario, an interface, a package or a procedure, and so forth) undertaken by an execution agent. A session is made up of steps which are themselves made up of tasks.

A step is the unit of execution found between a task and a session. It corresponds to a step in a package or in a scenario. When executing an interface or a single variable, for example, the resulting session has only one step.

The task is the smallest execution unit. It corresponds to a command in a KM, a procedure, etc.

Sessions can be grouped into session folders. Session folders automatically group sessions that were launched with certain keywords. Refer to Section 20.3.2.3, "Organizing the Log with Session Folders" for more information.

20.1.3 Scenarios and Schedules

A scenario is designed to put a source component (interface, package, procedure, variable) into production. A scenario results from the generation of code (SQL, shell, etc) for this component.

You can run from Operator Navigator a scenario that has been previously generated in Designer Navigator or that has been imported from a file. Refer to Section 20.3.3, "Managing Scenarios" for more details.

You can schedule the executions of your scenarios using Oracle Data Integrator's built-in scheduler or an external scheduler. Both methods are detailed in Section 19.5, "Scheduling Scenarios".

The schedules appear in Designer and Operator Navigator under the Scheduling node of the scenario. Each schedule allows a start date and a repetition cycle to be specified.

You can view the scheduling information in Operator Navigator. The Scheduling Information lets you visualize the agents' scheduled tasks.

20.2 Reviewing your Execution Results

In Oracle Data Integrator, an execution results in a session. A session is made up of steps which are made up of tasks. Sessions are viewed and managed in Operator Navigator.

20.2.1 Status

A session, step or task always has a status. Table 20-2 lists the six possible status values:

Table 20-2 Status Values

Status Name	Status Icon	Status Description
Done		The session, step or task was executed successfully
Error		The session, step or task has terminated due to an error.
Running		The session, step or task is being executed.
Waiting		The session, step or task is waiting to be executed
Warning (Sessions and tasks only)		For Sessions: The session has completed successfully but errors have been detected during the data quality check. For Tasks: The task has terminated in error, but since errors are allowed on this task, this did not stop the session.
Queued (Sessions only)		The session is waiting for an agent to be available for its execution

When finished, a session takes the status of the last executed step (Done or Error). When finished, the step, takes the status of the last executed task (Except if the task returned a Warning. In this case, the step takes the status Done)

20.2.2 Managing Errors

When your session ends in error or with a warning, you can analyze the error in Operator Navigator.

To analyze an error:

In the Operator Navigator, identify the session, the step and the task in error.
Double click the task in error. The Task editor opens.
On the Definition tab in the Execution Statistics section, the return code and message give the error that stopped the session.
On the Code tab, the source and target code for the task is displayed and can be reviewed and edited.
On the Connection tab, you can review the source and target connections against which the code is executed.

You can fix the code of the command in the Code tab and apply your changes. Restarting a Session is possible after performing this action. The session will restart from the task in error.

Note:

Fixing the code in the session's task does not fix the source object that was executed (interface, procedure, package or scenario). This source object must be fixed in Designer Navigator and the scenario (if any) must be regenerated. Modifying the code within the session is useful for debugging issues.

WARNING:

When a session fails, all connections and transactions to the source and target systems are rolled back. As a consequence, uncommitted statements on transactions are not applied.

20.2.3 Managing Successful Executions

When your session ends successfully, you can view the changes performed in Operator Navigator. These changes include record statistics such as the number of inserts, updates, deletes, errors, and the total number of rows as well as execution statistics indicating start and end time of the execution, the duration in seconds, the return code, and the message (if any).

Session level statistics aggregate the statistics of all the steps of this session, and each step's statistics aggregate the statistics of all the tasks within this step.

To review the execution statistics:

In the Operator Navigator, identify the session, the step or the task to review.
Double click the session, the step or the task. The corresponding editor opens.
The record and execution statistics are displayed on the Definition tab. Note that for session steps in which an interface has been executed or a datastore check has been performed also the target table details are displayed.

Record Statistics

Properties	Description
No. of Inserts	Number of rows inserted during the session/step/task.
No. of Updates	Number of rows updated during the session/step/task.
No. of Deletes	Number of rows deleted during the session/step/task.
No. of Errors	Number of rows in error in the session/step/task.
No. of Rows	Total number of rows handled during this session/step/task.

Execution Statistics

Properties	Description
Start	Start date and time of execution of the session/step/task.
End	End date and time of execution of the session/step/task.
Duration (seconds)	The time taken for execution of the session/step/task.
Return code	Return code for the session/step/task.

Target Table Details

Properties	Description
Table Name	Name of the target datastore.
Model Code	Code of the Model in which the target datastore is stored.
Resource Name	Resource name of the target datastore.
Logical Schema	Logical schema of this datastore.
Forced Context Code	The context of the target datastore.

20.3 Managing your Executions

Managing your development executions takes place in Operator Navigator. You can manage your executions during the execution process itself or once the execution has finished depending on the action that you wish to perform. The actions that you can perform are:

Managing Sessions
Managing the Log
Managing Scenarios
Managing Schedules

20.3.1 Managing Sessions

Managing sessions involves the following tasks

Starting a Session
Stopping a Session
Restarting a Session

20.3.1.1 Starting a Session

A session is created and started when an integration process is launched. How to run integration processes is covered in Chapter 19, "Running Integration Processes".

20.3.1.2 Stopping a Session

Any running or waiting session can be stopped. You may want to stop a session when you realize that for example your interface contains errors or when the execution takes a long time.

Note that there are two ways to stop a session:

Normal: The session is stopped once the current task is finished.
Immediate: The current task is immediately interrupted and the session is stopped. This mode allows to stop long-running tasks, as for example long SQL statements before they complete.

Note:

The immediate stop works only with technologies and drivers that support task interruption. It is supported if the statement.cancel method is implemented in the JDBC driver.

Note:

Only sessions that are running within a Java EE or standalone Agent can be stopped. Sessions running in the Studio built-in Agent or started with the startscen.sh or startscen.bat script without the AGENT_URL parameter, cannot be stopped. See Chapter 19, "Running Integration Processes" for more information.

To stop a session:

In Operator Navigator, select the running or waiting session to stop from the tree.
Right-click then select Stop Normal or Stop Immediate.
In the Stop Session Dialog, click OK.

The session is stopped and changed to Error status.

20.3.1.3 Restarting a Session

Any session that has encountered an error, or has been stopped by the user can be restarted.

How to restart a session is covered in Section 19.4.2, "Restarting a Session from a Command Line".

20.3.1.4 Cleaning Stale Sessions

Stale sessions are sessions that are incorrectly left in a running state after an agent or repository crash.

The Agent that started a session automatically detects when this session becomes stale and changes it to Error status. You can manually request specific Agents to clean stale sessions in Operator Navigator or Topology Navigator.

To clean stale sessions manually:

Do one of the following:
- From the Operator Navigator toolbar menu, select Clean Stale Sessions.
- In Topology Navigator, from the Physical Architecture accordion, select an Agent, right-click and select Clean Stale Sessions.
The Clean Stale Sessions Dialog opens.
In the Clean Stale Sessions Dialog specify the criteria for cleaning stale sessions:
- From the list, select the Agents that will clean their stale sessions.
  
  Select Clean all Agents if you want all Agents to clean their stale sessions.
- From the list, select the Work Repositories you want to clean.
  
  Select Clean all Work Repositories if you want to clean stale sessions in all Work Repositories.
Click Clean to start the cleaning process.

20.3.2 Managing the Log

Oracle Data Integrator provides several solutions for managing your log data:

Filtering Sessions to display only certain execution sessions in Operator Navigator
Purging the Log to remove the information of past sessions
Organizing the Log with Session Folders
Exporting and Importing Log Data for archiving purposes

20.3.2.1 Filtering Sessions

Filtering log sessions allows you to display only certain sessions in Operator Navigator, by filtering on parameters such as the user, status or duration of sessions. Sessions that do not meet the current filter are hidden from view, but they are not removed from the log.

To filter out sessions:

In the Operator Navigator toolbar menu, click Filter. The Define Filter editor opens.
In the Define Filter Editor, set the filter criteria according to your needs. Note that the default settings select all sessions.
- Session Number: Use blank to show all sessions.
- Session Name: Use % as a wildcard. For example DWH% matches any session whose name begins with DWH.
- Session's execution Context
- Agent used to execute the session
- User who launched the session
- Status: Running, Waiting etc.
- Date of execution: Specify either a date From or a date To, or both.
- Duration greater than a specified number of seconds
Click Apply for a preview of the current filter.
Click OK.

Sessions that do not match these criteria are hidden in the Session List accordion. The Filter button on the toolbar is activated.

To deactivate the filter click Filter in the Operator toolbar menu. The current filter is deactivated, and all sessions appear in the list.

20.3.2.2 Purging the Log

Purging the log allows you to remove past sessions from the log. This procedure is used to keeping a reasonable volume of sessions archived in the work repository. It is advised to perform a purge regularly. This purge can be automated using the OdiPurgeLog tool.

To purge the log:

From the Operator Navigator toolbar menu select Connect Navigator > Purge Log... The Purge Log editor opens.

In the Purge Log editor, set the criteria listed in Table 20-3 for the sessions you want to delete.

Table 20-3 Purge Log Parameters

Parameter	Description
From ... To	Sessions in this time range will be deleted.
Context	Sessions executed in this context will be deleted.
Agent	Sessions executed by this agent will be deleted.
Status	Session in this status will be deleted.
User	Sessions executed by this user will be deleted.
Session name	Sessions matching this session name will be deleted. Note that you can specify session name masks using % as a wildcard.
Purge scenario reports	If you select Purge scenario reports, the scenario reports (appearing under the execution node of each scenario) will also be purged.

Only the sessions matching the specified filters will be removed.

Click OK.

Oracle Data Integrator removes the sessions from the log.

Note:

It is also possible to delete sessions by selecting one or more sessions in Operator Navigator and pressing the Delete key.

20.3.2.3 Organizing the Log with Session Folders

You can use session folders to organize the log. Session folders automatically group sessions that were launched with certain keywords. Session folders are created under the Keywords node on the Session List accordion.

Each session folder has one or more keywords associated with it. Any session launched with all the keywords of a session folder is automatically categorized beneath it.

To create a new session folder:

In Operator Navigator, go to the Session List accordion.
Right-click the Keywords node and select New Session Folder.
Specify a Folder Name.
Click Add to add a keyword to the list. Repeat this step for every keyword you wish to add.

Note:

Only sessions with all the keywords of a given session folder will be shown below that session folder. Keyword matching is case sensitive.

Table 20-4 lists examples of how session folder keywords are matched.

Table 20-4 Matching of Session Folder Keywords

Session folder keywords	Session keywords	Matches?
DWH, Test, Batch	Batch	No - all keywords must be matched.
Batch	DWH, Batch	Yes - extra keywords on the session are ignored.
DWH, Test	Test, dwh	No - matching is case-sensitive.

To launch a session with keywords, you can for example start a scenario from a command line with the -KEYWORDS parameter. Refer to Chapter 19, "Running Integration Processes" for more information.

Note:

Session folder keyword matching is dynamic. If the keywords for a session folder are changed or if a new folder is created, existing sessions are immediately re-categorized.

20.3.2.4 Exporting and Importing Log Data

Export and import log data for archiving purposes.

Exporting Log Data

Exporting log data allows you to export log files for archiving purposes.

To export the log:

From the Operator Navigator toolbar menu select Connect Navigator > Export > Log

In the Export the log dialog, set the log export parameters as described in Table 20-5.

Table 20-5 Log Export Parameters

Properties	Description
Export to directory	Directory in which the export file will be created.
Exports to zip file	If this option is selected, a unique compressed file containing all log export files will be created. Otherwise, a set of log export files is created.
Zip File Name	Name given to the compressed export file.
Filters	This set of options allow to filter the log files to export according to the specified parameters.
From / To	Date of execution: specify either a date From or a date To, or both.
Agent	Agent used to execute the session.
Context	Session's execution Context
Session State	The possible states are `Done`, `Error`, `Queued`, `Running`, `Waiting` and `Warning`.
User	User who launched the session
Session Name	Use `%` as a wildcard. For example `DWH%` matches any session whose name begins with `DWH`.
Advanced options	This set of options allow to parameterize the output file format.
Character Set	Encoding specified in the export file. Parameter encoding in the XML file header. `<?xml version="1.0"` `encoding="ISO-8859-1"?>`
Java Character Set	Java character set used to generate the file.

Click OK.

The log data is exported into the specified location.

Note that you can also automate the log data export using the OdiExportLog tool.

Importing Log Data

Importing log data allows you to import into your work repository log files that have been exported for archiving purposes.

To import the log:

From the Operator Navigator toolbar menu select Connect Navigator > Import > Log
In the Import of the log dialog:
1. Select the Import Mode. Refer to Section 18.1.3, "Import Modes" for more information.
2. Select whether you want to import the files From a Folder or From a ZIP file.
3. Enter the file import folder or zip file.
4. Click OK.

The specified folder or ZIP file is imported into the work repository.

20.3.3 Managing Scenarios

You can also manage your developments executions in Operator Navigator by using scenarios.

Before running a scenario, you need to have the scenario generated from Designer Navigator or imported from a file. Refer to Chapter 13, "Working with Scenarios" for more information.

Launching a scenario from Operator Navigator is covered in Section 19.3.1, "Executing a Scenario from the Studio".

20.3.3.1 Scenario Folders

In Operator Navigator, scenarios can be grouped into scenario folders to facilitate organization. Scenario folders can contain other scenario folders.

To create a scenario folder:

In Operator Navigator go to the Scenarios accordion.
From the Scenarios toolbar menu, select New Scenario Folder > New Scenario Folder.
On the Definition tab of the Scenario Folder editor enter a name for your scenario folder.
From the File menu, select Save.

You can now reorganize your scenarios and drag and drop them into the scenario folder.

20.3.3.2 Importing Scenarios and Solutions in Production

A scenario generated from Designer can be exported and then imported into a development or production repository. This operation is used to deploy scenarios in a different repository, possibly in a different environment or site.

Importing a scenario in a development repository is performed via Designer or Operator Navigator. With a production repository, only Operator Navigator is available for this purpose.

How to import a scenario in production is covered in Section 13.6, "Importing Scenarios in Production".

Similarly, a solution containing several scenarios can be imported to easily transfer and restore a group of scenarios at once. See Chapter 17, "Working with Version Management" for more information. Note that when connected to a production repository, only scenarios may be restored from solutions.

20.3.4 Managing Schedules

A schedule is always attached to one scenario. Schedules can be created in Operator Navigator. See Section 19.5, "Scheduling Scenarios" for more information.

You can also import an already existing schedule during the scenario import. See Section 13.6, "Importing Scenarios in Production" for more information.

You can view the scheduled tasks of all your agents or you can view the scheduled tasks of one particular agent. See Section 19.5.1.3, "Displaying the Schedule" for more information.