7 High Availability for Oracle Data Integrator

Oracle Data Integrator provides a complete set of components for designing, deploying and managing data integration processes. Data integration processes move and transform data from source data servers to target data servers, using an Extract-Load-Transform approach that eliminates the need for a transformation engine by delegating all transformations to the source and target data servers.

This chapter provides a description of Oracle Data Integrator components from a high availability perspective. The sections in this chapter outline the single instance concepts that are important for designing high availability deployment.

This chapter includes the following topics:

7.1 Introduction to Oracle Data Integrator

As shown in Figure 7-1, Oracle Data Integrator provides a comprehensive set of features for developing, deploying and monitoring data integration processes. Oracle Data Integrator provides a unified run-time architecture based on Java components.

The run-time architecture consists of:

  • Repository: The repository stores the data integration design-time objects, the run-time objects (data integration scenarios to run) and the sessions corresponding to execution instances of these design-time and run-time objects.

  • Run-time agents: These are the standalone agent or Java EE agents deployed in an application server, which run the sessions. The agent connects the repository as well as the source and target data servers when processing the data integration sessions. The agent also provides a web service for starting and monitoring scenario executions from third party SOA-enabled applications.

  • Oracle Data Integrator Console: This web application enables users to browse, monitor and manage the artifacts stored in the repository. It also performs run-time operations, such as starting a scenario on run-time agents.

7.2 Oracle Data Integrator Single Instance Characteristics

Figure 7-1 shows the Oracle Data Integrator single instance architecture.

Figure 7-1 Oracle Data Integrator Single Instance Architecture

Description of Figure 7-1 follows
Description of "Figure 7-1 Oracle Data Integrator Single Instance Architecture"

Oracle Data Integrator run-time agents manage integration processes. Oracle Data Integrator agents are components that run the integration jobs deployed in a production configuration as scenarios stored in a repository.

Oracle Data Integrator agents process each scenario execution instance as a session. Each session exists in the agent as a separate thread of the agent Java process.

Agents store very basic information about the session they run. Most of the session data is stored in the repository. When a scenario runs on an agent, the agent creates a session in the repository that corresponds to this scenario's instance. The agent reads each task of this session from the repository, processes it, and writes the result - the return code, message and tasks metrics such as the duration or number of rows processed - into the repository.

The repository consists of two database schemas, one containing the master repository, and one containing the work repository. The master repository contains all topology and security related information (such as the source data server definition, target data server definition, and user credential). The work repository contains development and run-time data (such as sessions and scenarios). The master repository also contains the connection information to the work repository. To connect to a work repository, an agent first connects to the master repository, checks the Oracle Data Integrator user's credentials, reads the work repository connection information, and then connects to the work repository. A typical topology includes one master repository and possibly several work repositories (for example, for test and production).

Sessions can be initiated on the agent:

  • From another Oracle Data Integrator component (such as the agent or Oracle Data Integrator Studio) over HTTP.

  • Via the agent's web service interface.

  • From an external scheduler or from a command line.

  • From a Java program using the Agent Invocation SDK

The agent is always attached to a master repository. It connects to this master repository at startup and is able to start sessions on any of the work repositories attached to this master. It also acts as a scheduler. On startup, the agent reads from the different work repositories the schedules defined for the agent, and stores this scheduling information. The agent is able to initiate sessions from this in-memory schedule on the appropriate work repositories.

Agents can interact with one another through remote scenario startup (over HTTP) or via the load balancing feature. Load balancing enables defining hierarchies of parent/child agents. In this hierarchy, parent agents can delegate the processing of their sessions to their child agents.

The agent is a Java program that is provided as a Java EE agent and as a standalone agent. The Java EE agent is a web application that can be deployed in a Java EE application server, along with other web applications within the same JVM. This agent can use this server's data sources to connect the source, target and repository databases.

The standalone agent is provided as a standalone Java process started from a command line interface. This standalone agent is similar to the Java EE agent, but is embedded in a lightweight container. The main difference is that unlike the Java EE agent, the standalone agent can connect the source and target data servers using only direct JDBC connection.

This section includes the following topics:

7.2.1 Oracle Data Integrator Sessions Lifecycle and Recovery

When an execution request arrives to a run-time agent, the agent connects the master repository to check the user credentials and then the work repository to create the session and all its tasks, and marks them as waiting. Then it creates the connections to all the data servers that will be used during this session.

When execution starts, the agent reads the first task in the work repository to run and marks both the session and this task as running. This task can start an operation on the data servers or on the operating system. When the task is complete, the agent writes into the work repository the execution result for this task, moves it to a finished state (Done, Warning or Error) and proceeds to the next task in the session. Note that errors cases can be handled in the ODI packages, and an error does not always halt a session. When the session completes, either because of an unmanaged error, or by reaching a final step, the agent moves the session to a finished state and releases all the connections. At this point, the session is finished.

7.2.1.1 Sessions Interruption

Sessions can be interrupted when:

  • A user requests the agent to stop the session.

  • An agent is stopped by the administrator. All sessions for this agent are stopped, depending on the agent stop mode selected.

  • A critical event occurs on the agent or the repository.

Any session that is stopped due to user or administrator action is moved to an error state and marked as "Stopped."

In the case of an agent or repository failure, a session that cannot be stopped properly still appears in a running state in the repository. These sessions are stale sessions because they are marked as running, but are no longer handled by any agent. Stale sessions automatically move to an error state when an agent restarts and detects that these sessions are incorrectly marked in the repository as being run by this agent.

7.2.1.2 Recovering Sessions

Oracle Data Integrator uses JDBC transactions when interacting with source and target data servers, and any open transaction state is not persisted when a session finishes in error state. The appropriate restart point is the task that started the unfinished transaction(s). If such a restart point is not identifiable, it is recommended that you start a fresh session by running the scenario instead of restarting existing sessions that are in error state.

By default, a session restarts from the last task that failed to run (typically a task in error or in waiting state). A session may need to be restarted in order to proceed with existing staging tables and avoid re-running long loading phases. In that case the user should take into consideration transaction management, which is KM specific. A general guideline is: If a failure occurs during a loading task, you can restart from the loading task that failed. If a failure occurs during an integration phase, restart from the first integration task, because integration into the target is within a transaction. This guideline applies only to one interface at a time. If several interfaces are chained and only the last one performs the commit, then they should all be restarted because the transaction runs over several interfaces.

To restart from a specific task or step:

  1. In Operator Navigator, go to this task or step, edit it and switch it to Waiting state.

  2. Set all tasks and steps after this one in the Operator tree view to Waiting state.

  3. Right-click the session and click Restart.

The session restarts from the first task in waiting state.

7.2.2 Agent Startup and Shutdown Cycle

Figure 7-2 shows the agent startup cycle.

Figure 7-2 Oracle Data Integrator Agent Startup Cycle

Description of Figure 7-2 follows
Description of "Figure 7-2 Oracle Data Integrator Agent Startup Cycle"

When the Oracle Data Integrator agent starts, it first reads its configuration, which includes master repository connection information. Then the agent connects to each of the work repositories attached to this master repository and removes stale sessions. Stale sessions are sessions that are incorrectly indicated in the work repository as running on this given agent. Stale sessions may result from an agent being stopped without being able to stop these sessions gracefully. As the agent restarts, it identifies the stale sessions and moves them to an error state.

From that point, the agent can retrieve and compute the schedules available for it in each work repository. Once this phase is complete, the agent waits for incoming sessions requests to start their processing, and is also able to start sessions based on its schedules.

7.2.3 Oracle Data Integrator External Dependencies

Oracle Data Integrator depends on the Oracle Data Integrator master repository and work repository database schemas.

If advanced features are being used, these other dependencies may exist:

  • Other Oracle Data Integrator agents: If the load balancing feature is configured and the agent needs to delegate the execution of sessions to its child agents.

  • If External Password Storage is enabled for this agent's master repository, the agent depends on the credential store for retrieving the source and target data servers' passwords to connect these data servers during session execution.

  • If External Authentication is enabled for this agent's master repository, the run-time agents as well as Oracle Data Integrator Console depend on the Identity Store service that stores the Oracle Data Integrator user accounts.

These components must be available for the Oracle Data Integrator system to start and run properly.

7.2.4 Oracle Data Integrator Startup and Shutdown Process

The Oracle Data Integrator Java EE agent application is started by default whenever any Oracle WebLogic Managed Server to which it has been deployed is started. Normally, you should not need to stop Oracle Data Integrator or any of its components by themselves. Some operations may require the Oracle WebLogic Managed Server where Oracle Data Integrator runs to be rebooted. Only some patching scenarios should require stopping the application.

You can use Administration Console to verify the status and to start and stop Oracle WebLogic Server. You can also use the WebLogic Server WLST command line to control the application. Oracle Enterprise Manager Fusion Middleware Control also enables multiple operations and configuration of Oracle Data Integrator as well as monitoring its status.

7.2.5 Oracle Data Integrator Configuration Artifacts

This section describes Oracle Data Integrator configuration artifacts.

7.2.5.1 Java EE Agent Configuration

The configuration parameters for the Java EE agent are stored in both the agent application and in the master repository to which the agent is attached.

As shown in Figure 7-3, the agent configuration stored in the master repository can be edited from the Oracle Data Integrator Studio and includes the following key parameters:

  • Host

  • Port

  • Web Application Context

Figure 7-3 Agent Configuration in the Master Repository Displayed in Oracle Data Integrator Studio

Description of Figure 7-3 follows
Description of "Figure 7-3 Agent Configuration in the Master Repository Displayed in Oracle Data Integrator Studio"

Using this information and valid credentials, any component connecting the repository is able to request an execution from this agent on the <host>:<port>/<web_application_context> URL. Several agents can be started on the same host and port pair by specifying a different Web application context value.

In addition to this information, the Master Repository also contains:

  • The names of the data sources used by this agent to connect the source, target and repository database hosts.

  • The hierarchy of agents for the load balancing feature.

Oracle Data Integrator Studio provides a feature to generate an Oracle WebLogic template for a given agent. This template contains the agent binaries, configuration and data source definitions. Oracle WebLogic administrators can use these templates to deploy pre-configured agents over a cluster.

Other configuration options at the container level, such as creation or modification of data sources are available as WebLogic Server domain configuration, and are synchronized across a cluster of Oracle WebLogic Servers by the Oracle WebLogic Server core infrastructure.

See the chapter on setting up the topology in Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator for more information on the Java EE agent deployment options.

7.2.5.2 Standalone Agent Configuration

The configuration parameters for the standalone agent are stored both in the standalone agent configuration file as well as in the master repository to which this agent is attached. The repository part of the agent configuration is similar to the Java EE agent.

The agent configuration differs as it is stored in a configuration file, stored in its installation folder:

  • For UNIX: ODI_HOME/oracledi/agent/bin/odiparams.sh

  • For Windows: ODI_HOME/oracledi/agent/bin/odiparams.bat

This configuration file includes the following parameters that must be edited manually as part of the standalone agent configuration:

  • JDBC connection information to connect the master repository

  • ODI Supervisor user and encrypted password

The agent started in standalone mode is also configured with the command line used to start it. This command line can include the following parameters:

  • Listening Port

  • Agent Name

  • Trace Level

A component willing to send an execution request to this agent will send it on the following URL: <host>:<port>/oraclediagent. Note that the standalone deployed application name is set to oraclediagent. As a consequence, to start several standalone agents on a single host, different ports must be used.

See the chapter on setting up the topology in Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator for more information on the agent configuration details.

7.2.5.3 Oracle Data Integrator Console Configuration

Oracle Data Integrator Console configuration consists of connection definitions to the master and work repositories that can be browsed using this web application.

The list of connections is stored in the repositories.xml file in the following directory:

user_projects/domains/domainName/config/oracledi

Connections can be added, edited, or deleted from the Oracle Data Integrator Console management pages.

Note:

Oracle Data Integrator Console is used as the entry point for Enterprise Manager to discover Oracle Data Integrator targets in a domain. The discovery process works in the following way: Enterprise Manager identifies Oracle Data Integrator Console. Using the Oracle Data Integrator Console configuration, Enterprise Manager identifies the master and work repositories as well as the run-time agents in the domain.

7.2.5.4 Oracle Data Integrator Log Locations and Configuration

This section provides information about Oracle Data Integrator log locations and configuration.

7.2.5.4.1 Oracle Data Integrator Session Logs

Oracle Data Integrator session execution logs are stored in the work repositories against which the sessions are started. This session shows Oracle Data Integrator session details, such as the executed code and the number of processed rows. View the log from the Oracle Data Integrator Studio's Operator Navigator, in the Session List accordion, or from Oracle Data Integrator Console's Browse tab, under Run-Time > Sessions.

7.2.5.4.2 Java EE Agent Log Files

The operations performed by the Oracle Data Integrator Java EE agent are logged by Oracle WebLogic Managed Server where the agent application is running. You can find these logs at the following location:

DOMAIN_HOME/servers/WLS_ServerName/logs/oracledi/odiagent.log

The log files for the different Oracle WebLogic Server Managed Servers are also available from the Administration Console. To verify the logs, access Administration Console using the following URL: admin_server_host:port/console. Click Diagnostics-Log Files.

It is also important to verify the output of the Oracle WebLogic Managed Server where Oracle Data Integrator is running. This information is stored at the following location:

DOMAIN_HOME/servers/WLS_ServerName/logs/WLS_ServerName.out

Additionally, a diagnostic log is produced in the log directory for the managed server. This log's granularity and logging properties can be changed through the following file:

DOMAIN_HOME/config/fmwconfig/logging/oraclediagent-logging.xml
7.2.5.4.3 Standalone Agent Log Files

The operations performed by the Oracle Data Integrator standalone agent are logged by the lightweight container running the standalone agent. By default, logs are traced on the console and in the ODI_HOME/oracledi/log/ folder.

The logging method and the logging level can be configured by editing the ODI_HOME/oracledi/agent/bin/ODI-logging-config.xml file.

7.2.5.4.4 Oracle Data Integrator Console Log Files

Oracle Data Integrator Console logging operations are logged by Oracle WebLogic Managed Server where the agent application is running, like the Java EE agent log files described in Section 7.2.5.4.2, "Java EE Agent Log Files."

7.3 Oracle Data Integrator High Availability and Failover Considerations

This section describes Oracle Data Integrator high availability and failover considerations.

This section includes the following topics:

7.3.1 Oracle Data Integrator Clustered Deployment

Figure 7-4 shows a two-node Oracle Data Integrator cluster running on two Oracle WebLogic servers. Oracle WebLogic Servers are front ended by Oracle HTTP Servers, which load balance incoming requests to them.

Figure 7-4 Oracle Data Integrator High Availability Architecture

Description of Figure 7-4 follows
Description of "Figure 7-4 Oracle Data Integrator High Availability Architecture"

The main characteristics of this configuration are:

  • Oracle Data Integrator applications run on two clustered WebLogic Server managed servers. The WebLogic Server cluster synchronizes configuration for common artifacts of WebLogic Server used by Oracle Data Integrator, such as data sources.

  • To avoid duplicate schedule processing, only one of these agents behaves like a scheduler. A Coherence cache is used to handle scheduler service uniqueness and migration.

    The agent provides failover scheduling capabilities. For example, if a schedule is supposed to start at 9 AM, and the cycle is to run job X every hour for four hours, and the agent fails at 9:55 AM, it should compute where it was in the cycle and continue. However, if a single job is scheduled to start at 9 AM, and the agent fails at 8:59 AM, and then recovers at 9:01 AM, then it will not run the job that was scheduled at 9 AM.

  • Requests to the Oracle Data Integrator agent in a cluster must be routed via a load balancer or via an HTTP proxy server. The address of this fronting server is used by clients to connect transparently to any of the Oracle Data Integrator servers in the cluster. This address must be specified in the agent definition in the master repository. The scheduler singleton also routes all scheduled sessions startup requests to this address so that they are load balanced over the cluster.

  • Oracle Data Integrator's master and work repositories database is configured with Oracle Real Application Clusters (Oracle RAC) to protect from database failures. Oracle Data Integrator components perform the appropriate reconnection and operations retries if database instance failure occurs.

7.3.2 Standalone Agent High Availability with OPMN

The standalone agent is a standalone Java process started from a command line interface. This agent is typically deployed locally on the source or target machines for optimal integration flow performances. You can use OPMN to start, stop and protect the standalone agent in this situation.

To add a standalone agent to OPMN:

  1. Edit the agentcreate.properties file contained in the ODI_HOME/oracledi/agent/bin/ directory to match your agent and OPMN configuration.

  2. Run the script that adds this agent to the OPMN configuration.

    • For UNIX: ODI_HOME/oracledi/agent/bin/opmn_addagent.sh

    • For Windows: ODI_HOME/oracledi/agent/bin/opmn_addagent.bat

You can determine the status of Oracle HTTP Server using opmnctl:

opmnctl status

To start all agent components in an Oracle instance using opmnctl:

opmnctl startproc process-type=odiagent

To start a specific agent component, such as odiagent1, using opmnctl:

opmnctl startproc ias-component=odiagent1

To stop all agent components in an Oracle instance using opmnctl:

opmnctl stopproc process-type=odiagent

To stop a specific agent component, such as odiagent1, using opmnctl:

opmnctl stopproc ias-component=odiagent1

7.3.3 Oracle Data Integrator Protection from Failure and Expected Behavior

This section describes how an Oracle Data Integrator high availability cluster deployment and OPMN protects components from failure. This section also describes expected behavior in the event of component failure.

7.3.3.1 WebLogic Server or Standalone Agent Crash

If a WebLogic Server fails, Node Manager attempts to restart it locally. If repeated restarts fail, the WebLogic Server infrastructure attempts to perform a server migration of the server to the other node in the cluster. While the failover takes place, the other WebLogic instance becomes the scheduler and is able to read, compute, and run the schedule for all work repositories. A Coherence cache is used to handle the scheduler lifecycle. Locking guarantees the uniqueness of the scheduler, and event notification provides scheduler migration. Note that when an agent restarts and computes its schedule, it takes into account schedules in progress (those in the middle of an execution cycle). These are automatically continued in their execution cycle beyond the server startup time. New sessions will be triggered as if the scheduler was never stopped.

Stale sessions are moved to an error state and are treated as such when restarted. This session recovery/restart is described in Section 7.2.1.1, "Sessions Interruption" and Section 7.2.1.2, "Recovering Sessions."

If a standalone agent fails, OPMN restarts it locally. Existing sessions on the failing agent running become stale sessions, which are removed when the agent is restarted.

Oracle Data Integrator agents may be down due to failure in accessing resources, or other issues unrelated to whether the managed server is running. Therefore, Oracle recommends that administrators monitor the managed server logs for cluster errors caused by the application. For information about log file locations, see Section 7.2.5.4, "Oracle Data Integrator Log Locations and Configuration."

The Oracle Data Integrator Console does not support HTTP session failover. The user must log into the Oracle Data Integrator Console again after a failure.

7.3.3.2 Repository Database Failure

The Oracle Data Integrator repositories are protected against failures in the database by using multi data sources. These multi data sources are typically configured during the initial set up of the system (Oracle Fusion Middleware Configuration Wizard enables you to define these multi-pools directly at installation time) and guarantee that when an Oracle RAC database instance that hosts a repository fails, the connections are re-established with available database instances. The multi data source enables you to configure connections to multiple instances in an Oracle RAC database.

The Java EE agent uses WebLogic multi data sources that are configured during initial setup. The standalone agent uses the Oracle RAC JDBC connection string specified in odiparams.sh.

For additional information about multi data source configuration with Oracle RAC, see Section 5.1.3, "Using Multi Data Sources with Oracle RAC."

Oracle Data Integrator implements a retry logic that enables in-flight sessions to proceed if a repository instance becomes unavailable and is restored at a later time. In an Oracle RAC enabled configuration, both in-flight and incoming session execution requests are served as long as an Oracle RAC node is available. This is supported in both the standalone and Java EE agents using the Retry Connection Count number and Connection Retry Delay time parameters. Users can configure these parameters when generating the WebLogic Server template for the Java EE agent and by editing odiparams.sh or odiparams.bat for the standalone agent.

If Oracle Data Integrator Studio loses its connection to an Oracle RAC database, you will lose any Oracle Data Integrator Studio work performed since the last save operation. As a general practice, save your work on a regular basis when you use Oracle Data Integrator Studio.

7.3.3.3 Scheduler Node Failure

In an Oracle Data Integrator agent cluster, when the agent node that is the scheduler node fails, another node in the WebLogic Server cluster takes over as the scheduler node. The new scheduler node reinitializes all the schedules from that point and continues running the scheduled scenarios from that point forward.

However, an issue arises in this situation if a scheduled scenario with a repeatable execution cycle was running on the first scheduler node when that node crashed. When the new scheduler node takes over, the scheduler scenario that was running on the first scheduler node will not continue its iterations on the new scheduler node from the point at which the first scheduler node crashed.

For example, if the scheduled scenario is configured to repeat the execution ten times after an interval of two minutes, and the first scheduler node fails in the middle of the third execution, the new scheduler node should continue the execution of the scenario for the next eight executions. However, the new scheduler node does not continue the remaining executions of the scenario.

7.4 Configuring High Availability for Oracle Data Integrator

This section describes the installation and configuration steps for setting up an Oracle Data Integrator high availability configuration.

This section includes the following topics:

7.4.1 Running RCU to Create the Master and Work Repositories

Use the latest version of the Repository Creation Utility (RCU) to create the necessary schemas for Oracle Data Integrator in an Oracle RAC database repository.

See Oracle Fusion Middleware Repository Creation Utility User's Guide for more information about obtaining and running the latest version of RCU.

Start RCU by running this command:

RCU_HOME/bin/rcu

Then follow these steps:

  1. On the Create Repository page, choose Create and Load new schemas.

  2. On the Database Connection Details page, enter Database details. For Oracle RAC, only one mode is necessary.

  3. On the Select Components page, select Oracle Data Integrator. This causes Master and Work Repository to be automatically selected.

  4. On the Schema Passwords page, specify a password for the schema.

  5. On the Custom Variables page, specify at least the Supervisor and Work Repository passwords. All other fields can be left as default.

  6. On the Map Tablespaces screen, click OK.

  7. On the Summary screen, click OK and complete the repository installation.

7.4.2 Installation and Configuration of the First Oracle Data Integrator Host

This section describes the installation and configuration steps for the first Oracle Data Integrator host (for example, APPHOST1).

This section includes the following topics:

7.4.2.1 Installing the Oracle WebLogic Server on APPHOST1

See "Understanding Your Installation Starting Point" in Oracle Fusion Middleware Installation Planning Guide for the version of Oracle WebLogic Server to use with the latest version of Oracle Fusion Middleware.

Start the Oracle WebLogic Server installer.

Then follow these steps in the installer to install Oracle WebLogic Server on the computer:

  1. On the Welcome screen, click Next.

  2. On the Choose Middleware Home Directory screen, select Create a New Middleware Home. Then choose a directory on your computer into which the Oracle WebLogic software is to be installed.

    Click Next.

  3. On the Register for Security Updates screen, enter your "My Oracle Support" User Name and Password so that you can be notified of security updates

  4. On the Choose Install Type screen, the installation program prompts you to indicate whether you wish to perform a complete or a custom installation.

    Choose Typical.

    Click Next.

  5. On the Choose Product Installation Directories screen, accept the default locations for WebLogic Server and Coherence.

    Click Next.

  6. On the Installation Summary screen, the window contains a list of the components you selected for installation, along with the approximate amount of disk space to be used by the selected components once installation is complete.

    Click Next.

  7. Click Done to exit the installer.

7.4.2.2 Install Oracle Data Integrator on APPHOST1

Start the Oracle Data Integrator installer:

./runInstaller

When the installer prompts you for a JDK location, enter the full path to the SDK under the Middleware home.

Then perform these installation steps:

  1. If the Specify Inventory Directory screen appears, enter an Oracle inventory location and OS Group name, and click OK.

  2. On the Select Installation Type screen:

    • Select J2EE Installation.

    • Optionally, select Developer installation if Oracle Data Integrator Studio is required.

  3. On the Prerequisite Checks screen, verify that the checks complete successfully, then click Next.

  4. On the Specify Installation Location screen, select the Middleware Home from the list.

    Click Next.

  5. On the Repository Configuration screen, choose Skip Repository Configuration and click Next.

  6. On the Intallation Summary screen, click Install to start the installation.

  7. On the Installation Complete screen, click Finish.

7.4.2.3 Create the High Availability Domain

To start the Configuration Wizard, run config.sh in the ODI_HOME/common/bin directory. Then follow these steps:

  1. On the Welcome screen, choose Create a new WebLogic Domain.

  2. On the Select Domain Source screen, choose these components:

    • Oracle Data Integrator - SDK Web Services - 11.1.1.0

    • Oracle Data Integrator - Console - 11.1.1.0

    • Oracle Data Integrator - Agent - 11.1.1.0

    Oracle Enterprise Manager and the Oracle Enterprise Manager Plug-in for ODI should also optionally be selected.

  3. On the Specify Domain Name and Location screen, enter the domain name and click Next.

  4. On the Configure Administrator Username and Password screen, enter a preferred username and password for the domain.

  5. On the Configure Server Start Mode and JDK screen, choose Production Mode. Click Next.

  6. In the Configure JDBC Component Schema screen:

    1. Select all component schemas that appear in the table at the bottom: ODI Master Schema and ODI Work Schema.

    2. For the RAC configuration for component schemas, select Convert to GridLink or Convert to RAC multi data source. For single database configuration, select Don't convert.

    3. Ensure that the following data source appears on the screen. The user names in Table 7-1 assume that you used DEV as the prefix for schema creation from RCU.

      For information about GridLink data source configuration with Oracle RAC, see Section 5.1.2, "GridLink Data Sources and Oracle RAC."

      For information about multi data source configuration with Oracle RAC and the MDS repository, see Section 5.1.3, "Using Multi Data Sources with Oracle RAC."

      Table 7-1 Configuring Values for Data Sources

      Component Schema Schema Owner

      ODI Master Schema

      DEV_ODI_REPO

      ODI Work Schema

      DEV_ODI_REPO


    4. Click Next.

  7. In the Configure JDBC Component Schemas screen, enter values for the following fields, specifying connect information for the Oracle RAC database that was seeded with RCU:

    For multi data sources, enter values for these fields:

    Figure 7-5 Configure RAC Multi Data Source Component Schema Screen

    Description of Figure 7-5 follows
    Description of "Figure 7-5 Configure RAC Multi Data Source Component Schema Screen"

    For GridLink data sources, enter values for these fields:

    Figure 7-6 Configure GridLink RAC Component Schema

    Description of Figure 7-6 follows
    Description of "Figure 7-6 Configure GridLink RAC Component Schema"

    1. Driver: For RAC: Select Oracle driver (Thin) for RAC Service-Instance connections, Versions:10, 11. For GridLink: Oracle Driver (Thin) for GridLink Connections, Versions:10 and later.

    2. Service Name: Enter the service name of the database. For a GridLink data source, you must enter the RAC service name in lower-case letters followed by the domain name example.com. For example, <mydbservice>.example.com

      Note:

      An Oracle RAC Service name is defined on the database; it is not a fixed name. Oracle recommends that you register/add the RAC service name with the database domain name.

    3. Username prefix: Enter the prefix for the schemas. The user names in Table 7-1 assume that DEV was used as prefix for schema creation from RCU.

    4. Password and Confirm Password: Enter the password to access the schemas.

    5. For a GridLink data source, enter the SCAN address in the Listener Address field and SCAN port in the Port field. Enter the ONS host and port information in the ONS Host and Port fields, respectively.

      Note:

      Oracle recommends that you use SCAN addresses to specify the host and port for both the TNS listener and the ONS listener in the WebLogic console. You do not need to update a GridLink data source containing SCAN addresses if you add or remove Oracle RAC nodes. Contact your network administrator for appropriately configured SCAN urls for your environment. See SCAN Addresses in the Oracle Fusion Middleware Configuring and Managing JDBC Data Sources for Oracle WebLogic Server guide.

      For a multi data source, click Add. Enter details for Oracle RAC instances.

    6. Update each schema: select one data source at a time and add the appropriate details.

      Ensure that you enter information for all schemas: ODI Master and ODI Work.

    7. Click Next.

  8. In the Test JDBC Data Sources screen, the connections are tested automatically. The Status column shows the results. Ensure that all connections were successful. If not, click Previous to correct your entries.

    Click Next when all the connections are successful.

  9. On the Select Optional Configuration screen, check the following:

    • Administration Server

    • Managed Server, Clusters and Machines

    • Deployments and Services

    Click Next.

  10. On the Configure Administration Server screen, set the Listen Address and Listen Port for the Administration Server.

  11. On the Configure Managed Servers page:

    • Click to add a new server: odi_server2

    • Set the Listen Address for each server to the hostname on which each server will run.

    • Ensure that both servers have the same Listen Port.

  12. On the Configure Clusters page, create a cluster named odi_cluster and accept the default of unicast.

  13. On the Assign Servers to Cluster page, add both servers to the cluster.

  14. On the Configure Machines page, create two machines.

  15. On the Assign Machines page:

    • Add Admin Server and odi_server1 to Machine 1.

    • Add odi_server2 to Machine 2.

  16. On the Configuration Summary page, click Create to create the domain.

7.4.2.4 Start the Administration Server

Start the WebLogic Administrator Server on APPHOST1 using this command:

DOMAIN_HOME/bin/startWebLogic.sh

7.4.2.5 Configure the Credential Store

You can configure the credential store using WLST or Enterprise Manager.

7.4.2.5.1 Configuring Credentials Using WLST

Run the following script:

MW_HOME/oracle_common/common/bin/wlst.sh

and then issue these commands:

connect('weblogic','welcome1','t3://localhost:7001')
createCred(map="oracle.odi.credmap", key="SUPERVISOR", user="SUPERVISOR",
password="SUNOPSIS", desc="ODI SUPERVISOR Credential")
exit()

The user above is the Supervisor user that was created when the repository was created using RCU. The password to specify is the password for that Supervisor user.

7.4.2.5.2 Configuring Credentials using Enterprise Manager

To configure credentials using Enterprise Manager, follow these steps:

  1. Log into Oracle Enterprise Manager and go to Domain > Security > Credentials to display the Credentials page.

  2. Click Create Map to display the Create Map dialog.

  3. In the Create Map dialog, enter the name of the map for the credential being created: oracle.odi.credmap

  4. Click OK to return to the Credentials page. The new credential map name is displayed with a map icon in the table.

  5. Click Create Key to display the Create Key dialog.

  6. In this dialog, select a map of oracle.odi.credmap, enter a key in the text box Key of 'SUPERVISOR', select a type from the menu Type (the appearance of the dialog changes according to the type selected), enter the required data:

    user="SUPERVISOR", password="your_password"
    
  7. Click OK when finished to return to the Credentials page. The new key is shown under the map icon corresponding to the map you selected.

7.4.2.6 Configure the Default Agent

Before you start the Oracle Data Integrator managed servers, create the agent definition for Oracle Data Integrator Agent. Follow these steps:

  1. Start the Oracle Data Integrator Studio at ODI_HOME/oracledi/client/odi.sh

  2. After Oracle Data Integrator Studio comes up, click Connect to Repository on the left-hand pane.

  3. A Login window appears. Click on the pencil icon to edit the Master Repository logon information.

  4. On the 'Repository Connection Information' screen, enter the following:

    ODI Connection:

    • Login Name: Master Repository

    • User: SUPERVISOR

    • Password: (The password used when creating the credential store)

    DB Connection:

    • User: (ODI Schema user such as DEV_ODI_REPO)

    • Password: (ODI Schema password)

    • Drive Name: oracle.jdbc.driver.OracleDriver

    • URL: (connection URL in the form jdbc:oracle:thin:@host:port:sid)

  5. Click Test to test the connection.

  6. Click OK to return to the logon screen.

  7. Ensure the User/Password are correct and then click OK to connect to the Master Repository.

  8. Once connected, open up the Physical Architecture section on the left-hand pane.

  9. Select Agents and right-click to create a New Agent.

  10. Create a new Agent with the following properties:

    • Name: OracleDI Agent

    • Host: (The host where odi_server1 will be running)

    • Port: (The port that the odi_server1 managed server is running on)

    • Web Application Context: oraclediagent

  11. Open up the DataSources tab. On the left hand pane, expand the Repositories section and drag the Work Repository from there into the Datasources area of the Agent.

  12. Edit the Work Repository datasource to add the JNDI name used by the datasource in the WebLogic Server installer. This should be jdbc/odiWorkRepository.

  13. Save and exit Oracle Data Integrator Studio.

7.4.2.7 Configure Coherence for the Cluster

You must configure coherence to enable communication among the cluster members. To configure Coherence:

  1. In the Administration Console, select Environment > Servers from the left hand tab.

  2. Select odi_server1.

  3. Click the Server Startup tab.

  4. In the Arguments box, enter the following (all on a single line):

    -Doracle.odi.coherence.wka1=machine1 -Doracle.odi.coherence.wka1.port=9088
    -Doracle.odi.coherence.wka2=machine2 -Doracle.odi.coherence.wka2.port=9088
    -Dtangosol.coherence.localport=9088
    

    where machine1 and machine2 are the hostnames of the two machines in the cluster.

    Note:

    Use 9088 as the Coherence port if it is unused on the machine. Otherwise, choose another port to configure as the Coherence port.

  5. Click Save.

  6. Repeat the steps above for odi_server2.

7.4.2.8 Configure Node Manager and Start odi_server1

Configure and start Node Manager using these commands:

$ MW_HOME/oracle_common/common/bin/setNMProps.sh
$ MW_HOME/wlserver_10.3/server/bin/startNodeManager.sh

Start odi_server1 as follows:

  1. In the Administration Console, choose Environment > Servers > Control tab.

  2. Check odi_server1 and click Start.

7.4.2.9 Verify the Oracle Data Integrator Agent is Running

Verify that the Oracle Data Integrator agent is running by entering the following URL in a web browser:

http://APPHOST1:PORT/oraclediagent

7.4.3 Installation and Configuration of the Second Oracle Data Integrator Host

This section describes the installation and configuration steps for the second Oracle Data Integrator host (for example, APPHOST2).

7.4.3.1 Installing the Oracle WebLogic Server on APPHOST2

Follow the instructions in Section 7.4.2.1, "Installing the Oracle WebLogic Server on APPHOST1."

Use the same directory paths as you used on APPHOST1.

7.4.3.2 Pack and Unpack the Domain from APPHOST1 to APPHOST2

Use the following commands to pack the domain that was created on the first host (APPHOST1):

$ cd MW_HOME/oracle_common/common/bin
$ ./pack.sh -domain=<full path to the domain> -template=mytemplate.jar 
-template_name=<descriptive template name> -managed=true

Copy the jar file you created on the first host to the second host (APPHOST2) and unpack it:

$ cd MW_HOME/oracle_common/common/bin
$ ./unpack.sh -domain=<full path to the domain> -template=mytemplate.jar

7.4.3.3 Configure Node Manager and Start odi_server2

Configure and start Node Manager on APPHOST2 using these commands:

$ MW_HOME/oracle_common/common/bin/setNMProps.sh
$ MW_HOME/wlserver_10.3/server/bin/startNodeManager.sh

Start odi_server2 as follows:

  1. In the Administration Console, choose Environment > Servers > Control tab.

  2. Check odi_server2 and click Start.

7.4.3.4 Verify the Oracle Data Integrator Agent is Running

Verify that the Oracle Data Integrator agent is running by entering the following URL in a web browser:

http://APPHOST2:PORT/oraclediagent

7.4.4 Installing Oracle HTTP Server

Install Oracle HTTP Server on another host (for example, WEBHOST1).

Ensure that the system, patch, kernel and other requirements are met. These are listed in the Oracle Fusion Middleware Installation Guide for Oracle Web Tier in the Oracle Fusion Middleware documentation library for the platform and version you are using.

Start the installer for Oracle Web Tier components.

HOST> ./runInstaller

When the installer prompts you for a JRE/JDK location, enter the Oracle SDK location created in the Oracle WebLogic Server installation, for example, /u01/app/product/fmw/jrockit_160_<version>.

Then perform these installation steps:

  1. On the Prerequisite Checks screen, verify that the checks complete successfully, then click Next.

  2. On the Specify Installation Location screen, enter the following values:

    MW_HOME: Enter the value of the MW_HOME, for example:

    /u01/app/product/fmw
    

    Select the previously installed Middleware Home from the drop-down list. For the Oracle HTTP Server Oracle Home (OHS_ORACLE_HOME) directory, enter the directory name WEB.

    Click Next.

  3. On the Summary screen, click Install.

    When prompted, on Linux and UNIX installations, run the script oracleRoot.sh as the root user.

  4. On the Installation Complete screen, click Finish.

Repeat these steps to install the Oracle HTTP Server on another web tier host (for example, WEBHOST2).

7.4.4.1 Upgrading the Oracle HTTP Server Oracle Home

This section provides the steps to upgrade your Oracle HTTP Server software. For more information, see "Applying the Latest Oracle Fusion Middleware Patch Set" in the Oracle Fusion Middleware Patching Guide.

To upgrade the OHS_ORACLE_HOME:

  1. Start the Web Tier Upgrade Installer by running ./runinstaller.

  2. On the Welcome screen, click Next.

  3. On the Prerequisite Checks screen, click Next.

  4. On the Specify Install Location screen, provide the path to the Oracle Middleware Home and the name of the Oracle HTTP Server Oracle Home directory.

  5. On the Installation Summary screen, validate your selections, and then click Install.

  6. The Installation Progress screen shows the progress of the install.

    Once the installation is done, the oracleRoot.sh confirmation dialog box appears. This dialog box advises you that a configuration script needs to be run as root before the installation can proceed. Leaving the confirmation dialog box open, open another shell window, log in as root, and run this script file: /u01/app/oracle/product/fmw/id/oracleRoot.sh. After the script completes, click OK on the Confirmation Dialog box.

  7. On the Installation Complete screen, click Finish to exit.

These steps should be performed on each of the web tier hosts on which Oracle HTTP Server is installed (for example, WEBHOST1 and WEBHOST2).

7.4.4.2 Configuring Oracle HTTP Server

After the installation add the following lines to the OHS_HOME/instances/ohs_instance1/config/OHS/ohs11/mod_wl_ohs.conf file on the host or hosts on which you installed Oracle HTTP Server:

# ODI Agent
<Location /oraclediagent> 
    WebLogicCluster host1:8002,host2:8002 
    SetHandler weblogic-handler 
</Location> 
 
#ODI Explorere
<Location /odirepex> 
    WebLogicCluster host1:8002,host2:8002 
    SetHandler weblogic-handler 
</Location> 
 
#ODI Webservices
<Location /oracledisdkws> 
    WebLogicCluster host1:8002,host2:8002 
    SetHandler weblogic-handler 
</Location>

These steps should be performed on each of the web tier hosts on which Oracle HTTP Server is installed (for example, WEBHOST1 and WEBHOST2).

7.4.4.3 Configuring the Load Balancer

The load balancer should be configured to round-robin requests across the two HTTP Servers on WEBHOST1 and WEBHOST2.

Also, follow these load balancer configuration recommendations:

  • Persistence should not be enabled on the load balancer. This is provided by the HTTP Servers.

  • Monitors should be configured to monitor the HTTP Server ports so that the load balancer can provide failover.

7.4.4.4 Verify the Oracle Data Integrator Agent is Running

Verify that the Oracle Data Integrator agent is running by entering the following URL in a web browser:

http://LBRHOST:PORT/oraclediagent

7.4.4.5 Reconfigure Agents

Agent definitions should point to the load balancer address instead of the individual server addresses.

This should be kept in mind when creating new agents.

For the default agent, connect to Oracle Data Integrator Studio again:

  1. Start Oracle Data Integrator Studio at ODI_HOME/oracledi/client/odi.sh.

  2. After Oracle Data Integrator Studio comes up, click on Connect to Repository on the left-hand pane.

  3. When the Login window appears, click OK to logon.

  4. When you are connected, open the Physical Architecture section on the left-hand pane.

  5. Select Agents and then select the OracleDIAgent.

  6. Edit the following properties:

    • Host: The load balancer virtual server address.

    • Port: The load balancer virtual address listening port.

  7. Click Test to test the agent connection.

  8. Save and exit Oracle Data Integrator Studio.