What's New In Oracle Data Integrator?

This document describes the new and enhanced features introduced with Oracle Data Integrator 11g Release 1 (11.1.1).

This chapter includes the following sections:

Release Themes

While the new features of Oracle Data Integrator for this release cover a number of different areas, the most important changes for new and existing customers are:

New Architectures Supported for Enterprise-Scale Deployment Options

Oracle Data Integrator now provides several deployment options for lightweight standalone deployments and enhanced architectures for deployments based on cluster-able and fault tolerant application server frameworks. Features in this area include:

Core Design-Time Features for Enhanced Productivity and Performance

Oracle Data Integrator now provides a set of core features for increasing development productivity and the performance of the integration flows. Features in this area include:

Standard JDeveloper-Based IDE: Oracle Data Integrator Studio

The Oracle Data Integrator User Interface now uses the JDeveloper-based integrated development environment (IDE), and is renamed Oracle Data Integrator Studio.The user interface has been entirely redesigned in this release to improve developer productivity and make advanced features more accessible. This new IDE provides the following key features:

Developer Usability and Productivity Enhancements

In addition to the entire redesign of the development interface, features have been added to improve the developer's experience and productivity while working in Oracle Data Integrator. Features in this area include:

New Features for Administration

Features have been added to improve manageability of the Oracle Data Integrator components and sessions. Features in this area include:

Enhanced Diagnostic Features and Capabilities

Oracle Data Integrator has been improved with features to facilitate problems troubleshooting and fixing. Features in this area include:

Technologies and Knowledge Modules Enhancements

New technologies and knowledge modules planned for this release have been continuously delivered during the 10g lifecycle patch sets. In addition, existing knowledge modules and technologies have been enhanced to support new core product features and diagnostics.

Features added in 10g Release 3 patch sets include:

Features added for this release include:

New Features

Release 11.1.1 includes many new features. These features are listed below and grouped in the followings component and functional areas:

Runtime Agent

Oracle Data Integrator runtime agent has been enhanced with the features listed in this section.

Java EE Agent

The Runtime Agent can now be deployed as a Java EE component within an application server. It benefits in this configuration from the application server layer features such as clustering and connection pooling for large configurations. This Java EE Agent exposes an MBeans interface enabling lifecycle operations (start/stop) from the application server console and metrics that can be used by the application server console to monitor the agent activity and health.

Standalone Agent

In addition to the Java EE Agent, a Standalone Agent, similar to the one available in previous Oracle Data Integrator releases, is still available. It runs in a simple Java Virtual Machine and can be deployed where needed to perform the integration flows.

Connected Scheduler

Both agent flavors are now always connected to a master repository, and are started with the built-in scheduler service. This scheduler service takes its schedules from all the Work Repositories attached to the connected Master.

HTTP Protocol for Component Communication

Communications with the run-time agents (for example, when sending an execution request to a remote agent) now use standard HTTP protocol. This feature facilitates network management and security for Oracle Data Integrator components in distributed environments.

Oracle WebLogic Server Integration

Oracle Data Integrator components integrate seamlessly with Oracle's Java EE application server.

Java EE Agent Template Generation

Oracle Data Integrator provides a wizard to automatically generate templates for deploying Java EE agents in Oracle WebLogic Server. Such a template includes the Java EE Agent and its configuration, and can optionally include the JDBC datasources definitions required for this agent as well as the drivers and libraries files for these datasources to work.

By using the Oracle WebLogic Configuration Wizard, domain administrators can extend their domains or create a new domain for the Oracle Data Integrator Java EE runtime agents.

Automatic Datasource Creation for WebLogic Server

Java EE Components use JDBC datasources to connect to the repositories as well as to the source and target data servers, and benefit, when deployed in an application server, from the connection pooling feature of their container.

To facilitate the creation of these datasources in the application server, Oracle Data Integrator Studio provides an option to deploy a datasource into a remote Oracle WebLogic application server.

Pre-Packaged WebLogic Server Templates for Java EE Components

Oracle Data Integrator Java EE components that can be deployed in an application server are provided now with pre-packaged templates for Oracle WebLogic Server. Oracle Data Integrator provides templates for:

These templates are used to create a WLS domain for Oracle Data Integrator or extend an existing domain with Oracle Data Integrator components.

Web Services

Oracle Data Integrator web services support has been enhanced with the features listed in this section.

JAX-WS Support for Web Services

Oracle Data Integrator Web Services - including the Public Web Service as well as the generated Data Services - now support the market standard Java API for XML Web Services (JAX-WS 2.0). As a consequence, they can be deployed into any web service container that implements this API. The use of the Axis2 stack for these web services is deprecated.

Web Services Changes and Reorganization

The web services have been reorganized and the following run-time web operations are now part of the run-time agent application:

The Public Web Service application retains the following operations:

Advanced Security Capabilities

Security in Oracle Data Integrator can be hardened with the enterprise features listed in this section.

External Password Storage

Source and target data server passwords, as well as the context passwords, can optionally be stored in an external credential store instead of storing them in an encrypted form in the master repository. This credential store is accessed via the Java Platform Security (JPS) Credential Store Framework (CSF). The password storage method (internal or external with JPS) is defined at repository creation, and can be switched for existing repositories.

With this password storage approach, administrators can choose to rely on a corporate credential store for securing their data server passwords.

External Authentication and SSO

Oracle Data Integrator users can be authenticated using an external authentication service. Using Oracle Platform Security Services (OPSS), Oracle Data Integrator users authenticate against an external Enterprise Identity Store (LDAP, Oracle Internet Directory, Active Directory), which contains in a central place enterprise user and passwords.

With this feature, the master repository retains the Oracle Data Integrator-specific privileges and the user names, but passwords rely in a centralized identity store, and authentication always takes place against this external store. The authentication mode (internal or external) is defined at repository creation, and can be switched for existing repositories.

This feature enables Single Sign-On (SSO) for Oracle Data Integrator Console, and seamless authentication integration between Enterprise Manager and Oracle Data Integrator Console.

Default Password Policy

Oracle Data Integrator is now installed with a default password policy that prevents from setting passwords with a low security level.

Java EE Components Passwords in Credential Store

When deploying in Oracle WebLogic Server a Java EE component that requires a bootstrap connection to a repository (Java EE Agent, Oracle Data Integrator Console), the configuration of this component contains a Supervisor user login. To enforce a strong security policy this user's password is not stored within the application configuration, but centralized in the WLS Credential Store. The configuration will refer to this centralized store.

Production and Monitoring

Oracle Data Integrator provides new features for an enhanced experience in production.

Enhanced Error Messages

Error messages raised by Oracle Data Integrator Components and Sessions have been enhanced to provide administrators and production operators with precise information for troubleshooting and fixing the status of the architecture, and debugging the sessions. Enhanced messages cover:

These error messages are standardized with Oracle Data Integrator error codes.

Enhanced Notifications and Logging

Oracle Data Integrator components are now using the Oracle Logging Framework. Logging in any component can be configured to meet the requirements of development, test and production environments.

In addition to this logging capability, agent components can now raise status and session information in the form of Java Management Extension (JMX) notifications that propagate to any administration console.

Error Tables

Error tables can now be managed via Oracle Data Integrator Console. Production operators can review the content of the error tables and purge their content selectively.

Purge Log on Session Count

The OdiPurgeLog tool has been enhanced to support a purge of the log while retaining only a number of sessions in the log. Purged sessions can be automatically archived by the tool before performing the purge.

New Oracle Data Integrator Console

The Metadata Navigator UI has been replaced with the Oracle Data Integrator Console. This web interface for production operations has been rewritten using the ADF-Faces Ajax Framework for a rich user experience. Using this console, production users can set up an environment, export/import the repositories, manage run-time operations, monitor the sessions, diagnose the errors, browse through design-time artifacts, and generate lineage reports.

This web interface integrates seamlessly with Oracle Fusion Middleware Control Console and allows Fusion Middleware administrators to drill down into the details of Oracle Data Integrator components and sessions.

Oracle Fusion Middleware Control Console Integration

Oracle Data Integrator provides an extension integrated into the Oracle Fusion Middleware Control Console. The Oracle Data Integrator components can be monitored as a domain via this console and administrators can have a global view of these components along with other Fusion Middleware components from a single administration console.

This extension discovers Oracle Data Integrator components and allows administrators to:

Kill Sessions Immediate

Sessions can now be stopped in an immediate mode. This new mode attempts to abort the current operation (for example, SQL statements launched against a database engine) instead of waiting for its completion.

High Availability

For an enterprise scale deployment, the features enable high available of the Oracle Data Integrator components.

Stale Session Detection and Management

Oracle Data Integrator is now able to detect sessions pending due to an unexpected shutdown of the agent or repository. Such stale session are now managed and pushed to an error state.

Repository Connection Retry

The Agent, when connected to a repository based on Oracle RAC technology, can be configured with connection retry logic. If the one of the Oracle RAC nodes supporting sessions for an agent becomes unavailable, the agent is able to retry and continue its session on another node of the Oracle RAC infrastructure.

Support for WLS Clustering

Clustering is supported for the Java EE agents deployed on a WebLogic Server. Clustering includes schedule porting on a different cluster node. Unrecoverable running sessions are automatically moved to an error state.

OPMN Integration

Standalone agent can be now made highly available using Oracle Process Manager and Notification Server (OPMN). Scripts are provided to configure OPMN to protect standalone agents against failure.

Improved Integration Design

Integration interface design and performance are enhanced with the following features.

Partitioning

Oracle Data Integrator now supports partitioning features of the data servers. Partitions can be reverse-engineered using RKMs or manually created in models. When designing an interface, it is possible to define the partition to address on the sources and target datastores. Oracle Data Integrator code generation handles the partition usage syntax for each technology that supports this feature.

Lookups

A wizard is available in the interface editor to create lookups using a source as the driving table and a model or target datastore as the driving table. These lookups now appear as a compact graphical object in the Sources diagram of the interface. The user can choose how the lookup is generated: as a Left Outer Join in the FROM clause or as an expression in the SELECT clause (in-memory lookup with nested loop). This second syntax is sometimes more efficient on small lookup tables.

This feature simplifies the design and readability of interfaces using lookups, and allows for optimized code for executing lookups.

Datasets and Set-Based Operators

This major enhancement introduces the notion of dataset in interfaces. A dataset represents the data flow coming from a group of joined and filtered source datastores. Each dataset includes the target mappings for this group of sources. Several datasets can be merged into the interface target datastore using set-based operators such as Union and Intersect.

This feature accelerates the interface design and reduces the number of interfaces needed to merge several data flows into the same target datastore.

Derived Select for Temporary Interfaces

When using a temporary interface as a source or a lookup table in another interface, you can choose not to persist the target of the temporary interface, and generate instead a Derived Select (sub-select) statement corresponding to the loading of the temporary datastore. Consequently, the temporary interface no longer needs to be executed to load the temporary datastore. The code generated for the sub-select is either a default generated code, or a customized syntax defined in an IKM.

This feature eliminates the need for complex packages handling temporary interfaces and simplifies the execution of cascades of temporary interfaces.

Support for Native Sequences

Oracle Data Integrator now provides support for a new type of sequence that directly maps to database-defined sequences. When created, these can be picked from a list retrieved from the database. Native Sequences are used as regular Oracle Data Integrator sequences, and the code generation automatically handles technology-specific syntax for sequences.

This feature simplifies the use of native sequences in all expressions, and enables cross references when using such sequences.

Support for Natural Joins

Oracle Data Integrator now provides support for the Natural join, defined at technology level. This join does not require any join expression to be specified, and is handled by the engine that processes it. This engine matches automatically columns with the same name.

Automatic Temporary Index Management

When creating joins or filters on source tables, it is possible to have Oracle Data Integrator automatically generate temporary indexes for optimizing the execution of these joins or filters. The user selects the type of index that needs to be created in the list of index types for the technology. Knowledge modules automatically generate the code for handling indexes creation before executing the join and filters as well as deletion after usage.

This feature provides automated optimization of the joins and filters execution, and enables better performances for integration interfaces.

New Interface Editor

The interface editor, used to create the integration interfaces, has been entirely redesigned to use the JDeveloper diagramming framework.

The advantages of this new diagram include:

Quick-Edit

The new interface editor includes a new Quick-Edit tab to edit the interface diagram faster. Quick-Edit displays these components in a tabular form, supports mass-updates and intuitive keyboard navigation.

Auto fixing

When saving an interface or clicking the error button from the interface editor toolbar, a list of all the design errors in the interface is displayed with meaningful messages and tips. Automated fixes are suggested and can be applied with a single click.

Code Simulation

When performing an execution of design-time objects from the Oracle Data Integrator Studio (for example, when running an interface, a procedure, a package or a customized reverse-engineering process), it is possible to make a code simulation instead of a full execution.

Code simulation displays a session simulation report. This report includes complete session, step, and task information and contains the full generated code. The session simulation report can be reviewed and saved in XML or HTML format.

With this features Oracle Data Integrator developers can easily review the generated code for troubleshooting, debugging, optimization purposes, and save this generated code for documentation or archive purposes.

Reverse-Engineering Improvements

When a model is created the reverse-engineering context is automatically set to the default context, instead of having to select it manually. In addition, when performing a selective reverse-engineering, the system tables are now hidden from the display.

Scenario Naming Convention

When generating a scenario or a group of scenarios from the Studio or using a tool, the naming convention that is used for naming the scenario can be defined in a pattern (using the object name, folder path or project name) using the Scenario Naming Convention user parameter.

Object Name Length Extension

Object names have been extended to support long database object names (128 characters) and repository object labels (400 characters).

Oracle Data Integrator Java API

Oracle Data Integrator provides a Java API for managing run-time and design time artifacts. Using this API, Java developers can embed Oracle Data Integrator in their product and can drive integration process creation from their own user interface.

Oracle Data Integrator Studio

Oracle Data Integrator provides a new IDE called the Studio, based on JDeveloper. This component includes the following features:

New Navigator Organization

The new Oracle Data Integrator studio is used as a replacement for all Oracle Data Integrator modules (Designer, Topology, Operator and Security Manager). All the features of these modules now appear as Navigators within the Oracle Data Integrator Studio window.

This new Navigator organization provides the following features:

This feature provides a single user interface from which the user can perform all the tasks in a project lifecycle. It also provides a better productivity for the user.

New Look and Feel

The look and feel of Oracle Data Integrator has been enhanced with the use of the JDeveloper base IDE. This new look and feel is customizable with the Preferences menu option. Icons are being redesigned in a new, trendy style to enhance the overall visual appeal of Oracle Data Integrator

Redesigned Editors

All object editors in Oracle Data Integrator have been redesigned for better usability.

Main changes include:

Window Management

The windows, editors and navigators in the Oracle Data Integrator Studio benefit from the following JDeveloper IDE features:

Document Management and Navigation

Object edition has been enhanced in the Oracle Data Integrator Studio with improved document management. This includes:

Improved User Assistance

Oracle Data Integrator introduces intuitive new features that improve usability:

Export/Import

Export/import is enhanced in this new release with the following features:

Import Report

After objects have been imported, an import report displays the objects that have been imported or deleted in the target repository. In addition, missing objects referenced by the imported objects are indicated as missing references, and missing references fixed by the import are also indicated. Import reports can be saved in XML or HTML format

With this feature, importing objects becomes a very transparent operation as all changes can be identified and archived.

Repository Corruption Prevention

When importing objects across repositories, the following cases have been taken into account to avoid the risks of import errors and repository corruption:

In addition, import of objects that reference non-existing objects now create missing references, identified in the import report. Such references can be resolved by importing the missing object.

Repository Renumbering

It is now possible to change the identifier of a master or work repository after its creation. This operation automatically updates the internal identifier of the objects created in this repository to match the new identifier.

This feature facilitates configuration management and fixing import/export situations when multiple repositories have been created with the same identifier.

KM Enhancements in Release 10.1.3 Patch sets

The following improved and new knowledge modules have been delivered in 10gR3 patch sets and are available in this release.

Oracle GoldenGate Knowledge Modules

Oracle Data Integrator uses Oracle GoldenGate to replicate online data from a source to a staging database. A Journalizing KM manages the Oracle Data Integrator CDC infrastructure and automatically generates the configuration for Oracle GoldenGate.

Oracle E-Business Suite Knowledge Modules

Oracle Data Integrator Knowledge Modules for E-Business Suite provide comprehensive, bidirectional connectivity between Oracle Data Integrator and E-Business Suite, which enables you to extract and load data. The Knowledge Modules support all modules of E-Business Suite and provide bidirectional connectivity through EBS objects tables/views and interface tables.

Oracle OLAP Knowledge Modules

The Oracle Data Integrator Knowledge Modules for Oracle OLAP provide integration and connectivity between Oracle Data Integrator and Oracle OLAP cubes. The Oracle Data Integrator KMs for Oracle OLAP support reverse-engineering of Oracle OLAP data structures (all tables used by a ROLAP or a MOLAP cube) and data integration in an Oracle Analytical Workspace target in incremental update mode.

Oracle PeopleSoft Knowledge Modules

The Oracle Data Integrator Knowledge Modules for PeopleSoft provide integration and connectivity between Oracle Data Integrator and the PeopleSoft platform. These KMs enable Data-level integration for PeopleSoft and support reverse-engineering of PeopleSoft data structures (Business Objects, tables, views, columns, keys, and foreign keys) and data extraction from PeopleSoft.

Oracle Siebel Knowledge Modules

The Oracle Data Integrator Siebel Knowledge Modules support reverse-engineering Siebel data structures (Business Components and Business Objects) and Enterprise Integration Manager (EIM) tables, data extraction from Siebel using data-level integration and data extraction and integration with Siebel using the EIM tables

JDE EnterpriseOne Knowledge Modules

The Oracle Data Integrator Knowledge Modules for JDE EnterpriseOne provide connectivity and integration of the JDE EnterpriseOne platform with any database application through Oracle Data Integrator. These KM support reverse-engineering of JDE EnterpriseOne data structures, data extraction from JDE EnterpriseOne (Direct Database Integration) and integration through the Z-tables to an JDE Application (Interface Table Integration)

Oracle Changed Data Capture Adapters/Attunity Streams Knowledge Modules

The Oracle Data Integrator CDC Knowledge Module provides integration from Oracle Changed Data Capture Adapters/Attunity Streams Staging Areas via a JDBC interface. This KM reads changed data, loads this data into a staging area and handles the Oracle Changed Data Capture Adapters/Attunity Streams context to ensure consistent consumption of the changes read.

Hyperion Adapters

Knowledge Modules and technologies have been added for integrating the Hyperion technologies using Oracle Data Integrator.

These KMs support the following Hyperion products:

Row-By-Row KMs for Debugging

Knowledge modules supporting row-by-row loading (LKM SQL to SQL (row by row)) and integration (IKM SQL Incremental Update (row by row)) have been introduced for debugging purposes. These KMs allow logging of each row operation performed by the KM.

Teradata Optimizations

Teradata knowledge modules have been enhanced for Teradata to enable best performances.

This includes the following features:

SAP ERP Adapter

The SAP ERP Adapter allows extraction of data from SAP ERP systems. The Oracle Data Integrator SAP ABAP Knowledge Modules included in this adapter provide integration from SAP ERP systems using SAP JCo libraries and generated ABAP programs.

SAP BW Adapter

The SAP BW Adapter allows extraction of data from SAP BW systems. The Oracle Data Integrator SAP BW Knowledge Modules included in this adapter provide integration from SAP BW using SAP JCo libraries and generated ABAP programs. This adapter supports ODS, Info Objects, Info Cubes, Open Hub and Delta Extraction.

KM Enhancements in Release 11.1.1

The following knowledge modules enhancements are new to this release.

KM Enhancements for New Core Features

Knowledge modules have been enhanced to support the core features added in this version of Oracle Data Integrator. The following KMs have been updated to support these features:

Oracle Business Intelligence Enterprise Dietitian - Physical

Oracle Data Integrator provides the ability to reverse-engineer View Objects that are exposed in Oracle Business Intelligence Enterprise Edition (OBI-EE) physical layer. These objects can be used as sources of integration interfaces.

Oracle Multi-Table Inserts

A new Integration KM for Oracle allows populating several target tables from a single source, reading the data only once. It uses the INSERT ALL statement.

Teradata Multi-Statements

A new Teradata Integration KM provides support for Teradata Multi-Statements, allowing integration of several flows in parallel.