This section summarizes the new features and significant product changes for Oracle Data Integrator (ODI) in the Oracle Fusion Middleware 12c release.
This chapter includes the following sections:
New and Changed Features for Release 12c (126.96.36.199.1)
Oracle Data Integrator 12c (188.8.131.52.1) introduces new functionality in the following areas:
These key investment areas ensure that Oracle Data Integrator (ODI) will continue to accompany customers throughout their technological transformation and modernization process.
Oracle Object Storage and Oracle Object Storage Classic
Oracle Object Storage and Object Storage Classic offers fast, reliable and secure cloud storage and now Oracle Data Integrator can seamlessly integrate with them on Oracle Cloud Infrastructure (OCI).
ODI comes with a set of Knowledge Modules (KMs) and ODI Tools that can used in Mappings and Packages to connect to Oracle Object Storage and Object Storage Classic for uploading, downloading and deleting files/objects onto/from local directory or the Hadoop Distributed File System (HDFS).
Autonomous Databases: Autonomous Data Warehouse and Autonomous Transaction Processing
Oracle Data Integrator now comes with optimized Loading and Integration Knowledge Modules (KMs) that are certified with Oracle Autonomous databases:
Oracle Autonomous Data Warehouse Cloud (ADWC)
Oracle Autonomous Transaction Processing (ATP)
Oracle Data Integrator seamlessly integrates with ADWC and ATP. By integrating ODI with Autonomous databases, you can get the full performance of Oracle Data Integrator and Oracle databases, in a fully-managed environment that is tuned and optimized for various workloads. The same set of Knowledge Modules are used for both ADWC and ATP and also leverage the new native integration with Oracle Object Storage and Oracle Object Storage Classic.
In addition to being able to load data directly into ADWC or ATP, Oracle Data Integrator users can benefit from the native integration between Oracle Autonomous Data Warehouse and Oracle Object Storage to enable extremely fast data transfer into ADWC or ATP. ODI can then automate the complete loading process of Oracle Autonomous Data Warehouse Cloud (ADWC) and Oracle Autonomous Transaction Processing (ATP).
Oracle Enterprise Resource Planning (ERP) Cloud
This release also adds a new Technology and Knowledge Modules for Oracle Enterprise Resource Planning (ERP) Cloud, a suite of cloud applications for finance, project management, procurement, risk management and other core day-to-day activities important in every business, regardless of size, industry or geography. Oracle Data Integrator (ODI) seamlessly integrates with Oracle Enterprise Resource Planning (ERP) Cloud and helps organizations integrate their ERP data into their data warehouses, data marts or data lakes. This native integration also allows ODI users to load data into Oracle ERP Cloud.
For more details, see Oracle Enterprise Resource Planning Cloud Documentation.
New and Changed Features for Release 12c (184.108.40.206.0)
Oracle Data Integrator 12c (220.127.116.11.0) introduces the following enhancements:
Cloud and Big Data remain key investment areas and ensure that Oracle Data Integrator will continue to accompany customers throughout their technological transformation and modernization process.
Big Data continues to evolve within Oracle Data Integrator with advances including:
Spark Knowledge Modules Improvements — In this release, the focus has been on generating high performing and easily readable Spark code that will stand up to any handwritten scripts. In addition, the Spark Knowledge Modules (KMs) are now leveraging the latest Apache Spark 2.x features such as Dataframes. SparkSQL is also leveraged where applicable to speed up the ODI processes running on Spark.
Spark KMs Support in Knowledge Module Editor — The Spark KMs are now fully supported in the Knowledge Module editor and can be tailored to your own specific requirements.
Hadoop Complex Types Enhancements — ODI 18.104.22.168.6 introduced Complex Types support for Hive and HDFS (batch mode) technologies. ODI 22.214.171.124.0 further improves this functionality for Spark Streaming with Complex Types support in Apache Kafka as well as in HDFS.
Big Data Configuration Wizard — The Big Data Configuration Wizard has been improved with newer templates for the latest Cloudera’s Distribution including Apache Hadoop (CDH) releases.
Cloud improvements include:
Certification with Salesforce.com — Oracle Data Integrator is fully certified with Salesforce.com and now includes a JDBC driver for this technology out of the box.
New and Changed Features for Release 12c (126.96.36.199.6)
Oracle Data Integrator 12c (188.8.131.52.6) introduces the following enhancements:
Spark Streaming Support — Oracle Data Integrator (ODI) now supports Spark Streaming to fully enable the creation of Big Data streaming jobs easily without requiring end users to write a single line of code. In addition to Spark Streaming ODI already supports Hive, Pig and batch Spark when it comes to data processing. Through its unique decoupling of the Logical and Physical design of Mappings, Oracle Data Integrator is the only Data Integration tool on the market giving developers the flexibility to design Mappings with a generic business logic and then generate code for as many data processing technologies (Hive, Spark, Spark Streaming, etc.) as they want. This unique capability also helps future-proof Data Integration processes.
Support for Apache Kafka and Apache Cassandra — Apache Kafka and Cassandra are certified with the latest version of Oracle Data Integrator as both sources and targets.
Hadoop Complex Types and Storage Format— This release further extends the market leading Hadoop support in ODI with the ability to natively access data stored in various formats such as Avro, Parquet or JSON. In addition, new features were added to leverage complex types or nested types in Mappings such as Array, Struct or Map.
Enhancements to Big Data Configuration Wizard— The Big Data Configuration Wizard introduced with ODI 184.108.40.206.0 has been improved to support new Hadoop technologies such as Kafka and Cassandra. It also now helps users configure Oracle Data Integrator with Hadoop clusters secured with Kerberos.
RESTful Service Support— Oracle Data Integrator can now invoke RESTful Service. A RESTful Service connectivity, resource URL, methods and parameters can be configured in Topology configurations like any other data source connectivity. There are a number of parameters supported providing maximum flexibility to support widespread RESTful services. Data chunking and pagination are also supported for uploading or downloading larger payloads.
Business Intelligence Cloud Service (BICS) Knowledge Modules — Business Intelligence Cloud Service is now supported out of the box in Oracle Data Integrator. You can define Business Intelligence Cloud Service connectivity in Topology, reverse engineer metadata and load data into it just like any other target data server.
Git Support and Improvements to Lifecycle Management functionality — In addition to Apache Subversion, Oracle Data Integrator now also supports Git as an external version control system. A unified user experience is provided when using either of the supported version control systems. There are many advanced operations added for improved lifecycle management needs. You can now view all the pending changes and create version for set of objects from a single place. You can regenerate scenarios while creating versions or creating deployment archives ensuring scenarios always corresponds to the current object version. You can selectively populate objects from the branch or a tag. The SDK APIs are also enhanced to support various operations needed for continuous integration.
Enhanced Merge Capability — The merge capabilities have been enhanced to auto-merge changes based upon three-way merge and object change detection. Conflict resolution is simplified with brand new support to pick and choose properties or objects from the source or target in the Merge user interface.
Enhanced Knowledge Module Framework — Exciting new features have been added to the Knowledge Modules development framework to maximize flexibility and minimize maintenance. You can now inherit steps from a Knowledge Module into another Knowledge Module and override steps like in object-oriented programming languages. Brand new template languages and syntaxes have been introduced providing greater control over the generated code. Furthermore, several other productivity enhancements were added such as syntax highlighting, auto-complete, folding code blocks and more.
New and Changed Features for Release 12c (220.127.116.11)
Oracle Data Integrator 12c (18.104.22.168) introduces the following enhancements:
Hyperion Essbase and Hyperion Planning Knowledge Modules
Hyperion Essbase and Hyperion Planning Knowledge Modules are now available out of the box with Oracle Data Integrator and support the latest version (22.214.171.124) of these Hyperion applications.
Integration Capture and Delivery support in GoldenGate Knowledge Modules
The GoldenGate Journalization Knowledge Modules for Oracle databases have been updated and now support Integrated Capture and Delivery. These new capabilities can improve performance and provide better scalability and load balancing.
For more information, see the "Oracle GoldenGate" section in Connectivity and Knowledge Modules Guide for Oracle Data Integrator.
ETL enhancements to support Cubes and Dimensions
Oracle Data Integrator introduces support for two types of dimensional objects - Cubes and Dimensions. Users can now create and use Cubes and Dimensions objects directly in Mappings and reduce their development costs with out of the box patterns that automate the loading of dimensional objects.
For more information, see the "Creating and Using Dimensions and Cubes" section in Developing Integration Projects with Oracle Data Integrator.
Big Data Configuration Wizard
A brand new Big Data Configuration Wizard is now available in the ODI Studio Gallery and provides a single entry point to configure the Topology objects for Hadoop technologies such as Hadoop, Hive, or Spark.
For more information, see the "Configuring Big Data technologies using the Big Data Configuration Wizard" section in Integrating Big Data with Oracle Data Integrator.
New and Changed Features for Release 12c (12.2.1)
Oracle Data Integrator 12c (12.2.1) introduces the following enhancements:
Big Data Enhancement: Oozie Execution Modes
You can now choose between Task and Session execution modes for Oozie workflow generation. The new Session mode allows support for transactions, scripting, and loops in packages. ODI will automatically choose the correct mode based on the executed object or the mode can also be manually selected.
Please note that Big Data enhancements were made available in ODI 126.96.36.199.1 and improved upon in this release. For more information regarding Big Data enhancements, see "New and Changed Features for Release 12c (188.8.131.52.1)".
Lifecycle Management of ODI Objects
Oracle Data Integrator is now integrated with Subversion and this provides you with the ability to version control ODI Objects in Subversion. For more information on Subversion, visit the Subversion website.
Using the Subversion integration capabilities, you can create tags to take a snapshot of ODI object versions. You can create branches for parallel development from distributed locations or for parallel development for multiple releases.
Release management capabilities are introduced to provide a distinction between the development and deployment environments. You can create Deployment Archives (DA) from a development environment, which can be deployed in a QA environment for testing, and then delivered to the production environment. The DA can be created using ODI Studio or from a command line.
ODI Exchange for Sharing Global ODI Objects
You can now browse, download, and install global ODI objects made available to you by Oracle or other ODI users through Official or Third-Party Update Centers. This feature is available for Global Knowledge Modules, Global User Functions and Mapping Components. The Check for Updates menu item in the Help menu in ODI Studio enables you to connect to the Update Centers and obtain Global ODI Objects.
Complex File Enhancements
The Native Format Builder utility is now included with ODI Studio and allows you to create nXSD files without leaving the ODI user interface.
Complex File, File, LDAP, JMS Queue XML, JMS Topic XML, and XML Technology Enhancements
All JDBC properties for Complex File, File, LDAP, JMS Queue XML, JMS Topic XML, and XML technologies are now displayed at the Data Server level along with default values where applicable and a description of the properties, thereby enhancing usability.
Pre/Post Processing for XML and Complex JDBC Drivers
You can now customize the way data is fed into XML and Complex File drivers. This feature adds support for intermediate processing stages that may be added for processing data as it has either been retrieved from the external endpoint using Oracle Data Integrator, or it is to be written out to an external endpoint. This feature also provides support for complex configuration of intermediate processing stages as part of the configuration of data servers that use ODI XML or Complex File JDBC drivers.
Improved Web Service Support
A new SOAP Web Service technology is now available in Topology and allows the creation of data servers, physical schemas, and logical schemas for Web Services. Oracle Web Service Management (OWSM) policies can also be attached to Web Services data servers. In addition, the OdiInvokeWebService tool is enhanced to support Web Services data servers through Contexts and Logical Schemas.
Ability to Cancel Import/Export and Reverse Engineering Operations
You can now cancel import/export and reverse-engineering operations that may run for a long time.
Support for Analytic or Window Functions
Analytic or Window functions are now supported out of the box at the Mapping level. Analytic functions such as PERCENT_RANK, LAST, FIRST, or LAG can be used at the Mapping Expression level in any component.
Oracle Connectivity Enhancements
A new Knowledge Module to perform Partition Exchange Loading is now available allowing you to swap partitions when needed. In addition, improvements have been made to Loading Knowledge Modules using External Tables, which can now load more than one file at a time. Knowledge Modules using Data Pump have also been improved.
Enhanced Integration with Oracle Enterprise Data Quality
A new Oracle Enterprise Data Quality (EDQ) technology is available in Topology and allows the creation of data servers, physical schemas, and logical schemas for EDQ. Also, the OdiEnterpriseDataQuality tool is enhanced to support EDQ data servers through Contexts and Logical Schemas.
Ability to View a List of Users Connected to Studio/Repository
The Review User Activity menu item has been added to the Security menu. Using this you can view, purge, and save user activity record in the User Connections dialog. This feature is available in ODI Studio and ODI Console.
ODI Console Enhancements
The overall look and feel of ODI Console has been improved. Security tasks such as creating users or profiles can now be performed in ODI Console. Also, Release Management activities can now be performed in ODI Console and the functionality related to Topology activities has been enhanced.
New and Changed Features for Release 12c (184.108.40.206.1)
Oracle Data Integrator 12c (220.127.116.11.1) introduces the following enhancements:
Execution of ODI Mappings using Spark and Pig
ODI allows the defining of mappings through a logical design, which is independent of the implementation language. For Hadoop-based transformations, you can select between Hive, Spark, and Pig as the generated transformation language. This allows you to pick the best implementation based on the environment and use case; you can also choose different implementations simultaneously using multiple physical designs. This selection makes development for Big Data flexible and future-proof.
Generate Pig Latin transformations: You can choose Pig Latin as the transformation language and execution engine for ODI mappings. Apache Pig is a platform for analyzing large data sets in Hadoop and uses the high-level language, Pig Latin for expressing data analysis programs. Any Pig transformations can be executed either in Local or MapReduce mode. Custom Pig code can be added through user-defined functions or the table function component.
Generate Spark transformations: ODI mapping can also generate PySpark, which exposes the Spark programming model in the Python language. Apache Spark is a transformation engine for large-scale data processing. It provides fast in-memory processing of large data sets. Custom PySpark code can be added through user-defined functions or the table function component.
Orchestration of ODI Jobs using Oozie
You can now choose between the traditional ODI Agent or Apache Oozie as the orchestration engine for ODI jobs such as mappings, packages, scenarios, and procedures. Apache Oozie allows a fully native execution on a Hadoop infrastructure without installing an ODI environment for orchestration. You can utilize Oozie tools to schedule, manage, and monitor ODI jobs. ODI uses Oozie's native actions to execute Hadoop processes and conditional branching logic.
Enhanced Hive Driver and Knowledge Modules
ODI includes the WebLogic Hive JDBC driver that provides a number of advantages when compared to the Apache Hive driver, such as, total JDBC compliance and improved performance. All Hive Knowledge Modules have been rewritten to benefit from this new driver. Also, the Knowledge Modules whose main purpose is to load from a source are now provided as Load Knowledge Modules, enabling them to be combined in a single mapping with other Load Knowledge Modules. A new class of "direct load" Load Knowledge Modules also allows the loading of targets without intermediate staging. The table function component has been extended to support Hive constructs.
Retrieval of Hadoop Audit Logs
ODI integrates results from Hadoop Audit Logs in Operator tasks for executions of Oozie, Pig, and other tasks. The log results show MapReduce statistics and provide a link to Hadoop statistics in native web consoles.
HDFS access in ODI File Tools
The file based tools used in ODI packages and procedures have been enhanced to include Hadoop Distributed File System (HDFS) file processing. This includes copying, moving, appending, and deleting files, detecting file changes, managing folders, and transferring files using FTP directly into HDFS.
Flatten and Jagged Components
The new Flatten component for mappings allows complex sub-structures to be processed as part of a flat list of attributes. The new Jagged component converts key-value lists into named attributes for further processing.
New Big Data Guide added to the ODI Documentation Set
A new guide, Integrating Big Data with Oracle Data Integrator, has been added to the ODI documentation set. This guide provides information on how to integrate Big Data, deploy and execute Oozie workflows, and generate code in languages such as Pig Latin and Spark.
New and Changed Features for Release 12c (12.1.3)
Oracle Data Integrator 12c (12.1.3) introduces the following enhancements:
ODI FIPS Compliance
ODI now uses Advanced Encryption Standard (AES) as the standard encryption algorithm for encrypting Knowledge Modules, procedures, scenarios, actions, and passwords. You can configure the encryption algorithm and key length to meet requirements. Passwords and other sensitive information included in repository exports are now encrypted and secured by a password.
For more information, see "Advanced Encryption Standard".
ODI XML Driver Enhancements
The following XML Schema support enhancements have been added:
Recursion: ODI now supports recursion inside XML Schemas.
anyAttribute: Data defined by these types is stored in string type columns with XML markup from the original document.
Metadata annotations can be added inside an XML Schema to instruct the ODI XML Driver which table name, column name, type, length, and precision should be used.
For more information, see the "Oracle Data Integrator Driver for XML Reference" section in Connectivity and Knowledge Modules Guide for Oracle Data Integrator Developer's Guide.
The ODI Complex File Driver can now read and write files in JSON format. The JSON structure is defined through an nXSD schema.
For more information, see the "JSON Support" section in Connectivity and Knowledge Modules Guide for Oracle Data Integrator Developer's Guide.
Hadoop SQOOP Integration
ODI can now load the following sources and targets using Hadoop SQOOP:
From relational databases to HDFS, Hive, and HBase through Knowledge Module IKM File-Hive to SQL (SQOOP)
From HDFS and Hive to relational databases through Knowledge Module IKM SQL to Hive-HBase-File (SQOOP)
SQOOP enables load and unload mechanisms using parallel JDBC connections in Hadoop Map-Reduce processes.
Hadoop HBase Integration
ODI now supports Hadoop HBase through a new technology and the following knowledge modules:
LKM HBase to Hive (HBase-SerDe)
IKM Hive to HBase Incremental Update (HBase-SerDe)
Hive Append Optimization
Knowledge Modules writing to Hive now support the Hive 0.8+ capability and can append data to the existing data files rather than copying existing data into a new appended file.
Multi-threaded Target Table Load in ODI Engine
ODI can now load a target table using multiple parallel connections. This capability is controlled through the Degree of Parallelism for Target property in the data server.
For more information, see "Creating a Data Server".
Improved Control for Scenario and Load Plan Concurrent Execution
You can now limit concurrent executions in a scenario or load plan and force a concurrent execution to either wait or raise an execution error.
For more information, see the "Controlling Concurrent Execution of Scenarios and Load Plans" section in Developing Integration Projects with Oracle Data Integrator.
Create New Model and Topology Objects
The Create New Model and Topology Objects dialog in the Designer Navigator provides the ability to create a new model and associate it with new or existing topology objects, if connected to a work repository. This dialog enables you to create topology objects without having to use Topology editors unless more advanced options are required.
For more information, see the "Creating a Model and Topology Objects" section in Developing Integration Projects with Oracle Data Integrator.
The information that was previously available in the Oracle Data Integrator Developer's Guide is now reorganized. The following new guides have been added to the ODI documentation library:
Understanding Oracle Data Integrator
Administering Oracle Data Integrator
Oracle Data Integrator Tools Reference
For more information, see the "What's New In Oracle Data Integrator?" section in Developing Integration Projects with Oracle Data Integrator.
New and Changed Features for Release 12c (12.1.2)
Oracle Data Integrator 12c (12.1.2) introduces the following enhancements:
Declarative Flow-Based User Interface
The new declarative flow-based user interface combines the simplicity and ease-of-use of the declarative approach with the flexibility and extensibility of configurable flows. Mappings (the successor of the Interface concept in Oracle Data Integrator 11g) connect sources to targets through a flow of components such as Join, Filter, Aggregate, Set, Split, and so on.
Reusable Mappings can be used to encapsulate flow sections that can then be reused in multiple mappings. A reusable mapping can have input and output signatures to connect to an enclosing flow; it can also contain sources and targets that are encapsulated inside the reusable mapping.
Multiple Target Support
A mapping can now load multiple targets as part of a single flow. The order of target loading can be specified, and the Split component can be optionally used to route rows into different targets, based on one or several conditions.
Mappings, Packages, Procedures, and Scenarios can now be debugged in a step-by-step debugger. You can manually traverse task execution within these objects and set breakpoints to interrupt execution at pre-defined locations. Values of variables can be introspected and changed during a debugging session, and data of underlying sources and targets can be queried, including the content of uncommitted transactions.
Runtime Performance Enhancements
The runtime execution has been improved to enhance performance. Various changes have been made to reduce overhead of session execution, including the introduction of blueprints, which are cached execution plans for sessions.
Performance is improved by loading sources in parallel into the staging area. Parallelism of loads can be customized in the physical view of a map.
You also have the option to use unique names for temporary database objects, allowing parallel execution of the same mapping.
Oracle GoldenGate Integration Improvements
The integration of Oracle GoldenGate as a source for the Change Data Capture (CDC) framework has been improved in the following areas:
Oracle GoldenGate source and target systems are now configured as data servers in Topology. Extract and replicate processes are represented by physical and logical schemas. This representation in Topology allows separate configuration of multiple contexts, following the general context philosophy.
Most Oracle GoldenGate parameters can now be added to extract and replicate processes in the physical schema configuration. The UI provides support for selecting parameters from lists. This minimizes the need for the modification of Oracle GoldenGate parameter files after generation.
A single mapping can now be used for journalized CDC load and bulk load of a target. This is enabled by the Oracle GoldenGate JKM using the source model as opposed to the Oracle GoldenGate replication target, as well as configuration of journalizing in mapping as part of a deployment specification. Multiple deployment specifications can be used in a single mapping for journalized load and bulk load.
Oracle GoldenGate parameter files can now be automatically deployed and started to source and target Oracle GoldenGate instances through the JAgent technology.
Standalone Agent Management with WebLogic Management Framework
Oracle Data Integrator Standalone agents are now managed through the WebLogic Management Framework. This has the following advantages:
UI-driven configuration through Configuration Wizard
Multiple configurations can be maintained in separate domains
Node Manager can be used to control and automatically restart agents
Integration with OPSS Enterprise Roles
Oracle Data Integrator can now use the authorization model in Oracle Platform Security Services (OPSS) to control access to resources. Enterprise roles can be mapped into Oracle Data Integrator roles to authorize enterprise users across different tools.
The following XML Schema constructs are now supported:
list and union - List or union-based elements are mapped into VARCHAR columns.
substitutionGroup - Elements based on substitution groups create a table each for all types of the substitution group.
Mixed content - Elements with mixed content map into a VARCHAR column that contains text and markup content of the element.
Annotation - Content of XML Schema annotations are stored in the table metadata.
Oracle Warehouse Builder Integration
Oracle Warehouse Builder (OWB) jobs can now be executed in Oracle Data Integrator through the
OdiStartOwbJob tool. The OWB repository is configured as a data server in Topology. All the details of the OWB job execution are displayed as a session in the Operator tree.
Unique Repository IDs
Master and work repositories now use unique IDs following the GUID convention. This avoids collisions during import of artifacts and allows for easier management and consolidation of multiple repositories in an organization.