9.2 Working with Complex Files

Provides information on working with user defined metadata that drives the flatten component.

Oracle Data Integrator provides a Flatten component that can process input data with complex structure and produce flatten representation of the same data using standard data types. The input data may be in a database, in an XML, or any other source.

There are three check-box properties which are Include Nulls, Collection, and Structure. The collection check-box indicates whether the complex type attribute is a collection such as an array. It is automatically assigned if the complex type attribute has a data format defined. The structure check-box indicates whether the complex type is an object, record, or structure, not just a collection of scalar types. The Include Nulls check-box indicates whether null complex data should be processed. Some technologies, particularly Spark, can drop records containing null complex attributes.

Working with Complex Files with Improved Flattening for HDFS

The complex type attribute is an upstream attribute, of a collection or object type, to be flattened. Each flatten component can flatten only one complex type attribute, since just a single property is used to specify it.

Complex type member attributes that comprise a collection or object are automatically added to the flatten component when the complex type attribute is selected. The complex type member attributes can be added, changed, or deleted manually as well. The complex type (CT) attribute can be accessed as a data store column, and the collection or object attributes are determined from the data_format field.

Note:

The flatten component is only supported with Spark 1.3 and above.

For more information using the Flatten component and the KMs associated with it, see the following sections: