Transformation Configuration Templates

This topic explains the configuration parameters for the different transformations supported by the Oracle NoSQL Database Migrator. For the complete configuration file template, see Configuration File in Terminology used with Oracle NoSQL Database Migrator.

Oracle NoSQL Database Migrator lets you modify the data, that is, add data transformations as part of the migration activity. You can define multiple transformations in a single migration. In such a case, the order of transformations is vital because the source data undergoes each transformation in the given order. The output of one transformation becomes the input to the next one in the migrator pipeline.

The different transformations supported by the NoSQL Data Migrator are:

Table 5-2 Transformations

Transformation Config Attribute	You can use this transformation to ...
ignoreFields	Ignore the identified columns from the source row before writing to the sink.
includeFields	Include the identified columns from the source row before writing to the sink.
renameFields	Rename the identified columns from the source row before writing to the sink.
aggregateFields	Aggregate multiple columns from the source into a single column in the sink. As part of this transformation, you can also identify the columns that you want to exclude in the aggregation. Those fields will be skipped from the aggregated column.

You can find the configuration template for each supported transformation below.

ignoreFields

The configuration file format for the ignoreFields transformation is shown below.

Transformation Configuration Template

"transforms" : {
  "ignoreFields" : ["<field1>","<field2>",...]
}

Transformation Parameter

ignoreFields

• Purpose: An array of the column names to be ignored from the source records.

  Note:

  You can supply only top-level fields. Transformations can not be applied on the data in the nested fields.
• Data Type: array of strings
• Mandatory (Y/N): Y

• Example: To ignore the columns named "name" and "address" from the source record:

  "ignoreFields" : ["name","address"]

includeFields

The configuration file format for the includeFields transformation is shown below.

Transformation Configuration Template

"transforms" : {
  "includeFields" : ["<field1>","<field2>",...]
}

Transformation Parameter

includeFields

• Purpose: An array of the column names to be included from the source records. It only includes the fields specified in the array, the rest of the fields are ignored.

  Note:

  The NoSQL Database Migrator tool throws an error if you specify an empty array. Additionally, you can specify only the top-level fields. The NoSQL Database Migrator tool does not apply transformations to the data in the nested fields.
• Data Type: array of strings
• Mandatory (Y/N): Y

• Example: To ignore the columns named "age" and "gender" from the source record:

  "includeFields" : ["age","gender"]

renameFields

The configuration file format for the renameFields transformation is shown below.

Transformation Configuration Template

"transforms" : {
  "renameFields" : {
    "<old_name>" : "<new_name>",
    "<old_name>" : "<new_name>,"
    .....
  }
}

Transformation Parameter

renameFields

• Purpose: Key-Value pairs of the old and new names of the columns to be renamed.

  Note:

  You can supply only top-level fields. Transformations can not be applied on the data in the nested fields.
• Data Type: JSON object
• Mandatory (Y/N): Y

• Example: To rename the column named "residence" to "address" and the column named "_id" to "id":

  "renameFields" : { "residence" : "address", "_id" : "id" }

aggregateFields

The configuration file format for the aggregateFields transformation is shown below.

Transformation Configuration Template

"transforms" : {
  "aggregateFields" : {
    "fieldName" : "name of the new aggregate field",
    "skipFields" : ["<field1>","<field2">,...]
  }
}

Transformation Parameter

aggregateFields

• Purpose: Name of the aggregated field in the sink.

• Data Type: string
• Mandatory (Y/N): Y

• Example: If the given record is:

  {
    "id" : 100,
    "name" : "john",
    "address" : "USA",
    "age" : 20
  }

  If the aggregate transformation is:

  "aggregateFields" : {
    "fieldName" : "document",
    "skipFields" : ["id"]
  }

  The aggregated column in the sink looks like:

  {
    "id": 100,
    "document": {
      "name": "john",
      "address": "USA",
      "age": 20
    }
  }