Oracle NoSQL Database Migrator Reference

Learn about source, sink, and transformation configuration template parameters available for Oracle NoSQL Database Migrator.

This article has the following topics:

Related Topics

Parameters

The NoSQL Database Migrator requires a configuration file where you define all the parameters to perform the migration activity. A few parameters are common across several sources and sinks. This topic provides a list of these common parameters. For the list of other parameters that are unique to individual sources or sinks, see the corresponding configuration template sections.

Common Configuration Parameters

The following are the common configuration parameters. See the individual configuration template sections for examples.

bucket

Purpose: Specifies the name of the OCI Object Storage bucket, which contains the source/sink objects.

Ensure that the required bucket already exists in the OCI Object Storage instance and has read/write permissions.
Data Type: string
Mandatory (Y/N): Y

chunkSize

Purpose: Specifies the maximum size of a chunk of table data to be stored at the sink. The value is in MB. During migration, a table is split into chunkSize chunks and each chunk is written as a separate file to the sink. A new file is created when the source data that is being migrated exceeds the chunkSize value.

If not specified, defaults to 32MB. The valid value is an integer between 1 to 1024.
Data Type: integer
Mandatory (Y/N): N

credentials

Purpose: Specifies the absolute path to a file containing OCI credentials. The NoSQL Database Migrator uses this file to connect to the OCI service such as Oracle NoSQL Database Cloud Service, OCI Object Storage, and so on.

The default value is $HOME/.oci/config

See Example Configuration for an example of the credentials file.

Note:
The authentication parameters credentials, useInstancePrincipal, and useDelegationToken are mutually exclusive. Specify only one of these parameters in the configuration template.
Data Type: string
Mandatory (Y/N): N

credentialsProfile

Purpose: Specifies the name of the configuration profile to be used to connect to the OCI service such as Oracle NoSQL Database Cloud Service, OCI Object Storage, and so on. User account credentials are referred to as a profile.

If you do not specify this value, the NoSQL Database Migrator uses the DEFAULT profile.

Note:
This parameter is valid only if the credentials parameter is specified.
Data Type: string
Mandatory (Y/N): N

endpoint

Purpose: Specifies one of the following:
- The Service endpoint URL or the Region ID for the OCI Object Storage service.
  For the list of OCI Object Storage service endpoints, see Object Storage Endpoints.
- The Service endpoint URL or the Region ID for the Oracle NoSQL Database Cloud Service.
  You can either specify the complete URL or the Region ID alone. For the list of data regions supported for Oracle NoSQL Database Cloud Service, see Data Regions and Associated Service URLs in the Oracle NoSQL Database Cloud Service document.
Data Type: string
Mandatory (Y/N): Y

format

Purpose: Specifies the source/sink format.
Data Type: string
Mandatory (Y/N): Y

namespace

Purpose: Specifies the namespace of the OCI Object Storage service. This is an optional parameter. If you don't specify this parameter, the default namespace of the tenancy is used.
Data Type: string
Mandatory (Y/N): N

prefix

Purpose: The prefix acts as a logical container or directory for storing data in the OCI Object Storage bucket.
- Source configuration template: If the prefix parameter in specified, all the objects from the directory named in the prefix parameter are migrated. Else, all the objects present in the bucket are migrated.
- Sink configuration template: If the prefix parameter is specified, a directory with the given prefix is created in the bucket and the objects are migrated into this directory. Else, the table name from the source is used as the prefix. If any object with the same name already exists in the bucket, it is overwritten.
For more information about prefixes, see Object Naming Using Prefixes and Hierarchies.
Data Type: string
Mandatory (Y/N): N

requestTimeoutMs

Purpose: Specifies the time to wait for each read/write operation from/to the store to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer.
Data Type: integer
Mandatory (Y/N): N

security

Purpose: Specifies the absolute path to the security login file that contains your store credentials if your store is a secure store. For more information about the security login file, see Configuring Security with Remote Access in the Administrator's Guide.

You can use either password file based authentication or wallet based authentication. However, the wallet based authentication is supported only in the Enterprise Edition (EE) of Oracle NoSQL Database. For more information on wallet-based authentication, see Source and Sink Security .

The Community Edition(CE) edition supports password file based authentication only.
Data Type: string
Mandatory (Y/N): Y, for a secure store

type

Purpose: Identifies the source/sink type.
Data Type: string
Mandatory (Y/N): Y

useDelegationToken

Purpose: Specifies whether or not the NoSQL Database Migrator tool uses a delegation token authentication to connect to the OCI services. You must use the delegation token authentication to run the Migrator utility from the Cloud Shell. The delegation token is automatically created for the user when the Cloud Shell is invoked.

The default value is false.
Data Type: boolean

Mandatory (Y/N): N

Note:

The authentication with delegation token is supported only when the NoSQL Database Migrator tool is running from a Cloud Shell.
The authentication parameters credentials, useInstancePrincipal, and useDelegationToken are mutually exclusive. Specify only one of these parameters in the configuration template.

The Cloud Shell supports migration only between the following sources and sinks:

Type	Valid source	Valid sink
Oracle NoSQL Database Cloud Service (`nosqldb_cloud`)	Y	Y
File (JSON file in the home directory)	Y	Y
OCI Object Storage (JSON file) (`object_storage_oci`)	Y	Y
OCI Object Storage (Parquet file) (`object_storage_oci`)	N	Y

useInstancePrincipal

Purpose: Specifies whether or not the NoSQL Database Migrator tool uses instance principal authentication to connect to the OCI service such as Oracle NoSQL Database Cloud Service, OCI Object Storage, and so on. For more information on Instance Principal authentication method, see Source and Sink Security .

The default value is false.
Note:
- The authentication with Instance Principals is supported only when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, the NoSQL Database Migrator tool running in a VM hosted on OCI.
- The authentication parameters credentials, useInstancePrincipal and useDelegationToken are mutually exclusive. Specify only one of these parameters in the configuration template.
Data Type: boolean
Mandatory (Y/N): N

Source Configuration Templates

Learn about the source configuration file formats for each valid source and the purpose of each configuration parameter.

For the configuration file template, see Configuration File in Terminology used with NoSQL Data Migrator.

For details on valid sink formats for each of the source, see Sink Configuration Templates.

Topics

The following topics describe the source configuration templates referred by Oracle NoSQL Database Migrator to copy the data from the given source to a valid sink.

JSON File Source
Specified file or directory containing the JSON data.
JSON File in OCI Object Storage Bucket
Specified JSON file in the OCI Object Storage bucket.
MongoDB-Formatted JSON File
Specified file or directory containing the MongoDB formatted JSON data.
MongoDB-Formatted JSON File in OCI Object Storage bucket
Specified MongoDB exported JSON file stored in the OCI Object Storage bucket.
DynamoDB-Formatted JSON File stored in AWS S3
Specified DynamoDB exported JSON file stored in the AWS S3 storage.
DynamoDB-Formatted JSON File
Specified DynamoDB exported JSON file from a file system.
Oracle NoSQL Database
Specified table in Oracle NoSQL Database.
Oracle NoSQL Database Cloud Service
Specified table in Oracle NoSQL Database Cloud Service.
CSV File Source
Specified file or directory containing the CSV data.
CSV file in OCI Object Storage Bucket
Specified CSV file in the OCI Object Storage bucket.

JSON File Source

The configuration file format for JSON file as a source of NoSQL Database Migrator is shown below.

You can migrate a JSON source file by specifying the file path or a directory in the source configuration template.

A sample JSON source file is as follows:

{"id":6,"val_json":{"array":["q","r","s"],"date":"2023-02-04T02:38:57.520Z","nestarray":[[1,2,3],[10,20,30]],"nested":{"arrayofobjects":[{"datefield":"2023-03-04T02:38:57.520Z","numfield":30,"strfield":"foo54"},{"datefield":"2023-02-04T02:38:57.520Z","numfield":56,"strfield":"bar23"}],"nestNum":10,"nestString":"bar"},"num":1,"string":"foo"}}
{"id":3,"val_json":{"array":["g","h","i"],"date":"2023-02-02T02:38:57.520Z","nestarray":[[1,2,3],[10,20,30]],"nested":{"arrayofobjects":[{"datefield":"2023-02-02T02:38:57.520Z","numfield":28,"strfield":"foo3"},{"datefield":"2023-02-02T02:38:57.520Z","numfield":38,"strfield":"bar"}],"nestNum":10,"nestString":"bar"},"num":1,"string":"foo"}}

Source Configuration Template

"source": {
  "type": "file",
  "format": "json",
  "dataPath": "<path/to/JSON/[file|dir]>",
  "schemaInfo": {
    "schemaPath": "<path/to/schema/file>"
  }
},

Source Parameters

Common Configuration Parameters

type
Use "type" : "file"
format
Use "format" : "json"

Unique Configuration Parameters

dataPath
schemaInfo
schemaInfo.schemaPath

dataPath

Purpose: Specifies the absolute path to a file or directory containing the JSON data for migration.

You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported.
Data Type: string
Mandatory (Y/N): Y
Example:
- Specifying a JSON file
  
  "dataPath" : "/home/user/sample.json"
- Specifying a directory
  
  "dataPath" : "/home/user"

schemaInfo

Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.
Data Type: Object
Mandatory (Y/N): N

schemaInfo.schemaPath

Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated.
Data Type: string
Mandatory (Y/N): Y

Example:

"schemaInfo": {
  "schemaPath": "<path to the schema file>"
}

JSON File in OCI Object Storage Bucket

The configuration file format for JSON file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below.

You can migrate a JSON file in the OCI Object Storage bucket by specifying the name of the bucket in the source configuration template.

A sample JSON source file in the OCI Object Storage bucket is as follows:

{"id":6,"val_json":{"array":["q","r","s"],"date":"2023-02-04T02:38:57.520Z","nestarray":[[1,2,3],[10,20,30]],"nested":{"arrayofobjects":[{"datefield":"2023-03-04T02:38:57.520Z","numfield":30,"strfield":"foo54"},{"datefield":"2023-02-04T02:38:57.520Z","numfield":56,"strfield":"bar23"}],"nestNum":10,"nestString":"bar"},"num":1,"string":"foo"}}
{"id":3,"val_json":{"array":["g","h","i"],"date":"2023-02-02T02:38:57.520Z","nestarray":[[1,2,3],[10,20,30]],"nested":{"arrayofobjects":[{"datefield":"2023-02-02T02:38:57.520Z","numfield":28,"strfield":"foo3"},{"datefield":"2023-02-02T02:38:57.520Z","numfield":38,"strfield":"bar"}],"nestNum":10,"nestString":"bar"},"num":1,"string":"foo"}}

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Source Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "schemaInfo" : {
     "schemaObject" : "<object name>"
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
  "useDelegationToken" : <true|false>
}

Source Parameters

Common Configuration Parameters

type
Use "type" : "object_storage_oci"
format
Use "format" : "json"
endpoint
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"
namespace
Example: "namespace" : "my-namespace"
bucket
Example: "bucket" : "my-bucket"
prefix
Example:
1. "prefix" : "my_table/Data/000000.json" (migrates only 000000.json)
2. "prefix" : "my_table/Data" (migrates all the objects with prefix my_table/Data)
credentials
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"
credentialsProfile
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile" : "ADMIN_USER"
useInstancePrincipal
Example: "useInstancePrincipal" : true
useDelegationToken
Example: "useDelegationToken" : true

Note:
The authentication with delegation token is supported only when the NoSQL Database Migrator is running from a Cloud Shell.

Unique Configuration Parameters

schemaInfo
schemaInfo.schemaObject

schemaInfo

Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.
Data Type: Object
Mandatory (Y/N): N

schemaInfo.schemaObject

Purpose: Specifies the name of the object in the bucket where NoSQL table schema definitions for the data being migrated are stored.
Data Type: string
Mandatory (Y/N): Y

Example:

"schemaInfo": {
  "schemaObject": "mytable/Schema/schema.ddl"
},

MongoDB-Formatted JSON File

The configuration file format for MongoDB-formatted JSON File as a source of NoSQL Database Migrator is shown below.

You can migrate a MongoDB exported JSON data by specifying the file or directory in the source configuration template.

MongoDB supports two types of extensions to the JSON format of files, Canonical mode and Relaxed mode. You can supply the MongoDB-formatted JSON file that is generated using the mongoexport tool in either Canonical or Relaxed mode. Both the modes are supported by the NoSQL Database Migrator for migration.

For more information on the MongoDB Extended JSON (v2) file, See mongoexport_formats.

For more information on the generation of MongoDB-formatted JSON file, see mongoexport for more information.

A sample MongoDB-formatted Relaxed mode JSON file is as follows:

{"_id":0,"name":"Aimee Zank","scores":[{"score":1.463179736705023,"type":"exam"},{"score":11.78273309957772,"type":"quiz"},{"score":35.8740349954354,"type":"homework"}]}
{"_id":1,"name":"Aurelia Menendez","scores":[{"score":60.06045071030959,"type":"exam"},{"score":52.79790691903873,"type":"quiz"},{"score":71.76133439165544,"type":"homework"}]}
{"_id":2,"name":"Corliss Zuk","scores":[{"score":67.03077096065002,"type":"exam"},{"score":6.301851677835235,"type":"quiz"},{"score":66.28344683278382,"type":"homework"}]}
{"_id":3,"name":"Bao Ziglar","scores":[{"score":71.64343899778332,"type":"exam"},{"score":24.80221293650313,"type":"quiz"},{"score":42.26147058804812,"type":"homework"}]}
{"_id":4,"name":"Zachary Langlais","scores":[{"score":78.68385091304332,"type":"exam"},{"score":90.2963101368042,"type":"quiz"},{"score":34.41620148042529,"type":"homework"}]}

Source Configuration Template

"source": {
  "type": "file",
  "format": "mongodb_json",
  "dataPath": "</path/to/json/[file|dir]>",
  "schemaInfo": {
    "schemaPath": "</path/to/schema/file>"
  }
}

Source Parameters

Common Configuration Parameters

type
Use "type" : "file"
format
Use "format" : "mongodb_json"

Unique Configuration Parameters

dataPath
schemaInfo
schemaInfo.schemaPath

dataPath

Purpose: Specifies the absolute path to a file or directory containing the MongoDB exported JSON data for migration.

You can supply the MongoDB-formatted JSON file that is generated using the mongoexport tool.

If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported. You must ensure that this data matches with the NoSQL table schema defined at the sink.
Data Type: string
Mandatory (Y/N): Y
Example:
- Specifying a MongoDB formatted JSON file
  
  "dataPath" : "/home/user/sample.json"
- Specifying a directory
  
  "dataPath" : "/home/user"

schemaInfo

Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.
Data Type: Object
Mandatory (Y/N): N

schemaInfo.schemaPath

Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated.
Data Type: string
Mandatory (Y/N): Y

Example:

"schemaInfo" : {
  "schemaPath" : "/home/user/mytable/Schema/schema.ddl"
}

MongoDB-Formatted JSON File in OCI Object Storage bucket

The configuration file format for MongoDB-Formatted JSON file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below.

You can migrate the MongoDB exported JSON data in the OCI Object Storage bucket by specifying the name of the bucket in the source configuration template.

Extract the data from MongoDB using the mongoexport utility and upload it to the OCI Object Storage bucket. See mongoexport for more information. MongoDB supports two types of extensions to the JSON format of files, Canonical mode and Relaxed mode. Both formats are supported in the OCI Object Storage bucket.

A sample MongoDB-formatted Relaxed mode JSON File is as follows:

{"_id":0,"name":"Aimee Zank","scores":[{"score":1.463179736705023,"type":"exam"},{"score":11.78273309957772,"type":"quiz"},{"score":35.8740349954354,"type":"homework"}]}
{"_id":1,"name":"Aurelia Menendez","scores":[{"score":60.06045071030959,"type":"exam"},{"score":52.79790691903873,"type":"quiz"},{"score":71.76133439165544,"type":"homework"}]}
{"_id":2,"name":"Corliss Zuk","scores":[{"score":67.03077096065002,"type":"exam"},{"score":6.301851677835235,"type":"quiz"},{"score":66.28344683278382,"type":"homework"}]}
{"_id":3,"name":"Bao Ziglar","scores":[{"score":71.64343899778332,"type":"exam"},{"score":24.80221293650313,"type":"quiz"},{"score":42.26147058804812,"type":"homework"}]}
{"_id":4,"name":"Zachary Langlais","scores":[{"score":78.68385091304332,"type":"exam"},{"score":90.2963101368042,"type":"quiz"},{"score":34.41620148042529,"type":"homework"}]}

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Source Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "mongodb_json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "schemaInfo" : {
     "schemaObject" : "<object name>"
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

Source Parameters

Common Configuration Parameters

type
Use "type" : "object_storage_oci"
format
Use "format" : "mongodb_json"
endpoint
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"
namespace
Example: "namespace" : "my-namespace"
bucket
Example: "bucket" : "my-bucket"
prefix
Example:
1. "prefix" : "mongo_export/Data/table.json" (migrates only table.json)
2. "prefix" : "mongo_export/Data" (migrates all the objects with prefix mongo_export/Data)
Note:
If you do not provide any value, all the objects present in the bucket are migrated.
credentials
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"
credentialsProfile
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile" : "ADMIN_USER"
useInstancePrincipal
Example: "useInstancePrincipal" : true

Unique Configuration Parameters

schemaInfo
schemaInfo.schemaObject

schemaInfo

Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.
Data Type: Object
Mandatory (Y/N): N

schemaInfo.schemaObject

Purpose: Specifies the name of the object in the bucket where NoSQL table schema definitions for the data being migrated are stored.
Data Type: string
Mandatory (Y/N): Y

Example:

"schemaInfo": {
  "schemaObject": "mytable/Schema/schema.ddl"
}

DynamoDB-Formatted JSON File stored in AWS S3

The configuration file format for DynamoDB-formatted JSON File in AWS S3 as a source of NoSQL Database Migrator is shown below.

You can migrate a file containing the DynamoDB exported JSON data from the AWS S3 storage by specifying the path in the source configuration template.

A sample DynamoDB-formatted JSON File is as follows:

{"Item":{"Id":{"N":"101"},"Phones":{"L":[{"L":[{"S":"555-222"},{"S":"123-567"}]}]},"PremierCustomer":{"BOOL":false},"Address":{"M":{"Zip":{"N":"570004"},"Street":{"S":"21 main"},"DoorNum":{"N":"201"},"City":{"S":"London"}}},"FirstName":{"S":"Fred"},"FavNumbers":{"NS":["10"]},"LastName":{"S":"Smith"},"FavColors":{"SS":["Red","Green"]},"Age":{"N":"22"}}}
{"Item":{"Id":{"N":"102"},"Phones":{"L":[{"L":[{"S":"222-222"}]}]},"PremierCustomer":{"BOOL":false},"Address":{"M":{"Zip":{"N":"560014"},"Street":{"S":"32 main"},"DoorNum":{"N":"1024"},"City":{"S":"Wales"}}},"FirstName":{"S":"John"},"FavNumbers":{"NS":["10"]},"LastName":{"S":"White"},"FavColors":{"SS":["Blue"]},"Age":{"N":"48"}}}

You must export the DynamoDB table to AWS S3 storage as specified in Exporting DynamoDB table data to Amazon S3.

The valid sink types for DynamoDB-formatted JSON stored in AWS S3 are nosqldb and nosqldb_cloud.

Source Configuration Template

"source" : {
  "type" : "aws_s3",
  "format" : "dynamodb_json",
  "s3URL" : "<S3 object url>",
  "credentials" : "</path/to/aws/credentials/file>",
  "credentialsProfile" : <"profile name in aws credentials file">
}

Source Parameters

Common Configuration Parameters

type
Use "type" : "aws_s3"
format
Use "format" : "dynamodb_json"

Note:
If the value of the type parameter is aws_s3, then the format must be dynamodb_json.

Unique Configuration Parameters

s3URL
credentials
credentialsProfile

s3URL

Purpose: Specifies the URL of an exported DynamoDB table stored in AWS S3. You can obtain this URL from the AWS S3 console. The valid URL format is https://<bucket-name>.<s3_endpoint>/<prefix>. The NoSQL Database Migrator will look for json.gz files in the prefix during import.

Note:
You must export DynamoDB table as specified in Exporting DynamoDB table data to Amazon S3.
Data Type: string
Mandatory (Y/N): Y
Example: https://my-bucket.s3.ap-south-1.amazonaws.com/AWSDynamoDB/01649660790057-14f642be

credentials

Purpose: Specifies the absolute path to a file containing the AWS credentials. If not specified, it defaults to $HOME/.aws/credentials. For more details on the credentials file, see Configuration and credential file settings .
Data Type: string
Mandatory (Y/N): N
Example:
```
"credentials" : "/home/user/.aws/credentials"
"credentials" : "/home/user/security/credentials
```
Note:
The NoSQL Database Migrator does not log any of the credentials information. You must properly protect the credentials file from unauthorized access.

credentialsProfile

Purpose: Name of the profile in the AWS credentials file to be used to connect to AWS S3. User account credentials are referred to as a profile. If you do not specify this value, NoSQL Database Migrator uses the default profile. For more details on the credentials file, see Configuration and credential file settings .
Data Type: string
Mandatory (Y/N): N

Example:

"credentialsProfile" : "default"
"credentialsProfile" : "test"

DynamoDB-Formatted JSON File

The configuration file format for DynamoDB-formatted JSON File as a source of NoSQL Database Migrator is shown below.

You can migrate a file or directory containing the DynamoDB exported JSON data from a file system by specifying the path in the source configuration template.

A sample DynamoDB-formatted JSON File is as follows:

{"Item":{"Id":{"N":"101"},"Phones":{"L":[{"L":[{"S":"555-222"},{"S":"123-567"}]}]},"PremierCustomer":{"BOOL":false},"Address":{"M":{"Zip":{"N":"570004"},"Street":{"S":"21 main"},"DoorNum":{"N":"201"},"City":{"S":"London"}}},"FirstName":{"S":"Fred"},"FavNumbers":{"NS":["10"]},"LastName":{"S":"Smith"},"FavColors":{"SS":["Red","Green"]},"Age":{"N":"22"}}}
{"Item":{"Id":{"N":"102"},"Phones":{"L":[{"L":[{"S":"222-222"}]}]},"PremierCustomer":{"BOOL":false},"Address":{"M":{"Zip":{"N":"560014"},"Street":{"S":"32 main"},"DoorNum":{"N":"1024"},"City":{"S":"Wales"}}},"FirstName":{"S":"John"},"FavNumbers":{"NS":["10"]},"LastName":{"S":"White"},"FavColors":{"SS":["Blue"]},"Age":{"N":"48"}}}

You must copy the exported DynamoDB table data from AWS S3 storage to a local mounted file system.

The valid sink types for DynamoDB JSON file are nosqldb and nosqldb_cloud.

Source Configuration Template

"source" : {
  "type" : "file",
  "format" : "dynamodb_json",
  "dataPath" : "<path/to/[file|dir]/containing/exported/DDB/tabledata>"   
}

Source Parameters

Common Configuration Parameters

type
Use "type" : "file"
format
Use "format" : "dynamodb_json"

Unique Configuration Parameter

dataPath

Purpose: Specifies the absolute path to a file or directory containing the exported DynamoDB table data. You must copy exported DynamoDB table data from AWS S3 to a local mounted file system. You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json.gz extension in that directory and the data sub-directory.
Data Type: string
Mandatory (Y/N): Y

Example:

Specifying a file

"dataPath" : "/home/user/AWSDynamoDB/01639372501551-bb4dd8c3/data/zclclwucjy6v5mkefvckxzhfvq.json.gz"

Specifying a directory

"dataPath" : "/home/user/AWSDynamoDB/01639372501551-bb4dd8c3"

Oracle NoSQL Database

The configuration file format for Oracle NoSQL Database as a source of NoSQL Database Migrator is shown below.

You can migrate a table from Oracle NoSQL Database by specifying the table name in the source configuration template.

A sample Oracle NoSQL Database table is as follows:

{"id":20,"firstName":"Jane","lastName":"Smith","otherNames":[{"first":"Jane","last":"teacher"}],"age":25,"income":55000,"address":{"city":"San Jose","number":201,"phones":[{"area":608,"kind":"work","number":6538955},{"area":931,"kind":"home","number":9533341},{"area":931,"kind":"mobile","number":9533382}],"state":"CA","street":"Atlantic Ave","zip":95005},"connections":[40,75,63],"expenses":null}
{"id":10,"firstName":"John","lastName":"Smith","otherNames":[{"first":"Johny","last":"chef"}],"age":22,"income":45000,"address":{"city":"Santa Cruz","number":101,"phones":[{"area":408,"kind":"work","number":4538955},{"area":831,"kind":"home","number":7533341},{"area":831,"kind":"mobile","number":7533382}],"state":"CA","street":"Pacific Ave","zip":95008},"connections":[30,55,43],"expenses":null}
{"id":30,"firstName":"Adam","lastName":"Smith","otherNames":[{"first":"Adam","last":"handyman"}],"age":45,"income":75000,"address":{"city":"Houston","number":301,"phones":[{"area":618,"kind":"work","number":6618955},{"area":951,"kind":"home","number":9613341},{"area":981,"kind":"mobile","number":9613382}],"state":"TX","street":"Indian Ave","zip":95075},"connections":[60,45,73],"expenses":null}

Source Configuration Template

"source" : {
  "type": "nosqldb",
  "storeName" : "<store name>",
  "helperHosts" : ["hostname1:port1","hostname2:port2,..."],
  "table" : "<fully qualified table name>", 
  "includeTTL": <true|false>,    
  "security" : "</path/to/store/security/file>",
  "requestTimeoutMs" : 5000
}

Source Parameters

Common Configuration Parameter

type
Use "type" : "nosqldb"

security

Example:

"security" : "/home/user/client.credentials"

Example security file content for password file based authentication:

oracle.kv.password.noPrompt=true
oracle.kv.auth.username=admin
oracle.kv.auth.pwdfile.file=/home/nosql/login.passwd
oracle.kv.transport=ssl
oracle.kv.ssl.trustStore=/home/nosql/client.trust
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

Example security file content for wallet based authentication:

oracle.kv.password.noPrompt=true
oracle.kv.auth.username=admin
oracle.kv.auth.wallet.dir=/home/nosql/login.wallet
oracle.kv.transport=ssl
oracle.kv.ssl.trustStore=/home/nosql/client.trust
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

requestTimeoutMs
Example: "requestTimeoutMs" : 5000

Unique Configuration Parameters

storeName
helperHosts
table
includeTTL

storeName

Purpose: Name of the Oracle NoSQL Database store.
Data Type: string
Mandatory (Y/N): Y
Example: "storeName" : "kvstore"

helperHosts

Purpose: A list of host and registry port pairs in the hostname:port format. Delimit each item in the list using a comma. You must specify at least one helper host.
Data Type: array of strings
Mandatory (Y/N): Y
Example: "helperHosts" : ["localhost:5000","localhost:6000"]

table

Purpose: Fully qualified table name from which to migrate the data.

Format: [namespace_name:]<table_name>

If the table is in the DEFAULT namespace, you can omit the namespace_name. The table must exist in the store.
Data Type: string
Mandatory (Y/N): Y
Example:
- With the DEFAULT namespace "table" :"mytable"
- With a non-default namespace "table" : "mynamespace:mytable"
- To specify a child table "table" : "mytable.child"

includeTTL

Purpose: Specifies whether or not to include TTL metadata for table rows when exporting Oracle NoSQL Database tables. If set to true, the TTL data for rows also gets included in the data provided by the source. TTL is present in the _metadata JSON object associated with each row. The expiration time for each row gets exported as the number of milliseconds since the UNIX epoch (Jan 1st, 1970).

If you do not specify this parameter, it defaults to false.
Only the rows having a positive expiration value for TTL get included as part of the exported rows. If a row does not expire, which means TTL=0, then its TTL metadata is not included explicitly. For example, if ROW1 expires at 2021-10-19 00:00:00 and ROW2 does not expire, the exported data looks like as follows:
```
//ROW1
{
  "id" : 1,
  "name" : "abc",
  "_metadata" : {
    "expiration" : 1634601600000
  }
}

//ROW2
{
  "id" : 2,
  "name" : "xyz"
}
```
Data Type: boolean
Mandatory (Y/N): N
Example: "includeTTL" : true

Oracle NoSQL Database Cloud Service

The configuration file format for Oracle NoSQL Database Cloud Service as a source of NoSQL Database Migrator is shown below.

You can migrate a table from Oracle NoSQL Database Cloud Service by specifying the name or OCID of the compartment in which the table resides in the source configuration template.

A sample Oracle NoSQL Database Cloud Service table is as follows:

{"id":20,"firstName":"Jane","lastName":"Smith","otherNames":[{"first":"Jane","last":"teacher"}],"age":25,"income":55000,"address":{"city":"San Jose","number":201,"phones":[{"area":608,"kind":"work","number":6538955},{"area":931,"kind":"home","number":9533341},{"area":931,"kind":"mobile","number":9533382}],"state":"CA","street":"Atlantic Ave","zip":95005},"connections":[40,75,63],"expenses":null}
{"id":10,"firstName":"John","lastName":"Smith","otherNames":[{"first":"Johny","last":"chef"}],"age":22,"income":45000,"address":{"city":"Santa Cruz","number":101,"phones":[{"area":408,"kind":"work","number":4538955},{"area":831,"kind":"home","number":7533341},{"area":831,"kind":"mobile","number":7533382}],"state":"CA","street":"Pacific Ave","zip":95008},"connections":[30,55,43],"expenses":null}
{"id":30,"firstName":"Adam","lastName":"Smith","otherNames":[{"first":"Adam","last":"handyman"}],"age":45,"income":75000,"address":{"city":"Houston","number":301,"phones":[{"area":618,"kind":"work","number":6618955},{"area":951,"kind":"home","number":9613341},{"area":981,"kind":"mobile","number":9613382}],"state":"TX","street":"Indian Ave","zip":95075},"connections":[60,45,73],"expenses":null}

Source Configuration Template

"source" : {
  "type" : "nosqldb_cloud",
  "endpoint" : "<Oracle NoSQL Cloud Service endpoint URL or region ID>",
  "table" : "<table name>",
  "compartment" : "<OCI compartment name or id>",
  "credentials" : "<path/to/oci/credential/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
  "useDelegationToken" : <true|false>,
  "readUnitsPercent" : <table readunits percent>,
  "includeTTL": <true|false>,
  "requestTimeoutMs" : <timeout in milli seconds>
}

Source Parameters

Common Configuration Parameters

type
Use "type" : "nosqldb_cloud"
endpoint
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"
credentials
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"
credentialsProfile
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile" : "ADMIN_USER"
useInstancePrincipal
Example: "useInstancePrincipal" : true
useDelegationToken
Example: "useDelegationToken" : true

Note:
The authentication with delegation token is supported only when the NoSQL Database Migrator is running from a Cloud Shell.
requestTimeoutMs
Example: "requestTimeoutMs" : 5000

Unique Configuration Parameters

table
compartment
readUnitsPercent
includeTTL

table

Purpose: Name of the table from which to migrate the data.
Data Type: string
Mandatory (Y/N): Y
Example:
- To specify a table "table" : "myTable"
- To specify a child table "table" : "mytable.child"

compartment

Purpose: Specifies the name or OCID of the compartment in which the table resides.

If you do not provide any value, it defaults to the root compartment.

You can find your compartment's OCID from the Compartment Explorer window under Governance in the OCI Cloud Console.
Data Type: string
Mandatory (Y/N): Y, if the table is not in the root compartment of the tenancy OR when the useInstancePrincipal parameter is set to true.

Note:
If the useInstancePrincipal parameter is set to true, the compartment must specify the compartment OCID and not the name.
Example:
- Compartment name
  
  "compartment" : "mycompartment"
- Compartment name qualified with its parent compartment
  
  "compartment" : "parent.childcompartment"
- No value provided. Defaults to the root compartment.
  
  "compartment": ""
- Compartment OCID
  
  "compartment" : "ocid1.tenancy.oc1...4ksd"

readUnitsPercent

Purpose: Percentage of table read units to be used while migrating the NoSQL table.

The default value is 90. The valid range is any integer between 1 to 100. The amount of time required to migrate data is directly proportional to this attribute. It is better to increase the read throughput of the table for the migration activity. You can reduce the read throughput after the migration process completes.

To learn the daily limits on throughput changes, see Cloud Limits in the Oracle NoSQL Database Cloud Service document.

See Troubleshooting the Oracle NoSQL Database Migrator to learn how to use this attribute to improve the data migration speed.
Data Type: integer
Mandatory (Y/N): N
Example: "readUnitsPercent" : 90

includeTTL

Purpose: Specifies whether or not to include TTL metadata for table rows when exporting Oracle NoSQL Database Cloud Service tables. If set to true, the TTL data for rows also gets included in the data provided by the source. TTL is present in the _metadata JSON object associated with each row. The expiration time for each row gets exported as the number of milliseconds since the UNIX epoch (Jan 1st, 1970).

If you do not specify this parameter, it defaults to false.
Only the rows having a positive expiration value for TTL get included as part of the exported rows. If a row does not expire, which means TTL=0, then its TTL metadata is not included explicitly. For example, if ROW1 expires at 2021-10-19 00:00:00 and ROW2 does not expire, the exported data looks like as follows:
```
//ROW1
{
  "id" : 1,
  "name" : "abc",
  "_metadata" : {
    "expiration" : 1634601600000
  }
}

//ROW2
{
  "id" : 2,
  "name" : "xyz"
}
```
Data Type: boolean
Mandatory (Y/N): N
Example: "includeTTL" : true

CSV File Source

The configuration file format for the CSV file as a source of NoSQL Database Migrator is shown below. The CSV file must conform to the RFC4180 format.

You can migrate a CSV file or a directory containing the CSV data by specifying the file name or directory in the source configuration template.

A sample CSV file is as follows:

1,"Computer Science","San Francisco","2500"
2,"Bio-Technology","Los Angeles","1200"
3,"Journalism","Las Vegas","1500"
4,"Telecommunication","San Francisco","2500"

Source Configuration Template

"source" : {
  "type" : "file",
  "format" : "csv",
  "dataPath": "</path/to/a/csv/[file|dir]>",
  "hasHeader" : <true | false>,
  "columns" : ["column1", "column2", ....],
  "csvOptions": {
    "encoding": "<character set encoding>",
    "trim": "<true | false>"
 }
}

Source Parameters

Common Configuration Parameters

type
Use "type" : "file"
format
Use "format" : "csv"

Unique Configuration Parameters

dataPath
hasHeader
columns
csvOptions
csvOptions.encoding
csvOptions.trim

datapath

Purpose: Specifies the absolute path to a file or directory containing the CSV data for migration. If you specify a directory, NoSQL Database Migrator imports all the files with the .csv or .CSV extension in that directory. All the CSV files are copied into a single table, but not in any particular order.

CSV files must conform to the RFC4180 standard. You must ensure that the data in each CSV file matches with the NoSQL Database table schema defined in the sink table. Sub-directories are not supported.
Data Type: string
Mandatory (Y/N): Y
Example:
- Specifying a CSV file
  
  "dataPath" : "/home/user/sample.csv"
- Specifying a directory
  
  "dataPath" : "/home/user"

Note:

The CSV files must contain only scalar values. Importing CSV files containing complex types such as MAP, RECORD, ARRAY, and JSON is not supported. The NoSQL Database Migrator tool does not check for the correctness of the data in the input CSV file. The NoSQL Database Migrator tool supports the importing of CSV data that conforms to the RFC4180 format. CSV files containing data that does not conform to the RFC4180 standard may not get copied correctly or may result in an error. If the input data is corrupted, the NoSQL Database Migrator tool will not parse the CSV records. If any errors are encountered during migration, the NoSQL Database Migrator tool logs the information about the failed input records for debugging and informative purposes. For more details, see Logging Migrator Progress in Using Oracle NoSQL Data Migrator .

hasHeader

Purpose: Specifies if the CSV file has a header or not. If this is set to true, the first line is ignored. If it is set to false, the first line is considered a CSV record. The default value is false.
Data Type: Boolean
Mandatory (Y/N): N
Example: "hasHeader" : "false"

columns

Purpose: Specifies the list of NoSQL Database table column names. The order of the column names indicates the mapping of the CSV file fields with corresponding NoSQL Database table columns. If the order of the input CSV file columns does not match the existing or newly created NoSQL Database table columns, you can map the ordering using this parameter. Also, when importing into a table that has an Identity Column, you can skip the Identity column name in the columns parameter.
Note:
- If the NoSQL Database table has additional columns that are not available in the CSV file, the values of the missing columns are updated with the default value as defined in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration. For more information on default values, see Data Type Definitions section in the SQL Reference Guide.
- If the CSV file has additional columns that are not defined in the NoSQL Database table, the additional column information is ignored.
- If any value in the CSV record is empty, it is set to the default value of the corresponding columns in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration.
Data Type: Array of Strings
Mandatory (Y/N): N
Example: "columns" : ["table_column_1", "table_column_2"]

csvOptions

Purpose: Specifies the formatting options for a CSV file. Provide the character set encoding format of the CSV file and choose whether or not to trim the blank spaces.
Data Type: Object
Mandatory (Y/N): N

csvOptions.encoding

Purpose: Specifies the character set to decode the CSV file. The default value is UTF-8. The supported character sets are US-ASCII, ISO-8859-1, UTF-8, and UTF-16.
Data Type: String
Mandatory (Y/N): N
Example: "encoding" : "UTF-8"

csvOptions.trim

Purpose: Specifies if the leading and trailing blanks of a CSV field value must be trimmed. The default value is false.
Data Type: Boolean
Mandatory (Y/N): N
Example: "trim" : "true"

CSV file in OCI Object Storage Bucket

The configuration file format for the CSV file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below. The CSV file must conform to the RFC4180 format.

You can migrate a CSV file in the OCI Object Storage bucket by specifying the name of the bucket in the source configuration template.

A sample CSV file in the OCI Object Storage bucket is as follows:

1,"Computer Science","San Francisco","2500"
2,"Bio-Technology","Los Angeles","1200"
3,"Journalism","Las Vegas","1500"
4,"Telecommunication","San Francisco","2500"

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Source Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "csv",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
   "hasHeader" : <true | false>,
   "columns" : ["column1", "column2", ....],
   "csvOptions" : {         
     "encoding" : "<character set encoding>",
     "trim" : <true | false>
   }
 }

Source Parameters

Common Configuration Parameters

type
Use "type" : "object_storage_oci"
format
Use "format" : "csv"
endpoint
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"
namespace
Example: "namespace" : "my-namespace"
bucket
Example: "bucket" : "my-bucket"
Note:
- The NoSQL Database Migrator imports all the files with the .csv or .CSV extension object-wise and copies them into a single table in the same order.
- The CSV files must contain only scalar values. Importing CSV files containing complex types such as MAP, RECORD, ARRAY, and JSON is not supported. The NoSQL Database Migrator tool does not check for the correctness of the data in the input CSV file. The NoSQL Database Migrator tool supports the importing of CSV data that conforms to the RFC4180 format. CSV files containing data that does not conform to the RFC4180 standard may not get copied correctly or may result in an error. If the input data is corrupted, the NoSQL Database Migrator tool will not parse the CSV records. If any errors are encountered during migration, the NoSQL Database Migrator tool logs the information about the failed input records for debugging and informative purposes. For more details, see Logging Migrator Progress in Using Oracle NoSQL Data Migrator .
prefix
Example:
1. "prefix" : "my_table/Data/000000.csv" (migrates only 000000.csv)
2. "prefix" : "my_table/Data" (migrates all the objects with prefix my_table/Data)
credentials
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"
credentialsProfile
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile" : "ADMIN_USER"
useInstancePrincipal
Example: "useInstancePrincipal" : true

Unique Configuration Parameters

hasHeader
columns
csvOptions
csvOptions.encoding
csvOptions.trim

hasHeader

Purpose: Specifies if the CSV file has a header or not. If this is set to true, the first line is ignored. If it is set to false, the first line is considered a CSV record. The default value is false.
Data Type: Boolean
Mandatory (Y/N): N
Example: "hasHeader" : "false"

columns

Purpose: Specifies the list of NoSQL Database table column names. The order of the column names indicates the mapping of the CSV file fields with corresponding NoSQL Database table columns. If the order of the input CSV file columns does not match the existing or newly created NoSQL Database table columns, you can map the ordering using this parameter. Also, when importing into a table that has an Identity Column, you can skip the Identity column name in the columns parameter.
Note:
- If the NoSQL Database table has additional columns that are not available in the CSV file, the values of the missing columns are updated with the default value as defined in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration. For more information on default values, see Data Type Definitions section in the SQL Reference Guide.
- If the CSV file has additional columns that are not defined in the NoSQL Database table, the additional column information is ignored.
- If any value in the CSV record is empty, it is set to the default value of the corresponding columns in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration.
Data Type: Array of Strings
Mandatory (Y/N): N
Example: "columns" : ["table_column_1", "table_column_2"]

csvOptions

Purpose: Specifies the formatting options for a CSV file. Provide the character set encoding format of the CSV file and choose whether or not to trim the blank spaces.
Data Type: Object
Mandatory (Y/N): N

csvOptions.encoding

Purpose: Specifies the character set to decode the CSV file. The default value is UTF-8. The supported character sets are US-ASCII, ISO-8859-1, UTF-8, and UTF-16.
Data Type: String
Mandatory (Y/N): N
Example: "encoding" : "UTF-8"

csvOptions.trim

Purpose: Specifies if the leading and trailing blanks of a CSV field value must be trimmed. The default value is false.
Data Type: Boolean
Mandatory (Y/N): N
Example: "trim" : "true"

Sink Configuration Templates

Learn about the sink configuration file formats for each valid sink and the purpose of each configuration parameter.

For the configuration file template, see Configuration File in Terminology used with NoSQL Data Migrator.

For details on valid source formats for each of the sinks, see Source Configuration Templates.

Topics

The following topics describe the sink configuration templates referred by Oracle NoSQL Database Migrator to copy the data from a valid source to the given sink.

JSON File Sink
Specified JSON file.
Parquet File
Parquet file in the specified directory.
JSON File in OCI Object Storage Bucket
JSON file in the specified OCI Object Storage bucket.
Parquet File in OCI Object Storage Bucket
Parquet file in the specified OCI Object Storage bucket.
Oracle NoSQL Database
Specified table in Oracle NoSQL Database.
Oracle NoSQL Database Cloud Service
Specified table in Oracle NoSQL Database Cloud Service.

JSON File Sink

The configuration file format for JSON File as a sink of NoSQL Database Migrator is shown below.

Sink Configuration Template

"sink" : {
  "type" : "file",
  "format" : "json",
  "dataPath": "</path/to/a/file>",
  "schemaPath" : "<path/to/a/file>",
  "pretty" : <true|false>,
  "useMultiFiles" : <true|false>,
  "chunkSize" : <size in MB>
}

Sink Parameters

Common Configuration Parameters

type
Use "type" : "file"
format
Use "format" : "json"
chunkSize
Example: "chunkSize" : 40

Note:
This parameter is applicable ONLY when the useMultiFiles parameter is set to true.

Unique Configuration Parameters

dataPath
schemaPath
pretty
useMultiFiles

dataPath

Purpose: Specifies the absolute path to a file where the source data will be copied in the JSON format.

If the file does not exist in the specified data path, the NoSQL Database Migrator creates it. If it exists, the NoSQL Database Migrator will overwrite its contents with the source data.

You must ensure that the parent directory in the data path is valid for the specified file.

Note:
If the useMultiFiles parameter is set to true, specify the path to a directory else specifiy the path to the file.
Data Type: string
Mandatory (Y/N): Y
Example:
- With useMultiFiles parameter set to true
  "dataPath" :"/home/user/data"
- With useMultiFiles parameter not specified or it is set to false
  "dataPath" :"/home/user/sample.json"

schemaPath

Purpose: Specifies the absolute path to a file to write table schema information provided by the source.

If this value is not defined, the source schema information will not be migrated to the sink. If this value is specified, the migrator utility writes the schema of the source table into the file specified here.

The schema information is written as one DDL command per line in this file. If the file does not exist in the specified data path, NoSQL Database Migrator creates it. If it exists already, NoSQL Database Migrator will overwrite its contents with the source data. You must ensure that the parent directory in the data path is valid for the specified file.
Data Type: string
Mandatory (Y/N): N
Example: "schemaPath" : "/home/user/schema_file"

pretty

Purpose: Specifies whether or not to beautify the JSON output to increase readability.

If not specified, it defaults to false.
Data Type: boolean
Mandatory (Y/N): N
Example: "pretty" : true

useMultiFiles

Purpose: Specifies whether or not to split the NoSQL table data into multiple files when migrating source data to a file.

If not specified, it defaults to false.

If set to true, when migrating source data to a file, the NoSQL table data is split into multiple smaller files. For example, <chunk>.json, where chunk=000000, 000001, 000002, and so forth.
```
dataPath
         |--000000.json
         |--000001.json
```
Data Type: boolean
Mandatory (Y/N): N
Example: "useMultiFiles" : true

Parquet File

The configuration file format for Parquet File as a sink of NoSQL Database Migrator is shown below.

Sink Configuration Template

"sink" : {
  "type" : "file",
  "format" : "parquet",
  "dataPath": "</path/to/a/dir>",
  "chunkSize" : <size in MB>,
  "compression": "<SNAPPY|GZIP|NONE>",
  "parquetOptions": {
    "useLogicalJson": <true|false>,
    "useLogicalEnum": <true|false>,
    "useLogicalUUID": <true|false>,     
    "truncateDoubleSpecials": <true|false>
  }
}

Sink Parameters

Common Configuration Parameters

type
Use "type" : "file"
format
Use "format" : "parquet"
chunkSize
Example: "chunkSize" : 40

Unique Configuration Parameters

dataPath
compression
parquetOptions
parquetOptions.useLogicalJson
parquetOptions.useLogicalEnum
parquetOptions.useLogicalUUID
parquetOptions.truncateDoubleSpecials

dataPath

Purpose: Specifies the path to a directory for storing the migrated NoSQL table data. Ensure that the directory already exists and has read and write permissions.
Data Type: string
Mandatory (Y/N): Y
Example: "dataPath" : "/home/user/migrator/my_table"

compression

Purpose: Specifies the compression type to use to compress the Parquet data. Valid values are SNAPPY, GZIP, and NONE.

If not specified, it defaults to SNAPPY.
Data Type: string
Mandatory (Y/N): N
Example: "compression" : "GZIP"

parquetOptions

Purpose: Specifies the options to select Parquet logical types for NoSQL ENUM, JSON, and UUID columns.

If you do not specify this parameter, the NoSQL Database Migrator writes the data of ENUM, JSON, and UUID columns as String.
Data Type: object
Mandatory (Y/N): N

parquetOptions.useLogicalJson

Purpose: Specifies whether or not to write NoSQL JSON column data as Parquet logical JSON type. For more information, see Parquet Logical Type Definitions.

If not specified or set to false, NoSQL Database Migrator writes the NoSQL JSON column data as String.
Data Type: boolean
Mandatory (Y/N): N
Example: "useLogicalJson" : true

parquetOptions.useLogicalEnum

Purpose: Specifies whether or not to write NoSQL ENUM column data as Parquet logical ENUM type. For more information, see Parquet Logical Type Definitions.

If not specified or set to false, NoSQL Database Migrator writes the NoSQL ENUM column data as String.
Data Type: boolean
Mandatory (Y/N): N
Example: "useLogicalEnum" : true

parquetOptions.useLogicalUUID

Purpose: Specifies whether or not to write NoSQL UUID column data as Parquet logical UUID type. For more information, see Parquet Logical Type Definitions.

If not specified or set to false, NoSQL Database Migrator writes the NoSQL UUID column data as String.
Data Type: boolean
Mandatory (Y/N): N
Example: "useLogicalUUID" : true

parquetOptions.truncateDoubleSpecials

Purpose: Specifies whether or not to truncate the double +Infinity, -Infinity, and NaN values.
By default, it is set to false. If set to true,
- Positive_Infinity is truncated to Double.MAX_VALUE.
- NEGATIVE_INFINITY is truncated to -Double.MAX_VALUE.
- NaN is truncated to 9.9999999999999990E307.
Data Type: boolean
Mandatory (Y/N): N
Example: "truncateDoubleSpecials" : true

JSON File in OCI Object Storage Bucket

The configuration file format for JSON file in OCI Object Storage bucket as a sink of NoSQL Database Migrator is shown below.

Note:

The valid source types for OCI Object Storage as the sink are nosqldb and nosqldb_cloud.

Sink Configuration Template

"sink" : {
  "type" : "object_storage_oci",
  "format" : "json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "chunkSize" : <size in MB>,
  "pretty" : <true|false>,
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,  
  "useDelegationToken" : <true|false>
}

Sink Parameters

Common Configuration Parameters

type
Use "type" : "object_storage_oci"
format
Use "format" : "json"
endpoint
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"
namespace
Example: "namespace" : "my-namespace"
bucket
Example: "bucket" : "my-bucket"
prefix
Schema is migrated to the <prefix>/Schema/schema.ddl file and source data is migrated to the <prefix>/Data/<chunk>.json files, where chunk=000000.json, 000001.json, and so forth.
Example:
1. "prefix" : "my_export"
2. "prefix" : "my_export/2021-04-05/"
chunkSize
Example: "chunkSize" : 40
credentials
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"
credentialsProfile
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile" : "ADMIN_USER"
useInstancePrincipal
Example: "useInstancePrincipal" : true
useDelegationToken
Example: "useDelegationToken" : true

Note:
The authentication with delegation token is supported only when the NoSQL Database Migrator is running from a Cloud Shell.

Unique Configuration Parameter

pretty

Purpose: Specifies whether or not to beautify the JSON output to increase readability.

If not specified, it defaults to false.
Data Type: boolean
Mandatory (Y/N): N
Example: "pretty" : true

Parquet File in OCI Object Storage Bucket

The configuration file format for Parquet file in OCI Object Storage bucket as a sink of NoSQL Database Migrator is shown below.

Note:

The valid source types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Sink Configuration Template

"sink" : {
  "type" : "object_storage_oci",
  "format" : "parquet",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "chunkSize" : <size in MB>,
  "compression": "<SNAPPY|GZIP|NONE>",
  "parquetOptions": {
    "useLogicalJson": <true|false>,
    "useLogicalEnum": <true|false>,
    "useLogicalUUID": <true|false>,
    "truncateDoubleSpecials": <true|false>
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
  "useDelegationToken" : <true|false>
}

Sink Parameters

Common Configuration Parameters

type
Use "type" : "object_storage_oci"
format
Use "format" : "parquet"
endpoint
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"
namespace
Example: "namespace" : "my-namespace"
bucket
Example: "bucket" : "my-bucket"
prefix
Source data is migrated to the <prefix>/Data/<chunk>.parquet files, where chunk=000000.parquet, 000001.parquet, and so forth.
Example:
1. "prefix" : "my_export"
2. "prefix" : "my_export/2021-04-05/"
chunkSize
Example: "chunkSize" : 40
credentials
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"
credentialsProfile
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile" : "ADMIN_USER"
useInstancePrincipal
Example: "useInstancePrincipal" : true
useDelegationToken
Example: "useDelegationToken" : true

Note:
The authentication with delegation token is supported only when the NoSQL Database Migrator is running from a Cloud Shell.

Unique Configuration Parameter

compression
parquetOptions
parquetOptions.useLogicalJson
parquetOptions.useLogicalEnum
parquetOptions.useLogicalUUID
parquetOptions.truncateDoubleSpecials

compression

Purpose: Specifies the compression type to use to compress the Parquet data. Valid values are SNAPPY, GZIP, and NONE.

If not specified, it defaults to SNAPPY.
Data Type: string
Mandatory (Y/N): N
Example: "compression" : "GZIP"

parquetOptions

Purpose: Specifies the options to select Parquet logical types for NoSQL ENUM, JSON, and UUID columns.

If you do not specify this parameter, the NoSQL Database Migrator writes the data of ENUM, JSON, and UUID columns as String.
Data Type: object
Mandatory (Y/N): N

parquetOptions.useLogicalJson

Purpose: Specifies whether or not to write NoSQL JSON column data as Parquet logical JSON type. For more information, see Parquet Logical Type Definitions.

If not specified or set to false, NoSQL Database Migrator writes the NoSQL JSON column data as String.
Data Type: boolean
Mandatory (Y/N): N
Example: "useLogicalJson" : true

parquetOptions.useLogicalEnum

Purpose: Specifies whether or not to write NoSQL ENUM column data as Parquet logical ENUM type. For more information, see Parquet Logical Type Definitions.

If not specified or set to false, NoSQL Database Migrator writes the NoSQL ENUM column data as String.
Data Type: boolean
Mandatory (Y/N): N
Example: "useLogicalEnum" : true

parquetOptions.useLogicalUUID

Purpose: Specifies whether or not to write NoSQL UUID column data as Parquet logical UUID type. For more information, see Parquet Logical Type Definitions.

If not specified or set to false, NoSQL Database Migrator writes the NoSQL UUID column data as String.
Data Type: boolean
Mandatory (Y/N): N
Example: "useLogicalUUID" : true

parquetOptions.truncateDoubleSpecials

Purpose: Specifies whether or not to truncate the double +Infinity, -Infinity, and NaN values.
By default, it is set to false. If set to true,
- Positive_Infinity is truncated to Double.MAX_VALUE.
- NEGATIVE_INFINITY is truncated to -Double.MAX_VALUE.
- NaN is truncated to 9.9999999999999990E307.
Data Type: boolean
Mandatory (Y/N): N
Example: "truncateDoubleSpecials" : true

Oracle NoSQL Database

The configuration file format for Oracle NoSQL Database as a sink of NoSQL Database Migrator is shown below.

Sink Configuration Template

"sink" : {
  "type": "nosqldb",
  "storeName" : "<store name>",
  "helperHosts" : ["hostname1:port1","hostname2:port2,..."],
  "security" : "</path/to/store/credentials/file>",
  "table" : "<fully qualified table name>",
  "includeTTL": <true|false>,
  "ttlRelativeDate": "<date-to-use in UTC>",
  "schemaInfo" : {
    "schemaPath" : "</path/to/a/schema/file>",
    "defaultSchema" : <true|false>,
    "useSourceSchema" : <true|false>,
    "DDBPartitionKey" : <"name:type">,
    "DDBSortKey" : "<name:type>"
  },
  "overwrite" : <true|false>,
  "requestTimeoutMs" : <timeout in milli seconds>
}

Sink Parameters

Common Configuration Parameter

type
Use "type" : "nosqldb"

security

Example:

"security" : "/home/user/client.credentials"

Example security file content for password file based authentication:

oracle.kv.password.noPrompt=true
oracle.kv.auth.username=admin
oracle.kv.auth.pwdfile.file=/home/nosql/login.passwd
oracle.kv.transport=ssl
oracle.kv.ssl.trustStore=/home/nosql/client.trust
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

Example security file content for wallet based authentication:

oracle.kv.password.noPrompt=true
oracle.kv.auth.username=admin
oracle.kv.auth.wallet.dir=/home/nosql/login.wallet
oracle.kv.transport=ssl
oracle.kv.ssl.trustStore=/home/nosql/client.trust
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

requestTimeoutMs
Example: "requestTimeoutMs" : 5000

Unique Configuration Parameter

storeName
helperHosts
table
includeTTL
ttlRelativeDate
schemaInfo
schemaInfo.schemaPath
schemaInfo.defaultSchema
schemaInfo.useSourceSchema
schemaInfo.DDBPartitionKey
schemaInfo.DDBSortKey
overwrite

storeName

Purpose: Name of the Oracle NoSQL Database store.
Data Type: string
Mandatory (Y/N): Y
Example: "storeName" : "kvstore"

helperHosts

Purpose: A list of host and registry port pairs in the hostname:port format. Delimit each item in the list using a comma. You must specify at least one helper host.
Data Type: array of strings
Mandatory (Y/N): Y
Example: "helperHosts" : ["localhost:5000","localhost:6000"]

table

Purpose: Specifies the table name to store the migrated data.

Format: [namespace_name:]<table_name>

If the table is in the DEFAULT namespace, you can omit the namespace_name. The table must exist in the store during the migration, and its schema must match with the source data.

If the table is not available in the sink, you can use the schemaInfo parameter to instruct the NoSQL Database Migrator to create the table in the sink.
Data Type: string
Mandatory (Y/N): Y
Example:
- With the DEFAULT namespace "table" :"mytable"
- With a non-default namespace "table" : "mynamespace:mytable"
- To specify a child table "table" : "mytable.child"
  
  Note:
  You can migrate the child tables from a valid data source to Oracle NoSQL Database. The NoSQL Database Migrator copies only a single table in each execution. Ensure that the parent table is migrated before the child table.

includeTTL

Purpose: Specifies whether or not to include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.

If you do not specify this parameter, it defaults to false. In that case, the NoSQL Database Migrator does not include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.
If set to true, the NoSQL Database Migrator tool performs the following checks on the TTL metadata when importing a table row:
- If you import a row that does not have _metadata definition, the NoSQL Database Migrator tool sets the TTL to 0, which means the row never expires.
- If you import a row that has _metadata definition, the NoSQL Database Migrator tool compares the TTL value against a Reference Time when a row gets imported. If the row has already expired relative to the Reference Time, then it is skipped. If the row has not expired, then it is imported along with the TTL value. By default, the Reference Time of import operation is the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running. But you can also set a custom Reference Time using the ttlRelativeDate configuration parameter if you want to extend the expiration time and import rows that would otherwise expire immediately.
  The formula to calculate the expiration time of a row is as follows:
```
expiration = (TTL value of source row in milliseconds - Reference Time in milliseconds)
if (expiration <= 0) then it indicates that row has expired.
```
  Note:
  Since Oracle NoSQL TTL boundaries are in hours and days, in some cases, the TTL of the imported row might get adjusted to the nearest hour or day. For example, consider a row that has expiration value of 1629709200000 (2021-08-23 09:00:00) and Reference Time value is 1629707962582 (2021-08-23 08:39:22). Here, even though the row is not expired relative to the Reference Time when this data gets imported, the new TTL for the row is 1629712800000 (2021-08-23 10:00:00).
Data Type: boolean
Mandatory (Y/N): N
Example: "includeTTL" : true

ttlRelativeDate

Purpose: Specify a UTC date in the YYYY-MM-DD hh:mm:ss format used to set the TTL expiry of table rows during importing into the Oracle NoSQL Database.

If a table row in the data you are exporting has expired, you can set the ttlRelativeDate parameter to a date before the expiration time of the table row in the exported data.

If you do not specify this parameter, it defaults to the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running.
Data Type: date
Mandatory (Y/N): N
Example: "ttlRelativeDate" : "2021-01-03 04:31:17"
Let us consider a scenario where table rows expire after seven days from 1-Jan-2021. After exporting this table, on 7-Jan-2021, you run into an issue with your table and decide to import the data. The table rows are going to expire in one day (data expiration date minus the default value of ttlRelativedate configuration parameter, which is the current date). But if you want to extend the expiration date of table rows to five days instead of one day, use the ttlRelativeDate parameter and choose an earlier date. Therefore, in this scenario if you want to extend expiration time of the table rows by five days, set the value of ttlRelativeDate configuration parameters to 3-Jan-2021, which is used as Reference Time when table rows get imported.

schemainfo

Purpose: Specifies the schema for the data being migrated. If this is not specified, the NoSQL Database Migrator assumes that the table already exists in the sink's store.
Data Type: Object
Mandatory (Y/N): N

schemaInfo.schemaPath

Purpose: Specifies the absolute path to a file containing DDL statements for the NoSQL table.

The NoSQL Database Migrator executes the DDL commands listed in this file before migrating the data.

The NoSQL Database Migrator does not support more than one DDL statement per line in the schemaPath file.
Data Type: string
Mandatory (Y/N): N

Note:
defaultSchema and schemaPath are mutually exclusive.
Example: "schemaPath" : "/home/user/schema_file"

schemaInfo.defaultSchema

Purpose: Setting this parameter to true instructs the NoSQL Database Migrator to create a table with default schema. The default schema is defined by the migrator itself. For more information about default schema definitions, see Default Schema in Using Oracle NoSQL Data Migrator .
Data Type: boolean
Mandatory (Y/N): N

Note:
defaultSchema and schemaPath are mutually exclusive.

schemaInfo.useSourceSchema

Purpose: Specifies whether or not the sink uses the table schema definition provided by the source when migrating NoSQL tables.
Data Type: boolean
Mandatory (Y/N): N

Note:
defaultSchema, schemaPath, and useSourceSchema parameters are mutually exclusive. Specify only one of these parameters.

Example:

With Default Schema:

"schemaInfo" : {
  "defaultSchema" : true
}

With a pre-defined schema:

"schemaInfo" : {
 "schemaPath" : "<complete/path/to/the/schema/definition/file>"
}

With source schema:

"schemaInfo" : {
  "useSourceSchema" : true
}

schemaInfo.DDBPartitionKey

Purpose: Specifies the DynamoDB partition key and the corresponding Oracle NoSQL Database type to be used in the sink Oracle NoSQL Database table. This key will be used as a NoSQL DB table shard key. This is applicable only when defaultSchema is set to true and the source format is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.
Mandatory (Y/N): Y, if defaultSchema is true and the source is dynamodb_json.
Example: "DDBPartitionKey" : "PersonID:INTEGER"

Note:
If the partition key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

schemaInfo.DDBSortKey

Purpose: Specifies the DynamoDB sort key and its corresponding Oracle NoSQL Database type to be used in the target Oracle NoSQL Database table. If the importing DynamoDB table does not have a sort key, this attribute must not be set. This key will be used as a non-shard portion of the primary key in the NoSQL DB table. This is applicable only when defaultSchema is set to true and the source is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.
Mandatory (Y/N): N
Example: "DDBSortKey" : "Skey:STRING"

Note:
If the sort key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

overwrite

Purpose: Indicates the behavior of NoSQL Database Migrator when the record being migrated from the source is already present in the sink.

If the value is set to false, when migrating tables the NoSQL Database Migrator skips those records for which the same primary key already exists in the sink.

If the value is set to true, when migrating tables the NoSQL Database Migrator overwrites those records for which the same primary key already exists in the sink.

If not specified, it defaults to true.
Data Type: boolean
Mandatory (Y/N): N
Example: "overwrite" : false

Oracle NoSQL Database Cloud Service

The configuration file format for Oracle NoSQL Database Cloud Service as a sink of NoSQL Database Migrator is shown below.

Sink Configuration Template

"sink" : {
  "type" : "nosqldb_cloud",
  "endpoint" : "<Oracle NoSQL Cloud Service Endpoint>",
  "table" : "<table name>",
  "compartment" : "<OCI compartment name or id>",
  "includeTTL": <true|false>,
  "ttlRelativeDate" : "<date-to-use in UTC>",  
  "schemaInfo" : {
    "schemaPath" : "</path/to/a/schema/file>",
    "defaultSchema" : <true|false>,
    "useSourceSchema" : <true|false>,
    "DDBPartitionKey" : <"name:type">,
    "DDBSortKey" : "<name:type>",
    "onDemandThroughput" : <true|false>,
    "readUnits" : <table read units>,
    "writeUnits" : <table write units>,
    "storageSize" : <storage size in GB>
   },
  "credentials" : "</path/to/oci/credential/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
  "useDelegationToken" : <true|false>,
  "writeUnitsPercent" : <table writeunits percent>,
  "requestTimeoutMs" : <timeout in milli seconds>,
  "overwrite" : <true|false>
}

Sink Parameters

Common Configuration Parameters

type
Use "type" : "nosqldb_cloud"
endpoint
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"
credentials
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"
credentialsProfile
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile" : "ADMIN_USER"
useInstancePrincipal
Example: "useInstancePrincipal" : true
useDelegationToken
Example: "useDelegationToken" : true

Note:
The authentication with delegation token is supported only when the NoSQL Database Migrator is running from a Cloud Shell.
requestTimeoutMs
Example: "requestTimeoutMs" : 5000

Unique Configuration Parameter

table
compartment
includeTTL
ttlRelativeDate
schemaInfo
schemaInfo.schemaPath
schemaInfo.defaultSchema
schemaInfo.useSourceSchema
schemaInfo.DDBPartitionKey
schemaInfo.DDBSortKey
schemaInfo.onDemandThroughput
schemaInfo.readUnits
schemaInfo.writeUnits
schemaInfo.storageSize
writeUnitsPercent
overwrite

table

Purpose: Specifies the table name to store the migrated data.

You must ensure that this table exists in your Oracle NoSQL Database Cloud Service. Otherwise, you have to use the schemaInfo object in the sink configuration to instruct the NoSQL Database Migrator to create the table.

The schema of this table must match the source data.
Data Type: string
Mandatory (Y/N): Y
Example:
- To specify a table "table" : "mytable"
- To specify a child table "table" : "mytable.child"
  
  Note:
  You can migrate the child tables from a valid data source to Oracle NoSQL Database Cloud Service. The NoSQL Database Migrator copies only a single table in each execution. Ensure that the parent table is migrated before the child table.

compartment

Purpose: Specifies the name or OCID of the compartment in which the table resides.

If you do not provide any value, it defaults to the root compartment.

You can find your compartment's OCID from the Compartment Explorer window under Governance in the OCI Cloud Console.
Data Type: string
Mandatory (Y/N): Y, if the table is not in the root compartment of the tenancy OR when the useInstancePrincipal parameter is set to true.

Note:
If the useInstancePrincipal parameter is set to true, the compartment must specify the compartment OCID and not the name.
Example:
- Compartment name
  
  "compartment" : "mycompartment"
- Compartment name qualified with its parent compartment
  
  "compartment" : "parent.childcompartment"
- No value provided. Defaults to the root compartment.
  
  "compartment": ""
- Compartment OCID
  
  "compartment" : "ocid1.tenancy.oc1...4ksd"

includeTTL

Purpose: Specifies whether or not to include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.

If you do not specify this parameter, it defaults to false. In that case, the NoSQL Database Migrator does not include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.
If set to true, the NoSQL Database Migrator tool performs the following checks on the TTL metadata when importing a table row:
- If you import a row that does not have _metadata definition, the NoSQL Database Migrator tool sets the TTL to 0, which means the row never expires.
- If you import a row that has _metadata definition, the NoSQL Database Migrator tool compares the TTL value against a Reference Time when a row gets imported. If the row has already expired relative to the Reference Time, then it is skipped. If the row has not expired, then it is imported along with the TTL value. By default, the Reference Time of import operation is the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running. But you can also set a custom Reference Time using the ttlRelativeDate configuration parameter if you want to extend the expiration time and import rows that would otherwise expire immediately.
  The formula to calculate the expiration time of a row is as follows:
```
expiration = (TTL value of source row in milliseconds - Reference Time in milliseconds)
if (expiration <= 0) then it indicates that row has expired.
```
  Note:
  Since Oracle NoSQL TTL boundaries are in hours and days, in some cases, the TTL of the imported row might get adjusted to the nearest hour or day. For example, consider a row that has expiration value of 1629709200000 (2021-08-23 09:00:00) and Reference Time value is 1629707962582 (2021-08-23 08:39:22). Here, even though the row is not expired relative to the Reference Time when this data gets imported, the new TTL for the row is 1629712800000 (2021-08-23 10:00:00).
Data Type: boolean
Mandatory (Y/N): N
Example: "includeTTL" : true

ttlRelativeDate

Purpose: Specify a UTC date in the YYYY-MM-DD hh:mm:ss format used to set the TTL expiry of table rows during importing into the Oracle NoSQL Database.

If a table row in the data you are exporting has expired, you can set the ttlRelativeDate parameter to a date before the expiration time of the table row in the exported data.

If you do not specify this parameter, it defaults to the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running.
Data Type: date
Mandatory (Y/N): N
Example: "ttlRelativeDate" : "2021-01-03 04:31:17"
Let us consider a scenario where table rows expire after seven days from 1-Jan-2021. After exporting this table, on 7-Jan-2021, you run into an issue with your table and decide to import the data. The table rows are going to expire in one day (data expiration date minus the default value of ttlRelativedate configuration parameter, which is the current date). But if you want to extend the expiration date of table rows to five days instead of one day, use the ttlRelativeDate parameter and choose an earlier date. Therefore, in this scenario if you want to extend expiration time of the table rows by five days, set the value of ttlRelativeDate configuration parameters to 3-Jan-2021, which is used as Reference Time when table rows get imported.

schemaInfo

Purpose: Specifies the schema for the data being migrated.

If you do not specify this parameter, the NoSQL Database Migrator assumes that the table already exists in your Oracle NoSQL Database Cloud Service.

If this parameter is not specified and the table does not exist in the sink, the migration fails.
Data Type: Object
Mandatory (Y/N): N

schemaInfo.schemaPath

Purpose: Specifies the absolute path to a file containing DDL statements for the NoSQL table.

The NoSQL Database Migrator executes the DDL commands listed in this file before migrating the data.

The NoSQL Database Migrator does not support more than one DDL statement per line in the schemaPath file.
Data Type: string
Mandatory (Y/N): N

Note:
defaultSchema and schemaPath are mutually exclusive.
Example: "schemaPath" : "/home/user/schema_file"

schemaInfo.defaultSchema

Purpose: Setting this parameter to Yes instructs the NoSQL Database Migrator to create a table with default schema. The default schema is defined by the migrator itself. For more information about default schema definitions, see Default Schema in Using Oracle NoSQL Data Migrator .
Data Type: boolean
Mandatory (Y/N): N

Note:
defaultSchema and schemaPath are mutually exclusive.

schemaInfo.useSourceSchema

Purpose: Specifies whether or not the sink uses the table schema definition provided by the source when migrating NoSQL tables.
Data Type: boolean
Mandatory (Y/N): N

Note:
defaultSchema, schemaPath, and useSourceSchema parameters are mutually exclusive. Specify only one of these parameters.

Example:

With Default Schema:

"schemaInfo": {
  "defaultSchema": true,
  "readUnits": 100,
  "writeUnits": 60,
  "storageSize": 1
}

With a pre-defined schema:

"schemaInfo": {
  "schemaPath": "<complete/path/to/the/schema/definition/file>",
  "readUnits": 100,
  "writeUnits": 100,
  "storageSize": 1
}

With source schema:

"schemaInfo": {
  "useSourceSchema": true,
  "readUnits": 100,
  "writeUnits": 60,
  "storageSize": 1
}

schemaInfo.DDBPartitionKey

Purpose: Specifies the DynamoDB partition key and the corresponding Oracle NoSQL Database type to be used in the sink Oracle NoSQL Database table. This key will be used as a NoSQL DB table shard key. This is applicable only when defaultSchema is set to true and the source format is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.
Mandatory (Y/N): Y, if defaultSchema is true and the source is dynamodb_json.
Example: "DDBPartitionKey" : "PersonID:INTEGER"

Note:
If the partition key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

schemaInfo.DDBSortKey

Purpose: Specifies the DynamoDB sort key and its corresponding Oracle NoSQL Database type to be used in the target Oracle NoSQL Database table. If the importing DynamoDB table does not have a sort key, this attribute must not be set. This key will be used as a non-shard portion of the primary key in the NoSQL DB table. This is applicable only when defaultSchema is set to true and the source is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.
Mandatory (Y/N): N
Example: "DDBSortKey" : "Skey:STRING"

Note:
If the sort key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

schemaInfo.onDemandThroughput

Purpose: Specifies to create the table with on-demand read and write throughput. If this parameter is not set, the table is created with provisioned capacity.
The default value is false.

Note:
This parameter is not applicable for child tables as they share the throughput of the top-level parent table.
Data Type: Boolean
Mandatory (Y/N): N
Example: "onDemandThroughput" : "true"

schemaInfo.readUnits

Purpose: Specifies the read throughput of the new table.
Note:
- This parameter is not applicable for tables provisioned with on-demand capacity.
- This parameter is not applicable for child tables as they share the read throughput of the top-level parent table.
Data Type: integer
Mandatory (Y/N): Y, if the table is not a child table or if schemaInfo.onDemandThroughput parameter is set to false, else N.
Example: "readUnits" : 100

schemaInfo.writeUnits

Purpose: Specifies the write throughput of the new table.
Note:
- This parameter is not applicable for tables provisioned with on-demand capacity.
- This parameter is not applicable for child tables as they share the write throughput of the top-level parent table.
Data Type: integer
Mandatory (Y/N): Y, if the table is not a child table or if schemaInfo.onDemandThroughput parameter is set to false, else N.
Example: "writeUnits" : 100

schemaInfo.storageSize

Purpose: Specifies the storage size of the new table in GB.

Note:
This parameter is not applicable for child tables as they share the storage size of the top-level parent table.
Data Type: integer
Mandatory (Y/N): Y, if the table is not a child table, else N.

Example:

With schemaPath

"schemaInfo" : { 
  "schemaPath" : "</path/to/a/schema/file>",
  "readUnits" : 500,
  "writeUnits" : 1000,
  "storageSize" : 5 }

With defaultSchema

"schemaInfo" : {
  "defaultSchema" :Yes,
  "readUnits" : 500,   
  "writeUnits" : 1000,   
  "storageSize" : 5  
}

writeUnitsPercent

Purpose: Specifies the Percentage of table write units to be used during the migration activity. The amount of time required to migrate data is directly proportional to this attribute.

The default value is 90. The valid range is any integer between 1 to 100.

See Troubleshooting the Oracle NoSQL Database Migrator to learn how to use this attribute to improve the data migration speed.
Data Type: integer
Mandatory (Y/N): N
Example: "writeUnitsPercent" : 90

overwrite

Purpose: Indicates the behavior of NoSQL Database Migrator when the record being migrated from the source is already present in the sink.

If the value is set to false, when migrating tables the NoSQL Database Migrator skips those records for which the same primary key already exists in the sink.

If the value is set to true, when migrating tables the NoSQL Database Migrator overwrites those records for which the same primary key already exists in the sink.

If not specified, it defaults to true.
Data Type: boolean
Mandatory (Y/N): N
Example: "overwrite" : false

Transformation Configuration Templates

This topic explains the configuration parameters for the different transformations supported by the Oracle NoSQL Database Migrator. For the complete configuration file template, see Configuration File in Terminology used with NoSQL Data Migrator.

Oracle NoSQL Database Migrator lets you modify the data, that is, add data transformations as part of the migration activity. You can define multiple transformations in a single migration. In such a case, the order of transformations is vital because the source data undergoes each transformation in the given order. The output of one transformation becomes the input to the next one in the migrator pipeline.

The different transformations supported by the NoSQL Data Migrator are:

Table - Transformations

Transformation Config Attribute	You can use this transformation to ...
`ignoreFields`	Ignore the identified columns from the source row before writing to the sink.
`includeFields`	Include the identified columns from the source row before writing to the sink.
`renameFields`	Rename the identified columns from the source row before writing to the sink.
`aggregateFields`	Aggregate multiple columns from the source into a single column in the sink. As part of this transformation, you can also identify the columns that you want to exclude in the aggregation. Those fields will be skipped from the aggregated column.

You can find the configuration template for each supported transformation below.

ignoreFields

The configuration file format for the ignoreFields transformation is shown below.

Transformation Configuration Template

"transforms" : {
  "ignoreFields" : ["<field1>","<field2>",...]
}

Transformation Parameter

ignoreFields

Purpose: An array of the column names to be ignored from the source records.

Note:
You can supply only top-level fields. Transformations can not be applied on the data in the nested fields.
Data Type: array of strings
Mandatory (Y/N): Y
Example: To ignore the columns named "name" and "address" from the source record:

"ignoreFields" : ["name","address"]

includeFields

The configuration file format for the includeFields transformation is shown below.

Transformation Configuration Template

"transforms" : {
  "includeFields" : ["<field1>","<field2>",...]
}

Transformation Parameter

includeFields

Purpose: An array of the column names to be included from the source records. It only includes the fields specified in the array, the rest of the fields are ignored.

Note:
The NoSQL Database Migrator tool throws an error if you specify an empty array. Additionally, you can specify only the top-level fields. The NoSQL Database Migrator tool does not apply transformations to the data in the nested fields.
Data Type: array of strings
Mandatory (Y/N): Y
Example: To ignore the columns named "age" and "gender" from the source record:

"includeFields" : ["age","gender"]

renameFields

The configuration file format for the renameFields transformation is shown below.

Transformation Configuration Template

"transforms" : {
  "renameFields" : {
    "<old_name>" : "<new_name>",
    "<old_name>" : "<new_name>,"
    .....
  }
}

Transformation Parameter

renameFields

Purpose: Key-Value pairs of the old and new names of the columns to be renamed.

Note:
You can supply only top-level fields. Transformations can not be applied on the data in the nested fields.
Data Type: JSON object
Mandatory (Y/N): Y
Example: To rename the column named "residence" to "address" and the column named "_id" to "id":

"renameFields" : { "residence" : "address", "_id" : "id" }

aggregateFields

The configuration file format for the aggregateFields transformation is shown below.

Transformation Configuration Template

"transforms" : {
  "aggregateFields" : {
    "fieldName" : "name of the new aggregate field",
    "skipFields" : ["<field1>","<field2">,...]
  }
}

Transformation Parameter

aggregateFields

Purpose: Name of the aggregated field in the sink.
Data Type: string
Mandatory (Y/N): Y

Example: If the given record is:

{
  "id" : 100,
  "name" : "john",
  "address" : "USA",
  "age" : 20
}

If the aggregate transformation is:

"aggregateFields" : {
  "fieldName" : "document",
  "skipFields" : ["id"]
}

The aggregated column in the sink looks like:

{
  "id": 100,
  "document": {
    "name": "john",
    "address": "USA",
    "age": 20
  }
}

Mapping of DynamoDB types to Oracle NoSQL types

The table below shows the mapping of DynamoDB types to Oracle NoSQL types.

Table - Mapping DynamoDB type to Oracle NoSQL type

#	DynamoDB type	JSON type for NoSQL JSON column	Oracle NoSQL type
1	String (S)	JSON String	STRING
2	Number Type (N)	JSON Number	INTEGER/LONG/FLOAT/DOUBLE/NUMBER
3	Boolean (BOOL)	JSON Boolean	BOOLEAN
4	Binary type (B) - Byte buffer	BASE-64 encoded JSON String	BINARY
5	NULL	JSON null	NULL
6	String Set (SS)	JSON Array of Strings	ARRAY(STRING)
7	Number Set (NS)	JSON Array of Numbers	ARRAY(INTEGER/LONG/FLOAT/DOUBLE/NUMBER)
8	Binary Set (BS)	JSON Array of Base-64 encoded Strings	ARRAY(BINARY)
9	LIST (L)	Array of JSON	ARRAY(JSON)
10	MAP (M)	JSON Object	JSON
11	PARTITION KEY	NA	PRIMARY KEY and SHARD KEY
12	SORT KEY	NA	PRIMARY KEY
13	Attribute names with dash and dot	JSON field names with a underscore	Column names with underscore

Few additional points to consider while mapping DynamoDB types to Oracle NoSQL types:

DynamoDB Supports only one data type for Numbers and can have up to 38 digits of precision,on contrast Oracle NoSQL supports many types to choose from based on the range and precision of the data.You can select the appropriate Number type that fits the range of your input data. If you are not sure of the nature of the data, NoSQL NUMBER type can be used.
DynamoDB Supports only one data type for Numbers and can have up to 38 digits of precision,on contrast Oracle NoSQL supports many types to choose from based on the range and precision of the data.You can select the appropriate Number type that fits the range of your input data. If you are not sure of the nature of the data, NoSQL NUMBER type can be used.
Partition key in DynamoDB has a limit of 2048 bytes but Oracle NoSQL Cloud Service has a limit of 64 bytes for the Primary key/Shard key.
Sort key in DynamoDB has a limit of 1024 bytes but Oracle NoSQL Cloud Service has a limit of 64 bytes for the Primary key.
Attribute names in DynamoDB can be 64KB long but Oracle NoSQL Cloud service column names have a limit of 64 characters.

Oracle NoSQL to Parquet Data Type Mapping

Describes the mapping of Oracle NoSQL data types to Parquet data types.

NoSQL Type	Parquet Type
BOOLEAN	BOOLEAN
INTEGER	INT32
LONG	INT64
FLOAT	DOUBLE
DOUBLE	DOUBLE
BINARY	BINARY
FIXED_BINARY	BINARY
STRING	BINARY(STRING)
ENUM	BINARY(STRING) or BINARY(ENUM), if the logical ENUM is configured
UUID	BINARY(STRING) or FIXED_BINARY(16), if the logical UUID is configured
TIMESTAMP(p)	INT64(TIMESTAMP(p))
NUMBER	DOUBLE
field_name ARRAY(T)	`group field_name(LIST) { repeated group list { required T element } }`
field_name MAP(T)	`group field_name (MAP) { repeated group key_value (MAP_KEY_VALUE) { required binary key (STRING); required T value; } }`
field_name RECORD(K₁ T₁ N₁, Kٖ₂ T₂ N₂, ....) where: K = Key name T = Type N = Nullable or not	`group field_name { ni == true ? optional Ti ki : required Ti ki }`
JSON	BINARY(STRING) or BINARY(JSON), if logical JSON is configured

Note:

When the NoSQL Number type is converted to Parquet Double type, there may be some loss of precision in case the value cannot be represented in Double. If the number is too big to represent as Double, it is converted to Double.NEGATIVE_INFINITY or Double.POSITIVE_INFINITY.

Mapping of DynamoDB table to Oracle NoSQL table

In DynamoDB, a table is a collection of items, and each item is a collection of attributes. Each item in the table has a unique identifier, or a primary key. Other than the primary key, the table is schema-less. Each item can have its own distinct attributes.

DynamoDB supports two different kinds of primary keys:

Partition key – A simple primary key, composed of one attribute known as the partition key. DynamoDB uses the partition key's value as input to an internal hash function. The output from the hash function determines the partition in which the item will be stored.
Partition key and sort key – As a composite primary key, this type of key is composed of two attributes. The first attribute is the partition key, and the second attribute is the sort key. DynamoDB uses the partition key value as input to an internal hash function. The output from the hash function determines the partition in which the item will be stored. All items with the same partition key value are stored together, in sorted order by sort key value.

In contrast, Oracle NoSQL tables support flexible data models with both schema and schema-less design.

There are two different ways of modelling a DynamoDB table:

Modeling DynamoDB table as a JSON document(Recommended): In this modeling, you map all the attributes of the Dynamo DB tables into a JSON column of the NoSQL table except partition key and sort key. You will model partition key and sort key as the Primary Key columns of the NoSQL table. You will use AggregateFields transform in order to aggregate non-primary key data into a JSON column.

Note:
The Migrator provides a user-friendly configuration defaultSchema to automatically create a schema-less DDL table which also aggregates attributes into a JSON column.
Modeling DynamoDB table as fixed columns in NoSQL table: In this modeling, for each attribute of the DynamoDB table, you will create a column in the NoSQL table as specified in the Mapping of DynamoDB types to Oracle NoSQL types. You will model partition key and sort key attributes as Primary key(s). This should be used only when you are certain that importing DynamoDB table schema is fixed and each item has values for the most of the attributes. If DynamoDB items do not have common attributes, this can result in lot of NoSQL columns with empty values.

Note:
We highly recommend using schema-less tables when migrating data from DynamoDB to Oracle NoSQL Database due to the nature of DynamoDB tables being schema-less. This is especially for large tables where the content of each record may not be uniform across the table.

Troubleshooting the Oracle NoSQL Database Migrator

Learn about the general challenges that you may face while using the , and how to resolve them.

Migration has failed. How can I resolve this?

A failure of the data migration can be because of multiple underlying reasons. The important causes are listed below:

Table - Migration Failure Causes

Error Message	Meaning	Resolution
`Failed to connect to Oracle NoSQL Database`	The migrator could not establish a connection with the NoSQL Database.	Check if the values of the `storeName` and `helperHosts` attributes in the configuration JSON file are valid and that the hosts are reachable. For a secured store, verify if the security file is valid with correct user name and password values.
`Failed to connect to Oracle NoSQL Database Cloud Service`	The migrator could not establish a connection with the Oracle NoSQL Database Cloud Service.	Verify if the endpoint URL or region name specified in the configuration JSON file is correct. Check if the OCI credentials file is available in the path specified in the configuration JSON file. Ensure that the OCI credentials provided in the OCI credentials are valid.
`Table not found`	The table identified for the migration could not be located by the NoSQL Database Migrator.	For the Source: Verify if the table is present in the source database. Ensure that the table is qualified with its namespace in the configuration JSON file, if the table is created in a non-default namespace. Verify if you have the required read/write authorization to access the table. If the source is Oracle NoSQL Database Cloud Service, verify if the valid compartment name is specified in the configuration JSON file, and ensure that you have the required authorization to access the table. For the Sink: Verify if the table is present in the Sink. If it does not exist, you must either create the table manually or use the `schemaInfo` config to create it through the migration.
`DDL Execution failed`	The DDL commands provided in the input schema definition file is invalid.	Check the syntax of the DDL commands in the `schemaPath` file. Ensure that there is only one DDL statement per line in the `schemaPath` file.
`failed to write record to the sink table with java.lang.IllegalArgumentException`	The input record is not matching with the table schema of the sink.	Check if the data types and column names specified in the target sink table are matching with sink table schema. If you applied any transformation, check if the transformed records are matching with the sink table schema.
`Request timeout`	The source or sink's operation did not complete within the expected time.	Verify the network connection. Check if the NoSQL Database is up and running. Try to increase `requestTimeout` value in the configuration JSON file.

What should I consider before restarting a failed migration?

When a data migration task fails, the sink will be at an intermediate state containing the imported data until the point of failure. You can identify the error and failure details from the logs and restart the migration after diagnosing and correcting the error. A restarted migration starts over, processing all data from the beginning. There is no way to checkpoint and restart the migration from the point of failure. Therefore, NoSQL Database Migrator overwrites any record that was migrated to the sink already.

Migration is too slow. How can I speed it up?

The time taken for the data migration depends on multiple factors such as volume of data being migrated, network speed, current load on the database. In case of a cloud service, the speed of migration also depends on the read throughput and the write throughput provisioned. So, to improve the migration speed, you can:

Try to reduce the current workload on your Oracle NoSQL Database while migrating the data.
Ensure that the machine that is running the migration, source, and sink all are located in the same data center and the network latencies are minimal.
In case of Oracle NoSQL Database Cloud Service, provision high read/write throughput and verify if the storage allocated for table is sufficient or not. If the NoSQL Database Migrator is not creating the table, you can increase the write throughput. If the migrator is creating the table, consider specifying a higher value for the schemaInfo.writeUnits parameter in the sink configuration. Once the data migration completes, you can lower this value. Be aware of daily limits on throughput changes. see Cloud Limits and Sink Configuration Templates.

I have a long running migration involving huge datasets. How can I track the progress of the migration?

You can enable additional logging to track the progress of a long-running migration. To control the logging behavior of Oracle NoSQL Database Migrator, you must set the desired level of logging in the logging.properties file. This file is provided with the NoSQL Database Migrator package and available in the directory where the Oracle NoSQL Database Migrator was unpacked. The different levels of logging are OFF, SEVERE, WARNING, INFO, FINE, and ALL in the order of increasing verbosity. Setting the log level to OFF turns off all the logging information, whereas setting the log level to ALL provides the full log information. The default log level is WARNING. All the logging output is configured to go to the console by default. You can see comments in the logging.properties file to know about each log level.

Title and Copyright Information

Oracle NoSQL Database Migrator Reference

Documentation Accessibility

For information about Oracle's commitment to accessibility, visit the Oracle Accessibility Program website at http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc.

Access to Oracle Support

Oracle customers that have purchased support have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.