Sink Configuration Templates

Learn about the sink configuration file formats for each valid sink and the purpose of each configuration parameter.

For the configuration file template, see Configuration File in Terminology used with Oracle NoSQL Database Migrator.

For details on valid source formats for each of the sinks, see Source Configuration Templates.

Topics

JSON as the File Sink
The sink configuration template for the Oracle NoSQL Database Migrator to copy the data from a valid source to a JSON file as the sink.
Parquet File
The sink configuration template for the Oracle NoSQL Database Migrator to copy the data from a valid source to a Parquet file as the sink.
JSON File in OCI Object Storage Bucket
The sink configuration template for the Oracle NoSQL Database Migrator to copy the data from a valid source to a JSON file in the OCI Object Storage bucket as the sink.
Parquet File in OCI Object Storage Bucket
The sink configuration template for the Oracle NoSQL Database Migrator to copy the data from a valid source to a Parquet file in the OCI Object Storage bucket as the sink.
Oracle NoSQL Database
The sink configuration template for the Oracle NoSQL Database Migrator to copy the data from a valid source to Oracle NoSQL Database tables as the sink.
Oracle NoSQL Database Cloud Service
The sink configuration template for the Oracle NoSQL Database Migrator to copy the data from a valid source to Oracle NoSQL Database Cloud Service tables as the sink.

JSON as the File Sink

The configuration file format for JSON File as a sink of NoSQL Database Migrator is shown below.

Sink Configuration Template

"sink" : {
    "type" : "file",
    "format" : "json",
    "dataPath": "</path/to/a/file>",
    "schemaPath" : "<path/to/a/file>",
    "pretty" : <true|false>,
    "useMultiFiles" : <true|false>,
    "chunkSize" : <size in MB>
}

Sink Parameters

type

Purpose: Identifies the sink type.
Data Type: string
Mandatory (Y/N): Y
Example: "type" : "file"

format

Purpose: Specifies the sink format.
Data Type: string
Mandatory (Y/N): Y
Example: "format" : "json"

dataPath

Purpose: Specifies the absolute path to a file where the source data will be copied in the JSON format.

If the file does not exist in the specified data path, the NoSQL Database Migrator creates it. If it exists already, the NoSQL Database Migrator will overwrite its contents with the source data.

You must ensure that the parent directory for the file specified in the data path is valid.

Note:
If the useMultiFiles parameter is set to true, specify the path to a directory else specifies the path to the file.
Data Type: string
Mandatory (Y/N): Y
Example:
- With useMultiFiles parameter set to true
  "dataPath" :"/home/user/data"
- With useMultiFiles parameter not specified or it is set to false
  "dataPath" :"/home/user/sample.json"

schemaPath

Purpose: Specifies the absolute path to write schema information provided by the source.

If this value is not defined, the source schema information will not be migrated to the sink. If this value is specified, the migrator utility writes the schema of the source table into the file specified here.

The schema information is written as one DDL command per line in this file. If the file does not exist in the specified data path, NoSQL Database Migrator creates it. If it exists already, NoSQL Database Migrator will overwrite its contents with the source data. You must ensure that the parent directory for the file specified in the data path is valid.
Data Type: string
Mandatory (Y/N): N
Example: "schemaPath" : "/home/user/schema_file"

pretty

Purpose: Specifies whether to beautify the JSON output to increase readability or not.

If not specified, it defaults to false.
Data Type: boolean
Mandatory (Y/N): N
Example: "pretty" : true

useMultiFiles

Purpose: Specifies whether or not to split the NoSQL table data into multiple files when migrating source data to a file.

If not specified, it defaults to false.

If set to true, when migrating source data to a file, the NoSQL table data is split into multiple smaller files. For example, <chunk>.json, where chunk=000000,000001,000002, and so forth.
```
dataPath
         |--000000.json
         |--000001.json
```
Data Type: boolean
Mandatory (Y/N): N
Example: "useMultiFiles" : true

chunkSize

Purpose: Specifies the maximum size of a "chunk" of table data to be stored at the sink. During migration, a table is split into chunkSize chunks and each chunk is written as a separate file to the sink. When the source data being migrated exceeds this size, a new file is created.

If not specified, defaults to 32MB. The valid value is an integer between 1 to 1024.

Note:
This parameter is applicable ONLY when the useMultiFiles parameter is set to true.
Data Type: integer
Mandatory (Y/N): N
Example: "chunkSize" : 40

Parquet File

The configuration file format for Parquet File as a sink of NoSQL Database Migrator is shown below.

Sink Configuration Template

"sink" : {
    "type" : "file",
    "format" : "parquet",
    "dataPath": "</path/to/a/dir>",
    "chunkSize" : <size in MB>,
    "compression": "<SNAPPY|GZIP|NONE>",
    "parquetOptions": {
      "useLogicalJson": <true|false>,
      "useLogicalEnum": <true|false>,
      "useLogicalUUID": <true|false>,     
      "truncateDoubleSpecials": <true|false>
    }
}

Sink Parameters

type

Purpose: Identifies the sink type.
Data Type: string
Mandatory (Y/N): Y
Example: "type" : "file"

format

Purpose: Specifies the sink format.
Data Type: string
Mandatory (Y/N): Y
Example: "format" : "parquet"

dataPath

Purpose: Specifies the path to a directory to use for storing the migrated NoSQL table data. Ensure that the directory already exists and has read and write permissions.
Data Type: string
Mandatory (Y/N): Y
Example: "dataPath" : "/home/user/migrator/my_table"

chunkSize

Purpose: Specifies the maximum size of a "chunk" of table data to be stored at the sink. During migration, a table is split into chunkSize chunks and each chunk is written as a separate file to the sink. When the source data being migrated exceeds this size, a new file is created.

If not specified, defaults to 32MB. The valid value is an integer between 1 to 1024.
Data Type: integer
Mandatory (Y/N): N
Example: "chunkSize" : 40

compression

Purpose: Specifies the compression type to use to compress the Parquet data. Valid values are SNAPPY, GZIP, and NONE.

If not specified, it defaults to SNAPPY.
Data Type: string
Mandatory (Y/N): N
Example: "compression" : "GZIP"

parquetOptions

Purpose: Species the options to select Parquet logical types for NoSQL ENUM, JSON, and UUID columns.

If you do not specify this parameter, the NoSQL Database Migrator writes the data of ENUM, JSON, and UUID columns as String.
Data Type: object
Mandatory (Y/N): N

parquetOptions.useLogicalJson

Purpose: Specifies whether or not to write NoSQL JSON column data as Parquet logical JSON type. For more information see Parquet Logical Type Definitions.

If not specified or set to false, NoSQL Database Migrator writes the NoSQL JSON column data as String.
Data Type: boolean
Mandatory (Y/N): N
Example: "useLogicalJson" : true

parquetOptions.useLogicalEnum

Purpose: Specifies whether or not to write NoSQL ENUM column data as Parquet logical ENUM type. For more information see Parquet Logical Type Definitions.

If not specified or set to false, NoSQL Database Migrator writes the NoSQL ENUM column data as String.
Data Type: boolean
Mandatory (Y/N): N
Example: "useLogicalEnum" : true

parquetOptions.useLogicalUUID

Purpose: Specifies whether or not to write NoSQL UUID column data as Parquet logical UUID type. For more information see Parquet Logical Type Definitions.

If not specified or set to false, NoSQL Database Migrator writes the NoSQL UUID column data as String.
Data Type: boolean
Mandatory (Y/N): N
Example: "useLogicalUUID" : true

parquetOptions.truncateDoubleSpecials

Purpose: Specifies whether or not to truncate the double +Infinity, -Infinity, and Nan values.
By default, it is set to false. If set to true,
- Positive_Infinity is truncated to Double.MAX_VALUE.
- NEGATIVE_INFINITY is truncated to -Double.MAX_VALUE.
- NaN is truncated to 9.9999999999999990E307.
Data Type: boolean
Mandatory (Y/N): N
Example: "truncateDoubleSpecials" : true

JSON File in OCI Object Storage Bucket

The configuration file format for JSON file in OCI Object Storage bucket as a sink of NoSQL Database Migrator is shown below.

Note:

The valid source types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Sink Configuration Template

"sink" : {
  "type" : "object_storage_oci",
  "format" : "json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "chunkSize" : <size in MB>,
  "pretty" : <true|false>,
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

Source Parameters

type

Purpose: Identifies the sink type.
Data Type: string
Mandatory (Y/N): Y
Example: "type" : "object_storage_oci"

format

Purpose: Specifies the sink format.
Data Type: string
Mandatory (Y/N): Y
Example: "format" : "json"

endpoint

Purpose: Specifies the OCI Object Storage service endpoint URL or region ID.

You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.
Data Type: string
Mandatory (Y/N): Y
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

namespace

Purpose: Specifies the OCI Object Storage service namespace. This is an optional parameter. If you don't specify this parameter, the default namespace of the tenancy is used.
Data Type: string
Mandatory (Y/N): Y
Example: "namespace" : "my-namespace"

bucket

Purpose: Specifies the bucket name to use for storing the migrated data. Ensure that the required bucket already exists in the OCI Object Storage instance and has write permissions.
Data Type: string
Mandatory (Y/N): Y
Example: "bucket" : "staging_bucket"

prefix

Purpose: Specifies the prefix that is added to the object name when objects are created in the bucket. The prefix acts as a logical container or directory for storing data. For more information about prefixes, see Object Naming Using Prefixes and Hierarchies.

If not specified, the table name from the source is used as the prefix. If any object with the same name already exists in the bucket, it is overwritten.

Schema is migrated to the <prefix>/Schema /schema.ddl file and source data is migrated to the <prefix>/Data/<chunk>.json file(s), where chunk=000000.json, 000001.json, and so forth.
Data Type: string
Mandatory (Y/N): N
Example:
1. "prefix" : "my_export"
2. "prefix" : "my_export/2021-04-05/"

chunkSize

Purpose: Specifies the maximum size of a "chunk" of table data to be stored at the sink. During migration, a table is split into chunkSize chunks and each chunk is written as a separate file to the sink. When the source data being migrated exceeds this size, a new file is created.

If not specified, defaults to 32MB. The valid value is an integer between 1 to 1024.
Data Type: integer
Mandatory (Y/N): N
Example: "chunkSize" : 40

pretty

Purpose: Specifies whether to beautify the JSON output to increase readability or not.

If not specified, it defaults to false.
Data Type: boolean
Mandatory (Y/N): N
Example: "pretty" : true

credentials

Purpose: Absolute path to a file containing OCI credentials.

If not specified, it defaults to $HOME/.oci/config

See Example Configuration for an example of the credentials file.

Note:
Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
Data Type: string
Mandatory (Y/N): N
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"

credentialsProfile

Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

If you do not specify this value, it defaults to the DEFAULT profile.

Note:
This parameter is valid ONLY if the credentials parameter is specified.
Data Type: string
Mandatory (Y/N): N
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile": "ADMIN_USER"

useInstancePrincipal

Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security.

If not specified, it defaults to false.
Note:
- It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
- Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
Data Type: boolean
Mandatory (Y/N): N
Example: "useInstancePrincipal" : true

Parquet File in OCI Object Storage Bucket

The configuration file format for Parquet file in OCI Object Storage bucket as a sink of NoSQL Database Migrator is shown below.

Note:

The valid source types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Sink Configuration Template

"sink" : {
  "type" : "object_storage_oci",
  "format" : "parquet",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "chunkSize" : <size in MB>,
  "compression": "<SNAPPY|GZIP|NONE>",
  "parquetOptions": {
    "useLogicalJson": <true|false>,
    "useLogicalEnum": <true|false>,
    "useLogicalUUID": <true|false>,
    "truncateDoubleSpecials": <true|false>
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

Source Parameters

type

Purpose: Identifies the sink type.
Data Type: string
Mandatory (Y/N): Y
Example: "type" : "object_storage_oci"

format

Purpose: Specifies the sink format.
Data Type: string
Mandatory (Y/N): Y
Example: "format" : "parquet"

endpoint

Purpose: Specifies the OCI Object Storage service endpoint URL or region ID.

You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.
Data Type: string
Mandatory (Y/N): Y
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

namespace

Purpose: Specifies the OCI Object Storage service namespace. This is an optional parameter. If you don't specify this parameter, the default namespace of the tenancy is used.
Data Type: string
Mandatory (Y/N): Y
Example: "namespace" : "my-namespace"

bucket

Purpose: Specifies the bucket name to use for storing the migrated data. Ensure that the required bucket already exists in the OCI Object Storage instance and has write permissions.
Data Type: string
Mandatory (Y/N): Y
Example: "bucket" : "staging_bucket"

prefix

Purpose: Specifies the prefix that is added to the object name when objects are created in the bucket. The prefix acts as a logical container or directory for storing data. For more information about prefixes, see Object Naming Using Prefixes and Hierarchies.

If not specified, the table name from the source is used as the prefix. If any object with the same name already exists in the bucket, it is overwritten.

Source data is migrated to the <prefix>/Data/<chunk>.parquet file(s), where chunk=000000.parquet, 000001.parquet, and so forth.
Data Type: string
Mandatory (Y/N): N
Example:
1. "prefix" : "my_export"
2. "prefix" : "my_export/2021-04-05/"

chunkSize

Purpose: Specifies the maximum size of a "chunk" of table data to be stored at the sink. During migration, a table is split into chunkSize chunks and each chunk is written as a separate file to the sink. When the source data being migrated exceeds this size, a new file is created.

If not specified, defaults to 32MB. The valid value is an integer between 1 to 1024.
Data Type: integer
Mandatory (Y/N): N
Example: "chunkSize" : 40

compression

Purpose: Specifies the compression type to use to compress the Parquet data. Valid values are SNAPPY, GZIP, and NONE.

If not specified, it defaults to SNAPPY.
Data Type: string
Mandatory (Y/N): N
Example: "compression" : "GZIP"

parquetOptions

Purpose: Species the options to select Parquet logical types for NoSQL ENUM, JSON, and UUID columns.

If you do not specify this parameter, the NoSQL Database Migrator writes the data of ENUM, JSON, and UUID columns as String.
Data Type: object
Mandatory (Y/N): N

parquetOptions.useLogicalJson

Purpose: Specifies whether or not to write NoSQL JSON column data as Parquet logical JSON type. For more information see Parquet Logical Type Definitions.

If not specified or set to false, NoSQL Database Migrator writes the NoSQL JSON column data as String.
Data Type: boolean
Mandatory (Y/N): N
Example: "useLogicalJson" : true

parquetOptions.useLogicalEnum

Purpose: Specifies whether or not to write NoSQL ENUM column data as Parquet logical ENUM type. For more information see Parquet Logical Type Definitions.

If not specified or set to false, NoSQL Database Migrator writes the NoSQL ENUM column data as String.
Data Type: boolean
Mandatory (Y/N): N
Example: "useLogicalEnum" : true

parquetOptions.useLogicalUUID

Purpose: Specifies whether or not to write NoSQL UUID column data as Parquet logical UUID type. For more information see Parquet Logical Type Definitions.

If not specified or set to false, NoSQL Database Migrator writes the NoSQL UUID column data as String.
Data Type: boolean
Mandatory (Y/N): N
Example: "useLogicalUUID" : true

parquetOptions.truncateDoubleSpecials

Purpose: Specifies whether or not to truncate the double +Infinity, -Infinity, and Nan values.
By default, it is set to false. If set to true,
- Positive_Infinity is truncated to Double.MAX_VALUE.
- NEGATIVE_INFINITY is truncated to -Double.MAX_VALUE.
- NaN is truncated to 9.9999999999999990E307.
Data Type: boolean
Mandatory (Y/N): N
Example: "truncateDoubleSpecials" : true

credentials

Purpose: Absolute path to a file containing OCI credentials.

If not specified, it defaults to $HOME/.oci/config

See Example Configuration for an example of the credentials file.

Note:
Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
Data Type: string
Mandatory (Y/N): N
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"

credentialsProfile

Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

If you do not specify this value, it defaults to the DEFAULT profile.

Note:
This parameter is valid ONLY if the credentials parameter is specified.
Data Type: string
Mandatory (Y/N): N
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile": "ADMIN_USER"

useInstancePrincipal

Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security.

If not specified, it defaults to false.
Note:
- It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
- Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
Data Type: boolean
Mandatory (Y/N): N
Example: "useInstancePrincipal" : true

Oracle NoSQL Database

The configuration file format for Oracle NoSQL Database as a sink of NoSQL Database Migrator is shown below.

Sink Configuration Template

"sink" : {
  "type": "nosqldb",
  "table" : "<fully qualified table name>",
  "schemaInfo" : {
    "schemaPath" : "</path/to/a/schema/file>",
    "defaultSchema" : <true|false>,
    "useSourceSchema" : <true|false>,
    "DDBPartitionKey" : <"name:type">,
    "DDBSortKey" : "<name:type>"
  },
  "overwrite" : <true|false>,
  "storeName" : "<store name>",
  "helperHosts" : ["hostname1:port1","hostname2:port2,..."],
  "security" : "</path/to/store/credentials/file>",
  "requestTimeoutMs" : <timeout in milli seconds>,
  "includeTTL": <true|false>,
  "ttlRelativeDate": "<date-to-use in UTC>"
}

Sink Parameters

type

Purpose: Identifies the sink type.
Data Type: string
Mandatory (Y/N): Y
Example: "type" : "nosqldb"

table

Purpose: Fully qualified table name from which to migrate the data.

Format: [namespace_name:]<table_name>

If the table is in the DEFAULT namespace, you can omit the namespace_name. The table must exist in the store during the migration, and its schema must match with the source data.

If the table is not available in the sink, you can use the schemaInfo parameter to instruct the NoSQL Database Migrator to create the table also in the sink.
Data Type: string
Mandatory (Y/N): Y
Example:
- With the DEFAULT namespace "table" :"mytable"
- With a non-default namespace "table" : "mynamespace:mytable"
- To specify a child table "table" : "mytable.child"
  
  Note:
  You can migrate the child tables from a valid data source to Oracle NoSQL Database. The NoSQL Database Migrator copies only a single table in each execution. Ensure that the parent table is migrated before the child table.

schemainfo

Purpose: Specifies the schema for the data being migrated. If this is not specified, the assumes that the table already exists in the sink's store.
Data Type: Object
Mandatory (Y/N): N

schemaInfo.schemaPath

Purpose: Specifies the absolute path to a file containing DDL statements for the NoSQL table.

The NoSQL Database Migrator executes the DDL commands listed in this file before migrating the data.

The NoSQL Database Migrator does not support more than one DDL statement per line in the schemaPath file.
Data Type: string
Mandatory: Y, only when the schemaInfo.defaultSchema parameter is set to No.

schemaInfo.defaultSchema

Purpose: Setting this parameter to true instructs the NoSQL Database Migrator to create a table with default schema. The default schema is defined by the migrator itself. For more information about default schema definitions, see Default Schema in Using Oracle NoSQL Data Migrator.
Data Type: boolean
Mandatory: N

Note:
defaultSchema and schemaPath are mutually exclusive

Example:

With Default Schema:

"schemaInfo" : {
      "defaultSchema" : true
    }

With a pre-defined schema:

"schemaInfo" : {
      "schemaPath" : "<complete/path/to/the/schema/definition/file>"
    }

schemaInfo.useSourceSchema

Purpose: Specifies whether or not the sink uses the table schema definition provided by the source when migrating NoSQL tables.
Data Type: boolean
Mandatory (Y/N): N

Note:
defaultSchema, schemaPath, and useSourceSchema parameters are mutually exclusive. Specify ONLY one of these parameters.

Example:

With Default Schema:

"schemaInfo" : {
      "defaultSchema" : true
    }

With a pre-defined schema:

"schemaInfo" : {
      "schemaPath" : "<complete/path/to/the/schema/definition/file>"
    }

With source schema:

"schemaInfo" : {
      "useSourceSchema" : true
    }

schemaInfo.DDBPartitionKey

Purpose: Specifies the DynamoDB partition key and the corresponding Oracle NoSQL Database type to be used in the sink Oracle NoSQL Database table. This key will be used as a NoSQL DB table shard key. This is applicable only when defaultSchema is set to true and the source format is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.
Mandatory: Yes if defaultSchema is true and the source is dynamodb_json.
Example: "DDBPartitionKey" : "PersonID:INTEGER"

Note:
If the partition key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

schemaInfo.DDBSortKey

Purpose: Specifies the DynamoDB sort key and its corresponding Oracle NoSQL Database type to be used in the target Oracle NoSQL Database table. If the importing DynamoDB table does not have a sort key this should not be set. This key will be used as a non-shard portion of the primary key in the NoSQL DB table. This is applicable only when defaultSchema is set to true and the source is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.
Mandatory: No
Example:"DDBSortKey" : "Skey:STRING"

Note:
If the sort key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

overwrite

Purpose: Indicates the behavior of NoSQL Database Migrator when the record being migrated from the source is already present in the sink.

If the value is set to false, when migrating tables the NoSQL Database Migrator skips those records for which the same primary key already exists in the sink.

If the value is set to true, when migrating tables the NoSQL Database Migrator overwrites those records for which the same primary key already exists in the sink.

If not specified, it defaults to true.
Data Type: boolean
Mandatory (Y/N): N
Example: "overwrite" : false

storeName

Purpose: Name of the Oracle NoSQL Database store.
Data Type: string
Mandatory (Y/N): Y
Example: "storeName" : "kvstore"

helperHosts

Purpose: A list of host and registry port pairs in the hostname:port format. Delimit each item in the list using a comma. You must specify at least one helper host.
Data Type: array of strings
Mandatory (Y/N): Y
Example: "helperHosts" : ["localhost:5000","localhost:6000"]

security

Purpose:

If your store is a secure store, provide the absolute path to the security login file that contains your store credentials. See Configuring Security with Remote Access in Administrator's Guide to know more about the security login file.

You can use either password file based authentication or wallet based authentication. However, the wallet based authentication is supported only in the Enterprise Edition (EE) of Oracle NoSQL Database. For more information on wallet-based authentication, see Source and Sink Security.
Data Type: string
Mandatory (Y/N): Y for a secure store

Example:

"security" : "/home/user/client.credentials"

Example security file content for password file based authentication:

oracle.kv.password.noPrompt=true
oracle.kv.auth.username=admin
oracle.kv.auth.pwdfile.file=/home/nosql/login.passwd
oracle.kv.transport=ssl
oracle.kv.ssl.trustStore=/home/nosql/client.trust
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

Example security file content for wallet based authentication:

oracle.kv.password.noPrompt=true
oracle.kv.auth.username=admin
oracle.kv.auth.wallet.dir=/home/nosql/login.wallet
oracle.kv.transport=ssl
oracle.kv.ssl.trustStore=/home/nosql/client.trust
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

requestTimeoutMs

Purpose: Specifies the time to wait for each write operation in the sink to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer.
Data Type: integer
Mandatory (Y/N): N
Example: "requestTimeoutMs" : 5000

includeTTL

Purpose: Specifies whether or not to include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.

If you do not specify this parameter, it defaults to false. In that case, the NoSQL Database Migrator does not include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.
If set to true, the NoSQL Database Migrator tool performs the following checks on the TTL metadata when importing a table row:
- If you import a row that does not have _metadata definition, the NoSQL Database Migrator tool sets the TTL to 0, which means the row never expires.
- If you import a row that has _metadata definition, the NoSQL Database Migrator tool compares the TTL value against a Reference Time when a row gets imported. If the row has already expired relative to the Reference Time, then it is skipped. If the row has not expired, then it is imported along with the TTL value. By default, the Reference Time of import operation is the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running. But you can also set a custom Reference Time using the ttlRelativeDate configuration parameter if you want to extend the expiration time and import rows that would otherwise expire immediately.
  The formula to calculate the expiration time of a row is as follows:
```
expiration = (TTL value of source row in milliseconds - Reference Time in milliseconds)
if (expiration <= 0) then it indicates that row has expired.
```
  Note:
  Since Oracle NoSQL TTL boundaries are in hours and days, in some cases, the TTL of the imported row might get adjusted to the nearest hour or day. For example, consider a row that has expiration value of 1629709200000 (2021-08-23 09:00:00) and Reference Time value is 1629707962582 (2021-08-23 08:39:22). Here, even though the row is not expired relative to the Reference Time when this data gets imported, the new TTL for the row is 1629712800000 (2021-08-23 10:00:00).
Data Type: boolean
Mandatory (Y/N): N
Example: "includeTTL" : true

ttlRelativeDate

Purpose: Specify a UTC date in the YYYY-MM-DD hh:mm:ss format used to set the TTL expiry of table rows during importing into the Oracle NoSQL Database.

If a table row in the data you are exporting has expired, you can set the ttlRelativeDate parameter to a date before the expiration time of the table row in the exported data.

If you do not specify this parameter, it defaults to the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running.
Data Type: date
Mandatory (Y/N): N
Example: "ttlRelativeDate" : "2021-01-03 04:31:17"
Let us consider a scenario where table rows expire after seven days from 1-Jan-2021. After exporting this table, on 7-Jan-2021, you run into an issue with your table and decide to import the data. The table rows are going to expire in one day (data expiration date minus the default value of ttlRelativedate configuration parameter, which is the current date).But if you want to extend the expiration date of table rows to five days instead of one day, use the ttlRelativeDate parameter and choose an earlier date. Therefore, in this scenario if you want to extend expiration time of the table rows by five days, set the value of ttlRelativeDate configuration parameters to 3-Jan-2021, which is used as Reference Time when table rows get imported.

Oracle NoSQL Database Cloud Service

The configuration file format for Oracle NoSQL Database Cloud Service as a sink of NoSQL Database Migrator is shown below.

Sink Configuration Template

"sink" : {
    "type" : "nosqldb_cloud",
    "endpoint" : "<Oracle NoSQL Cloud Service Endpoint>",
    "table" : "<table name>",
    "compartment" : "<OCI compartment name or id>",
    "schemaInfo" : {
      "schemaPath" : "</path/to/a/schema/file>",
      "defaultSchema" : <true|false>,
      "useSourceSchema" : <true|false>,
      "DDBPartitionKey" : <"name:type">,
      "DDBSortKey" : "<name:type>",
      "onDemandThroughput" : <true|false>,
      "readUnits" : <table read units>,
      "writeUnits" : <table write units>,
      "storageSize" : <storage size in GB>
    },
    "credentials" : "</path/to/oci/credential/file>",
    "credentialsProfile" : "<oci credentials profile name>",
    "writeUnitsPercent" : <table writeunits percent>,
    "requestTimeoutMs" : <timeout in milli seconds>,
    "useInstancePrincipal" : <true|false>,
    "overwrite" : <true|false>,
    "includeTTL": <true|false>,
    "ttlRelativeDate" : "<date-to-use in UTC>"  
}

Sink Parameters

type

Purpose: Identifies the sink type.
Data Type: string
Mandatory (Y/N): Y
Example: "type" : "nosqldb_cloud"

endpoint

Purpose: Specifies the Service Endpoint of the Oracle NoSQL Database Cloud Service.

You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.
Data Type: string
Mandatory (Y/N): Y
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://nosql.us-ashburn-1.oci.oraclecloud.com/"

table

Purpose: Name of the table to which to migrate the data.

You must ensure that this table exists in your Oracle NoSQL Database Cloud Service. Otherwise, you have to use the schemaInfo object in the sink configuration to instruct the NoSQL Database Migrator to create the table.

The schema of this table must match the source data.
Data Type: string
Mandatory (Y/N): Y
Example:
- To specify a table "table" : "mytable"
- To specify a child table "table" : "mytable.child"
  
  Note:
  You can migrate the child tables from a valid data source to Oracle NoSQL Database Cloud Service. The NoSQL Database Migrator copies only a single table in each execution. Ensure that the parent table is migrated before the child table.

compartment

Purpose: Specifies the name or OCID of the compartment in which the table resides.

If you do not provide any value, it defaults to the root compartment.

You can find your compartment's OCID from the Compartment Explorer window under Governance in the OCI Cloud Console.
Data Type: string
Mandatory (Y/N): Y if the table is not in the root compartment of the tenancy OR when the useInstancePrincipal parameter is set to true.

Note:
If the useInstancePrincipal parameter is set to true, the compartment must specify the compartment OCID and not the name.
Example:
- Compartment name
  
  "compartment" : "mycompartment"
- Compartment name qualified with its parent compartment
  
  "compartment" : "parent.childcompartment"
- No value provided. Defaults to the root compartment.
  
  "compartment": ""
- Compartment OCID
  
  "compartment" : "ocid1.tenancy.oc1...4ksd"

schemaInfo

Purpose: Specifies the schema for the data being migrated.

If you do not specify this parameter, the NoSQL Database Migrator assumes that the table already exists in your Oracle NoSQL Database Cloud Service.

If this parameter is not specified and the table does not exist in the sink, the migration fails.
Data Type: Object
Mandatory (Y/N): N

schemaInfo.schemaPath

Purpose: Specifies the absolute path to a file containing DDL statements for the NoSQL table.

The NoSQL Database Migrator executes the DDL commands listed in this file before migrating the data.

The NoSQL Database Migrator does not support more than one DDL statement per line in the schemaPath file.
Data Type: string
Mandatory: Y, only when the schemaInfo.defaultSchema parameter is set to No.

schemaInfo.defaultSchema

Purpose: Setting this parameter to Yes instructs the NoSQL Database Migrator to create a table with default schema. The default schema is defined by the migrator itself. For more information about default schema definitions, see Default Schema in Using Oracle NoSQL Data Migrator.
Data Type: boolean
Mandatory: Y, only when the schemaInfo.defaultSchema parameter is set to No.

Note:
defaultSchema and schemaPath are mutually exclusive

schemaInfo.useSourceSchema

Purpose: Specifies whether or not the sink uses the table schema definition provided by the source when migrating NoSQL tables.
Data Type: boolean
Mandatory (Y/N): N

Note:
defaultSchema, schemaPath, and useSourceSchema parameters are mutually exclusive. Specify ONLY one of these parameters.

Example:

With Default Schema:

"schemaInfo" : {
      "defaultSchema" : true,
      "readUnits" : 100,
      "writeUnits" : 60,
      "storageSize" : 1 
    }

With a pre-defined schema:

"schemaInfo" : {
      "schemaPath" : "<complete/path/to/the/schema/definition/file>",
      "readUnits" : 100,
      "writeUnits" : 100,
      "storageSize" : 1
    }

With source schema:

"schemaInfo" : {
      "useSourceSchema" : true,
      "readUnits" : 100,
      "writeUnits" : 60,
      "storageSize" : 1
    }

schemaInfo.DDBPartitionKey

Purpose: Specifies the DynamoDB partition key and the corresponding Oracle NoSQL Database type to be used in the sink Oracle NoSQL Database table. This key will be used as a NoSQL DB table shard key. This is applicable only when defaultSchema is set to true and the source format is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.
Mandatory: Yes if defaultSchema is true and the source is dynamodb_json.
Example: "DDBPartitionKey" : "PersonID:INTEGER"

Note:
If the partition key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

schemaInfo.DDBSortKey

Purpose: Specifies the DynamoDB sort key and its corresponding Oracle NoSQL Database type to be used in the target Oracle NoSQL Database table. If the importing DynamoDB table does not have a sort key this should not be set. This key will be used as a non-shard portion of the primary key in the NoSQL DB table. This is applicable only when defaultSchema is set to true and the source is dynamodb_json.See Mapping of DynamoDB types to Oracle NoSQL types for more details.
Mandatory: No
Example:"DDBSortKey" : "Skey:STRING"

Note:
If the sort key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

schemaInfo.onDemandThroughput

Purpose: Specifies to create the table with on-demand read and write throughput. If this parameter is not set, the table is created with provisioned capacity.
The default value is false.

Note:
This parameter is not applicable for child tables as they share the throughput of the top-level parent table.
Data Type: Boolean
Mandatory: N
Example:"onDemandThroughput" : "true"

schemaInfo.readUnits

Purpose: Specifies the read throughput of the new table.
Note:
- This parameter is not applicable for tables provisioned with on-demand capacity.
- This parameter is not applicable for child tables as they share the read throughput of the top-level parent table.
Data Type: integer
Mandatory: Y when the table is not a child table or when schemaInfo.onDemandThroughput parameter is set to false, else N.
Example:"readUnits" : 100

schemaInfo.writeUnits

Purpose: Specifies the write throughput of the new table.
Note:
- This parameter is not applicable for tables provisioned with on-demand capacity.
- This parameter is not applicable for child tables as they share the write throughput of the top-level parent table.
Data Type: integer
Mandatory: Y when the table is not a child table or when schemaInfo.onDemandThroughput parameter is set to false, else N.
Example:"writeUnits" : 100

schemaInfo.storageSize

Purpose: Specifies the storage size of the new table in GB

Note:
This parameter is not applicable for child tables as they share the storage size of the top-level parent table.
Data Type: integer
Mandatory: Y when the table is not a child table, else N.

Example:

With schemaPath

"schemaInfo" : { 
  "schemaPath" : "</path/to/a/schema/file>",
  "readUnits" : 500,
  "writeUnits" : 1000,
  "storageSize" : 5 }

With defaultSchema

"schemaInfo" : {
  "defaultSchema" :Yes,
  "readUnits" : 500,   
  "writeUnits" : 1000,   
  "storageSize" : 5  
}

credentials

Purpose: Absolute path to a file containing OCI credentials.

If not specified, it defaults to $HOME/.oci/config

See Example Configuration for an example of the credentials file.

Note:
Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
Data Type: string
Mandatory (Y/N): N
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"

credentialsProfile

Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service.

If you do not specify this value, it defaults to the DEFAULT profile.

Note:
This parameter is valid ONLY if the credentials parameter is specified.
Data Type: string
Mandatory (Y/N): N
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile": "ADMIN_USER"

writeUnitsPercent

Purpose: Specifies the Percentage of table write units to be used during the migration activity.

The default value is 90. The valid range is any integer between 1 to 100.

Note:
The time required for the data migration is directly proportional to the writeUnitsPercent value.

See Troubleshooting the Oracle NoSQL Database Migrator to learn how to use this attribute to improve the data migration speed.
Data Type: integer
Mandatory (Y/N): N
Example: "writeUnitsPercent" : 90

requestTimeoutMs

Purpose: Specifies the time to wait for each write operation in the sink to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer.
Data Type: integer
Mandatory (Y/N): N
Example: "requestTimeoutMs" : 5000

useInstancePrincipal

Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security

If not specified, it defaults to false.
Note:
- It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
- Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
Data Type: boolean
Mandatory (Y/N): N
Example: "useInstancePrincipal" : true

overwrite

Purpose: Indicates the behavior of NoSQL Database Migrator when the record being migrated from the source is already present in the sink.

If the value is set to false, when migrating tables the NoSQL Database Migrator skips those records for which the same primary key already exists in the sink.

If the value is set to true, when migrating tables the NoSQL Database Migrator overwrites those records for which the same primary key already exists in the sink.

If not specified, it defaults to true.
Data Type: boolean
Mandatory (Y/N): N
Example: "overwrite" : false

includeTTL

Purpose: Specifies whether or not to include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.

If you do not specify this parameter, it defaults to false. In that case, the NoSQL Database Migrator does not include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.
If set to true, the NoSQL Database Migrator tool performs the following checks on the TTL metadata when importing a table row:
- If you import a row that does not have _metadata definition, the NoSQL Database Migrator tool sets the TTL to 0, which means the row never expires.
- If you import a row that has _metadata definition, the NoSQL Database Migrator tool compares the TTL value against a Reference Time when a row gets imported. If the row has already expired relative to the Reference Time, then it is skipped. If the row has not expired, then it is imported along with the TTL value. By default, the Reference Time of import operation is the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running. But you can also set a custom Reference Time using the ttlRelativeDate configuration parameter if you want to extend the expiration time and import rows that would otherwise expire immediately.
  The formula to calculate the expiration time of a row is as follows:
```
expiration = (TTL value of source row in milliseconds - Reference Time in milliseconds)
if (expiration <= 0) then it indicates that row has expired.
```
  Note:
  Since Oracle NoSQL TTL boundaries are in hours and days, in some cases, the TTL of the imported row might get adjusted to the nearest hour or day. For example, consider a row that has expiration value of 1629709200000 (2021-08-23 09:00:00) and Reference Time value is 1629707962582 (2021-08-23 08:39:22). Here, even though the row is not expired relative to the Reference Time when this data gets imported, the new TTL for the row is 1629712800000 (2021-08-23 10:00:00).
Data Type: boolean
Mandatory (Y/N): N
Example: "includeTTL" : true

ttlRelativeDate

Purpose: Specify a UTC date in the YYYY-MM-DD hh:mm:ss format used to set the TTL expiry of table rows during importing into the Oracle NoSQL Database.

If a table row in the data you are exporting has expired, you can set the ttlRelativeDate parameter to a date before the expiration time of the table row in the exported data.

If you do not specify this parameter, it defaults to the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running.
Data Type: date
Mandatory (Y/N): N
Example: "ttlRelativeDate" : "2021-01-03 04:31:17"
Let us consider a scenario where table rows expire after seven days from 1-Jan-2021. After exporting this table, on 7-Jan-2021, you run into an issue with your table and decide to import the data. The table rows are going to expire in one day (data expiration date minus the default value of ttlRelativedate configuration parameter, which is the current date).But if you want to extend the expiration date of table rows to five days instead of one day, use the ttlRelativeDate parameter and choose an earlier date. Therefore, in this scenario if you want to extend expiration time of the table rows by five days, set the value of ttlRelativeDate configuration parameters to 3-Jan-2021, which is used as Reference Time when table rows get imported.