Sink Configuration Templates

Learn about the sink configuration file formats for each valid sink and the purpose of each configuration parameter.

For the configuration file template, see Configuration File in Terminology used with Oracle NoSQL Database Migrator.

For details on valid source formats for each of the sinks, see Source Configuration Templates.

Topics

The following topics describe the sink configuration templates referred by Oracle NoSQL Database Migrator to copy the data from a valid source to the given sink.

• JSON File Sink

  Specified JSON file.

• Parquet File

  Parquet file in the specified directory.

• JSON File in OCI Object Storage Bucket

  JSON file in the specified OCI Object Storage bucket.

• Parquet File in OCI Object Storage Bucket

  Parquet file in the specified OCI Object Storage bucket.

• Oracle NoSQL Database

  Specified table in Oracle NoSQL Database.

• Oracle NoSQL Database Cloud Service

  Specified table in Oracle NoSQL Database Cloud Service.

JSON File Sink

The configuration file format for JSON File as a sink of NoSQL Database Migrator is shown below.

Sink Configuration Template

"sink" : {
  "type" : "file",
  "format" : "json",
  "dataPath": "</path/to/a/directory>",
  "schemaPath" : "<path/to/a/file>",
  "pretty" : <true|false>,
  "useMultiFiles" : <true|false>,
  "chunkSize" : <size in MB>
}

Sink Parameters

Common Configuration Parameters

• type

  Use "type" : "file"

• format

  Use "format" : "json"

• chunkSize

  Example: "chunkSize" : 40

Note: This parameter is applicable ONLY when the useMultiFiles parameter is set to true.

Unique Configuration Parameters

• dataPath

• schemaPath

• pretty

• useMultiFiles

dataPath

• Purpose: Specifies the path to a directory where NoSQL Database Migrator copies the source data in the JSON format.

  NoSQL Database Migrator creates JSON files in the specified directory. If the files exist, NoSQL Database Migrator overwrites their content with source data.

  Ensure that the directory already exists and has read and write permissions.

• Data Type: string

• Mandatory (Y/N): Y

• Example: "dataPath" : "/home/user/data"

  After successful migration, the directory specified in the dataPath parameter will include exported files as shown in the following sample:

    |--<Table_name>_1_5.json
    |--<Table_name>_6_10.json
    ...

schemaPath

• Purpose: Specifies the absolute path to a file to write table schema information provided by the source.

  If this value is not defined, the source schema information will not be migrated to the sink. If this value is specified, the migrator utility writes the schema of the source table into the file specified here.

  The schema information is written as one DDL command per line in this file. If the file does not exist in the specified data path, NoSQL Database Migrator creates it. If it exists already, NoSQL Database Migrator will overwrite its contents with the source data. You must ensure that the parent directory in the data path is valid for the specified file.

• Data Type: string

• Mandatory (Y/N): N

• Example: "schemaPath" : "/home/user/schema_file"

pretty

• Purpose: Specifies whether or not to beautify the JSON output to increase readability.

  If not specified, it defaults to false.

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "pretty" : true

useMultiFiles

• Purpose: Specifies whether or not to further split the exported files (created under the directory specified in the dataPath parameter) into multiple sub-files of a specific size while migrating NoSQL Database table data to a directory. The useMultiFiles parameter defaults to true.

  NoSQL Database Migrator splits the NoSQL Database table data into multiple files while exporting data. If useMultiFiles parameter is set to true, each exported file is further split into sub-files of size specified in the chunkSize parameter.

  Example: After successful migration, the directory specified in the dataPath parameter will include exported files as shown in the following sample:

  |--<Table_name>_1_5_0.json
  |--<Table_name>_1_5_1.json
  |--<Table_name>_6_10_0.json
  |--<Table_name>_6_10_1.json
  |--<Table_name>_6_10_2.json
  ...

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "useMultiFiles" : true

Parquet File

The configuration file format for Parquet File as a sink of NoSQL Database Migrator is shown below.

Sink Configuration Template

"sink" : {
  "type" : "file",
  "format" : "parquet",
  "dataPath": "</path/to/a/dir>",
  "chunkSize" : <size in MB>,
  "compression": "<SNAPPY|GZIP|NONE>",
  "parquetOptions": {
    "useLogicalJson": <true|false>,
    "useLogicalEnum": <true|false>,
    "useLogicalUUID": <true|false>,
    "truncateDoubleSpecials": <true|false>
  }
}

Sink Parameters

Common Configuration Parameters

• type

  Use "type" : "file"

• format

  Use "format" : "parquet"

• chunkSize

  Example: "chunkSize" : 40

Unique Configuration Parameters

• dataPath

• compression

• parquetOptions

• parquetOptions.useLogicalJson

• parquetOptions.useLogicalEnum

• parquetOptions.useLogicalUUID

• parquetOptions.truncateDoubleSpecials

dataPath

• Purpose: Specifies the path to a directory for storing the migrated NoSQL table data. Ensure that the directory already exists and has read and write permissions.

• Data Type: string

• Mandatory (Y/N): Y

• Example: "dataPath" : "/home/user/migrator/my_table"

compression

• Purpose: Specifies the compression type to use to compress the Parquet data. Valid values are SNAPPY, GZIP, and NONE.

  If not specified, it defaults to SNAPPY.

• Data Type: string

• Mandatory (Y/N): N

• Example: "compression" : "GZIP"

parquetOptions

• Purpose: Specifies the options to select Parquet logical types for NoSQL ENUM, JSON, and UUID columns.

  If you do not specify this parameter, the NoSQL Database Migrator writes the data of ENUM, JSON, and UUID columns as String.

• Data Type: object

• Mandatory (Y/N): N

parquetOptions.useLogicalJson

• Purpose: Specifies whether or not to write NoSQL JSON column data as Parquet logical JSON type. For more information, see Parquet Logical Type Definitions.

  If not specified or set to false, NoSQL Database Migrator writes the NoSQL JSON column data as String.

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "useLogicalJson" : true

parquetOptions.useLogicalEnum

• Purpose: Specifies whether or not to write NoSQL ENUM column data as Parquet logical ENUM type. For more information, see Parquet Logical Type Definitions.

  If not specified or set to false, NoSQL Database Migrator writes the NoSQL ENUM column data as String.

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "useLogicalEnum" : true

parquetOptions.useLogicalUUID

• Purpose: Specifies whether or not to write NoSQL UUID column data as Parquet logical UUID type. For more information, see Parquet Logical Type Definitions.

  If not specified or set to false, NoSQL Database Migrator writes the NoSQL UUID column data as String.

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "useLogicalUUID" : true

parquetOptions.truncateDoubleSpecials

• Purpose: Specifies whether or not to truncate the double +Infinity, -Infinity, and NaN values.

  By default, it is set to false. If set to true,

  • Positive_Infinity is truncated to Double.MAX_VALUE.

  • NEGATIVE_INFINITY is truncated to -Double.MAX_VALUE.

  • NaN is truncated to 9.9999999999999990E307.

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "truncateDoubleSpecials" : true

JSON File in OCI Object Storage Bucket

The configuration file format for JSON file in OCI Object Storage bucket as a sink of NoSQL Database Migrator is shown below.

Note: The valid source types for OCI Object Storage as the sink are nosqldb and nosqldb_cloud.

Sink Configuration Template

"sink" : {
  "type" : "object_storage_oci",
  "format" : "json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "chunkSize" : <size in MB>,
  "pretty" : <true|false>,
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
  "useDelegationToken" : <true|false>,
  "useSessionToken" : <true|false>,
  "useOKEWorkloadIdentity" : <true|false>
}

Sink Parameters

Common Configuration Parameters

• type

  Use "type" : "object_storage_oci"

• format

  Use "format" : "json"

• endpoint Example:

  • Region ID: "endpoint" : "us-ashburn-1"

  • URL format: "endpoint" : "https://objectstorage.us-ashburn- 1.oraclecloud.com"

• namespace

  Example: "namespace" : "my-namespace"

• bucket

  Example: "bucket" : "my-bucket"

• prefix

  Schema is migrated to the <prefix>/Schema/schema.ddl file and source data is migrated to the <prefix>/Data/<chunk>.json files, where chunk=000000.json, 000001.json, and so forth.

  Example:

  1. "prefix" : "my_export"

  2. "prefix" : "my_export/2021-04-05/"

• chunkSize

  Example: "chunkSize" : 40

• credentials Example:

  1. "credentials" : "/home/user/.oci/config"

  2. "credentials" : "/home/user/security/config"

• credentialsProfile Example:

  1. "credentialsProfile" : "DEFAULT"

  2. "credentialsProfile" : "ADMIN_USER"

• useInstancePrincipal

  Example: "useInstancePrincipal" : true

• useDelegationToken

  Example: "useDelegationToken" : true

Note: The authentication with delegation token is supported only when the NoSQL Database Migrator is running from a Cloud Shell.

• useOKEWorkloadIdentity

  Example: "useOKEWorkloadIdentity" : true

• useSessionToken

  Example: "useSessionToken" : true

Unique Configuration Parameter

pretty

• Purpose: Specifies whether or not to beautify the JSON output to increase readability.

  If not specified, it defaults to false.

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "pretty" : true

Parquet File in OCI Object Storage Bucket

The configuration file format for Parquet file in OCI Object Storage bucket as a sink of NoSQL Database Migrator is shown below.

Note: The valid source types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Sink Configuration Template

"sink" : {
  "type" : "object_storage_oci",
  "format" : "parquet",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "chunkSize" : <size in MB>,
  "compression": "<SNAPPY|GZIP|NONE>",
  "parquetOptions": {
    "useLogicalJson": <true|false>,
    "useLogicalEnum": <true|false>,
    "useLogicalUUID": <true|false>,
    "truncateDoubleSpecials": <true|false>
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
  "useDelegationToken" : <true|false>,
  "useSessionToken" : <true|false>,
  "useOKEWorkloadIdentity" : <true|false>
}

Sink Parameters

Common Configuration Parameters

• type

  Use "type" : "object_storage_oci"

• format

  Use "format" : "parquet"

• endpoint

  Example:

  • Region ID: "endpoint" : "us-ashburn-1"

  • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

• namespace

  Example: "namespace" : "my-namespace"

• bucket

  Example: "bucket" : "my-bucket"

• prefix

  Source data is migrated to the <prefix>/Data/<chunk>.parquet files, where chunk=000000.parquet, 000001.parquet, and so forth.

  Example:

  1. "prefix" : "my_export"

  2. "prefix" : "my_export/2021-04-05/"

• chunkSize

  Example: "chunkSize" : 40

• credentials

  Example:

  1. "credentials" : "/home/user/.oci/config"

  2. "credentials" : "/home/user/security/config"

• credentialsProfile

  Example:

  1. "credentialsProfile" : "DEFAULT"

  2. "credentialsProfile" : "ADMIN_USER"

• useInstancePrincipal

  Example: "useInstancePrincipal" : true

• useDelegationToken

  Example: "useDelegationToken" : true

Note: The authentication with delegation token is supported only when the NoSQL Database Migrator is running from a Cloud Shell.

• useOKEWorkloadIdentity

  Example: "useOKEWorkloadIdentity" : true

• useSessionToken

  Example: "useSessionToken" : true

Unique Configuration Parameter

• compression

• parquetOptions

• parquetOptions.useLogicalJson

• parquetOptions.useLogicalEnum

• parquetOptions.useLogicalUUID

• parquetOptions.truncateDoubleSpecials

compression

• Purpose: Specifies the compression type to use to compress the Parquet data. Valid values are SNAPPY, GZIP, and NONE.

  If not specified, it defaults to SNAPPY.

• Data Type: string

• Mandatory (Y/N): N

• Example: "compression" : "GZIP"

parquetOptions

• Purpose: Specifies the options to select Parquet logical types for NoSQL ENUM, JSON, and UUID columns.

  If you do not specify this parameter, the NoSQL Database Migrator writes the data of ENUM, JSON, and UUID columns as String.

• Data Type: object

• Mandatory (Y/N): N

parquetOptions.useLogicalJson

• Purpose: Specifies whether or not to write NoSQL JSON column data as Parquet logical JSON type. For more information, see Parquet Logical Type Definitions.

  If not specified or set to false, NoSQL Database Migrator writes the NoSQL JSON column data as String.

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "useLogicalJson" : true

parquetOptions.useLogicalEnum

• Purpose: Specifies whether or not to write NoSQL ENUM column data as Parquet logical ENUM type. For more information, see Parquet Logical Type Definitions.

  If not specified or set to false, NoSQL Database Migrator writes the NoSQL ENUM column data as String.

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "useLogicalEnum" : true

parquetOptions.useLogicalUUID

• Purpose: Specifies whether or not to write NoSQL UUID column data as Parquet logical UUID type. For more information, see Parquet Logical Type Definitions.

  If not specified or set to false, NoSQL Database Migrator writes the NoSQL UUID column data as String.

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "useLogicalUUID" : true

parquetOptions.truncateDoubleSpecials

• Purpose: Specifies whether or not to truncate the double +Infinity, -Infinity, and NaN values.

  By default, it is set to false. If set to true,

  • Positive_Infinity is truncated to Double.MAX_VALUE.

  • NEGATIVE_INFINITY is truncated to -Double.MAX_VALUE.

  • NaN is truncated to 9.9999999999999990E307.

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "truncateDoubleSpecials" : true

Oracle NoSQL Database

The configuration file format for Oracle NoSQL Database as a sink of NoSQL Database Migrator is shown below.

Sink Configuration Template

"sink" : {
  "type": "nosqldb",
  "storeName" : "<store name>",
  "helperHosts" : ["hostname1:port1","hostname2:port2,..."],
  "security" : "</path/to/store/credentials/file>",
  "table" : "<fully qualified table name>",
  "includeTTL": <true|false>,
  "ttlRelativeDate": "<date-to-use in UTC>",
  "schemaInfo" : {
    "schemaPath" : "</path/to/a/schema/file>",
    "defaultSchema" : <true|false>,
    "useSourceSchema" : <true|false>,
    "DDBPartitionKey" : <"name:type">,
    "DDBSortKey" : "<name:type>"
  },
  "overwrite" : <true|false>,
  "requestTimeoutMs" : <timeout in milli seconds>
}

Sink Parameters

Common Configuration Parameter

• type

  Use "type" : "nosqldb"

• security

  Example:

  "security" : "/home/user/client.credentials"

  Example security file content for password file based authentication:

  oracle.kv.password.noPrompt=true
  oracle.kv.auth.username=admin
  oracle.kv.auth.pwdfile.file=/home/nosql/login.passwd
  oracle.kv.transport=ssl
  oracle.kv.ssl.trustStore=/home/nosql/client.trust
  oracle.kv.ssl.protocols=TLSv1.2
  oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

  Example security file content for wallet based authentication:

  oracle.kv.password.noPrompt=true
  oracle.kv.auth.username=admin
  oracle.kv.auth.wallet.dir=/home/nosql/login.wallet
  oracle.kv.transport=ssl
  oracle.kv.ssl.trustStore=/home/nosql/client.trust
  oracle.kv.ssl.protocols=TLSv1.2
  oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

• requestTimeoutMs

  Example: "requestTimeoutMs" : 5000

Unique Configuration Parameter

• storeName

• helperHosts

• table

• includeTTL

• ttlRelativeDate

• schemaInfo

• schemaInfo.schemaPath

• schemaInfo.defaultSchema

• schemaInfo.useSourceSchema

• schemaInfo.DDBPartitionKey

• schemaInfo.DDBSortKey

• overwrite

storeName

• Purpose: Name of the Oracle NoSQL Database store.

• Data Type: string

• Mandatory (Y/N): Y

• Example: "storeName" : "kvstore"

helperHosts

• Purpose: A list of host and registry port pairs in the hostname:port format. Delimit each item in the list using a comma. You must specify at least one helper host.

• Data Type: array of strings

• Mandatory (Y/N): Y

• Example: "helperHosts" : ["localhost:5000","localhost:6000"]

table

• Purpose: Specifies the table name to store the migrated data.

  Format: [namespace_name:]<table_name>

  If the table is in the DEFAULT namespace, you can omit the namespace_name. The table must exist in the store during the migration, and its schema must match with the source data.

  If the table is not available in the sink, you can use the schemaInfo parameter to instruct the NoSQL Database Migrator to create the table in the sink.

• Data Type: string

• Mandatory (Y/N): Y

• Example:

  • With the DEFAULT namespace "table" :"mytable"

  • With a non-default namespace "table" : "mynamespace:mytable"

  • To specify a child table "table" : "mytable.child"

  Note: You can migrate the child tables from a valid data source to Oracle NoSQL Database. The NoSQL Database Migrator copies only a single table in each execution. Ensure that the parent table is migrated before the child table.

includeTTL

• Purpose: Specifies whether or not to include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.

  If you do not specify this parameter, it defaults to false. In that case, the NoSQL Database Migrator does not include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.

  If set to true, the NoSQL Database Migrator tool performs the following checks on the TTL metadata when importing a table row:

  • If you import a row that does not have _metadata definition, the NoSQL Database Migrator tool sets the TTL to 0, which means the row never expires.

  • If you import a row that has _metadata definition, the NoSQL Database Migrator tool compares the TTL value against a Reference Time when a row gets imported. If the row has already expired relative to the Reference Time, then it is skipped. If the row has not expired, then it is imported along with the TTL value. By default, the Reference Time of import operation is the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running. But you can also set a custom Reference Time using the ttlRelativeDate configuration parameter if you want to extend the expiration time and import rows that would otherwise expire immediately.

    The formula to calculate the expiration time of a row is as follows:

    expiration = (TTL value of source row in milliseconds - Reference Time in milliseconds)
    if (expiration <= 0) then it indicates that row has expired.

  Note: Since Oracle NoSQL TTL boundaries are in hours and days, in some cases, the TTL of the imported row might get adjusted to the nearest hour or day. For example, consider a row that has expiration value of 1629709200000 (2021-08-23 09:00:00) and Reference Time value is 1629707962582 (2021-08-23 08:39:22). Here, even though the row is not expired relative to the Reference Time when this data gets imported, the new TTL for the row is 1629712800000 (2021-08-23 10:00:00).

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "includeTTL" : true

ttlRelativeDate

• Purpose: Specify a UTC date in the YYYY-MM-DD hh:mm:ss format used to set the TTL expiry of table rows during importing into the Oracle NoSQL Database.

  If a table row in the data you are exporting has expired, you can set the ttlRelativeDate parameter to a date before the expiration time of the table row in the exported data.

  If you do not specify this parameter, it defaults to the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running.

• Data Type: date

• Mandatory (Y/N): N

• Example: "ttlRelativeDate" : "2021-01-03 04:31:17"

  Let us consider a scenario where table rows expire after seven days from 1-Jan-2021. After exporting this table, on 7-Jan-2021, you run into an issue with your table and decide to import the data. The table rows are going to expire in one day (data expiration date minus the default value of ttlRelativedate configuration parameter, which is the current date). But if you want to extend the expiration date of table rows to five days instead of one day, use the ttlRelativeDate parameter and choose an earlier date. Therefore, in this scenario if you want to extend expiration time of the table rows by five days, set the value of ttlRelativeDate configuration parameters to 3-Jan-2021, which is used as Reference Time when table rows get imported.

schemainfo

• Purpose: Specifies the schema for the data being migrated. If this is not specified, the NoSQL Database Migrator assumes that the table already exists in the sink’s store.

• Data Type: Object

• Mandatory (Y/N): N

schemaInfo.schemaPath

• Purpose: Specifies the absolute path to a file containing DDL statements for the NoSQL table.

  The NoSQL Database Migrator executes the DDL commands listed in this file before migrating the data.

  The NoSQL Database Migrator does not support more than one DDL statement per line in the schemaPath file.

• Data Type: string

• Mandatory (Y/N): N

  Note: defaultSchema and schemaPath are mutually exclusive.

• Example: "schemaPath" : "/home/user/schema_file"

schemaInfo.defaultSchema

• Purpose: Setting this parameter to true instructs the NoSQL Database Migrator to create a table with default schema. The default schema is defined by the migrator itself. For more information about default schema definitions, see Default Schema in Using Oracle NoSQL Data Migrator.

• Data Type: boolean

• Mandatory (Y/N): N

Note: defaultSchema and schemaPath are mutually exclusive.

schemaInfo.useSourceSchema

• Purpose: Specifies whether or not the sink uses the table schema definition provided by the source when migrating NoSQL tables.

• Data Type: boolean

• Mandatory (Y/N): N

  Note: defaultSchema, schemaPath, and useSourceSchema parameters are mutually exclusive. Specify only one of these parameters.

• Example:

  • With Default Schema:

    "schemaInfo" : {
      "defaultSchema" : true
    }

  • With a pre-defined schema:

    "schemaInfo" : {
     "schemaPath" : "<complete/path/to/the/schema/definition/file>"
    }

  • With source schema:

    "schemaInfo" : {
      "useSourceSchema" : true
    }

schemaInfo.DDBPartitionKey

• Purpose: Specifies the DynamoDB partition key and the corresponding Oracle NoSQL Database type to be used in the sink Oracle NoSQL Database table. This key will be used as a NoSQL DB table shard key. This is applicable only when defaultSchema is set to true and the source format is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.

• Mandatory (Y/N): Y, if defaultSchema is true and the source is dynamodb_json.

• Example: "DDBPartitionKey" : "PersonID:INTEGER"

Note: If the partition key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

schemaInfo.DDBSortKey

• Purpose: Specifies the DynamoDB sort key and its corresponding Oracle NoSQL Database type to be used in the target Oracle NoSQL Database table. If the importing DynamoDB table does not have a sort key, this attribute must not be set. This key will be used as a non-shard portion of the primary key in the NoSQL DB table. This is applicable only when defaultSchema is set to true and the source is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.

• Mandatory (Y/N): N

• Example: "DDBSortKey" : "Skey:STRING"

Note: If the sort key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

overwrite

• Purpose: Indicates the behavior of NoSQL Database Migrator when the record being migrated from the source is already present in the sink.

  If the value is set to false, when migrating tables the NoSQL Database Migrator skips those records for which the same primary key already exists in the sink.

  If the value is set to true, when migrating tables the NoSQL Database Migrator overwrites those records for which the same primary key already exists in the sink.

  If not specified, it defaults to true.

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "overwrite" : false

Oracle NoSQL Database Cloud Service

The configuration file format for Oracle NoSQL Database Cloud Service as a sink of NoSQL Database Migrator is shown below.

Sink Configuration Template

"sink" : {
  "type" : "nosqldb_cloud",
  "endpoint" : "<Oracle NoSQL Cloud Service Endpoint>",
  "table" : "<table name>",
  "compartment" : "<OCI compartment name or id>",
  "includeTTL": <true|false>,
  "ttlRelativeDate" : "<date-to-use in UTC>",
  "schemaInfo" : {
    "schemaPath" : "</path/to/a/schema/file>",
    "defaultSchema" : <true|false>,
    "useSourceSchema" : <true|false>,
    "DDBPartitionKey" : <"name:type">,
    "DDBSortKey" : "<name:type>",
    "onDemandThroughput" : <true|false>,
    "readUnits" : <table read units>,
    "writeUnits" : <table write units>,
    "storageSize" : <storage size in GB>
   },
  "credentials" : "</path/to/oci/credential/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
  "useDelegationToken" : <true|false>,
  "useSessionToken" : <true|false>,
  "useOKEWorkloadIdentity" : <true|false>,
  "writeUnitsPercent" : <table writeunits percent>,
  "requestTimeoutMs" : <timeout in milli seconds>,
  "overwrite" : <true|false>
}

Sink Parameters

Common Configuration Parameters

• type

  Use "type" : "nosqldb_cloud"

• endpoint

  Example:

  • Region ID: "endpoint" : "us-ashburn-1"

  • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

• credentials

  Example:

  1. "credentials" : "/home/user/.oci/config"

  2. "credentials" : "/home/user/security/config"

• credentialsProfile

  Example:

  1. "credentialsProfile" : "DEFAULT"

  2. "credentialsProfile" : "ADMIN_USER"

• useInstancePrincipal

  Example: "useInstancePrincipal" : true

• useDelegationToken

  Example: "useDelegationToken" : true

  Note: The authentication with delegation token is supported only when the NoSQL Database Migrator is running from a Cloud Shell.

• useOKEWorkloadIdentity

  Example: "useOKEWorkloadIdentity" : true

• useSessionToken

  Example: "useSessionToken" : true

• requestTimeoutMs

  Example: "requestTimeoutMs" : 5000

Unique Configuration Parameter

• table

• compartment

• includeTTL

• ttlRelativeDate

• schemaInfo

• schemaInfo.schemaPath

• schemaInfo.defaultSchema

• schemaInfo.useSourceSchema

• schemaInfo.DDBPartitionKey

• schemaInfo.DDBSortKey

• schemaInfo.onDemandThroughput

• schemaInfo.readUnits

• schemaInfo.writeUnits

• schemaInfo.storageSize

• writeUnitsPercent

• overwrite

table

• Purpose: Specifies the table name to store the migrated data.

  You must ensure that this table exists in your Oracle NoSQL Database Cloud Service. Otherwise, you have to use the schemaInfo object in the sink configuration to instruct the NoSQL Database Migrator to create the table.

  The schema of this table must match the source data.

• Data Type: string

• Mandatory (Y/N): Y

• Example:

  • To specify a table "table" : "mytable"

  • To specify a child table "table" : "mytable.child"

Note: You can migrate the child tables from a valid data source to Oracle NoSQL Database Cloud Service. The NoSQL Database Migrator copies only a single table in each execution. Ensure that the parent table is migrated before the child table.

compartment

• Purpose: Specifies the name or OCID of the compartment in which the table resides.

  If you do not provide any value, it defaults to the root compartment.

  You can find your compartment’s OCID from the Compartment Explorer window under Governance in the OCI Cloud Console.

• Data Type: string

• Mandatory (Y/N): Y, if the table is not in the root compartment of the tenancy OR when the useInstancePrincipal parameter is set to true.

Note: If the useInstancePrincipal parameter is set to true, the compartment must specify the compartment OCID and not the name.

• Example:

  • Compartment name

    "compartment" : "mycompartment"

  • Compartment name qualified with its parent compartment

    "compartment" : "parent.childcompartment"

  • No value provided. Defaults to the root compartment.

    "compartment": ""

  • Compartment OCID

    "compartment" : "ocid1.tenancy.oc1...4ksd"

includeTTL

• Purpose: Specifies whether or not to include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.

  If you do not specify this parameter, it defaults to false. In that case, the NoSQL Database Migrator does not include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.

  If set to true, the NoSQL Database Migrator tool performs the following checks on the TTL metadata when importing a table row:

  • If you import a row that does not have _metadata definition, the NoSQL Database Migrator tool sets the TTL to 0, which means the row never expires.

  • If you import a row that has _metadata definition, the NoSQL Database Migrator tool compares the TTL value against a Reference Time when a row gets imported. If the row has already expired relative to the Reference Time, then it is skipped. If the row has not expired, then it is imported along with the TTL value. By default, the Reference Time of import operation is the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running. But you can also set a custom Reference Time using the ttlRelativeDate configuration parameter if you want to extend the expiration time and import rows that would otherwise expire immediately.

    The formula to calculate the expiration time of a row is as follows:

    expiration = (TTL value of source row in milliseconds - Reference Time in milliseconds)
    if (expiration <= 0) then it indicates that row has expired.

  Note: Since Oracle NoSQL TTL boundaries are in hours and days, in some cases, the TTL of the imported row might get adjusted to the nearest hour or day. For example, consider a row that has expiration value of 1629709200000 (2021-08-23 09:00:00) and Reference Time value is 1629707962582 (2021-08-23 08:39:22). Here, even though the row is not expired relative to the Reference Time when this data gets imported, the new TTL for the row is 1629712800000 (2021-08-23 10:00:00).

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "includeTTL" : true

ttlRelativeDate

• Purpose: Specify a UTC date in the YYYY-MM-DD hh:mm:ss format used to set the TTL expiry of table rows during importing into the Oracle NoSQL Database.

  If a table row in the data you are exporting has expired, you can set the ttlRelativeDate parameter to a date before the expiration time of the table row in the exported data.

  If you do not specify this parameter, it defaults to the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running.

• Data Type: date

• Mandatory (Y/N): N

• Example: "ttlRelativeDate" : "2021-01-03 04:31:17"

  Let us consider a scenario where table rows expire after seven days from 1-Jan-2021. After exporting this table, on 7-Jan-2021, you run into an issue with your table and decide to import the data. The table rows are going to expire in one day (data expiration date minus the default value of ttlRelativedate configuration parameter, which is the current date). But if you want to extend the expiration date of table rows to five days instead of one day, use the ttlRelativeDate parameter and choose an earlier date. Therefore, in this scenario if you want to extend expiration time of the table rows by five days, set the value of ttlRelativeDate configuration parameters to 3-Jan-2021, which is used as Reference Time when table rows get imported.

schemaInfo

• Purpose: Specifies the schema for the data being migrated.

  If you do not specify this parameter, the NoSQL Database Migrator assumes that the table already exists in your Oracle NoSQL Database Cloud Service.

  If this parameter is not specified and the table does not exist in the sink, the migration fails.

• Data Type: Object

• Mandatory (Y/N): N

schemaInfo.schemaPath

• Purpose: Specifies the absolute path to a file containing DDL statements for the NoSQL table.

  The NoSQL Database Migrator executes the DDL commands listed in this file before migrating the data.

  The NoSQL Database Migrator does not support more than one DDL statement per line in the schemaPath file.

• Data Type: string

• Mandatory (Y/N): N

  Note: defaultSchema and schemaPath are mutually exclusive.

• Example: "schemaPath" : "/home/user/schema_file"

schemaInfo.defaultSchema

• Purpose: Setting this parameter to Yes instructs the NoSQL Database Migrator to create a table with default schema. The default schema is defined by the migrator itself. For more information about default schema definitions, see Default Schema in Using Oracle NoSQL Data Migrator.

• Data Type: boolean

• Mandatory (Y/N): N

Note: defaultSchema and schemaPath are mutually exclusive.

schemaInfo.useSourceSchema

• Purpose: Specifies whether or not the sink uses the table schema definition provided by the source when migrating NoSQL tables.

• Data Type: boolean

• Mandatory (Y/N): N

  Note: defaultSchema, schemaPath, and useSourceSchema parameters are mutually exclusive. Specify only one of these parameters.

• Example:

  • With Default Schema:

    "schemaInfo": {
      "defaultSchema": true,
      "readUnits": 100,
      "writeUnits": 60,
      "storageSize": 1
    }

  • With a pre-defined schema:

    "schemaInfo": {
      "schemaPath": "<complete/path/to/the/schema/definition/file>",
      "readUnits": 100,
      "writeUnits": 100,
      "storageSize": 1
    }

  • With source schema:

    "schemaInfo": {
      "useSourceSchema": true,
      "readUnits": 100,
      "writeUnits": 60,
      "storageSize": 1
    }

schemaInfo.DDBPartitionKey

• Purpose: Specifies the DynamoDB partition key and the corresponding Oracle NoSQL Database type to be used in the sink Oracle NoSQL Database table. This key will be used as a NoSQL DB table shard key. This is applicable only when defaultSchema is set to true and the source format is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.

• Mandatory (Y/N): Y, if defaultSchema is true and the source is dynamodb_json.

• Example: "DDBPartitionKey" : "PersonID:INTEGER"

Note: If the partition key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

schemaInfo.DDBSortKey

• Purpose: Specifies the DynamoDB sort key and its corresponding Oracle NoSQL Database type to be used in the target Oracle NoSQL Database table. If the importing DynamoDB table does not have a sort key, this attribute must not be set. This key will be used as a non-shard portion of the primary key in the NoSQL DB table. This is applicable only when defaultSchema is set to true and the source is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.

• Mandatory (Y/N): N

• Example: "DDBSortKey" : "Skey:STRING"

Note: If the sort key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

schemaInfo.onDemandThroughput

• Purpose: Specifies to create the table with on-demand read and write throughput. If this parameter is not set, the table is created with provisioned capacity.

  The default value is false.

  Note: This parameter is not applicable for child tables as they share the throughput of the top-level parent table.

• Data Type: Boolean

• Mandatory (Y/N): N

• Example: "onDemandThroughput" : "true"

schemaInfo.readUnits

• Purpose: Specifies the read throughput of the new table.

  Note:

  • This parameter is not applicable for tables provisioned with on-demand capacity.
  • This parameter is not applicable for child tables as they share the read throughput of the top-level parent table.

• Data Type: integer

• Mandatory (Y/N): Y, if the table is not a child table or if schemaInfo.onDemandThroughput parameter is set to false, else N.

• Example: "readUnits" : 100

schemaInfo.writeUnits

• Purpose: Specifies the write throughput of the new table.

  Note:

  • This parameter is not applicable for tables provisioned with on-demand capacity.
  • This parameter is not applicable for child tables as they share the write throughput of the top-level parent table.

• Data Type: integer

• Mandatory (Y/N): Y, if the table is not a child table or if schemaInfo.onDemandThroughput parameter is set to false, else N.

• Example: "writeUnits" : 100

schemaInfo.storageSize

• Purpose: Specifies the storage size of the new table in GB.

  Note: This parameter is not applicable for child tables as they share the storage size of the top-level parent table.

• Data Type: integer

• Mandatory (Y/N): Y, if the table is not a child table, else N.

• Example:

  • With schemaPath

    "schemaInfo" : {
      "schemaPath" : "</path/to/a/schema/file>",
      "readUnits" : 500,
      "writeUnits" : 1000,
      "storageSize" : 5 }

  • With defaultSchema

    "schemaInfo" : {
      "defaultSchema" :Yes,
      "readUnits" : 500,
      "writeUnits" : 1000,
      "storageSize" : 5
    }

writeUnitsPercent

• Purpose: Specifies the Percentage of table write units to be used during the migration activity. The amount of time required to migrate data is directly proportional to this attribute.

  The default value is 90. The valid range is any integer between 1 to 100.

  See Troubleshooting the Oracle NoSQL Database Migrator to learn how to use this attribute to improve the data migration speed.

• Data Type: integer

• Mandatory (Y/N): N

• Example: "writeUnitsPercent" : 90

overwrite

• Purpose: Indicates the behavior of NoSQL Database Migrator when the record being migrated from the source is already present in the sink.

  If the value is set to false, when migrating tables the NoSQL Database Migrator skips those records for which the same primary key already exists in the sink.

  If the value is set to true, when migrating tables the NoSQL Database Migrator overwrites those records for which the same primary key already exists in the sink.

  If not specified, it defaults to true.

• Data Type: boolean

• Mandatory (Y/N): N

• Example: "overwrite" : false