Oracle NoSQL Database Migrator Reference

Learn about source, sink, and transformation configuration template parameters available for Oracle NoSQL Database Migrator.

This article has the following topics:

Source Configuration Templates

Learn about the configuration file formats for each valid source and the purpose of each configuration parameter.

JSON File

The configuration file format for JSON File as a source of NoSQL Database Migrator is shown below.

Configuration Template

"source" : {
  "type" : "file",
  "format" : "json",
  "dataPath": "</path/to/a/json/file>",
  "schemaInfo": {
               "schemaPath": "</path/to/schema/file>"
  }
}

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "type" : "file"

format

  • Purpose: Specifies the source format.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "format" : "json"

dataPath

  • Purpose: Specifies the absolute path to a file or directory containing the JSON data for migration.

    You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • Specifying a JSON file

      "dataPath" : "/home/user/sample.json"

    • Specifying a directory

      "dataPath" : "/home/user"

schemaInfo

  • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaPath

  • Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    "schemaInfo" : {
                "schemaPath" : "/home/user/mytable/Schema/schema.ddl"
                 }

JSON File in OCI Object Storage Bucket

The configuration file format for JSON file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below.

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "schemaInfo" : {
      "schemaObject" : "<object name>"
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "type" : "object_storage_oci"

format

  • Purpose: Specifies the source format.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "format" : "json"

endpoint

  • Purpose: Specifies the OCI Object Storage service endpoint URL or region ID.

    You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

bucket

  • Purpose: Specifies the name of the bucket, which contains the source JSON files. Ensure that the required bucket already exists in the OCI Object Storage instance and has read permissions.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "bucket" : "staging_bucket"

prefix

  • Purpose: Used for filtering the objects that are being migrated from the bucket. All the objects with the given prefix present in the bucket are migrated. For more information about prefixes, see Object Naming Using Prefixes and Hierarchies.

    If you do not provide any value, no filter is applied and all the objects present in the bucket are migrated.

  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "prefix" : "my_table/Data/000000.json" (migrates only 000000.json)
    2. "prefix" : "my_table/Data" (migrates all the objects with prefix my_table/Data)

schemaInfo

  • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaObject

  • Purpose: Specifies the name of the object in the bucket where NoSQL table schema definitions for the data being migrated are stored.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    "schemaInfo" : {
                "schemaObject" : "mytable/Schema/schema.ddl"
                 }

credentials

  • Purpose: Absolute path to a file containing OCI credentials.

    If not specified, it defaults to $HOME/.oci/config

    See Example Configuration for an example of the credentials file.

    Note:

    Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"

credentialsProfile

  • Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

    If you do not specify this value, it defaults to the DEFAULT profile.

    Note:

    This parameter is valid ONLY if the credentials parameter is specified.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile": "ADMIN_USER"

useInstancePrincipal

  • Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security.

    If not specified, it defaults to false.

    Note:

    • It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
    • Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useInstancePrincipal" : true

MongoDB-Formatted JSON File

The configuration file format for MongoDB-formatted JSON File as a source of NoSQL Database Migrator is shown below.

Configuration Template

"source" : {
    "type" : "file",
    "format" : "mongodb_json",
    "dataPath": "</path/to/a/json/file>",
    "schemaInfo": {
               "schemaPath": "</path/to/schema/file>"
    }
}

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "type" : "file"

format

  • Purpose: Specifies the source format.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "format" : "mongodb_json"

dataPath

  • Purpose: Specifies the absolute path to a file or directory containing the MongoDB exported JSON data for migration.

    You must have generated these files using the mongoexport tool. See mongoexport for more information.

    You can supply the MongoDB-formatted JSON file that is generated using the mongoexport tool in either canonical or relaxed mode. Both the modes are supported by the NoSQL Database Migrator for migration.

    If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported. You must ensure that this data matches with the NoSQL table schema defined at the sink.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • Specifying a MongoDB formatted JSON file

      "dataPath" : "/home/user/sample.json"

    • Specifying a directory

      "dataPath" : "/home/user"

schemaInfo

  • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaPath

  • Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    "schemaInfo" : {
                "schemaPath" : "/home/user/mytable/Schema/schema.ddl"
                 }

MongoDB-Formatted JSON File in OCI Object Storage bucket

The configuration file format for MongoDB-Formatted JSON file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below.

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "mongodb_json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "schemaInfo" : {
      "schemaObject" : "<object name>"
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "type" : "object_storage_oci"

format

  • Purpose: Specifies the source format.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "format" : "mongodb_json"

endpoint

  • Purpose: Specifies the OCI Object Storage service endpoint URL or region ID.

    You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

bucket

  • Purpose: Specifies the name of the bucket, which contains the source MongoDB-Formatted JSON files. Ensure that the required bucket already exists in the OCI Object Storage instance and has read permissions.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "bucket" : "staging_bucket"

prefix

  • Purpose: Used for filtering the objects that are being migrated from the bucket. All the objects with the given prefix present in the bucket are migrated. For more information about prefixes, see Object Naming Using Prefixes and Hierarchies.

    If you do not provide any value, no filter is applied and all the MongoDB JSON formatted objects present in the bucket are migrated. Extract the data from MongoDB using the mongoexport utility and upload it to the OCI Object Storage bucket. See mongoexport for more information.

    If you do not provide any value, no filter is applied and all the objects present in the bucket are migrated.

  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "prefix" : "mongo_export/Data/table.json" (migrates only table.json)
    2. "prefix" : "mongo_export/Data" (migrates all the objects with prefix mongo_export/Data)

schemaInfo

  • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaObject

  • Purpose: Specifies the name of the object in the bucket where NoSQL table schema definitions for the data being migrated are stored.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    "schemaInfo" : {
                "schemaObject" : "mytable/Schema/schema.ddl"
                 }

credentials

  • Purpose: Absolute path to a file containing OCI credentials.

    If not specified, it defaults to $HOME/.oci/config

    See Example Configuration for an example of the credentials file.

    Note:

    Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"

credentialsProfile

  • Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

    If you do not specify this value, it defaults to the DEFAULT profile.

    Note:

    This parameter is valid ONLY if the credentials parameter is specified.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile": "ADMIN_USER"

useInstancePrincipal

  • Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security.

    If not specified, it defaults to false.

    Note:

    • It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
    • Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useInstancePrincipal" : true

DynamoDB-Formatted JSON File stored in AWS S3

The configuration file format for DynamoDB-formatted JSON File in AWS S3 as a source of NoSQL Database Migrator is shown below.

The valid sink types for DynamoDB-formatted JSON stored in AWS S3 are nosqldb and nosqldb_cloud.

Configuration Template
"source" : {
    "type" : "aws_s3",
    "format" : "dynamodb_json",
    "s3URL" : "<S3 object url>",
    "credentials" : "</path/to/aws/credentials/file>",
    "credentialsProfile" : <"profile name in aws credentials file">
}
Source Parameters:
  • type
  • format
  • s3URL
  • credentials
  • credentialsProfile
type
  • Purpose:Identifies the source type.
  • Data Type: string
  • Mandatory (Y/N):Y
  • Example: "type" : "aws_s3"
format
  • Purpose:Specifies the source format.
  • Data Type: string
  • Mandatory (Y/N):Y
  • Example: "format" : "dynamodb_json"

Note:

If the value of the "type" is aws_s3, then format must be dynamodb_json.
s3URL
  • Purpose: Specifies the URL of an exported DynamoDB table stored in AWS S3. You can obtain this URL from the AWS S3 console. Valid URL format is https://<bucket-name>.<s3_endpoint>/<prefix>. The migrator will look for json.gz files in the prefix for import.

    Note:

    You must export DynamoDB table as specified in Exporting DynamoDB table data to Amazon S3.
  • Data Type: string
  • Mandatory: Yes
  • Example: https://my-bucket.s3.ap-south-1.amazonaws.com/AWSDynamoDB/01649660790057-14f642be
credentials
  • Purpose: Specifies the absolute path to a file containing the AWS credentials. If not specified, it defaults to $HOME/.aws/credentials. Please refer to Configuration and credential file settings for more details on the credentials file.
  • Data Type: string
  • Mandatory (Y/N):N
  • Example:
    "credentials" : "/home/user/.aws/credentials"
    "credentials" : "/home/user/security/credentials

Note:

The Migrator does not log any of the credentials information. You should properly protect the credentials file from unauthorized access.
credentialsProfile
  • Purpose: Name of the profile in the AWS credentials file to be used to connect to AWS S3. User account credentials are referred to as a profile. If you do not specify this value, it defaults to the default profile. Please refer to Configuration and credential file settings for more details on the credentials file.
  • Data Type: string
  • Mandatory (Y/N):N
  • Example:
    "credentialsProfile" : "default"
            "credentialsProfile": "test"

DynamoDB-Formatted JSON File

The configuration file format for DynamoDB-formatted JSON File as a source of NoSQL Database Migrator is shown below.

The valid sink types for DynamoDB JSON file are nosqldb and nosqldb_cloud.

Configuration Template
"source" : {
    "type" : "file",
    "format" : "dynamodb_json",
    "dataPath" : "<path to a file or directory containing exported DDB table data>"   
}
Source Parameters:
  • type
  • format
  • dataPath
type
  • Purpose:Identifies the source type.
  • Data Type: string
  • Mandatory (Y/N):Y
  • Example: "type" : "file"
format
  • Purpose:Specifies the source format.
  • Data Type: string
  • Mandatory (Y/N):Y
  • Example: "format" : "dynamodb_json"
dataPath
  • Purpose: Specifies the absolute path to a file or directory containing the exported DynamoDB table data. You shall copy exported DynamoDB table data from AWS S3 to a local mounted file system. You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json.gz extension in that directory and the datasub-directory.
  • Data Type: string
  • Mandatory (Y/N):Y
  • Example:
    • Specifying a file
      "dataPath" : "/home/user/AWSDynamoDB/01639372501551-bb4dd8c3/data/zclclwucjy6v5mkefvckxzhfvq.json.gz"
    • Specifying a directory
      "dataPath" : "/home/user/AWSDynamoDB/01639372501551-bb4dd8c3"

Oracle NoSQL Database

The configuration file format for Oracle NoSQL Database as a source of NoSQL Database Migrator is shown below.

Configuration Template

"source" : {
    "type": "nosqldb",
    "table" : "<fully qualified table name>",
    "storeName" : "<store name>",
    "helperHosts" : ["hostname1:port1","hostname2:port2,..."],
    "security" : "</path/to/store/security/file>",
    "requestTimeoutMs" : 5000,
    "includeTTL": <true|false>
}

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "type" : "nosqldb"

table

  • Purpose: Fully qualified table name from which to migrate the data.

    Format: [namespace_name:]<table_name>

    If the table is in the DEFAULT namespace, you can omit the namespace_name. The table must exist in the store.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • With the DEFAULT namespace "table" :"mytable"

    • With a non-default namespace "table" : "mynamespace:mytable"

storeName

  • Purpose: Name of the Oracle NoSQL Database store.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "storeName" : "kvstore"

helperHosts

  • Purpose: A list of host and registry port pairs in the hostname:port format. Delimit each item in the list using a comma. You must specify at least one helper host.

  • Data Type: array of strings
  • Mandatory (Y/N): Y
  • Example: "helperHosts" : ["localhost:5000","localhost:6000"]

security

  • Purpose:

    If your store is a secure store, provide the absolute path to the security login file that contains your store credentials. See Configuring Security with Remote Access in Administrator's Guide to know more about the security login file.

    You can use either password file based authentication or wallet based authentication. However, the wallet based authentication is supported only in the Enterprise Edition (EE) of Oracle NoSQL Database. For more information on wallet-based authentication, see Source and Sink Security.

    The Community Edition(CE) edition supports password file based authentication only.

  • Data Type: string
  • Mandatory (Y/N): Y for a secure store
  • Example:

    "security" : "/home/user/client.credentials"

    Example security file content for password file based authentication:

    oracle.kv.password.noPrompt=true
    oracle.kv.auth.username=admin
    oracle.kv.auth.pwdfile.file=/home/nosql/login.passwd
    oracle.kv.transport=ssl
    oracle.kv.ssl.trustStore=/home/nosql/client.trust
    oracle.kv.ssl.protocols=TLSv1.2
    oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

    Example security file content for wallet based authentication:

    oracle.kv.password.noPrompt=true
    oracle.kv.auth.username=admin
    oracle.kv.auth.wallet.dir=/home/nosql/login.wallet
    oracle.kv.transport=ssl
    oracle.kv.ssl.trustStore=/home/nosql/client.trust
    oracle.kv.ssl.protocols=TLSv1.2
    oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

requestTimeoutMs

  • Purpose: Specifies the time to wait for each read operation from the store to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "requestTimeoutMs" : 5000

includeTTL

  • Purpose: Specifies whether or not to include TTL metadata for table rows when exporting Oracle NoSQL Database tables. If set to true, the TTL data for rows also gets included in the data provided by the source. TTL is present in the _metadata JSON object associated with each row. The expiration time for each row gets exported as the number of milliseconds since the UNIX epoch (Jan 1st, 1970).

    If you do not specify this parameter, it defaults to false.

    Only the rows having a positive expiration value for TTL get included as part of the exported rows. If a row does not expire, which means TTL=0, then its TTL metadata is not included explicitly. For example, if ROW1 expires at 2021-10-19 00:00:00 and ROW2 does not expire, the exported data looks like as follows:
    //ROW1
    {
      "id" : 1,
      "name" : "abc",
      "_metadata" : {
        "expiration" : 1634601600000
      }
    }
    
    //ROW2
    {
      "id" : 2,
      "name" : "xyz"
    }
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "includeTTL" : true

Oracle NoSQL Database Cloud Service

The configuration file format for Oracle NoSQL Database Cloud Service as a source of NoSQL Database Migrator is shown below.

Configuration Template

"source" : {
  "type" : "nosqldb_cloud",
  "endpoint" : "<Oracle NoSQL Cloud Service Endpoint. You can either specify  the complete URL or the Region ID alone>",
  "table" : "<table name>",
  "compartment" : "<OCI compartment name or id>",
  "credentials" : "</path/to/oci/credential/file>",
  "credentialsProfile" : "<oci credentials profile name>",
  "readUnitsPercent" : <table readunits percent>,
  "requestTimeoutMs" : <timeout in milli seconds>,
  "useInstancePrincipal" : <true|false>,
  "includeTTL": <true|false>
}

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "type" : "nosqldb_cloud"

endpoint

  • Purpose: Specifies the Service Endpoint of the Oracle NoSQL Database Cloud Service.

    You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://nosql.us-ashburn-1.oci.oraclecloud.com/"

table

  • Purpose: Name of the table from which to migrate the data.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "table" :"myTable"

compartment

  • Purpose: Specifies the name or OCID of the compartment in which the table resides.

    If you do not provide any value, it defaults to the root compartment.

    You can find your compartment's OCID from the Compartment Explorer window under Governance in the OCI Cloud Console.

  • Data Type: string
  • Mandatory (Y/N): Yes, if the table is not in the root compartment of the tenancy OR when the useInstancePrincipal parameter is set to true.

    Note:

    If the useInstancePrincipal parameter is set to true, the compartment must specify the compartment OCID and not the name.
  • Example:
    • Compartment name

      "compartment" : "mycompartment"

    • Compartment name qualified with its parent compartment

      "compartment" : "parent.childcompartment"

    • No value provided. Defaults to the root compartment.

      "compartment": ""

    • Compartment OCID

      "compartment" : "ocid1.tenancy.oc1...4ksd"

credentials

  • Purpose: Absolute path to a file containing OCI credentials.

    If not specified, it defaults to $HOME/.oci/config

    See Example Configuration for an example of the credentials file.

    Note:

    Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"

credentialsProfile

  • Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

    If you do not specify this value, it defaults to the DEFAULT profile.

    Note:

    This parameter is valid ONLY if the credentials parameter is specified.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile": "ADMIN_USER"

readUnitsPercent

  • Purpose: Percentage of table read units to be used while migrating the NoSQL table.

    The default value is 90. The valid range is any integer between 1 to 100. Please note that amount of time required to migrate data is directly proportional to this attribute. It's better to increase the read throughput of the table for the migration activity. You can reduce the read throughput after the migration process completes.

    To learn the daily limits on throughput changes, see Cloud Limits in Using Oracle NoSQL Database Cloud Service.

    The default value is 90. The valid range is any integer between 1 to 100.

    Note:

    The time required for the data migration is directly proportional to the writeUnitsPercent value.

    See Troubleshooting the Oracle NoSQL Database Migrator to learn how to use this attribute to improve the data migration speed.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "readUnitsPercent" : 90

requestTimeoutMs

  • Purpose: Specifies the time to wait for each read operation in the sink to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "requestTimeoutMs" : 5000

useInstancePrincipal

  • Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security

    If not specified, it defaults to false.

    Note:

    • It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
    • Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useInstancePrincipal" : true

includeTTL

  • Purpose: Specifies whether or not to include TTL metadata for table rows when exporting Oracle NoSQL Database Cloud Service tables. If set to true, the TTL data for rows also gets included in the data provided by the source. TTL is present in the _metadata JSON object associated with each row. The expiration time for each row gets exported as the number of milliseconds since the UNIX epoch (Jan 1st, 1970).

    If you do not specify this parameter, it defaults to false.

    Only the rows having a positive expiration value for TTL get included as part of the exported rows. If a row does not expire, which means TTL=0, then its TTL metadata is not included explicitly. For example, if ROW1 expires at 2021-10-19 00:00:00 and ROW2 does not expire, the exported data looks like as follows:
    //ROW1
    {
      "id" : 1,
      "name" : "abc",
      "_metadata" : {
        "expiration" : 1634601600000
      }
    }
    
    //ROW2
    {
      "id" : 2,
      "name" : "xyz"
    }
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "includeTTL" : true

Sink Configuration Templates

Learn about the configuration file formats for each valid sink and the purpose of each configuration parameter.

JSON File

The configuration file format for JSON File as a sink of NoSQL Database Migrator is shown below.

Configuration Template

"sink" : {
    "type" : "file",
    "format" : "json",
    "dataPath": "</path/to/a/file>",
    "schemaPath" : "<path/to/a/file>",
    "pretty" : <true|false>,
    "useMultiFiles" : <true|false>,
    "chunkSize" : <size in MB>
}

type

  • Purpose: Identifies the sink type.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "type" : "file"

format

  • Purpose: Specifies the sink format.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "format" : "json"

dataPath

  • Purpose: Specifies the absolute path to a file where the source data will be copied in the JSON format.

    If the file does not exist in the specified data path, the NoSQL Database Migrator creates it. If it exists already, the NoSQL Database Migrator will overwrite its contents with the source data.

    You must ensure that the parent directory for the file specified in the data path is valid.

    Note:

    If the useMultiFiles parameter is set to true, specify the path to a directory else specifies the path to the file.
  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • With useMultiFiles parameter set to true

      "dataPath" :"/home/user/data"

    • With useMultiFiles parameter not specified or it is set to false

      "dataPath" :"/home/user/sample.json"

schemaPath

  • Purpose: Specifies the absolute path to write schema information provided by the source.

    If this value is not defined, the source schema information will not be migrated to the sink. If this value is specified, the migrator utility writes the schema of the source table into the file specified here.

    The schema information is written as one DDL command per line in this file. If the file does not exist in the specified data path, NoSQL Database Migrator creates it. If it exists already, NoSQL Database Migrator will overwrite its contents with the source data. You must ensure that the parent directory for the file specified in the data path is valid.

  • Data Type: string
  • Mandatory (Y/N): N
  • Example: "schemaPath" : "/home/user/schema_file"

pretty

  • Purpose: Specifies whether to beautify the JSON output to increase readability or not.

    If not specified, it defaults to false.

  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "pretty" : true

useMultiFiles

  • Purpose: Specifies whether or not to split the NoSQL table data into multiple files when migrating source data to a file.

    If not specified, it defaults to false.

    If set to true, when migrating source data to a file, the NoSQL table data is split into multiple smaller files. For example, <chunk>.json, where chunk=000000,000001,000002, and so forth.

    dataPath
             |--000000.json
             |--000001.json
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useMultiFiles" : true

chunkSize

  • Purpose: Specifies the maximum size of a "chunk" of table data to be stored at the sink. During migration, a table is split into chunkSize chunks and each chunk is written as a separate file to the sink. When the source data being migrated exceeds this size, a new file is created.

    If not specified, defaults to 32MB. The valid value is an integer between 1 to 1024.

    Note:

    This parameter is applicable ONLY when the useMultiFiles parameter is set to true.
  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "chunkSize" : 40

Parquet File

The configuration file format for Parquet File as a sink of NoSQL Database Migrator is shown below.

Configuration Template

"sink" : {
    "type" : "file",
    "format" : "parquet",
    "dataPath": "</path/to/a/dir>",
    "chunkSize" : <size in MB>,
    "compression": "<SNAPPY|GZIP|NONE>",
    "parquetOptions": {
      "useLogicalJson": <true|false>,
      "useLogicalEnum": <true|false>,
      "useLogicalUUID": <true|false>,     
      "truncateDoubleSpecials": <true|false>
    }
}

type

  • Purpose: Identifies the sink type.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "type" : "file"

format

  • Purpose: Specifies the sink format.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "format" : "parquet"

dataPath

  • Purpose: Specifies the path to a directory to use for storing the migrated NoSQL table data. Ensure that the directory already exists and has read and write permissions.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "dataPath" : "/home/user/migrator/my_table"

chunkSize

  • Purpose: Specifies the maximum size of a "chunk" of table data to be stored at the sink. During migration, a table is split into chunkSize chunks and each chunk is written as a separate file to the sink. When the source data being migrated exceeds this size, a new file is created.

    If not specified, defaults to 32MB. The valid value is an integer between 1 to 1024.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "chunkSize" : 40

compression

  • Purpose: Specifies the compression type to use to compress the Parquet data. Valid values are SNAPPY, GZIP, and NONE.

    If not specified, it defaults to SNAPPY.

  • Data Type: string
  • Mandatory (Y/N): N
  • Example: "compression" : "GZIP"

parquetOptions

  • Purpose: Species the options to select Parquet logical types for NoSQL ENUM, JSON, and UUID columns.

    If you do not specify this parameter, the NoSQL Database Migrator writes the data of ENUM, JSON, and UUID columns as String.

  • Data Type: object
  • Mandatory (Y/N): N

parquetOptions.useLogicalJson

  • Purpose: Specifies whether or not to write NoSQL JSON column data as Parquet logical JSON type. For more information see Parquet Logical Type Definitions.

    If not specified or set to false, NoSQL Database Migrator writes the NoSQL JSON column data as String.

  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useLogicalJson" : true

parquetOptions.useLogicalEnum

  • Purpose: Specifies whether or not to write NoSQL ENUM column data as Parquet logical ENUM type. For more information see Parquet Logical Type Definitions.

    If not specified or set to false, NoSQL Database Migrator writes the NoSQL ENUM column data as String.

  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useLogicalEnum" : true

parquetOptions.useLogicalUUID

  • Purpose: Specifies whether or not to write NoSQL UUID column data as Parquet logical UUID type. For more information see Parquet Logical Type Definitions.

    If not specified or set to false, NoSQL Database Migrator writes the NoSQL UUID column data as String.

  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useLogicalUUID" : true

parquetOptions.truncateDoubleSpecials

  • Purpose: Specifies whether or not to truncate the double +Infinity, -Infinity, and Nan values.

    By default, it is set to false. If set to true,
    • Positive_Infinity is truncated to Double.MAX_VALUE.
    • NEGATIVE_INFINITY is truncated to -Double.MAX_VALUE.
    • NaN is truncated to 9.9999999999999990E307.
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "truncateDoubleSpecials" : true

JSON File in OCI Object Storage Bucket

The configuration file format for JSON file in OCI Object Storage bucket as a sink of NoSQL Database Migrator is shown below.

Note:

The valid source types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Configuration Template

"sink" : {
  "type" : "object_storage_oci",
  "format" : "json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "chunkSize" : <size in MB>,
  "pretty" : <true|false>,
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

type

  • Purpose: Identifies the sink type.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "type" : "object_storage_oci"

format

  • Purpose: Specifies the sink format.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "format" : "json"

endpoint

  • Purpose: Specifies the OCI Object Storage service endpoint URL or region ID.

    You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

bucket

  • Purpose: Specifies the bucket name to use for storing the migrated data. Ensure that the required bucket already exists in the OCI Object Storage instance and has write permissions.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "bucket" : "staging_bucket"

prefix

  • Purpose: Specifies the prefix that is added to the object name when objects are created in the bucket. The prefix acts as a logical container or directory for storing data. For more information about prefixes, see Object Naming Using Prefixes and Hierarchies.

    If not specified, the table name from the source is used as the prefix. If any object with the same name already exists in the bucket, it is overwritten.

    Schema is migrated to the <prefix>/Schema /schema.ddl file and source data is migrated to the <prefix>/Data/<chunk>.json file(s), where chunk=000000.json, 000001.json, and so forth.

  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "prefix" : "my_export"
    2. "prefix" : "my_export/2021-04-05/"

chunkSize

  • Purpose: Specifies the maximum size of a "chunk" of table data to be stored at the sink. During migration, a table is split into chunkSize chunks and each chunk is written as a separate file to the sink. When the source data being migrated exceeds this size, a new file is created.

    If not specified, defaults to 32MB. The valid value is an integer between 1 to 1024.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "chunkSize" : 40

pretty

  • Purpose: Specifies whether to beautify the JSON output to increase readability or not.

    If not specified, it defaults to false.

  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "pretty" : true

credentials

  • Purpose: Absolute path to a file containing OCI credentials.

    If not specified, it defaults to $HOME/.oci/config

    See Example Configuration for an example of the credentials file.

    Note:

    Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"

credentialsProfile

  • Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

    If you do not specify this value, it defaults to the DEFAULT profile.

    Note:

    This parameter is valid ONLY if the credentials parameter is specified.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile": "ADMIN_USER"

useInstancePrincipal

  • Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security.

    If not specified, it defaults to false.

    Note:

    • It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
    • Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useInstancePrincipal" : true

Parquet File in OCI Object Storage Bucket

The configuration file format for Parquet file in OCI Object Storage bucket as a sink of NoSQL Database Migrator is shown below.

Note:

The valid source types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Configuration Template

"sink" : {
  "type" : "object_storage_oci",
  "format" : "parquet",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "chunkSize" : <size in MB>,
  "compression": "<SNAPPY|GZIP|NONE>",
  "parquetOptions": {
    "useLogicalJson": <true|false>,
    "useLogicalEnum": <true|false>,
    "useLogicalUUID": <true|false>,
    "truncateDoubleSpecials": <true|false>
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

type

  • Purpose: Identifies the sink type.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "type" : "object_storage_oci"

format

  • Purpose: Specifies the sink format.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "format" : "parquet"

endpoint

  • Purpose: Specifies the OCI Object Storage service endpoint URL or region ID.

    You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

bucket

  • Purpose: Specifies the bucket name to use for storing the migrated data. Ensure that the required bucket already exists in the OCI Object Storage instance and has write permissions.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "bucket" : "staging_bucket"

prefix

  • Purpose: Specifies the prefix that is added to the object name when objects are created in the bucket. The prefix acts as a logical container or directory for storing data. For more information about prefixes, see Object Naming Using Prefixes and Hierarchies.

    If not specified, the table name from the source is used as the prefix. If any object with the same name already exists in the bucket, it is overwritten.

    Source data is migrated to the <prefix>/Data/<chunk>.parquet file(s), where chunk=000000.parquet, 000001.parquet, and so forth.

  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "prefix" : "my_export"
    2. "prefix" : "my_export/2021-04-05/"

chunkSize

  • Purpose: Specifies the maximum size of a "chunk" of table data to be stored at the sink. During migration, a table is split into chunkSize chunks and each chunk is written as a separate file to the sink. When the source data being migrated exceeds this size, a new file is created.

    If not specified, defaults to 32MB. The valid value is an integer between 1 to 1024.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "chunkSize" : 40

compression

  • Purpose: Specifies the compression type to use to compress the Parquet data. Valid values are SNAPPY, GZIP, and NONE.

    If not specified, it defaults to SNAPPY.

  • Data Type: string
  • Mandatory (Y/N): N
  • Example: "compression" : "GZIP"

parquetOptions

  • Purpose: Species the options to select Parquet logical types for NoSQL ENUM, JSON, and UUID columns.

    If you do not specify this parameter, the NoSQL Database Migrator writes the data of ENUM, JSON, and UUID columns as String.

  • Data Type: object
  • Mandatory (Y/N): N

parquetOptions.useLogicalJson

  • Purpose: Specifies whether or not to write NoSQL JSON column data as Parquet logical JSON type. For more information see Parquet Logical Type Definitions.

    If not specified or set to false, NoSQL Database Migrator writes the NoSQL JSON column data as String.

  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useLogicalJson" : true

parquetOptions.useLogicalEnum

  • Purpose: Specifies whether or not to write NoSQL ENUM column data as Parquet logical ENUM type. For more information see Parquet Logical Type Definitions.

    If not specified or set to false, NoSQL Database Migrator writes the NoSQL ENUM column data as String.

  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useLogicalEnum" : true

parquetOptions.useLogicalUUID

  • Purpose: Specifies whether or not to write NoSQL UUID column data as Parquet logical UUID type. For more information see Parquet Logical Type Definitions.

    If not specified or set to false, NoSQL Database Migrator writes the NoSQL UUID column data as String.

  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useLogicalUUID" : true

parquetOptions.truncateDoubleSpecials

  • Purpose: Specifies whether or not to truncate the double +Infinity, -Infinity, and Nan values.

    By default, it is set to false. If set to true,
    • Positive_Infinity is truncated to Double.MAX_VALUE.
    • NEGATIVE_INFINITY is truncated to -Double.MAX_VALUE.
    • NaN is truncated to 9.9999999999999990E307.
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "truncateDoubleSpecials" : true

credentials

  • Purpose: Absolute path to a file containing OCI credentials.

    If not specified, it defaults to $HOME/.oci/config

    See Example Configuration for an example of the credentials file.

    Note:

    Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"

credentialsProfile

  • Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

    If you do not specify this value, it defaults to the DEFAULT profile.

    Note:

    This parameter is valid ONLY if the credentials parameter is specified.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile": "ADMIN_USER"

useInstancePrincipal

  • Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security.

    If not specified, it defaults to false.

    Note:

    • It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
    • Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useInstancePrincipal" : true

Oracle NoSQL Database

The configuration file format for Oracle NoSQL Database as a sink of NoSQL Database Migrator is shown below.

Configuration Template

"sink" : {
  "type": "nosqldb",
  "table" : "<fully qualified table name>",
  "schemaInfo" : {
    "schemaPath" : "</path/to/a/schema/file>",
    "defaultSchema" : <true|false>,
    "useSourceSchema" : <true|false>,
    "DDBPartitionKey" : <"name:type">,
    "DDBSortKey" : "<name:type>"
  },
  "overwrite" : <true|false>,
  "storeName" : "<store name>",
  "helperHosts" : ["hostname1:port1","hostname2:port2,..."],
  "security" : "</path/to/store/credentials/file>",
  "requestTimeoutMs" : <timeout in milli seconds>,
  "includeTTL": <true|false>,
  "ttlRelativeDate": "<date-to-use in UTC>"
}

type

  • Purpose: Identifies the sink type.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "type" : "nosqldb"

table

  • Purpose: Fully qualified table name from which to migrate the data.

    Format: [namespace_name:]<table_name>

    If the table is in the DEFAULT namespace, you can omit the namespace_name. The table must exist in the store during the migration, and its schema must match with the source data.

    If the table is not available in the sink, you can use the schemaInfo parameter to instruct the NoSQL Database Migrator to create the table also in the sink.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • With the DEFAULT namespace "table" :"mytable"

    • With a non-default namespace "table" : "mynamespace:mytable"

schemainfo

  • Purpose: Specifies the schema for the data being migrated. If this is not specified, the assumes that the table already exists in the sink's store.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaPath

  • Purpose: Specifies the absolute path to a file containing DDL statements for the NoSQL table.

    The NoSQL Database Migrator executes the DDL commands listed in this file before migrating the data.

    The NoSQL Database Migrator does not support more than one DDL statement per line in the schemaPath file.

  • Data Type: string

  • Mandatory: Y, only when the schemaInfo.defaultSchema parameter is set to No.

schemaInfo.defaultSchema

  • Purpose: Setting this parameter to true instructs the NoSQL Database Migrator to create a table with default schema. The default schema is defined by the migrator itself. For more information about default schema definitions, see Default Schema.

  • Data Type: boolean

  • Mandatory: N

    Note:

    defaultSchema and schemaPath are mutually exclusive
  • Example:
    • With Default Schema:
      "schemaInfo" : {
            "defaultSchema" : true
          }
    • With a pre-defined schema:
      "schemaInfo" : {
            "schemaPath" : "<complete/path/to/the/schema/definition/file>"
          }

schemaInfo.useSourceSchema

  • Purpose: Specifies whether or not the sink uses the table schema definition provided by the source when migrating NoSQL tables.

  • Data Type: boolean

  • Mandatory (Y/N): N

    Note:

    defaultSchema, schemaPath, and useSourceSchema parameters are mutually exclusive. Specify ONLY one of these parameters.
  • Example:
    • With Default Schema:
      "schemaInfo" : {
            "defaultSchema" : true
          }
    • With a pre-defined schema:
      "schemaInfo" : {
            "schemaPath" : "<complete/path/to/the/schema/definition/file>"
          }
    • With source schema:
      "schemaInfo" : {
            "useSourceSchema" : true
          }

schemaInfo.DDBPartitionKey

  • Purpose: Specifies the DynamoDB partition key and the corresponding Oracle NoSQL Database type to be used in the sink Oracle NoSQL Database table. This key will be used as a NoSQL DB table shard key. This is applicable only when defaultSchema is set to true and the source format is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.
  • Mandatory: Yes if defaultSchema is true and the source is dynamodb_json.
  • Example: "DDBPartitionKey" : "PersonID:INTEGER"

    Note:

    If the partition key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

schemaInfo.DDBSortKey

  • Purpose: Specifies the DynamoDB sort key and its corresponding Oracle NoSQL Database type to be used in the target Oracle NoSQL Database table. If the importing DynamoDB table does not have a sort key this should not be set. This key will be used as a non-shard portion of the primary key in the NoSQL DB table. This is applicable only when defaultSchema is set to true and the source is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.
  • Mandatory: No
  • Example:"DDBSortKey" : "Skey:STRING"

    Note:

    If the sort key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

overwrite

  • Purpose: Indicates the behavior of NoSQL Database Migrator when the record being migrated from the source is already present in the sink.

    If the value is set to false, when migrating tables the NoSQL Database Migrator skips those records for which the same primary key already exists in the sink.

    If the value is set to true, when migrating tables the NoSQL Database Migrator overwrites those records for which the same primary key already exists in the sink.

    If not specified, it defaults to true.

  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "overwrite" : false

storeName

  • Purpose: Name of the Oracle NoSQL Database store.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "storeName" : "kvstore"

helperHosts

  • Purpose: A list of host and registry port pairs in the hostname:port format. Delimit each item in the list using a comma. You must specify at least one helper host.

  • Data Type: array of strings
  • Mandatory (Y/N): Y
  • Example: "helperHosts" : ["localhost:5000","localhost:6000"]

security

  • Purpose:

    If your store is a secure store, provide the absolute path to the security login file that contains your store credentials. See Configuring Security with Remote Access in Administrator's Guide to know more about the security login file.

    You can use either password file based authentication or wallet based authentication. However, the wallet based authentication is supported only in the Enterprise Edition (EE) of Oracle NoSQL Database. For more information on wallet-based authentication, see Source and Sink Security.

  • Data Type: string
  • Mandatory (Y/N): Y for a secure store
  • Example:

    "security" : "/home/user/client.credentials"

    Example security file content for password file based authentication:

    oracle.kv.password.noPrompt=true
    oracle.kv.auth.username=admin
    oracle.kv.auth.pwdfile.file=/home/nosql/login.passwd
    oracle.kv.transport=ssl
    oracle.kv.ssl.trustStore=/home/nosql/client.trust
    oracle.kv.ssl.protocols=TLSv1.2
    oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

    Example security file content for wallet based authentication:

    oracle.kv.password.noPrompt=true
    oracle.kv.auth.username=admin
    oracle.kv.auth.wallet.dir=/home/nosql/login.wallet
    oracle.kv.transport=ssl
    oracle.kv.ssl.trustStore=/home/nosql/client.trust
    oracle.kv.ssl.protocols=TLSv1.2
    oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

requestTimeoutMs

  • Purpose: Specifies the time to wait for each write operation in the sink to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "requestTimeoutMs" : 5000

includeTTL

  • Purpose: Specifies whether or not to include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.

    If you do not specify this parameter, it defaults to false. In that case, the NoSQL Database Migrator does not include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.

    If set to true, the NoSQL Database Migrator tool performs the following checks on the TTL metadata when importing a table row:
    • If you import a row that does not have _metadata definition, the NoSQL Database Migrator tool sets the TTL to 0, which means the row never expires.
    • If you import a row that has _metadata definition, the NoSQL Database Migrator tool compares the TTL value against a Reference Time when a row gets imported. If the row has already expired relative to the Reference Time, then it is skipped. If the row has not expired, then it is imported along with the TTL value. By default, the Reference Time of import operation is the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running. But you can also set a custom Reference Time using the ttlRelativeDate configuration parameter if you want to extend the expiration time and import rows that would otherwise expire immediately.
      The formula to calculate the expiration time of a row is as follows:
      expiration = (TTL value of source row in milliseconds - Reference Time in milliseconds)
      if (expiration <= 0) then it indicates that row has expired.

      Note:

      Since Oracle NoSQL TTL boundaries are in hours and days, in some cases, the TTL of the imported row might get adjusted to the nearest hour or day. For example, consider a row that has expiration value of 1629709200000 (2021-08-23 09:00:00) and Reference Time value is 1629707962582 (2021-08-23 08:39:22). Here, even though the row is not expired relative to the Reference Time when this data gets imported, the new TTL for the row is 1629712800000 (2021-08-23 10:00:00).
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "includeTTL" : true

ttlRelativeDate

  • Purpose: Specify a UTC date in the YYYY-MM-DD hh:mm:ss format used to set the TTL expiry of table rows during importing into the Oracle NoSQL Database.

    If a table row in the data you are exporting has expired, you can set the ttlRelativeDate parameter to a date before the expiration time of the table row in the exported data.

    If you do not specify this parameter, it defaults to the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running.

  • Data Type: date
  • Mandatory (Y/N): N
  • Example: "ttlRelativeDate" : "2021-01-03 04:31:17"

    Let us consider a scenario where table rows expire after seven days from 1-Jan-2021. After exporting this table, on 7-Jan-2021, you run into an issue with your table and decide to import the data. The table rows are going to expire in one day (data expiration date minus the default value of ttlRelativedate configuration parameter, which is the current date).But if you want to extend the expiration date of table rows to five days instead of one day, use the ttlRelativeDate parameter and choose an earlier date. Therefore, in this scenario if you want to extend expiration time of the table rows by five days, set the value of ttlRelativeDate configuration parameters to 3-Jan-2021, which is used as Reference Time when table rows get imported.

Oracle NoSQL Database Cloud Service

The configuration file format for Oracle NoSQL Database Cloud Service as a sink of NoSQL Database Migrator is shown below.

Configuration Template

"sink" : {
    "type" : "nosqldb_cloud",
    "endpoint" : "<Oracle NoSQL Cloud Service Endpoint>",
    "table" : "<table name>",
    "compartment" : "<OCI compartment name or id>",
    "schemaInfo" : {
      "schemaPath" : "</path/to/a/schema/file>",
      "defaultSchema" : <true|false>,
      "useSourceSchema" : <true|false>,
      "DDBPartitionKey" : <"name:type">,
      "DDBSortKey" : "<name:type>",
      "readUnits" : <table read units>,
      "writeUnits" : <table write units>,
      "storageSize" : <storage size in GB>
    },
    "credentials" : "</path/to/oci/credential/file>",
    "credentialsProfile" : "<oci credentials profile name>",
    "writeUnitsPercent" : <table writeunits percent>,
    "requestTimeoutMs" : <timeout in milli seconds>,
    "useInstancePrincipal" : <true|false>,
    "overwrite" : <true|false>,
    "includeTTL": <true|false>,
    "ttlRelativeDate" : "<date-to-use in UTC>"  
}

type

  • Purpose: Identifies the sink type.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "type" : "nosqldb_cloud"

endpoint

  • Purpose: Specifies the Service Endpoint of the Oracle NoSQL Database Cloud Service.

    You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://nosql.us-ashburn-1.oci.oraclecloud.com/"

table

  • Purpose: Name of the table to which to migrate the data.

    You must ensure that this table exists in your Oracle NoSQL Database Cloud Service. Otherwise, you have to use the schemaInfo object in the sink configuration to instruct the NoSQL Database Migrator to create the table.

    The schema of this table must match the source data.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "table" :"myTable"

compartment

  • Purpose: Specifies the name or OCID of the compartment in which the table resides.

    If you do not provide any value, it defaults to the root compartment.

    You can find your compartment's OCID from the Compartment Explorer window under Governance in the OCI Cloud Console.

  • Data Type: string
  • Mandatory (Y/N): Y if the table is not in the root compartment of the tenancy OR when the useInstancePrincipal parameter is set to true.

    Note:

    If the useInstancePrincipal parameter is set to true, the compartment must specify the compartment OCID and not the name.
  • Example:
    • Compartment name

      "compartment" : "mycompartment"

    • Compartment name qualified with its parent compartment

      "compartment" : "parent.childcompartment"

    • No value provided. Defaults to the root compartment.

      "compartment": ""

    • Compartment OCID

      "compartment" : "ocid1.tenancy.oc1...4ksd"

schemaInfo

  • Purpose: Specifies the schema for the data being migrated.

    If you do not specify this parameter, the NoSQL Database Migrator assumes that the table already exists in your Oracle NoSQL Database Cloud Service.

    If this parameter is not specified and the table does not exist in the sink, the migration fails.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaPath

  • Purpose: Specifies the absolute path to a file containing DDL statements for the NoSQL table.

    The NoSQL Database Migrator executes the DDL commands listed in this file before migrating the data.

    The NoSQL Database Migrator does not support more than one DDL statement per line in the schemaPath file.

  • Data Type: string

  • Mandatory: Y, only when the schemaInfo.defaultSchema parameter is set to No.

schemaInfo.defaultSchema

  • Purpose: Setting this parameter to Yes instructs the NoSQL Database Migrator to create a table with default schema. The default schema is defined by the migrator itself. For more information about default schema definitions, see Default Schema.

  • Data Type: boolean

  • Mandatory: Y, only when the schemaInfo.defaultSchema parameter is set to No.

    Note:

    defaultSchema and schemaPath are mutually exclusive

schemaInfo.useSourceSchema

  • Purpose: Specifies whether or not the sink uses the table schema definition provided by the source when migrating NoSQL tables.

  • Data Type: boolean

  • Mandatory (Y/N): N

    Note:

    defaultSchema, schemaPath, and useSourceSchema parameters are mutually exclusive. Specify ONLY one of these parameters.
  • Example:
    • With Default Schema:
      "schemaInfo" : {
            "defaultSchema" : true,
            "readUnits" : 100,
            "writeUnits" : 60,
            "storageSize" : 1 
          }
    • With a pre-defined schema:
      "schemaInfo" : {
            "schemaPath" : "<complete/path/to/the/schema/definition/file>",
            "readUnits" : 100,
            "writeUnits" : 100,
            "storageSize" : 1
          }
    • With source schema:
      "schemaInfo" : {
            "useSourceSchema" : true,
            "readUnits" : 100,
            "writeUnits" : 60,
            "storageSize" : 1
          }

schemaInfo.DDBPartitionKey

  • Purpose: Specifies the DynamoDB partition key and the corresponding Oracle NoSQL Database type to be used in the sink Oracle NoSQL Database table. This key will be used as a NoSQL DB table shard key. This is applicable only when defaultSchema is set to true and the source format is dynamodb_json. See Mapping of DynamoDB types to Oracle NoSQL types for more details.
  • Mandatory: Yes if defaultSchema is true and the source is dynamodb_json.
  • Example: "DDBPartitionKey" : "PersonID:INTEGER"

    Note:

    If the partition key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

schemaInfo.DDBSortKey

  • Purpose: Specifies the DynamoDB sort key and its corresponding Oracle NoSQL Database type to be used in the target Oracle NoSQL Database table. If the importing DynamoDB table does not have a sort key this should not be set. This key will be used as a non-shard portion of the primary key in the NoSQL DB table. This is applicable only when defaultSchema is set to true and the source is dynamodb_json.See Mapping of DynamoDB types to Oracle NoSQL types for more details.
  • Mandatory: No
  • Example:"DDBSortKey" : "Skey:STRING"

    Note:

    If the sort key contains dash(-) or dot(.), Migrator will replace it with underscore(_) as NoSQL column name does not support dot and dash.

schemaInfo.readUnits

  • Purpose: Specifies the read throughput of the new table.
  • Data Type: integer

  • Mandatory: Y

schemaInfo.writeUnits

  • Purpose: Specifies the write throughput of the new table.
  • Data Type: integer

  • Mandatory: Y

schemaInfo.storageSize

  • Purpose: Specifies the storage size of the new table in GB
  • Data Type: integer

  • Mandatory: Y

  • Example:
    • With schemaPath

      "schemaInfo" : { 
        "schemaPath" : "</path/to/a/schema/file>",
        "readUnits" : 500,
        "writeUnits" : 1000,
        "storageSize" : 5 }
    • With defaultSchema

      "schemaInfo" : {
        "defaultSchema" :Yes,
        "readUnits" : 500,   
        "writeUnits" : 1000,   
        "storageSize" : 5  
      }

credentials

  • Purpose: Absolute path to a file containing OCI credentials.

    If not specified, it defaults to $HOME/.oci/config

    See Example Configuration for an example of the credentials file.

    Note:

    Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"

credentialsProfile

  • Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service.

    If you do not specify this value, it defaults to the DEFAULT profile.

    Note:

    This parameter is valid ONLY if the credentials parameter is specified.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile": "ADMIN_USER"

writeUnitsPercent

  • Purpose: Specifies the Percentage of table write units to be used during the migration activity.

    The default value is 90. The valid range is any integer between 1 to 100.

    Note:

    The time required for the data migration is directly proportional to the writeUnitsPercent value.

    See Troubleshooting the Oracle NoSQL Database Migrator to learn how to use this attribute to improve the data migration speed.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "writeUnitsPercent" : 90

requestTimeoutMs

  • Purpose: Specifies the time to wait for each write operation in the sink to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "requestTimeoutMs" : 5000

useInstancePrincipal

  • Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security

    If not specified, it defaults to false.

    Note:

    • It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
    • Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "useInstancePrincipal" : true

overwrite

  • Purpose: Indicates the behavior of NoSQL Database Migrator when the record being migrated from the source is already present in the sink.

    If the value is set to false, when migrating tables the NoSQL Database Migrator skips those records for which the same primary key already exists in the sink.

    If the value is set to true, when migrating tables the NoSQL Database Migrator overwrites those records for which the same primary key already exists in the sink.

    If not specified, it defaults to true.

  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "overwrite" : false

includeTTL

  • Purpose: Specifies whether or not to include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.

    If you do not specify this parameter, it defaults to false. In that case, the NoSQL Database Migrator does not include TTL metadata for table rows provided by the source when importing Oracle NoSQL Database tables.

    If set to true, the NoSQL Database Migrator tool performs the following checks on the TTL metadata when importing a table row:
    • If you import a row that does not have _metadata definition, the NoSQL Database Migrator tool sets the TTL to 0, which means the row never expires.
    • If you import a row that has _metadata definition, the NoSQL Database Migrator tool compares the TTL value against a Reference Time when a row gets imported. If the row has already expired relative to the Reference Time, then it is skipped. If the row has not expired, then it is imported along with the TTL value. By default, the Reference Time of import operation is the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running. But you can also set a custom Reference Time using the ttlRelativeDate configuration parameter if you want to extend the expiration time and import rows that would otherwise expire immediately.
      The formula to calculate the expiration time of a row is as follows:
      expiration = (TTL value of source row in milliseconds - Reference Time in milliseconds)
      if (expiration <= 0) then it indicates that row has expired.

      Note:

      Since Oracle NoSQL TTL boundaries are in hours and days, in some cases, the TTL of the imported row might get adjusted to the nearest hour or day. For example, consider a row that has expiration value of 1629709200000 (2021-08-23 09:00:00) and Reference Time value is 1629707962582 (2021-08-23 08:39:22). Here, even though the row is not expired relative to the Reference Time when this data gets imported, the new TTL for the row is 1629712800000 (2021-08-23 10:00:00).
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "includeTTL" : true

ttlRelativeDate

  • Purpose: Specify a UTC date in the YYYY-MM-DD hh:mm:ss format used to set the TTL expiry of table rows during importing into the Oracle NoSQL Database.

    If a table row in the data you are exporting has expired, you can set the ttlRelativeDate parameter to a date before the expiration time of the table row in the exported data.

    If you do not specify this parameter, it defaults to the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running.

  • Data Type: date
  • Mandatory (Y/N): N
  • Example: "ttlRelativeDate" : "2021-01-03 04:31:17"

    Let us consider a scenario where table rows expire after seven days from 1-Jan-2021. After exporting this table, on 7-Jan-2021, you run into an issue with your table and decide to import the data. The table rows are going to expire in one day (data expiration date minus the default value of ttlRelativedate configuration parameter, which is the current date).But if you want to extend the expiration date of table rows to five days instead of one day, use the ttlRelativeDate parameter and choose an earlier date. Therefore, in this scenario if you want to extend expiration time of the table rows by five days, set the value of ttlRelativeDate configuration parameters to 3-Jan-2021, which is used as Reference Time when table rows get imported.

Transformation Configuration Templates

This topic explains the configuration parameters for the different transformations supported by the Oracle NoSQL Database Migrator.

Oracle NoSQL Database Migrator lets you modify the data, that is, add data transformations as part of the migration activity. You can define multiple transformations in a single migration. In such a case, the order of transformations is vital because the source data undergoes each transformation in the given order. The output of one transformation becomes the input to the next one in the migrator pipeline.

The different transformations supported by the NoSQL Data Migrator are:

Table 9-7 Transformations

Transformation Config Attribute You can use this transformation to ...
ignoreFields Ignore the identified columns from the source row before writing to the sink.
includeFields Include the identified columns from the source row before writing to the sink.
renameFields Rename the identified columns from the source row before writing to the sink.
aggregateFields Aggregate multiple columns from the source into a single column in the sink. As part of this transformation, you can also identify the columns that you want to exclude in the aggregation. Those fields will be skipped from the aggregated column.

You can find the configuration template for each supported transformation below.

ignoreFields

The configuration file format for the ignoreFields transformation is shown below.

Configuration Template

"transforms" : {
  "ignoreFields" : ["<field1>","<field2>",...]
}

Transformation Parameter

ignoreFields

  • Purpose: An array of the column names to be ignored from the source records.

    Note:

    You can supply only top-level fields. Transformations can not be applied on the data in the nested fields.
  • Data Type: array of strings
  • Mandatory (Y/N): Y
  • Example: To ignore the columns named "name" and "address" from the source record:

    "ignoreFields" : ["name","address"]

includeFields

The configuration file format for the includeFields transformation is shown below.

Configuration Template

"transforms" : {
  "includeFields" : ["<field1>","<field2>",...]
}

Transformation Parameter

includeFields

  • Purpose: An array of the column names to be included from the source records. It only includes the fields specified in the array, the rest of the fields are ignored.

    Note:

    The NoSQL Database Migrator tool throws an error if you specify an empty array. Additionally, you can specify only the top-level fields. The NoSQL Database Migrator tool does not apply transformations to the data in the nested fields.
  • Data Type: array of strings
  • Mandatory (Y/N): Y
  • Example: To ignore the columns named "age" and "gender" from the source record:

    "includeFields" : ["age","gender"]

renameFields

The configuration file format for the renameFields transformation is shown below.

Configuration Template

"transforms" : {
  "renameFields" : {
    "<old_name>" : "<new_name>",
    "<old_name>" : "<new_name>,"
    .....
  }
}

Transformation Parameter

renameFields

  • Purpose: Key-Value pairs of the old and new names of the columns to be renamed.

    Note:

    You can supply only top-level fields. Transformations can not be applied on the data in the nested fields.
  • Data Type: JSON object
  • Mandatory (Y/N): Y
  • Example: To rename the column named "residence" to "address" and the column named "_id" to "id":

    "renameFields" : { "residence" : "address", "_id" : "id" }

aggregateFields

The configuration file format for the aggregateFields transformation is shown below.

Configuration Template

"transforms" : {
  "aggregateFields" : {
    "fieldName" : "name of the new aggregate field",
    "skipFields" : ["<field1>","<field2">,...]
  }
}

Transformation Parameter

aggregateFields

  • Purpose: Name of the aggregated field in the sink.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: If the given record is:

    {
      "id" : 100,
      "name" : "john",
      "address" : "USA",
      "age" : 20
    }

    If the aggregate transformation is:

    "aggregateFields" : {
      "fieldName" : "document",
      "skipFields" : "id"
    }

    The aggregated column in the sink looks like:

    {
      "id": 100,
      "document": {
        "name": "john",
        "address": "USA",
        "age": 20
      }
    }

Mapping of DynamoDB types to Oracle NoSQL types

The table below shows the mapping of DynamoDB types to Oracle NoSQL types.

Table 9-8 Mapping DynamoDB type to Oracle NoSQL type

# DynamoDB type JSON type for NoSQL JSON column Oracle NoSQL type
1 String (S) JSON String STRING
2 Number Type (N) JSON Number INTEGER/LONG/FLOAT/DOUBLE/NUMBER

Note:

DynamoDB Supports only one data type for Numbers and can have up to 38 digits of precision,on contrast Oracle NoSQL supports many types to choose from based on the range and precision of the data.You can select the appropriate Number type that fits the range of your input data. If you are not sure of the nature of the data, NoSQL NUMBER type can be used.
3 Boolean (BOOL) JSON Boolean BOOLEAN
4 Binary type (B) - Byte buffer BASE-64 encoded JSON String BINARY
5 NULL JSON null NULL
6 String Set (SS) JSON Array of Strings ARRAY(STRING)
7 Number Set (NS) JSON Array of Numbers ARRAY(INTEGER/LONG/FLOAT/DOUBLE/NUMBER)

Note:

DynamoDB Supports only one data type for Numbers and can have up to 38 digits of precision,on contrast Oracle NoSQL supports many types to choose from based on the range and precision of the data.You can select the appropriate Number type that fits the range of your input data. If you are not sure of the nature of the data, NoSQL NUMBER type can be used.
8 Binary Set (BS) JSON Array of Base-64 encoded Strings ARRAY(BINARY)
9 LIST (L) Array of JSON ARRAY(JSON)
10 MAP (M) JSON Object JSON
11 PARTITION KEY NA PRIMARY KEY and SHARD KEY

Note:

Partition key in DynamoDB has a limit of 2048 bytes but Oracle NoSQL Cloud Service has a limit of 64 bytes for the Primary key/Shard key.
12 SORT KEY NA PRIMARY KEY

Note:

Sort key in DynamoDB has a limit of 1024 bytes but Oracle NoSQL Cloud Service has a limit of 64 bytes for the Primary key.
13 Attribute names with dash and dot JSON field names with a underscore Column names with underscore

Note:

Attribute names in DynamoDB can be 64KB long but Oracle NoSQL Cloud service column names have a limit of 64 characters.

Oracle NoSQL to Parquet Data Type Mapping

Describes the mapping of Oracle NoSQL data types to Parquet data types.

NoSQL Type Parquet Type
BOOLEAN BOOLEAN
INTEGER INT32
LONG INT64
FLOAT DOUBLE
DOUBLE DOUBLE
BINARY BINARY
FIXED_BINARY BINARY
STRING BINARY(STRING)
ENUM BINARY(STRING)

or

BINARY(ENUM), if the logical ENUM is configured

UUID BINARY(STRING)

or

FIXED_BINARY(16), if the logical UUID is configured

TIMESTAMP(p) INT64(TIMESTAMP(p))
NUMBER DOUBLE

Note:

When the NoSQL Number type is converted to Parquet Double type, there may be some loss of precision in case the value cannot be represented in Double. If the number is too big to represent as Double, it is converted to Double.NEGATIVE_INFINITY or Double.POSITIVE_INFINITY.
field_name ARRAY(T)
group field_name(LIST) {
  repeated group list {
      required T element
  }
}
field_name MAP(T)
group field_name (MAP) {
    repeated group key_value (MAP_KEY_VALUE) {
       required binary key (STRING);
        required T value;
    }
}
field_name RECORD(K₁ T₁ N₁, Kٖ₂ T₂ N₂, ....)

where:

K = Key name

T = Type

N = Nullable or not

group field_name {
    ni == true ? optional Ti ki : required Ti ki   
}
JSON BINARY(STRING)

or

BINARY(JSON), if logical JSON is configured

Mapping of DynamoDB table to Oracle NoSQL table

In DynamoDB, a table is a collection of items, and each item is a collection of attributes. Each item in the table has a unique identifier, or a primary key. Other than the primary key, the table is schema-less. Each item can have its own distinct attributes.

DynamoDB supports two different kinds of primary keys:
  • Partition key – A simple primary key, composed of one attribute known as the partition key. DynamoDB uses the partition key's value as input to an internal hash function. The output from the hash function determines the partition in which the item will be stored.
  • Partition key and sort key – As a composite primary key, this type of key is composed of two attributes. The first attribute is the partition key, and the second attribute is the sort key. DynamoDB uses the partition key value as input to an internal hash function. The output from the hash function determines the partition in which the item will be stored. All items with the same partition key value are stored together, in sorted order by sort key value.

In contrast, Oracle NoSQL tables support flexible data models with both schema and schema-less design.

There are two different ways of modelling a DynamoDB table:
  1. Modeling DynamoDB table as a JSON document(Recommended): In this modeling, you map all the attributes of the Dynamo DB tables into a JSON column of the NoSQL table except partition key and sort key. You will model partition key and sort key as the Primary Key columns of the NoSQL table. You will use AggregateFields transform in order to aggregate non-primary key data into a JSON column.

    Note:

    The Migrator provides a user-friendly configuration defaultSchema to automatically create a schema-less DDL table which also aggregates attributes into a JSON column.
  2. Modeling DynamoDB table as fixed columns in NoSQL table: In this modeling, for each attribute of the DynamoDB table, you will create a column in the NoSQL table as specified in the Mapping of DynamoDB types to Oracle NoSQL types. You will model partition key and sort key attributes as Primary key(s). This should be used only when you are certain that importing DynamoDB table schema is fixed and each item has values for the most of the attributes. If DynamoDB items do not have common attributes, this can result in lot of NoSQL columns with empty values.

    Note:

    We highly recommend using schema-less tables when migrating data from DynamoDB to Oracle NoSQL Database due to the nature of DynamoDB tables being schema-less. This is especially for large tables where the content of each record may not be uniform across the table.

Troubleshooting the Oracle NoSQL Database Migrator

Learn about the general challenges that you may face while using the , and how to resolve them.

Migration has failed. How can I resolve this?

A failure of the data migration can be because of multiple underlying reasons. The important causes are listed below:

Table 9-9 Migration Failure Causes

Error Message Meaning Resolution
Failed to connect to Oracle NoSQL Database The migrator could not establish a connection with the NoSQL Database.
  • Check if the values of the storeName and helperHosts attributes in the configuration JSON file are valid and that the hosts are reachable.
  • For a secured store, verify if the security file is valid with correct user name and password values.
Failed to connect to Oracle NoSQL Database Cloud Service The migrator could not establish a connection with the Oracle NoSQL Database Cloud Service.
  • Verify if the endpoint URL or region name specified in the configuration JSON file is correct.
  • Check if the OCI credentials file is available in the path specified in the configuration JSON file.
  • Ensure that the OCI credentials provided in the OCI credentials are valid.
Table not found The table identified for the migration could not be located by the NoSQL Database Migrator.

For the Source:

  • Verify if the table is present in the source database.
  • Ensure that the table is qualified with its namespace in the configuration JSON file, if the table is created in a non-default namespace.
  • Verify if you have the required read/write authorization to access the table.
  • If the source is Oracle NoSQL Database Cloud Service, verify if the valid compartment name is specified in the configuration JSON file, and ensure that you have the required authorization to access the table.

For the Sink:

  • Verify if the table is present in the Sink. If it does not exist, you must either create the table manually or use the schemaInfo config to create it through the migration.
DDL Execution failed The DDL commands provided in the input schema definition file is invalid.
  • Check the syntax of the DDL commands in the schemaPath file.
  • Ensure that there is only one DDL statement per line in the schemaPath file.
failed to write record to the sink table with java.lang.IllegalArgumentException The input record is not matching with the table schema of the sink.
  • Check if the data types and column names specified in the target sink table are matching with sink table schema.
  • If you applied any transformation, check if the transformed records are matching with the sink table schema.
Request timeout The source or sink's operation did not complete within the expected time.
  • Verify the network connection.
  • Check if the NoSQL Database is up and running.
  • Try to increase requestTimeout value in the configuration JSON file.

What should I consider before restarting a failed migration?

When a data migration task fails, the sink will be at an intermediate state containing the imported data until the point of failure. You can identify the error and failure details from the logs and restart the migration after diagnosing and correcting the error. A restarted migration starts over, processing all data from the beginning. There is no way to checkpoint and restart the migration from the point of failure. Therefore, NoSQL Database Migrator overwrites any record that was migrated to the sink already.

Migration is too slow. How can I speed it up?

The time taken for the data migration depends on multiple factors such as volume of data being migrated, network speed, current load on the database. In case of a cloud service, the speed of migration also depends on the read throughput and the write throughput provisioned. So, to improve the migration speed, you can:
  • Try to reduce the current workload on your Oracle NoSQL Database while migrating the data.
  • Ensure that the machine that is running the migration, source, and sink all are located in the same data center and the network latencies are minimal.
  • In case of Oracle NoSQL Database Cloud Service, provision high read/write throughput and verify if the storage allocated for table is sufficient or not. If the NoSQL Database Migrator is not creating the table, you can increase the write throughput. If the migrator is creating the table, consider specifying a higher value for the schemaInfo.writeUnits parameter in the sink configuration. Once the data migration completes, you can lower this value. Be aware of daily limits on throughput changes. see Cloud Limits and Sink Configuration Templates.

I have a long running migration involving huge datasets. How can I track the progress of the migration?

You can enable additional logging to track the progress of a long-running migration. To control the logging behavior of Oracle NoSQL Database Migrator, you must set the desired level of logging in the logging.properties file. This file is provided with the NoSQL Database Migrator package and available in the directory where the Oracle NoSQL Database Migrator was unpacked. The different levels of logging are OFF, SEVERE, WARNING, INFO, FINE, and ALL in the order of increasing verbosity. Setting the log level to OFF turns off all the logging information, whereas setting the log level to ALL provides the full log information. The default log level is WARNING. All the logging output is configured to go to the console by default. You can see comments in the logging.properties file to know about each log level.