1. Administrator's Guide
  2. Tools and Utilities
  3. Using Oracle NoSQL Database Migrator
  4. Sources and Sinks
  5. Source Configuration Templates

Source Configuration Templates

Learn about the source configuration file formats for each valid source and the purpose of each configuration parameter.

For the configuration file template, see Configuration File in Terminology used with Oracle NoSQL Database Migrator.

For details on valid sink formats for each of the source, see Sink Configuration Templates.

Topics

The following topics describe the source configuration templates referred by Oracle NoSQL Database Migrator to copy the data from the given source to a valid sink.

JSON File Source

The configuration file format for JSON file as a source of NoSQL Database Migrator is shown below.

You can migrate a JSON source file by specifying the file path or a directory in the source configuration template.

A sample JSON source file is as follows:
{"id":6,"val_json":{"array":["q","r","s"],"date":"2023-02-04T02:38:57.520Z","nestarray":[[1,2,3],[10,20,30]],"nested":{"arrayofobjects":[{"datefield":"2023-03-04T02:38:57.520Z","numfield":30,"strfield":"foo54"},{"datefield":"2023-02-04T02:38:57.520Z","numfield":56,"strfield":"bar23"}],"nestNum":10,"nestString":"bar"},"num":1,"string":"foo"}}
{"id":3,"val_json":{"array":["g","h","i"],"date":"2023-02-02T02:38:57.520Z","nestarray":[[1,2,3],[10,20,30]],"nested":{"arrayofobjects":[{"datefield":"2023-02-02T02:38:57.520Z","numfield":28,"strfield":"foo3"},{"datefield":"2023-02-02T02:38:57.520Z","numfield":38,"strfield":"bar"}],"nestNum":10,"nestString":"bar"},"num":1,"string":"foo"}}

Source Configuration Template

"source": {
  "type": "file",
  "format": "json",
  "dataPath": "<path/to/JSON/[file|dir]>",
  "schemaInfo": {
    "schemaPath": "<path/to/schema/file>"
  }
},

Source Parameters

Common Configuration Parameters

  • type

    Use "type" : "file"

  • format

    Use "format" : "json"

Unique Configuration Parameters

dataPath

  • Purpose: Specifies the absolute path to a file or directory containing the JSON data for migration.

    You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:

    • Specifying a JSON file

      "dataPath" : "/home/user/sample.json"

    • Specifying a directory

      "dataPath" : "/home/user"

schemaInfo

  • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaPath

  • Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: 
    "schemaInfo": {
  "schemaPath": "<path to the schema file>"
}

JSON File in OCI Object Storage Bucket

The configuration file format for JSON file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below.

You can migrate a JSON file in the OCI Object Storage bucket by specifying the name of the bucket in the source configuration template.

A sample JSON source file in the OCI Object Storage bucket is as follows:
{"id":6,"val_json":{"array":["q","r","s"],"date":"2023-02-04T02:38:57.520Z","nestarray":[[1,2,3],[10,20,30]],"nested":{"arrayofobjects":[{"datefield":"2023-03-04T02:38:57.520Z","numfield":30,"strfield":"foo54"},{"datefield":"2023-02-04T02:38:57.520Z","numfield":56,"strfield":"bar23"}],"nestNum":10,"nestString":"bar"},"num":1,"string":"foo"}}
{"id":3,"val_json":{"array":["g","h","i"],"date":"2023-02-02T02:38:57.520Z","nestarray":[[1,2,3],[10,20,30]],"nested":{"arrayofobjects":[{"datefield":"2023-02-02T02:38:57.520Z","numfield":28,"strfield":"foo3"},{"datefield":"2023-02-02T02:38:57.520Z","numfield":38,"strfield":"bar"}],"nestNum":10,"nestString":"bar"},"num":1,"string":"foo"}}

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Source Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "schemaInfo" : {
     "schemaObject" : "<object name>"
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
  "useDelegationToken" : <true|false>
}

Source Parameters

Common Configuration Parameters

  • type

    Use "type" : "object_storage_oci"

  • format

    Use "format" : "json"

  • endpoint
    Example:

    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

  • namespace

    Example: "namespace" : "my-namespace"

  • bucket

    Example: "bucket" : "my-bucket"

  • prefix
    Example:
    1. "prefix" : "my_table/Data/000000.json" (migrates only 000000.json)
    2. "prefix" : "my_table/Data" (migrates all the objects with prefix my_table/Data)
  • credentials
    Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"
  • credentialsProfile
    Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile" : "ADMIN_USER"
  • useInstancePrincipal

    Example: "useInstancePrincipal" : true

  • useDelegationToken

    Example: "useDelegationToken" : true

    Note:

    The authentication with delegation token is supported only when the NoSQL Database Migrator is running from a Cloud Shell.

Unique Configuration Parameters

schemaInfo

  • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaObject

  • Purpose: Specifies the name of the object in the bucket where NoSQL table schema definitions for the data being migrated are stored.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: 
    "schemaInfo": {
  "schemaObject": "mytable/Schema/schema.ddl"
},

MongoDB-Formatted JSON File

The configuration file format for MongoDB-formatted JSON File as a source of NoSQL Database Migrator is shown below.

You can migrate a MongoDB exported JSON data by specifying the file or directory in the source configuration template.

MongoDB supports two types of extensions to the JSON format of files, Canonical mode and Relaxed mode. You can supply the MongoDB-formatted JSON file that is generated using the mongoexport tool in either Canonical or Relaxed mode. Both the modes are supported by the NoSQL Database Migrator for migration.

For more information on the MongoDB Extended JSON (v2) file, See mongoexport_formats.

For more information on the generation of MongoDB-formatted JSON file, see mongoexport for more information.

A sample MongoDB-formatted Relaxed mode JSON file is as follows:
{"_id":0,"name":"Aimee Zank","scores":[{"score":1.463179736705023,"type":"exam"},{"score":11.78273309957772,"type":"quiz"},{"score":35.8740349954354,"type":"homework"}]}
{"_id":1,"name":"Aurelia Menendez","scores":[{"score":60.06045071030959,"type":"exam"},{"score":52.79790691903873,"type":"quiz"},{"score":71.76133439165544,"type":"homework"}]}
{"_id":2,"name":"Corliss Zuk","scores":[{"score":67.03077096065002,"type":"exam"},{"score":6.301851677835235,"type":"quiz"},{"score":66.28344683278382,"type":"homework"}]}
{"_id":3,"name":"Bao Ziglar","scores":[{"score":71.64343899778332,"type":"exam"},{"score":24.80221293650313,"type":"quiz"},{"score":42.26147058804812,"type":"homework"}]}
{"_id":4,"name":"Zachary Langlais","scores":[{"score":78.68385091304332,"type":"exam"},{"score":90.2963101368042,"type":"quiz"},{"score":34.41620148042529,"type":"homework"}]}

Source Configuration Template

"source": {
  "type": "file",
  "format": "mongodb_json",
  "dataPath": "</path/to/json/[file|dir]>",
  "schemaInfo": {
    "schemaPath": "</path/to/schema/file>"
  }
}

Source Parameters

Common Configuration Parameters

  • type

    Use "type" : "file"

  • format

    Use "format" : "mongodb_json"

Unique Configuration Parameters

dataPath

  • Purpose: Specifies the absolute path to a file or directory containing the MongoDB exported JSON data for migration.

    You can supply the MongoDB-formatted JSON file that is generated using the mongoexport tool.

    If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported. You must ensure that this data matches with the NoSQL table schema defined at the sink.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:

    • Specifying a MongoDB formatted JSON file

      "dataPath" : "/home/user/sample.json"

    • Specifying a directory

      "dataPath" : "/home/user"

schemaInfo

  • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaPath

  • Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: 
    "schemaInfo" : {
  "schemaPath" : "/home/user/mytable/Schema/schema.ddl"
}

MongoDB-Formatted JSON File in OCI Object Storage bucket

The configuration file format for MongoDB-Formatted JSON file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below.

You can migrate the MongoDB exported JSON data in the OCI Object Storage bucket by specifying the name of the bucket in the source configuration template.

Extract the data from MongoDB using the mongoexport utility and upload it to the OCI Object Storage bucket. See mongoexport for more information. MongoDB supports two types of extensions to the JSON format of files, Canonical mode and Relaxed mode. Both formats are supported in the OCI Object Storage bucket.

A sample MongoDB-formatted Relaxed mode JSON File is as follows:
{"_id":0,"name":"Aimee Zank","scores":[{"score":1.463179736705023,"type":"exam"},{"score":11.78273309957772,"type":"quiz"},{"score":35.8740349954354,"type":"homework"}]}
{"_id":1,"name":"Aurelia Menendez","scores":[{"score":60.06045071030959,"type":"exam"},{"score":52.79790691903873,"type":"quiz"},{"score":71.76133439165544,"type":"homework"}]}
{"_id":2,"name":"Corliss Zuk","scores":[{"score":67.03077096065002,"type":"exam"},{"score":6.301851677835235,"type":"quiz"},{"score":66.28344683278382,"type":"homework"}]}
{"_id":3,"name":"Bao Ziglar","scores":[{"score":71.64343899778332,"type":"exam"},{"score":24.80221293650313,"type":"quiz"},{"score":42.26147058804812,"type":"homework"}]}
{"_id":4,"name":"Zachary Langlais","scores":[{"score":78.68385091304332,"type":"exam"},{"score":90.2963101368042,"type":"quiz"},{"score":34.41620148042529,"type":"homework"}]}

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Source Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "mongodb_json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "schemaInfo" : {
     "schemaObject" : "<object name>"
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

Source Parameters

Common Configuration Parameters

  • type

    Use "type" : "object_storage_oci"

  • format

    Use "format" : "mongodb_json"

  • endpoint
    Example:

    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

  • namespace

    Example: "namespace" : "my-namespace"

  • bucket

    Example: "bucket" : "my-bucket"

  • prefix
    Example:
    1. "prefix" : "mongo_export/Data/table.json" (migrates only table.json)
    2. "prefix" : "mongo_export/Data" (migrates all the objects with prefix mongo_export/Data)

    Note:

    If you do not provide any value, all the objects present in the bucket are migrated.
  • credentials
    Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"
  • credentialsProfile
    Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile" : "ADMIN_USER"
  • useInstancePrincipal

    Example: "useInstancePrincipal" : true

Unique Configuration Parameters

schemaInfo

  • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaObject

  • Purpose: Specifies the name of the object in the bucket where NoSQL table schema definitions for the data being migrated are stored.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: 
    "schemaInfo": {
  "schemaObject": "mytable/Schema/schema.ddl"
}

DynamoDB-Formatted JSON File stored in AWS S3

The configuration file format for DynamoDB-formatted JSON File in AWS S3 as a source of NoSQL Database Migrator is shown below.

You can migrate a file containing the DynamoDB exported JSON data from the AWS S3 storage by specifying the path in the source configuration template.

A sample DynamoDB-formatted JSON File is as follows:
{"Item":{"Id":{"N":"101"},"Phones":{"L":[{"L":[{"S":"555-222"},{"S":"123-567"}]}]},"PremierCustomer":{"BOOL":false},"Address":{"M":{"Zip":{"N":"570004"},"Street":{"S":"21 main"},"DoorNum":{"N":"201"},"City":{"S":"London"}}},"FirstName":{"S":"Fred"},"FavNumbers":{"NS":["10"]},"LastName":{"S":"Smith"},"FavColors":{"SS":["Red","Green"]},"Age":{"N":"22"}}}
{"Item":{"Id":{"N":"102"},"Phones":{"L":[{"L":[{"S":"222-222"}]}]},"PremierCustomer":{"BOOL":false},"Address":{"M":{"Zip":{"N":"560014"},"Street":{"S":"32 main"},"DoorNum":{"N":"1024"},"City":{"S":"Wales"}}},"FirstName":{"S":"John"},"FavNumbers":{"NS":["10"]},"LastName":{"S":"White"},"FavColors":{"SS":["Blue"]},"Age":{"N":"48"}}}

You must export the DynamoDB table to AWS S3 storage as specified in Exporting DynamoDB table data to Amazon S3.

The valid sink types for DynamoDB-formatted JSON stored in AWS S3 are nosqldb and nosqldb_cloud.

Source Configuration Template
"source" : {
  "type" : "aws_s3",
  "format" : "dynamodb_json",
  "s3URL" : "<S3 object url>",
  "credentials" : "</path/to/aws/credentials/file>",
  "credentialsProfile" : <"profile name in aws credentials file">
}

Source Parameters

Common Configuration Parameters

  • type

    Use "type" : "aws_s3"

  • format

    Use "format" : "dynamodb_json"

    Note:

    If the value of the type parameter is aws_s3, then the format must be dynamodb_json.

Unique Configuration Parameters

s3URL

  • Purpose: Specifies the URL of an exported DynamoDB table stored in AWS S3. You can obtain this URL from the AWS S3 console. The valid URL format is https://<bucket-name>.<s3_endpoint>/<prefix>. The NoSQL Database Migrator will look for json.gz files in the prefix during import.

    Note:

    You must export DynamoDB table as specified in Exporting DynamoDB table data to Amazon S3.
  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: https://my-bucket.s3.ap-south-1.amazonaws.com/AWSDynamoDB/01649660790057-14f642be

credentials

  • Purpose: Specifies the absolute path to a file containing the AWS credentials. If not specified, it defaults to $HOME/.aws/credentials. For more details on the credentials file, see Configuration and credential file settings .
  • Data Type: string
  • Mandatory (Y/N): N
  • Example: 
    "credentials" : "/home/user/.aws/credentials"
"credentials" : "/home/user/security/credentials

    Note:

    The NoSQL Database Migrator does not log any of the credentials information. You must properly protect the credentials file from unauthorized access.

credentialsProfile

  • Purpose: Name of the profile in the AWS credentials file to be used to connect to AWS S3. User account credentials are referred to as a profile. If you do not specify this value, NoSQL Database Migrator uses the default profile. For more details on the credentials file, see Configuration and credential file settings .
  • Data Type: string
  • Mandatory (Y/N): N
  • Example: 
    "credentialsProfile" : "default"
"credentialsProfile" : "test"

DynamoDB-Formatted JSON File

The configuration file format for DynamoDB-formatted JSON File as a source of NoSQL Database Migrator is shown below.

You can migrate a file or directory containing the DynamoDB exported JSON data from a file system by specifying the path in the source configuration template.

A sample DynamoDB-formatted JSON File is as follows:
{"Item":{"Id":{"N":"101"},"Phones":{"L":[{"L":[{"S":"555-222"},{"S":"123-567"}]}]},"PremierCustomer":{"BOOL":false},"Address":{"M":{"Zip":{"N":"570004"},"Street":{"S":"21 main"},"DoorNum":{"N":"201"},"City":{"S":"London"}}},"FirstName":{"S":"Fred"},"FavNumbers":{"NS":["10"]},"LastName":{"S":"Smith"},"FavColors":{"SS":["Red","Green"]},"Age":{"N":"22"}}}
{"Item":{"Id":{"N":"102"},"Phones":{"L":[{"L":[{"S":"222-222"}]}]},"PremierCustomer":{"BOOL":false},"Address":{"M":{"Zip":{"N":"560014"},"Street":{"S":"32 main"},"DoorNum":{"N":"1024"},"City":{"S":"Wales"}}},"FirstName":{"S":"John"},"FavNumbers":{"NS":["10"]},"LastName":{"S":"White"},"FavColors":{"SS":["Blue"]},"Age":{"N":"48"}}}

You must copy the exported DynamoDB table data from AWS S3 storage to a local mounted file system.

The valid sink types for DynamoDB JSON file are nosqldb and nosqldb_cloud.

Source Configuration Template
"source" : {
  "type" : "file",
  "format" : "dynamodb_json",
  "dataPath" : "<path/to/[file|dir]/containing/exported/DDB/tabledata>"   
}

Source Parameters

Common Configuration Parameters

  • type

    Use "type" : "file"

  • format

    Use "format" : "dynamodb_json"

Unique Configuration Parameter

dataPath

  • Purpose: Specifies the absolute path to a file or directory containing the exported DynamoDB table data. You must copy exported DynamoDB table data from AWS S3 to a local mounted file system. You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json.gz extension in that directory and the data sub-directory.
  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • Specifying a file
      "dataPath" : "/home/user/AWSDynamoDB/01639372501551-bb4dd8c3/data/zclclwucjy6v5mkefvckxzhfvq.json.gz"
    • Specifying a directory
      "dataPath" : "/home/user/AWSDynamoDB/01639372501551-bb4dd8c3"

Oracle NoSQL Database

The configuration file format for Oracle NoSQL Database as a source of NoSQL Database Migrator is shown below.

You can migrate a table from Oracle NoSQL Database by specifying the table name in the source configuration template.

A sample Oracle NoSQL Database table is as follows:
{"id":20,"firstName":"Jane","lastName":"Smith","otherNames":[{"first":"Jane","last":"teacher"}],"age":25,"income":55000,"address":{"city":"San Jose","number":201,"phones":[{"area":608,"kind":"work","number":6538955},{"area":931,"kind":"home","number":9533341},{"area":931,"kind":"mobile","number":9533382}],"state":"CA","street":"Atlantic Ave","zip":95005},"connections":[40,75,63],"expenses":null}
{"id":10,"firstName":"John","lastName":"Smith","otherNames":[{"first":"Johny","last":"chef"}],"age":22,"income":45000,"address":{"city":"Santa Cruz","number":101,"phones":[{"area":408,"kind":"work","number":4538955},{"area":831,"kind":"home","number":7533341},{"area":831,"kind":"mobile","number":7533382}],"state":"CA","street":"Pacific Ave","zip":95008},"connections":[30,55,43],"expenses":null}
{"id":30,"firstName":"Adam","lastName":"Smith","otherNames":[{"first":"Adam","last":"handyman"}],"age":45,"income":75000,"address":{"city":"Houston","number":301,"phones":[{"area":618,"kind":"work","number":6618955},{"area":951,"kind":"home","number":9613341},{"area":981,"kind":"mobile","number":9613382}],"state":"TX","street":"Indian Ave","zip":95075},"connections":[60,45,73],"expenses":null}

Source Configuration Template

"source" : {
  "type": "nosqldb",
  "storeName" : "<store name>",
  "helperHosts" : ["hostname1:port1","hostname2:port2,..."],
  "table" : "<fully qualified table name>", 
  "includeTTL": <true|false>,    
  "security" : "</path/to/store/security/file>",
  "requestTimeoutMs" : 5000
}

Source Parameters

Common Configuration Parameter

  • type

    Use "type" : "nosqldb"

  • security

    Example:

    "security" : "/home/user/client.credentials"

    Example security file content for password file based authentication:

    oracle.kv.password.noPrompt=true
oracle.kv.auth.username=admin
oracle.kv.auth.pwdfile.file=/home/nosql/login.passwd
oracle.kv.transport=ssl
oracle.kv.ssl.trustStore=/home/nosql/client.trust
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

    Example security file content for wallet based authentication: 

    oracle.kv.password.noPrompt=true
oracle.kv.auth.username=admin
oracle.kv.auth.wallet.dir=/home/nosql/login.wallet
oracle.kv.transport=ssl
oracle.kv.ssl.trustStore=/home/nosql/client.trust
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)
  • requestTimeoutMs

    Example: "requestTimeoutMs" : 5000

Unique Configuration Parameters

storeName

  • Purpose: Name of the Oracle NoSQL Database store.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "storeName" : "kvstore"

helperHosts

  • Purpose: A list of host and registry port pairs in the hostname:port format. Delimit each item in the list using a comma. You must specify at least one helper host.

  • Data Type: array of strings
  • Mandatory (Y/N): Y
  • Example: "helperHosts" : ["localhost:5000","localhost:6000"]

table

  • Purpose: Fully qualified table name from which to migrate the data.

    Format: [namespace_name:]<table_name>

    If the table is in the DEFAULT namespace, you can omit the namespace_name. The table must exist in the store.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:

    • With the DEFAULT namespace "table" :"mytable"

    • With a non-default namespace "table" : "mynamespace:mytable"

    • To specify a child table "table" : "mytable.child"

includeTTL

  • Purpose: Specifies whether or not to include TTL metadata for table rows when exporting Oracle NoSQL Database tables. If set to true, the TTL data for rows also gets included in the data provided by the source. TTL is present in the _metadata JSON object associated with each row. The expiration time for each row gets exported as the number of milliseconds since the UNIX epoch (Jan 1st, 1970).

    If you do not specify this parameter, it defaults to false.

    Only the rows having a positive expiration value for TTL get included as part of the exported rows. If a row does not expire, which means TTL=0, then its TTL metadata is not included explicitly. For example, if ROW1 expires at 2021-10-19 00:00:00 and ROW2 does not expire, the exported data looks like as follows:
    //ROW1
{
  "id" : 1,
  "name" : "abc",
  "_metadata" : {
    "expiration" : 1634601600000
  }
}

//ROW2
{
  "id" : 2,
  "name" : "xyz"
}
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "includeTTL" : true

Oracle NoSQL Database Cloud Service

The configuration file format for Oracle NoSQL Database Cloud Service as a source of NoSQL Database Migrator is shown below.

You can migrate a table from Oracle NoSQL Database Cloud Service by specifying the name or OCID of the compartment in which the table resides in the source configuration template.

A sample Oracle NoSQL Database Cloud Service table is as follows:
{"id":20,"firstName":"Jane","lastName":"Smith","otherNames":[{"first":"Jane","last":"teacher"}],"age":25,"income":55000,"address":{"city":"San Jose","number":201,"phones":[{"area":608,"kind":"work","number":6538955},{"area":931,"kind":"home","number":9533341},{"area":931,"kind":"mobile","number":9533382}],"state":"CA","street":"Atlantic Ave","zip":95005},"connections":[40,75,63],"expenses":null}
{"id":10,"firstName":"John","lastName":"Smith","otherNames":[{"first":"Johny","last":"chef"}],"age":22,"income":45000,"address":{"city":"Santa Cruz","number":101,"phones":[{"area":408,"kind":"work","number":4538955},{"area":831,"kind":"home","number":7533341},{"area":831,"kind":"mobile","number":7533382}],"state":"CA","street":"Pacific Ave","zip":95008},"connections":[30,55,43],"expenses":null}
{"id":30,"firstName":"Adam","lastName":"Smith","otherNames":[{"first":"Adam","last":"handyman"}],"age":45,"income":75000,"address":{"city":"Houston","number":301,"phones":[{"area":618,"kind":"work","number":6618955},{"area":951,"kind":"home","number":9613341},{"area":981,"kind":"mobile","number":9613382}],"state":"TX","street":"Indian Ave","zip":95075},"connections":[60,45,73],"expenses":null}

Source Configuration Template

"source" : {
  "type" : "nosqldb_cloud",
  "endpoint" : "<Oracle NoSQL Cloud Service endpoint URL or region ID>",
  "table" : "<table name>",
  "compartment" : "<OCI compartment name or id>",
  "credentials" : "<path/to/oci/credential/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
  "useDelegationToken" : <true|false>,
  "readUnitsPercent" : <table readunits percent>,
  "includeTTL": <true|false>,
  "requestTimeoutMs" : <timeout in milli seconds>
}

Source Parameters

Common Configuration Parameters

  • type

    Use "type" : "nosqldb_cloud"

  • endpoint
    Example:

    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

  • credentials
    Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"
  • credentialsProfile
    Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile" : "ADMIN_USER"
  • useInstancePrincipal

    Example: "useInstancePrincipal" : true

  • useDelegationToken

    Example: "useDelegationToken" : true

    Note:

    The authentication with delegation token is supported only when the NoSQL Database Migrator is running from a Cloud Shell.
  • requestTimeoutMs

    Example: "requestTimeoutMs" : 5000

Unique Configuration Parameters

table

  • Purpose: Name of the table from which to migrate the data.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • To specify a table "table" : "myTable"
    • To specify a child table "table" : "mytable.child"

compartment

  • Purpose: Specifies the name or OCID of the compartment in which the table resides.

    If you do not provide any value, it defaults to the root compartment.

    You can find your compartment's OCID from the Compartment Explorer window under Governance in the OCI Cloud Console.

  • Data Type: string
  • Mandatory (Y/N): Y, if the table is not in the root compartment of the tenancy OR when the useInstancePrincipal parameter is set to true.

    Note:

    If the useInstancePrincipal parameter is set to true, the compartment must specify the compartment OCID and not the name.
  • Example:

    • Compartment name

      "compartment" : "mycompartment"

    • Compartment name qualified with its parent compartment

      "compartment" : "parent.childcompartment"

    • No value provided. Defaults to the root compartment.

      "compartment": ""

    • Compartment OCID

      "compartment" : "ocid1.tenancy.oc1...4ksd"

readUnitsPercent

  • Purpose: Percentage of table read units to be used while migrating the NoSQL table.

    The default value is 90. The valid range is any integer between 1 to 100. The amount of time required to migrate data is directly proportional to this attribute. It is better to increase the read throughput of the table for the migration activity. You can reduce the read throughput after the migration process completes.

    To learn the daily limits on throughput changes, see Cloud Limits in the Oracle NoSQL Database Cloud Service document.

    See Troubleshooting the Oracle NoSQL Database Migrator to learn how to use this attribute to improve the data migration speed.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "readUnitsPercent" : 90

includeTTL

  • Purpose: Specifies whether or not to include TTL metadata for table rows when exporting Oracle NoSQL Database tables. If set to true, the TTL data for rows also gets included in the data provided by the source. TTL is present in the _metadata JSON object associated with each row. The expiration time for each row gets exported as the number of milliseconds since the UNIX epoch (Jan 1st, 1970).

    If you do not specify this parameter, it defaults to false.

    Only the rows having a positive expiration value for TTL get included as part of the exported rows. If a row does not expire, which means TTL=0, then its TTL metadata is not included explicitly. For example, if ROW1 expires at 2021-10-19 00:00:00 and ROW2 does not expire, the exported data looks like as follows:
    //ROW1
{
  "id" : 1,
  "name" : "abc",
  "_metadata" : {
    "expiration" : 1634601600000
  }
}

//ROW2
{
  "id" : 2,
  "name" : "xyz"
}
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "includeTTL" : true

CSV File Source

The configuration file format for the CSV file as a source of NoSQL Database Migrator is shown below. The CSV file must conform to the RFC4180 format.

You can migrate a CSV file or a directory containing the CSV data by specifying the file name or directory in the source configuration template.

A sample CSV file is as follows:
1,"Computer Science","San Francisco","2500"
2,"Bio-Technology","Los Angeles","1200"
3,"Journalism","Las Vegas","1500"
4,"Telecommunication","San Francisco","2500"

Source Configuration Template

"source" : {
  "type" : "file",
  "format" : "csv",
  "dataPath": "</path/to/a/csv/[file|dir]>",
  "hasHeader" : <true | false>,
  "columns" : ["column1", "column2", ....],
  "csvOptions": {
    "encoding": "<character set encoding>",
    "trim": "<true | false>"
 }
}

Source Parameters

Common Configuration Parameters

  • type

    Use "type" : "file"

  • format

    Use "format" : "csv"

Unique Configuration Parameters

datapath

  • Purpose: Specifies the absolute path to a file or directory containing the CSV data for migration. If you specify a directory, NoSQL Database Migrator imports all the files with the .csv or .CSV extension in that directory. All the CSV files are copied into a single table, but not in any particular order.

    CSV files must conform to the RFC4180 standard. You must ensure that the data in each CSV file matches with the NoSQL Database table schema defined in the sink table. Sub-directories are not supported.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:

    • Specifying a CSV file

      "dataPath" : "/home/user/sample.csv"

    • Specifying a directory

      "dataPath" : "/home/user"

Note:

The CSV files must contain only scalar values. Importing CSV files containing complex types such as MAP, RECORD, ARRAY, and JSON is not supported. The NoSQL Database Migrator tool does not check for the correctness of the data in the input CSV file. The NoSQL Database Migrator tool supports the importing of CSV data that conforms to the RFC4180 format. CSV files containing data that does not conform to the RFC4180 standard may not get copied correctly or may result in an error. If the input data is corrupted, the NoSQL Database Migrator tool will not parse the CSV records. If any errors are encountered during migration, the NoSQL Database Migrator tool logs the information about the failed input records for debugging and informative purposes. For more details, see Logging Migrator Progress in Using Oracle NoSQL Data Migrator.

hasHeader

  • Purpose: Specifies if the CSV file has a header or not. If this is set to true, the first line is ignored. If it is set to false, the first line is considered a CSV record. The default value is false.

  • Data Type: Boolean
  • Mandatory (Y/N): N

  • Example: "hasHeader" : "false"

columns

  • Purpose: Specifies the list of NoSQL Database table column names. The order of the column names indicates the mapping of the CSV file fields with corresponding NoSQL Database table columns. If the order of the input CSV file columns does not match the existing or newly created NoSQL Database table columns, you can map the ordering using this parameter. Also, when importing into a table that has an Identity Column, you can skip the Identity column name in the columns parameter.

    Note:

    • If the NoSQL Database table has additional columns that are not available in the CSV file, the values of the missing columns are updated with the default value as defined in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration. For more information on default values, see Data Type Definitions section in the SQL Reference Guide.
    • If the CSV file has additional columns that are not defined in the NoSQL Database table, the additional column information is ignored.
    • If any value in the CSV record is empty, it is set to the default value of the corresponding columns in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration.
  • Data Type: Array of Strings
  • Mandatory (Y/N): N

  • Example: "columns" : ["table_column_1", "table_column_2"]

csvOptions

  • Purpose: Specifies the formatting options for a CSV file. Provide the character set encoding format of the CSV file and choose whether or not to trim the blank spaces.

  • Data Type: Object
  • Mandatory (Y/N): N

csvOptions.encoding

  • Purpose: Specifies the character set to decode the CSV file. The default value is UTF-8. The supported character sets are US-ASCII, ISO-8859-1, UTF-8, and UTF-16.

  • Data Type: String
  • Mandatory (Y/N): N

  • Example: "encoding" : "UTF-8"

csvOptions.trim

  • Purpose: Specifies if the leading and trailing blanks of a CSV field value must be trimmed. The default value is false.

  • Data Type: Boolean
  • Mandatory (Y/N): N

  • Example: "trim" : "true"

CSV file in OCI Object Storage Bucket

The configuration file format for the CSV file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below. The CSV file must conform to the RFC4180 format.

You can migrate a CSV file in the OCI Object Storage bucket by specifying the name of the bucket in the source configuration template.

A sample CSV file in the OCI Object Storage bucket is as follows:
1,"Computer Science","San Francisco","2500"
2,"Bio-Technology","Los Angeles","1200"
3,"Journalism","Las Vegas","1500"
4,"Telecommunication","San Francisco","2500"

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Source Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "csv",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
   "hasHeader" : <true | false>,
   "columns" : ["column1", "column2", ....],
   "csvOptions" : {         
     "encoding" : "<character set encoding>",
     "trim" : <true | false>
   }
 }

Source Parameters

Common Configuration Parameters

  • type

    Use "type" : "object_storage_oci"

  • format

    Use "format" : "csv"

  • endpoint
    Example:

    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

  • namespace

    Example: "namespace" : "my-namespace"

  • bucket

    Example: "bucket" : "my-bucket"

    Note:

    • The NoSQL Database Migrator imports all the files with the .csv or .CSV extension object-wise and copies them into a single table in the same order.

    • The CSV files must contain only scalar values. Importing CSV files containing complex types such as MAP, RECORD, ARRAY, and JSON is not supported. The NoSQL Database Migrator tool does not check for the correctness of the data in the input CSV file. The NoSQL Database Migrator tool supports the importing of CSV data that conforms to the RFC4180 format. CSV files containing data that does not conform to the RFC4180 standard may not get copied correctly or may result in an error. If the input data is corrupted, the NoSQL Database Migrator tool will not parse the CSV records. If any errors are encountered during migration, the NoSQL Database Migrator tool logs the information about the failed input records for debugging and informative purposes. For more details, see Logging Migrator Progress in Using Oracle NoSQL Data Migrator.

  • prefix
    Example:
    1. "prefix" : "my_table/Data/000000.csv" (migrates only 000000.csv)
    2. "prefix" : "my_table/Data" (migrates all the objects with prefix my_table/Data)
  • credentials
    Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"
  • credentialsProfile
    Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile" : "ADMIN_USER"
  • useInstancePrincipal

    Example: "useInstancePrincipal" : true

Unique Configuration Parameters

hasHeader

  • Purpose: Specifies if the CSV file has a header or not. If this is set to true, the first line is ignored. If it is set to false, the first line is considered a CSV record. The default value is false.

  • Data Type: Boolean
  • Mandatory (Y/N): N

  • Example: "hasHeader" : "false"

columns

  • Purpose: Specifies the list of NoSQL Database table column names. The order of the column names indicates the mapping of the CSV file fields with corresponding NoSQL Database table columns. If the order of the input CSV file columns does not match the existing or newly created NoSQL Database table columns, you can map the ordering using this parameter. Also, when importing into a table that has an Identity Column, you can skip the Identity column name in the columns parameter.

    Note:

    • If the NoSQL Database table has additional columns that are not available in the CSV file, the values of the missing columns are updated with the default value as defined in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration. For more information on default values, see Data Type Definitions section in the SQL Reference Guide.
    • If the CSV file has additional columns that are not defined in the NoSQL Database table, the additional column information is ignored.
    • If any value in the CSV record is empty, it is set to the default value of the corresponding columns in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration.
  • Data Type: Array of Strings
  • Mandatory (Y/N): N

  • Example: "columns" : ["table_column_1", "table_column_2"]

csvOptions

  • Purpose: Specifies the formatting options for a CSV file. Provide the character set encoding format of the CSV file and choose whether or not to trim the blank spaces.

  • Data Type: Object
  • Mandatory (Y/N): N

csvOptions.encoding

  • Purpose: Specifies the character set to decode the CSV file. The default value is UTF-8. The supported character sets are US-ASCII, ISO-8859-1, UTF-8, and UTF-16.

  • Data Type: String
  • Mandatory (Y/N): N

  • Example: "encoding" : "UTF-8"

csvOptions.trim

  • Purpose: Specifies if the leading and trailing blanks of a CSV field value must be trimmed. The default value is false.

  • Data Type: Boolean
  • Mandatory (Y/N): N

  • Example: "trim" : "true"