1. Administrator's Guide
  2. Administrative Procedures
  3. Using Oracle NoSQL Database Migrator
  4. Sources and Sinks
  5. Source Configuration Templates

Source Configuration Templates

Learn about the source configuration file formats for each valid source and the purpose of each configuration parameter.

For the configuration file template, see Configuration File in Terminology used with Oracle NoSQL Database Migrator.

For details on valid sink formats for each of the source, see Sink Configuration Templates.

Topics

  • JSON as the File Source

    The source configuration template for the Oracle NoSQL Database Migrator to copy the data from a JSON file as a source to a valid sink.

  • JSON File in OCI Object Storage Bucket

    The source configuration template for the Oracle NoSQL Database Migrator to copy the data from a JSON file in the OCI Object Storage bucket as a source to a valid sink.

  • MongoDB-Formatted JSON File

    The source configuration template for the Oracle NoSQL Database Migrator to copy the data from a MongoDB-Formatted JSON file as a source to a valid sink.

  • MongoDB-Formatted JSON File in OCI Object Storage bucket

    The source configuration template for the Oracle NoSQL Database Migrator to copy the data from a MongoDB-Formatted JSON file in the OCI Object Storage bucket as a source to a valid sink.

  • DynamoDB-Formatted JSON File stored in AWS S3

    The source configuration template for the Oracle NoSQL Database Migrator to copy the data from a DynamoDB-Formatted JSON file in the AWS S3 storage as a source to a valid sink.

  • DynamoDB-Formatted JSON File

    The source configuration template for the Oracle NoSQL Database Migrator to copy the data from a DynamoDB-Formatted JSON file as a source to a valid sink.

  • Oracle NoSQL Database

    The source configuration template for the Oracle NoSQL Database Migrator to copy the data from Oracle NoSQL Database tables as a source to a valid sink.

  • Oracle NoSQL Database Cloud Service

    The source configuration template for the Oracle NoSQL Database Migrator to copy the data from Oracle NoSQL Database Cloud Service tables as a source to a valid sink.

  • CSV as the File Source

    The source configuration template for the Oracle NoSQL Database Migrator to copy the data from CSV file as a source to a valid sink.

  • CSV file in OCI Object Storage Bucket

    The source configuration template for the Oracle NoSQL Database Migrator to copy the data from CSV file stored in OCI Object Storage bucket as a source to a valid sink.

JSON as the File Source

The configuration file format for JSON File as a source of NoSQL Database Migrator is shown below.

You can migrate a JSON source file from a file path or a directory by specifying the file path or directory in the source configuration template.

A sample JSON source file is as follows:
{"Item":{"PK":{"S":"ACCT#82691500"},"SK":{"S":"ACCT#82691500"},"AccountIndexId":{"S":"ACCT#82691500"},"Emailid":{"S":"alejandro.rosalez11@example.org"},"AccountId":{"N":"82691500"},"PlasticCardNumber":{"S":"9610432116466295"},"FirstName":{"S":"Alejandro"},"Addresses":{"M":{"RESIDENCE":{"M":{"city":{"S":"Any Town"},"country":{"S":"USA"},"street":{"S":"123 Any Street"}}},"BUSINESS":{"M":{"city":{"S":"Anytown"},"country":{"S":"country"},"street":{"S":"221 Main Street"}}}}},"LastName":{"S":"Rosalez"}}}
{"Item":{"PK":{"S":"ACCT#76584123"},"SK":{"S":"ACCT#76584123"},"AccountIndexId":{"S":"ACCT#76584123"},"Emailid":{"S":"zhang.wei@example.com"},"AccountId":{"N":"76584123"},"PlasticCardNumber":{"S":"4235400034568756"},"FirstName":{"S":"Zhang"},"Addresses":{"M":{"RESIDENCE":{"M":{"city":{"S":"Any Town"},"country":{"S":"USA"},"street":{"S":"135 Any Street"}}},"BUSINESS":{"M":{"city":{"S":"AnyTown"},"country":{"S":"country"},"street":{"S":"100 Main Street"}}}}},"LastName":{"S":"Wei"},"AuthUsers":{"M":{"AUTHUSER-2":{"M":{"Name":{"S":"Mateo Jackson"},"PlasticCardNumber":{"S":"4036516984267960"}}},"AUTHUSER-1":{"M":{"Name":{"S":"Paulo Santos"},"PlasticCardNumber":{"S":"4036546984262340"}}}}}}}

Source Configuration Template

"source" : {
  "type" : "file",
  "format" : "json",
  "dataPath": "</path/to/a/json/file>",
  "schemaInfo": {
               "schemaPath": "</path/to/schema/file>"
  }
}

Source Parameters

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "type" : "file"

format

  • Purpose: Specifies the source format.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "format" : "json"

dataPath

  • Purpose: Specifies the absolute path to a file or directory containing the JSON data for migration.

    You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:

    • Specifying a JSON file

      "dataPath" : "/home/user/sample.json"

    • Specifying a directory

      "dataPath" : "/home/user"

schemaInfo

  • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaPath

  • Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: 
    "schemaInfo" : {
            "schemaPath" : "/home/user/mytable/Schema/schema.ddl"
             }

JSON File in OCI Object Storage Bucket

The configuration file format for JSON file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below.

You can migrate a JSON file in the OCI Object Storage bucket by specifying the name of the bucket in the source configuration template.

A sample JSON source file in the OCI Object Storage bucket is as follows:
{"Item":{"PK":{"S":"ACCT#82691500"},"SK":{"S":"ACCT#82691500"},"AccountIndexId":{"S":"ACCT#82691500"},"Emailid":{"S":"alejandro.rosalez11@example.org"},"AccountId":{"N":"82691500"},"PlasticCardNumber":{"S":"9610432116466295"},"FirstName":{"S":"Alejandro"},"Addresses":{"M":{"RESIDENCE":{"M":{"city":{"S":"Any Town"},"country":{"S":"USA"},"street":{"S":"123 Any Street"}}},"BUSINESS":{"M":{"city":{"S":"Anytown"},"country":{"S":"country"},"street":{"S":"221 Main Street"}}}}},"LastName":{"S":"Rosalez"}}}
{"Item":{"PK":{"S":"ACCT#76584123"},"SK":{"S":"ACCT#76584123"},"AccountIndexId":{"S":"ACCT#76584123"},"Emailid":{"S":"zhang.wei@example.com"},"AccountId":{"N":"76584123"},"PlasticCardNumber":{"S":"4235400034568756"},"FirstName":{"S":"Zhang"},"Addresses":{"M":{"RESIDENCE":{"M":{"city":{"S":"Any Town"},"country":{"S":"USA"},"street":{"S":"135 Any Street"}}},"BUSINESS":{"M":{"city":{"S":"AnyTown"},"country":{"S":"country"},"street":{"S":"100 Main Street"}}}}},"LastName":{"S":"Wei"},"AuthUsers":{"M":{"AUTHUSER-2":{"M":{"Name":{"S":"Mateo Jackson"},"PlasticCardNumber":{"S":"4036516984267960"}}},"AUTHUSER-1":{"M":{"Name":{"S":"Paulo Santos"},"PlasticCardNumber":{"S":"4036546984262340"}}}}}}}

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Source Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "schemaInfo" : {
      "schemaObject" : "<object name>"
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

Source Parameters

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "type" : "object_storage_oci"

format

  • Purpose: Specifies the source format.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "format" : "json"

endpoint

  • Purpose: Specifies the OCI Object Storage service endpoint URL or region ID.

    You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:

    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

namespace

  • Purpose: Specifies the OCI Object Storage service namespace. This is an optional parameter. If you don't specify this parameter, the default namespace of the tenancy is used.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "namespace" : "my-namespace"

bucket

  • Purpose: Specifies the name of the bucket, which contains the source JSON files. Ensure that the required bucket already exists in the OCI Object Storage instance and has read permissions.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "bucket" : "staging_bucket"

prefix

  • Purpose: Used for filtering the objects that are being migrated from the bucket. All the objects with the given prefix present in the bucket are migrated. For more information about prefixes, see Object Naming Using Prefixes and Hierarchies.

    If you do not provide any value, no filter is applied and all the objects present in the bucket are migrated.

  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "prefix" : "my_table/Data/000000.json" (migrates only 000000.json)
    2. "prefix" : "my_table/Data" (migrates all the objects with prefix my_table/Data)

schemaInfo

  • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaObject

  • Purpose: Specifies the name of the object in the bucket where NoSQL table schema definitions for the data being migrated are stored.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: 
    "schemaInfo" : {
            "schemaObject" : "mytable/Schema/schema.ddl"
             }

credentials

  • Purpose: Absolute path to a file containing OCI credentials.

    If not specified, it defaults to $HOME/.oci/config

    See Example Configuration for an example of the credentials file.

    Note:

    Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"

credentialsProfile

  • Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

    If you do not specify this value, it defaults to the DEFAULT profile.

    Note:

    This parameter is valid ONLY if the credentials parameter is specified.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile": "ADMIN_USER"

useInstancePrincipal

  • Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security.

    If not specified, it defaults to false.

    Note:

    • It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
    • Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: boolean
  • Mandatory (Y/N): N

  • Example: "useInstancePrincipal" : true

MongoDB-Formatted JSON File

The configuration file format for MongoDB-formatted JSON File as a source of NoSQL Database Migrator is shown below.

You can copy a file or directory containing the MongoDB exported JSON data for migration by specifying the file or directory in the source configuration template.

A sample MongoDB-formatted JSON File is as follows:
{"_id":0,"name":"Aimee Zank","scores":[{"score":1.463179736705023,"type":"exam"},{"score":11.78273309957772,"type":"quiz"},{"score":35.8740349954354,"type":"homework"}]}
{"_id":1,"name":"Aurelia Menendez","scores":[{"score":60.06045071030959,"type":"exam"},{"score":52.79790691903873,"type":"quiz"},{"score":71.76133439165544,"type":"homework"}]}
{"_id":2,"name":"Corliss Zuk","scores":[{"score":67.03077096065002,"type":"exam"},{"score":6.301851677835235,"type":"quiz"},{"score":66.28344683278382,"type":"homework"}]}
{"_id":3,"name":"Bao Ziglar","scores":[{"score":71.64343899778332,"type":"exam"},{"score":24.80221293650313,"type":"quiz"},{"score":42.26147058804812,"type":"homework"}]}
{"_id":4,"name":"Zachary Langlais","scores":[{"score":78.68385091304332,"type":"exam"},{"score":90.2963101368042,"type":"quiz"},{"score":34.41620148042529,"type":"homework"}]}

MongoDB supports two types of extensions to the JSON format of files, Canonical mode and Relaxed mode. You can supply the MongoDB-formatted JSON file that is generated using the mongoexport tool in either Canonical or Relaxed mode. Both the modes are supported by the NoSQL Database Migrator for migration.

For more information on the MongoDB Extended JSON (v2) file, See mongoexport_formats.

For more information on the generation of MongoDB-formatted JSON file, see mongoexport for more information.

Source Configuration Template

"source" : {
    "type" : "file",
    "format" : "mongodb_json",
    "dataPath": "</path/to/a/json/file>",
    "schemaInfo": {
               "schemaPath": "</path/to/schema/file>"
    }
}

Source Parameters

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "type" : "file"

format

  • Purpose: Specifies the source format.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "format" : "mongodb_json"

dataPath

  • Purpose: Specifies the absolute path to a file or directory containing the MongoDB exported JSON data for migration.

    You must have generated these files using the mongoexport tool. See mongoexport for more information.

    You can supply the MongoDB-formatted JSON file that is generated using the mongoexport tool in either canonical or relaxed mode. Both the modes are supported by the NoSQL Database Migrator for migration.

    For more information on the MongoDB Extended JSON (v2) file, See mongoexport_formats.

    If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported. You must ensure that this data matches with the NoSQL table schema defined at the sink.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:

    • Specifying a MongoDB formatted JSON file

      "dataPath" : "/home/user/sample.json"

    • Specifying a directory

      "dataPath" : "/home/user"

schemaInfo

  • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaPath

  • Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: 
    "schemaInfo" : {
            "schemaPath" : "/home/user/mytable/Schema/schema.ddl"
             }

MongoDB-Formatted JSON File in OCI Object Storage bucket

The configuration file format for MongoDB-Formatted JSON file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below.

You can copy the MongoDB exported JSON data in the OCI Object Storage bucket for migration by specifying the name of the bucket in the source configuration template.

A sample MongoDB-formatted JSON File is as follows:
{"_id":0,"name":"Aimee Zank","scores":[{"score":1.463179736705023,"type":"exam"},{"score":11.78273309957772,"type":"quiz"},{"score":35.8740349954354,"type":"homework"}]}
{"_id":1,"name":"Aurelia Menendez","scores":[{"score":60.06045071030959,"type":"exam"},{"score":52.79790691903873,"type":"quiz"},{"score":71.76133439165544,"type":"homework"}]}
{"_id":2,"name":"Corliss Zuk","scores":[{"score":67.03077096065002,"type":"exam"},{"score":6.301851677835235,"type":"quiz"},{"score":66.28344683278382,"type":"homework"}]}
{"_id":3,"name":"Bao Ziglar","scores":[{"score":71.64343899778332,"type":"exam"},{"score":24.80221293650313,"type":"quiz"},{"score":42.26147058804812,"type":"homework"}]}
{"_id":4,"name":"Zachary Langlais","scores":[{"score":78.68385091304332,"type":"exam"},{"score":90.2963101368042,"type":"quiz"},{"score":34.41620148042529,"type":"homework"}]}

Extract the data from MongoDB using the mongoexport utility and upload it to the OCI Object Storage bucket. See mongoexport for more information.

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Source Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "mongodb_json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "schemaInfo" : {
      "schemaObject" : "<object name>"
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

Source Parameters

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "type" : "object_storage_oci"

format

  • Purpose: Specifies the source format.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "format" : "mongodb_json"

endpoint

  • Purpose: Specifies the OCI Object Storage service endpoint URL or region ID.

    You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:

    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

namespace

  • Purpose: Specifies the OCI Object Storage service namespace. This is an optional parameter. If you don't specify this parameter, the default namespace of the tenancy is used.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "namespace" : "my-namespace"

bucket

  • Purpose: Specifies the name of the bucket, which contains the source MongoDB-Formatted JSON files. Ensure that the required bucket already exists in the OCI Object Storage instance and has read permissions.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "bucket" : "staging_bucket"

prefix

  • Purpose: Used for filtering the objects that are being migrated from the bucket. All the objects with the given prefix present in the bucket are migrated. For more information about prefixes, see Object Naming Using Prefixes and Hierarchies.

    If you do not provide any value, no filter is applied and all the MongoDB JSON formatted objects present in the bucket are migrated. Extract the data from MongoDB using the mongoexport utility and upload it to the OCI Object Storage bucket. See mongoexport for more information.

    If you do not provide any value, no filter is applied and all the objects present in the bucket are migrated.

  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "prefix" : "mongo_export/Data/table.json" (migrates only table.json)
    2. "prefix" : "mongo_export/Data" (migrates all the objects with prefix mongo_export/Data)

schemaInfo

  • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.

  • Data Type: Object
  • Mandatory (Y/N): N

schemaInfo.schemaObject

  • Purpose: Specifies the name of the object in the bucket where NoSQL table schema definitions for the data being migrated are stored.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: 
    "schemaInfo" : {
            "schemaObject" : "mytable/Schema/schema.ddl"
             }

credentials

  • Purpose: Absolute path to a file containing OCI credentials.

    If not specified, it defaults to $HOME/.oci/config

    See Example Configuration for an example of the credentials file.

    Note:

    Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"

credentialsProfile

  • Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

    If you do not specify this value, it defaults to the DEFAULT profile.

    Note:

    This parameter is valid ONLY if the credentials parameter is specified.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile": "ADMIN_USER"

useInstancePrincipal

  • Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security.

    If not specified, it defaults to false.

    Note:

    • It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
    • Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: boolean
  • Mandatory (Y/N): N

  • Example: "useInstancePrincipal" : true

DynamoDB-Formatted JSON File stored in AWS S3

The configuration file format for DynamoDB-formatted JSON File in AWS S3 as a source of NoSQL Database Migrator is shown below.

You can migrate a file containing the DynamoDB exported JSON data from the AWS S3 storage by specifying the path in the source configuration template.

A sample DynamoDB-formatted JSON File is as follows:
{"Item":{"Id":{"N":"101"},"Phones":{"L":[{"L":[{"S":"555-222"},{"S":"123-567"}]}]},"PremierCustomer":{"BOOL":false},"Address":{"M":{"Zip":{"N":"570004"},"Street":{"S":"21 main"},"DoorNum":{"N":"201"},"City":{"S":"London"}}},"FirstName":{"S":"Fred"},"FavNumbers":{"NS":["10"]},"LastName":{"S":"Smith"},"FavColors":{"SS":["Red","Green"]},"Age":{"N":"22"}}}
{"Item":{"Id":{"N":"102"},"Phones":{"L":[{"L":[{"S":"222-222"}]}]},"PremierCustomer":{"BOOL":false},"Address":{"M":{"Zip":{"N":"560014"},"Street":{"S":"32 main"},"DoorNum":{"N":"1024"},"City":{"S":"Wales"}}},"FirstName":{"S":"John"},"FavNumbers":{"NS":["10"]},"LastName":{"S":"White"},"FavColors":{"SS":["Blue"]},"Age":{"N":"48"}}}

You must export the DynamoDB table to AWS S3 storage as specified in Exporting DynamoDB table data to Amazon S3.

The valid sink types for DynamoDB-formatted JSON stored in AWS S3 are nosqldb and nosqldb_cloud.

Source Configuration Template
"source" : {
    "type" : "aws_s3",
    "format" : "dynamodb_json",
    "s3URL" : "<S3 object url>",
    "credentials" : "</path/to/aws/credentials/file>",
    "credentialsProfile" : <"profile name in aws credentials file">
}
Source Parameters:
  • type
  • format
  • s3URL
  • credentials
  • credentialsProfile
type
  • Purpose:Identifies the source type.
  • Data Type: string
  • Mandatory (Y/N):Y
  • Example: "type" : "aws_s3"
format
  • Purpose:Specifies the source format.
  • Data Type: string
  • Mandatory (Y/N):Y
  • Example: "format" : "dynamodb_json"

Note:

If the value of the "type" is aws_s3, then format must be dynamodb_json.
s3URL
  • Purpose: Specifies the URL of an exported DynamoDB table stored in AWS S3. You can obtain this URL from the AWS S3 console. Valid URL format is https://<bucket-name>.<s3_endpoint>/<prefix>. The migrator will look for json.gz files in the prefix for import.

    Note:

    You must export DynamoDB table as specified in Exporting DynamoDB table data to Amazon S3.
  • Data Type: string
  • Mandatory: Yes
  • Example: https://my-bucket.s3.ap-south-1.amazonaws.com/AWSDynamoDB/01649660790057-14f642be
credentials
  • Purpose: Specifies the absolute path to a file containing the AWS credentials. If not specified, it defaults to $HOME/.aws/credentials. Please refer to Configuration and credential file settings for more details on the credentials file.
  • Data Type: string
  • Mandatory (Y/N):N
  • Example: 
    "credentials" : "/home/user/.aws/credentials"
"credentials" : "/home/user/security/credentials

Note:

The Migrator does not log any of the credentials information. You should properly protect the credentials file from unauthorized access.
credentialsProfile
  • Purpose: Name of the profile in the AWS credentials file to be used to connect to AWS S3. User account credentials are referred to as a profile. If you do not specify this value, it defaults to the default profile. Please refer to Configuration and credential file settings for more details on the credentials file.
  • Data Type: string
  • Mandatory (Y/N):N
  • Example:
    "credentialsProfile" : "default"
        "credentialsProfile": "test"

DynamoDB-Formatted JSON File

The configuration file format for DynamoDB-formatted JSON File as a source of NoSQL Database Migrator is shown below.

You can migrate a file or directory containing the DynamoDB exported JSON data from a file system by specifying the path in the source configuration template.

A sample DynamoDB-formatted JSON File is as follows:
{"Item":{"Id":{"N":"101"},"Phones":{"L":[{"L":[{"S":"555-222"},{"S":"123-567"}]}]},"PremierCustomer":{"BOOL":false},"Address":{"M":{"Zip":{"N":"570004"},"Street":{"S":"21 main"},"DoorNum":{"N":"201"},"City":{"S":"London"}}},"FirstName":{"S":"Fred"},"FavNumbers":{"NS":["10"]},"LastName":{"S":"Smith"},"FavColors":{"SS":["Red","Green"]},"Age":{"N":"22"}}}
{"Item":{"Id":{"N":"102"},"Phones":{"L":[{"L":[{"S":"222-222"}]}]},"PremierCustomer":{"BOOL":false},"Address":{"M":{"Zip":{"N":"560014"},"Street":{"S":"32 main"},"DoorNum":{"N":"1024"},"City":{"S":"Wales"}}},"FirstName":{"S":"John"},"FavNumbers":{"NS":["10"]},"LastName":{"S":"White"},"FavColors":{"SS":["Blue"]},"Age":{"N":"48"}}}

You must copy the exported DynamoDB table data from AWS S3 storage to a local mounted file system.

The valid sink types for DynamoDB JSON file are nosqldb and nosqldb_cloud.

Source Configuration Template
"source" : {
    "type" : "file",
    "format" : "dynamodb_json",
    "dataPath" : "<path to a file or directory containing exported DDB table data>"   
}
Source Parameters:
  • type
  • format
  • dataPath
type
  • Purpose:Identifies the source type.
  • Data Type: string
  • Mandatory (Y/N):Y
  • Example: "type" : "file"
format
  • Purpose:Specifies the source format.
  • Data Type: string
  • Mandatory (Y/N):Y
  • Example: "format" : "dynamodb_json"
dataPath
  • Purpose: Specifies the absolute path to a file or directory containing the exported DynamoDB table data. You must copy exported DynamoDB table data from AWS S3 to a local mounted file system. You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json.gz extension in that directory and the datasub-directory.
  • Data Type: string
  • Mandatory (Y/N):Y
  • Example:
    • Specifying a file
      "dataPath" : "/home/user/AWSDynamoDB/01639372501551-bb4dd8c3/data/zclclwucjy6v5mkefvckxzhfvq.json.gz"
    • Specifying a directory
      "dataPath" : "/home/user/AWSDynamoDB/01639372501551-bb4dd8c3"

Oracle NoSQL Database

The configuration file format for Oracle NoSQL Database as a source of NoSQL Database Migrator is shown below.

You can migrate a table from Oracle NoSQL Database by specifying the table name in the source configuration template.

A sample Oracle NoSQL Database table is as follows:
{"id":20,"firstName":"Jane","lastName":"Smith","otherNames":[{"first":"Jane","last":"teacher"}],"age":25,"income":55000,"address":{"city":"San Jose","number":201,"phones":[{"area":608,"kind":"work","number":6538955},{"area":931,"kind":"home","number":9533341},{"area":931,"kind":"mobile","number":9533382}],"state":"CA","street":"Atlantic Ave","zip":95005},"connections":[40,75,63],"expenses":null}
{"id":10,"firstName":"John","lastName":"Smith","otherNames":[{"first":"Johny","last":"chef"}],"age":22,"income":45000,"address":{"city":"Santa Cruz","number":101,"phones":[{"area":408,"kind":"work","number":4538955},{"area":831,"kind":"home","number":7533341},{"area":831,"kind":"mobile","number":7533382}],"state":"CA","street":"Pacific Ave","zip":95008},"connections":[30,55,43],"expenses":null}
{"id":30,"firstName":"Adam","lastName":"Smith","otherNames":[{"first":"Adam","last":"handyman"}],"age":45,"income":75000,"address":{"city":"Houston","number":301,"phones":[{"area":618,"kind":"work","number":6618955},{"area":951,"kind":"home","number":9613341},{"area":981,"kind":"mobile","number":9613382}],"state":"TX","street":"Indian Ave","zip":95075},"connections":[60,45,73],"expenses":null}

Source Configuration Template

"source" : {
    "type": "nosqldb",
    "table" : "<fully qualified table name>",
    "storeName" : "<store name>",
    "helperHosts" : ["hostname1:port1","hostname2:port2,..."],
    "security" : "</path/to/store/security/file>",
    "requestTimeoutMs" : 5000,
    "includeTTL": <true|false>
}

Source Parameters

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "type" : "nosqldb"

table

  • Purpose: Fully qualified table name from which to migrate the data.

    Format: [namespace_name:]<table_name>

    If the table is in the DEFAULT namespace, you can omit the namespace_name. The table must exist in the store.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:

    • With the DEFAULT namespace "table" :"mytable"

    • With a non-default namespace "table" : "mynamespace:mytable"

    • To specify a child table "table" : "mytable.child"

storeName

  • Purpose: Name of the Oracle NoSQL Database store.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example: "storeName" : "kvstore"

helperHosts

  • Purpose: A list of host and registry port pairs in the hostname:port format. Delimit each item in the list using a comma. You must specify at least one helper host.

  • Data Type: array of strings
  • Mandatory (Y/N): Y
  • Example: "helperHosts" : ["localhost:5000","localhost:6000"]

security

  • Purpose:

    If your store is a secure store, provide the absolute path to the security login file that contains your store credentials. See Configuring Security with Remote Access in Administrator's Guide to know more about the security login file.

    You can use either password file based authentication or wallet based authentication. However, the wallet based authentication is supported only in the Enterprise Edition (EE) of Oracle NoSQL Database. For more information on wallet-based authentication, see Source and Sink Security.

    The Community Edition(CE) edition supports password file based authentication only.

  • Data Type: string
  • Mandatory (Y/N): Y for a secure store
  • Example:

    "security" : "/home/user/client.credentials"

    Example security file content for password file based authentication:

    oracle.kv.password.noPrompt=true
oracle.kv.auth.username=admin
oracle.kv.auth.pwdfile.file=/home/nosql/login.passwd
oracle.kv.transport=ssl
oracle.kv.ssl.trustStore=/home/nosql/client.trust
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

    Example security file content for wallet based authentication: 

    oracle.kv.password.noPrompt=true
oracle.kv.auth.username=admin
oracle.kv.auth.wallet.dir=/home/nosql/login.wallet
oracle.kv.transport=ssl
oracle.kv.ssl.trustStore=/home/nosql/client.trust
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

requestTimeoutMs

  • Purpose: Specifies the time to wait for each read operation from the store to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "requestTimeoutMs" : 5000

includeTTL

  • Purpose: Specifies whether or not to include TTL metadata for table rows when exporting Oracle NoSQL Database tables. If set to true, the TTL data for rows also gets included in the data provided by the source. TTL is present in the _metadata JSON object associated with each row. The expiration time for each row gets exported as the number of milliseconds since the UNIX epoch (Jan 1st, 1970).

    If you do not specify this parameter, it defaults to false.

    Only the rows having a positive expiration value for TTL get included as part of the exported rows. If a row does not expire, which means TTL=0, then its TTL metadata is not included explicitly. For example, if ROW1 expires at 2021-10-19 00:00:00 and ROW2 does not expire, the exported data looks like as follows:
    //ROW1
{
  "id" : 1,
  "name" : "abc",
  "_metadata" : {
    "expiration" : 1634601600000
  }
}

//ROW2
{
  "id" : 2,
  "name" : "xyz"
}
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "includeTTL" : true

Oracle NoSQL Database Cloud Service

The configuration file format for Oracle NoSQL Database Cloud Service as a source of NoSQL Database Migrator is shown below.

You can migrate a table from Oracle NoSQL Database Cloud Service by specifying the name or OCID of the compartment in which the table resides in the source configuration template.

A sample Oracle NoSQL Database Cloud Service table is as follows:
{"id":20,"firstName":"Jane","lastName":"Smith","otherNames":[{"first":"Jane","last":"teacher"}],"age":25,"income":55000,"address":{"city":"San Jose","number":201,"phones":[{"area":608,"kind":"work","number":6538955},{"area":931,"kind":"home","number":9533341},{"area":931,"kind":"mobile","number":9533382}],"state":"CA","street":"Atlantic Ave","zip":95005},"connections":[40,75,63],"expenses":null}
{"id":10,"firstName":"John","lastName":"Smith","otherNames":[{"first":"Johny","last":"chef"}],"age":22,"income":45000,"address":{"city":"Santa Cruz","number":101,"phones":[{"area":408,"kind":"work","number":4538955},{"area":831,"kind":"home","number":7533341},{"area":831,"kind":"mobile","number":7533382}],"state":"CA","street":"Pacific Ave","zip":95008},"connections":[30,55,43],"expenses":null}
{"id":30,"firstName":"Adam","lastName":"Smith","otherNames":[{"first":"Adam","last":"handyman"}],"age":45,"income":75000,"address":{"city":"Houston","number":301,"phones":[{"area":618,"kind":"work","number":6618955},{"area":951,"kind":"home","number":9613341},{"area":981,"kind":"mobile","number":9613382}],"state":"TX","street":"Indian Ave","zip":95075},"connections":[60,45,73],"expenses":null}

Source Configuration Template

"source" : {
  "type" : "nosqldb_cloud",
  "endpoint" : "<Oracle NoSQL Cloud Service Endpoint. You can either specify  the complete URL or the Region ID alone>",
  "table" : "<table name>",
  "compartment" : "<OCI compartment name or id>",
  "credentials" : "</path/to/oci/credential/file>",
  "credentialsProfile" : "<oci credentials profile name>",
  "readUnitsPercent" : <table readunits percent>,
  "requestTimeoutMs" : <timeout in milli seconds>,
  "useInstancePrincipal" : <true|false>,
  "includeTTL": <true|false>
}

Source Parameters

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "type" : "nosqldb_cloud"

endpoint

  • Purpose: Specifies the Service Endpoint of the Oracle NoSQL Database Cloud Service.

    You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:

    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://nosql.us-ashburn-1.oci.oraclecloud.com/"

table

  • Purpose: Name of the table from which to migrate the data.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:
    • To specify a table "table" : "myTable"
    • To specify a child table "table" : "mytable.child"

compartment

  • Purpose: Specifies the name or OCID of the compartment in which the table resides.

    If you do not provide any value, it defaults to the root compartment.

    You can find your compartment's OCID from the Compartment Explorer window under Governance in the OCI Cloud Console.

  • Data Type: string
  • Mandatory (Y/N): Yes, if the table is not in the root compartment of the tenancy OR when the useInstancePrincipal parameter is set to true.

    Note:

    If the useInstancePrincipal parameter is set to true, the compartment must specify the compartment OCID and not the name.
  • Example:

    • Compartment name

      "compartment" : "mycompartment"

    • Compartment name qualified with its parent compartment

      "compartment" : "parent.childcompartment"

    • No value provided. Defaults to the root compartment.

      "compartment": ""

    • Compartment OCID

      "compartment" : "ocid1.tenancy.oc1...4ksd"

credentials

  • Purpose: Absolute path to a file containing OCI credentials.

    If not specified, it defaults to $HOME/.oci/config

    See Example Configuration for an example of the credentials file.

    Note:

    Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"

credentialsProfile

  • Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

    If you do not specify this value, it defaults to the DEFAULT profile.

    Note:

    This parameter is valid ONLY if the credentials parameter is specified.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile": "ADMIN_USER"

readUnitsPercent

  • Purpose: Percentage of table read units to be used while migrating the NoSQL table.

    The default value is 90. The valid range is any integer between 1 to 100. Please note that amount of time required to migrate data is directly proportional to this attribute. It's better to increase the read throughput of the table for the migration activity. You can reduce the read throughput after the migration process completes.

    To learn the daily limits on throughput changes, see Cloud Limits in Using Oracle NoSQL Database Cloud Service.

    The default value is 90. The valid range is any integer between 1 to 100.

    Note:

    The time required for the data migration is directly proportional to the writeUnitsPercent value.

    See Troubleshooting the Oracle NoSQL Database Migrator to learn how to use this attribute to improve the data migration speed.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "readUnitsPercent" : 90

requestTimeoutMs

  • Purpose: Specifies the time to wait for each read operation in the sink to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer.

  • Data Type: integer
  • Mandatory (Y/N): N
  • Example: "requestTimeoutMs" : 5000

useInstancePrincipal

  • Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security.

    If not specified, it defaults to false.

    Note:

    • It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
    • Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
  • Data Type: boolean
  • Mandatory (Y/N): N

  • Example: "useInstancePrincipal" : true

includeTTL

  • Purpose: Specifies whether or not to include TTL metadata for table rows when exporting Oracle NoSQL Database tables. If set to true, the TTL data for rows also gets included in the data provided by the source. TTL is present in the _metadata JSON object associated with each row. The expiration time for each row gets exported as the number of milliseconds since the UNIX epoch (Jan 1st, 1970).

    If you do not specify this parameter, it defaults to false.

    Only the rows having a positive expiration value for TTL get included as part of the exported rows. If a row does not expire, which means TTL=0, then its TTL metadata is not included explicitly. For example, if ROW1 expires at 2021-10-19 00:00:00 and ROW2 does not expire, the exported data looks like as follows:
    //ROW1
{
  "id" : 1,
  "name" : "abc",
  "_metadata" : {
    "expiration" : 1634601600000
  }
}

//ROW2
{
  "id" : 2,
  "name" : "xyz"
}
  • Data Type: boolean
  • Mandatory (Y/N): N
  • Example: "includeTTL" : true

CSV as the File Source

The configuration file format for the CSV file as a source of NoSQL Database Migrator is shown below. The CSV file must conform to the RFC4180 format.

You can migrate a CSV file or a directory containing the CSV data by specifying the file name or directory in the source configuration template.

A sample CSV file is as follows:
1,"Computer Science","San Francisco","2500"
2,"Bio-Technology","Los Angeles","1200"
3,"Journalism","Las Vegas","1500"
4,"Telecommunication","San Francisco","2500"

Source Configuration Template

"source" : {
    "type" : "file",
    "format" : "csv",
    "dataPath": "</path/to/a/csv/file-or-directory>",
    "hasHeader" : <true | false>,
    "columns" : ["column1", "column2", ....],
    "csvOptions" : {         
                 "trim" : <true | false>,
                 "encoding" : "<character set encoding>"
    }
}

Source Parameters

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "type" : "file"

format

  • Purpose: Specifies the source format.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "format" : "csv"

datapath

  • Purpose: Specifies the absolute path to a file or directory containing the CSV data for migration. If you specify a directory, NoSQL Database Migrator imports all the files with the .csv or .CSV extension in that directory. All the CSV files are copied into a single table, but not in any particular order.

    CSV files must conform to the RFC4180 standard. You must ensure that the data in each CSV file matches with the NoSQL Database table schema defined in the sink table. Sub-directories are not supported.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:

    • Specifying a CSV file

      "dataPath" : "/home/user/sample.csv"

    • Specifying a directory

      "dataPath" : "/home/user"

Note:

The CSV files must contain only scalar values. Importing CSV files containing complex types such as MAP, RECORD, ARRAY, and JSON is not supported. The NoSQL Database Migrator tool does not check for the correctness of the data in the input CSV file. The NoSQL Database Migrator tool supports the importing of CSV data that conforms to the RFC4180 format. CSV files containing data that does not conform to the RFC4180 standard may not get copied correctly or may result in an error. If the input data is corrupted, the NoSQL Database Migrator tool will not parse the CSV records. If any errors are encountered during migration, the NoSQL Database Migrator tool logs the information about the failed input records for debugging and informative purposes. For more details, see Logging Migrator Progress in Using Oracle NoSQL Data Migrator.

hasHeader

  • Purpose: Specifies if the CSV file has a header or not. If this is set to true, the first line is ignored. If it is set to false, the first line is considered a CSV record. The default value is false.

  • Data Type: Boolean
  • Mandatory (Y/N): N

  • Example: "hasHeader" : "false"

columns

  • Purpose: Specifies the list of NoSQL Database table column names. The order of the column names indicates the mapping of the CSV file fields with corresponding NoSQL Database table columns. If the order of the input CSV file columns does not match the existing or newly created NoSQL Database table columns, you can map the ordering using this parameter. Also, when importing into a table that has an Identity Column, you can skip the Identity column name in the columns configuration.

    Note:

    • If the NoSQL Database table has additional columns that are not available in the CSV file, the values of the missing columns are updated with the default value as defined in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration. For more information on default values, see Data Type Definitions section in the SQL Reference Guide.
    • If the CSV file has additional columns that are not defined in the NoSQL Database table, the additional column information is ignored.
    • If any value in the CSV record is empty, it is set to the default value of the corresponding columns in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration.
  • Data Type: Array of Strings
  • Mandatory (Y/N): N

  • Example: "columns" : ["table_column_1", "table_column_2"]

csvOptions

  • Purpose: Specifies the formatting options for a CSV file. Provide the character set encoding format of the CSV file and choose whether or not to trim the blank spaces.

  • Data Type: Object
  • Mandatory (Y/N): N

csvOptions.trim

  • Purpose: Specifies if the leading and trailing blanks of a CSV field value must be trimmed. The default value is false.

  • Data Type: Boolean
  • Mandatory (Y/N): N

  • Example: "trim" : "true"

csvOptions.encoding

  • Purpose: Specifies the character set to decode the CSV file. The default value is UTF-8. The supported character sets are US-ASCII, ISO-8859-1, UTF-8, and UTF-16.

  • Data Type: String
  • Mandatory (Y/N): N

  • Example: "encoding" : "UTF-8"

CSV file in OCI Object Storage Bucket

The configuration file format for the CSV file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below. The CSV file must conform to the RFC4180 format.

You can migrate a CSV file in the OCI Object Storage bucket by specifying the name of the bucket in the source configuration template.

A sample CSV file in the OCI Object Storage bucket is as follows:
1,"Computer Science","San Francisco","2500"
2,"Bio-Technology","Los Angeles","1200"
3,"Journalism","Las Vegas","1500"
4,"Telecommunication","San Francisco","2500"

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Source Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "csv",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "namespace" : "<OCI Object Storage namespace>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>,
   "hasHeader" : <true | false>,
   "columns" : ["column1", "column2", ....],
   "csvOptions" : {         
                 "trim" : <true | false>,
                 "encoding" : "<character set encoding>"
   }
 }

Source Parameters

type

  • Purpose: Identifies the source type.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "type" : "object_storage_oci"

format

  • Purpose: Specifies the source format.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "format" : "csv"

endpoint

  • Purpose: Specifies the OCI Object Storage service endpoint URL or region ID.

    You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.

  • Data Type: string
  • Mandatory (Y/N): Y
  • Example:

    • Region ID: "endpoint" : "us-ashburn-1"

    • URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

namespace

  • Purpose: Specifies the OCI Object Storage service namespace. This is an optional parameter. If you don't specify this parameter, the default namespace of the tenancy is used.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "namespace" : "my-namespace"

bucket

  • Purpose: Specifies the name of the bucket, which contains the source CSV files. The NoSQL Database Migrator imports all the files with the .csv or .CSV extension object-wise and copies them into a single table in the same order.

    Ensure that the required bucket already exists in the OCI Object Storage instance and has read permissions.

  • Data Type: string
  • Mandatory (Y/N): Y

  • Example: "bucket" : "staging_bucket"

    Note:

    The CSV files must contain only scalar values. Importing CSV files containing complex types such as MAP, RECORD, ARRAY, and JSON is not supported. The NoSQL Database Migrator tool does not check for the correctness of the data in the input CSV file. The NoSQL Database Migrator tool supports the importing of CSV data that conforms to the RFC4180 format. CSV files containing data that does not conform to the RFC4180 standard may not get copied correctly or may result in an error. If the input data is corrupted, the NoSQL Database Migrator tool will not parse the CSV records. If any errors are encountered during migration, the NoSQL Database Migrator tool logs the information about the failed input records for debugging and informative purposes. For more details, see Logging Migrator Progress in Using Oracle NoSQL Data Migrator.

prefix

  • Purpose: Used for filtering the objects that are being migrated from the bucket. All the objects with the given prefix present in the bucket are migrated. For more information about prefixes, see Object Naming Using Prefixes and Hierarchies.

    If you do not provide any value, filter is not applied and all the objects present in the bucket are migrated.

  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "prefix" : "my_table/Data/000000.json" (migrates only 000000.json)
    2. "prefix" : "my_table/Data" (migrates all the objects with prefix my_table/Data)

credentials

  • Purpose: Absolute path to a file containing OCI credentials.

    If not specified, it defaults to $HOME/.oci/config

    See Example Configuration for an example of the credentials file.

    Note:

    You must specify either credentials or useInstancePrincipal parameters in the configuration template.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentials" : "/home/user/.oci/config"
    2. "credentials" : "/home/user/security/config"

credentialsProfile

  • Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a profile.

    If you do not specify this value, it defaults to the DEFAULT profile.

    Note:

    This parameter is valid only if the credentials parameter is specified.
  • Data Type: string
  • Mandatory (Y/N): N
  • Example:
    1. "credentialsProfile" : "DEFAULT"
    2. "credentialsProfile": "ADMIN_USER"

useInstancePrincipal

  • Purpose: Specifies whether or not the NoSQL Database Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security.

    The default value is false.

    Note:

    • The authentication with Instance Principals is supported only when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
    • You must specify either credentials or useInstancePrincipal parameters in the configuration template.
  • Data Type: boolean
  • Mandatory (Y/N): N

  • Example: "useInstancePrincipal" : true

hasHeader

  • Purpose: Specifies if the CSV file has a header or not. If this is set to true, the first line is ignored. If it is set to false, the first line is considered a CSV record. The default value is false.

  • Data Type: Boolean
  • Mandatory (Y/N): N

  • Example: "hasHeader" : "false"

columns

  • Purpose: Specifies the list of NoSQL Database table column names. The order of the column names indicates the mapping of the CSV file fields with corresponding NoSQL Database table columns. If the order of the input CSV file columns does not match the existing or newly created NoSQL Database table columns, you can map the ordering using this parameter. Also, when importing into a table that has an Identity Column, you can skip the Identity column name in the columns configuration.

    Note:

    • If the NoSQL Database table has additional columns that are not available in the CSV file, the values of the missing columns are updated with the default value as defined in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration. For more information on default values, see Data Type Definitions section in the SQL Reference Guide.
    • If the CSV file has additional columns that are not defined in the NoSQL Database table, the additional column information is ignored.
    • If any value in the CSV record is empty, it is set to the default value of the corresponding columns in the NoSQL Database table. If a default value is not provided, a Null value is inserted during migration.
  • Data Type: Array of Strings
  • Mandatory (Y/N): N

  • Example: "columns" : ["table_column_1", "table_column_2"]

csvOptions

  • Purpose: Specifies the formatting options for a CSV file. Provide the character set encoding format of the CSV file and choose whether or not to trim the blank spaces.

  • Data Type: Object
  • Mandatory (Y/N): N

csvOptions.trim

  • Purpose: Specifies if the leading and trailing blanks of a CSV field value must be trimmed. The default value is false.

  • Data Type: Boolean
  • Mandatory (Y/N): N

  • Example: "trim" : "true"

csvOptions.encoding

  • Purpose: Specifies the character set to decode the CSV file. The default value is UTF-8. The supported character sets are US-ASCII, ISO-8859-1, UTF-8, and UTF-16.

  • Data Type: String
  • Mandatory (Y/N): N

  • Example: "encoding" : "UTF-8"