Source Configuration Templates

Learn about the configuration file formats for each valid source and the purpose of each configuration parameter.

Topics

JSON File

The configuration file format for JSON File as a source of NoSQL Database Migrator is shown below.

Configuration Template

"source" : {
  "type" : "file",
  "format" : "json",
  "dataPath": "</path/to/a/json/file>",
  "schemaInfo": {
               "schemaPath": "</path/to/schema/file>"
  }
}

Source Parameters

type

Purpose: Identifies the source type.
Data Type: string
Mandatory (Y/N): Y
Example: "type" : "file"

format

Purpose: Specifies the source format.
Data Type: string
Mandatory (Y/N): Y
Example: "format" : "json"

dataPath

Purpose: Specifies the absolute path to a file or directory containing the JSON data for migration.

You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported.
Data Type: string
Mandatory (Y/N): Y
Example:
- Specifying a JSON file
  
  "dataPath" : "/home/user/sample.json"
- Specifying a directory
  
  "dataPath" : "/home/user"

schemaInfo

Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.
Data Type: Object
Mandatory (Y/N): N

schemaInfo.schemaPath

Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated.
Data Type: string
Mandatory (Y/N): Y

Example:

"schemaInfo" : {
            "schemaPath" : "/home/user/mytable/Schema/schema.ddl"
             }

JSON File in OCI Object Storage Bucket

The configuration file format for JSON file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below.

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "schemaInfo" : {
      "schemaObject" : "<object name>"
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

Source Parameters

type

Purpose: Identifies the source type.
Data Type: string
Mandatory (Y/N): Y
Example: "type" : "object_storage_oci"

format

Purpose: Specifies the source format.
Data Type: string
Mandatory (Y/N): Y
Example: "format" : "json"

endpoint

Purpose: Specifies the OCI Object Storage service endpoint URL or region ID.

You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.
Data Type: string
Mandatory (Y/N): Y
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

bucket

Purpose: Specifies the name of the bucket, which contains the source JSON files. Ensure that the required bucket already exists in the OCI Object Storage instance and has read permissions.
Data Type: string
Mandatory (Y/N): Y
Example: "bucket" : "staging_bucket"

prefix

Purpose: Used for filtering the objects that are being migrated from the bucket. All the objects with the given prefix present in the bucket are migrated. For more information about prefixes, see Object Naming Using Prefixes and Hierarchies.

If you do not provide any value, no filter is applied and all the objects present in the bucket are migrated.
Data Type: string
Mandatory (Y/N): N
Example:
1. "prefix" : "my_table/Data/000000.json" (migrates only 000000.json)
2. "prefix" : "my_table/Data" (migrates all the objects with prefix my_table/Data)

schemaInfo

Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.
Data Type: Object
Mandatory (Y/N): N

schemaInfo.schemaObject

Purpose: Specifies the name of the object in the bucket where NoSQL table schema definitions for the data being migrated are stored.
Data Type: string
Mandatory (Y/N): Y

Example:

"schemaInfo" : {
            "schemaObject" : "mytable/Schema/schema.ddl"
             }

credentials

Purpose: Absolute path to a file containing OCI credentials.

If not specified, it defaults to $HOME/.oci/config

See Example Configuration for an example of the credentials file.

Note:
Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
Data Type: string
Mandatory (Y/N): N
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"

credentialsProfile

Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

If you do not specify this value, it defaults to the DEFAULT profile.

Note:
This parameter is valid ONLY if the credentials parameter is specified.
Data Type: string
Mandatory (Y/N): N
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile": "ADMIN_USER"

useInstancePrincipal

Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security.

If not specified, it defaults to false.
Note:
- It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
- Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
Data Type: boolean
Mandatory (Y/N): N
Example: "useInstancePrincipal" : true

MongoDB-Formatted JSON File

The configuration file format for MongoDB-formatted JSON File as a source of NoSQL Database Migrator is shown below.

Configuration Template

"source" : {
    "type" : "file",
    "format" : "mongodb_json",
    "dataPath": "</path/to/a/json/file>",
    "schemaInfo": {
               "schemaPath": "</path/to/schema/file>"
    }
}

Source Parameters

type

Purpose: Identifies the source type.
Data Type: string
Mandatory (Y/N): Y
Example: "type" : "file"

format

Purpose: Specifies the source format.
Data Type: string
Mandatory (Y/N): Y
Example: "format" : "mongodb_json"

dataPath

Purpose: Specifies the absolute path to a file or directory containing the MongoDB exported JSON data for migration.

You must have generated these files using the mongoexport tool. See mongoexport for more information.

You can supply the MongoDB-formatted JSON file that is generated using the mongoexport tool in either canonical or relaxed mode. Both the modes are supported by the NoSQL Database Migrator for migration.

If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported. You must ensure that this data matches with the NoSQL table schema defined at the sink.
Data Type: string
Mandatory (Y/N): Y
Example:
- Specifying a MongoDB formatted JSON file
  
  "dataPath" : "/home/user/sample.json"
- Specifying a directory
  
  "dataPath" : "/home/user"

schemaInfo

Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.
Data Type: Object
Mandatory (Y/N): N

schemaInfo.schemaPath

Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated.
Data Type: string
Mandatory (Y/N): Y

Example:

"schemaInfo" : {
            "schemaPath" : "/home/user/mytable/Schema/schema.ddl"
             }

MongoDB-Formatted JSON File in OCI Object Storage bucket

The configuration file format for MongoDB-Formatted JSON file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below.

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Configuration Template

"source" : {
  "type" : "object_storage_oci",
  "format" : "mongodb_json",
  "endpoint" : "<OCI Object Storage service endpoint URL or region ID>",
  "bucket" : "<bucket name>",
  "prefix" : "<object prefix>",
  "schemaInfo" : {
      "schemaObject" : "<object name>"
  },
  "credentials" : "</path/to/oci/config/file>",
  "credentialsProfile" : "<profile name in oci config file>",
  "useInstancePrincipal" : <true|false>
}

Source Parameters

type

Purpose: Identifies the source type.
Data Type: string
Mandatory (Y/N): Y
Example: "type" : "object_storage_oci"

format

Purpose: Specifies the source format.
Data Type: string
Mandatory (Y/N): Y
Example: "format" : "mongodb_json"

endpoint

Purpose: Specifies the OCI Object Storage service endpoint URL or region ID.

You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.
Data Type: string
Mandatory (Y/N): Y
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://objectstorage.us-ashburn-1.oraclecloud.com"

bucket

Purpose: Specifies the name of the bucket, which contains the source MongoDB-Formatted JSON files. Ensure that the required bucket already exists in the OCI Object Storage instance and has read permissions.
Data Type: string
Mandatory (Y/N): Y
Example: "bucket" : "staging_bucket"

prefix

Purpose: Used for filtering the objects that are being migrated from the bucket. All the objects with the given prefix present in the bucket are migrated. For more information about prefixes, see Object Naming Using Prefixes and Hierarchies.

If you do not provide any value, no filter is applied and all the MongoDB JSON formatted objects present in the bucket are migrated. Extract the data from MongoDB using the mongoexport utility and upload it to the OCI Object Storage bucket. See mongoexport for more information.

If you do not provide any value, no filter is applied and all the objects present in the bucket are migrated.
Data Type: string
Mandatory (Y/N): N
Example:
1. "prefix" : "mongo_export/Data/table.json" (migrates only table.json)
2. "prefix" : "mongo_export/Data" (migrates all the objects with prefix mongo_export/Data)

schemaInfo

Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink.
Data Type: Object
Mandatory (Y/N): N

schemaInfo.schemaObject

Purpose: Specifies the name of the object in the bucket where NoSQL table schema definitions for the data being migrated are stored.
Data Type: string
Mandatory (Y/N): Y

Example:

"schemaInfo" : {
            "schemaObject" : "mytable/Schema/schema.ddl"
             }

credentials

Purpose: Absolute path to a file containing OCI credentials.

If not specified, it defaults to $HOME/.oci/config

See Example Configuration for an example of the credentials file.

Note:
Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
Data Type: string
Mandatory (Y/N): N
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"

credentialsProfile

Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

If you do not specify this value, it defaults to the DEFAULT profile.

Note:
This parameter is valid ONLY if the credentials parameter is specified.
Data Type: string
Mandatory (Y/N): N
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile": "ADMIN_USER"

useInstancePrincipal

Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security.

If not specified, it defaults to false.
Note:
- It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
- Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
Data Type: boolean
Mandatory (Y/N): N
Example: "useInstancePrincipal" : true

DynamoDB-Formatted JSON File stored in AWS S3

The configuration file format for DynamoDB-formatted JSON File in AWS S3 as a source of NoSQL Database Migrator is shown below.

The valid sink types for DynamoDB-formatted JSON stored in AWS S3 are nosqldb and nosqldb_cloud.

Configuration Template

"source" : {
    "type" : "aws_s3",
    "format" : "dynamodb_json",
    "s3URL" : "<S3 object url>",
    "credentials" : "</path/to/aws/credentials/file>",
    "credentialsProfile" : <"profile name in aws credentials file">
}

Source Parameters:

type
format
s3URL
credentials
credentialsProfile

type

Purpose:Identifies the source type.
Data Type: string
Mandatory (Y/N):Y
Example: "type" : "aws_s3"

format

Purpose:Specifies the source format.
Data Type: string
Mandatory (Y/N):Y
Example: "format" : "dynamodb_json"

Note:

If the value of the "type" is aws_s3, then format must be dynamodb_json.

s3URL

Purpose: Specifies the URL of an exported DynamoDB table stored in AWS S3. You can obtain this URL from the AWS S3 console. Valid URL format is https://<bucket-name>.<s3_endpoint>/<prefix>. The migrator will look for json.gz files in the prefix for import.

Note:
You must export DynamoDB table as specified in Exporting DynamoDB table data to Amazon S3.
Data Type: string
Mandatory: Yes
Example: https://my-bucket.s3.ap-south-1.amazonaws.com/AWSDynamoDB/01649660790057-14f642be

credentials

Purpose: Specifies the absolute path to a file containing the AWS credentials. If not specified, it defaults to $HOME/.aws/credentials. Please refer to Configuration and credential file settings for more details on the credentials file.
Data Type: string
Mandatory (Y/N):N

Example:

"credentials" : "/home/user/.aws/credentials"
"credentials" : "/home/user/security/credentials

Note:

The Migrator does not log any of the credentials information. You should properly protect the credentials file from unauthorized access.

credentialsProfile

Purpose: Name of the profile in the AWS credentials file to be used to connect to AWS S3. User account credentials are referred to as a profile. If you do not specify this value, it defaults to the default profile. Please refer to Configuration and credential file settings for more details on the credentials file.
Data Type: string
Mandatory (Y/N):N

Example:

"credentialsProfile" : "default"
        "credentialsProfile": "test"

DynamoDB-Formatted JSON File

The configuration file format for DynamoDB-formatted JSON File as a source of NoSQL Database Migrator is shown below.

The valid sink types for DynamoDB JSON file are nosqldb and nosqldb_cloud.

Configuration Template

"source" : {
    "type" : "file",
    "format" : "dynamodb_json",
    "dataPath" : "<path to a file or directory containing exported DDB table data>"   
}

Source Parameters:

type
format
dataPath

type

Purpose:Identifies the source type.
Data Type: string
Mandatory (Y/N):Y
Example: "type" : "file"

format

Purpose:Specifies the source format.
Data Type: string
Mandatory (Y/N):Y
Example: "format" : "dynamodb_json"

dataPath

Purpose: Specifies the absolute path to a file or directory containing the exported DynamoDB table data. You shall copy exported DynamoDB table data from AWS S3 to a local mounted file system. You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json.gz extension in that directory and the datasub-directory.
Data Type: string
Mandatory (Y/N):Y

Example:

Specifying a file

"dataPath" : "/home/user/AWSDynamoDB/01639372501551-bb4dd8c3/data/zclclwucjy6v5mkefvckxzhfvq.json.gz"

Specifying a directory
```
"dataPath" : "/home/user/AWSDynamoDB/01639372501551-bb4dd8c3"
```

Oracle NoSQL Database

The configuration file format for Oracle NoSQL Database as a source of NoSQL Database Migrator is shown below.

Configuration Template


"source" : {
    "type": "nosqldb",
    "table" : "<fully qualified table name>",
    "storeName" : "<store name>",
    "helperHosts" : ["hostname1:port1","hostname2:port2,..."],
    "security" : "</path/to/store/security/file>",
    "requestTimeoutMs" : 5000
}

Source Parameters

type

Purpose: Identifies the source type.
Data Type: string
Mandatory (Y/N): Y
Example: "type" : "nosqldb"

table

Purpose: Fully qualified table name from which to migrate the data.

Format: [namespace_name:]<table_name>

If the table is in the DEFAULT namespace, you can omit the namespace_name. The table must exist in the store.
Data Type: string
Mandatory (Y/N): Y
Example:
- With the DEFAULT namespace "table" :"mytable"
- With a non-default namespace "table" : "mynamespace:mytable"

storeName

Purpose: Name of the Oracle NoSQL Database store.
Data Type: string
Mandatory (Y/N): Y
Example: "storeName" : "kvstore"

helperHosts

Purpose: A list of host and registry port pairs in the hostname:port format. Delimit each item in the list using a comma. You must specify at least one helper host.
Data Type: array of strings
Mandatory (Y/N): Y
Example: "helperHosts" : ["localhost:5000","localhost:6000"]

security

Purpose:

If your store is a secure store, provide the absolute path to the security login file that contains your store credentials. See Configuring Security with Remote Access in Administrator's Guide to know more about the security login file.

You can use either password file based authentication or wallet based authentication. However, the wallet based authentication is supported only in the Enterprise Edition (EE) of Oracle NoSQL Database. For more information on wallet-based authentication, see Source and Sink Security.

The Community Edition(CE) edition supports password file based authentication only.
Data Type: string
Mandatory (Y/N): Y for a secure store

Example:

"security" : "/home/user/client.credentials"

Example security file content for password file based authentication:

oracle.kv.password.noPrompt=true
oracle.kv.auth.username=admin
oracle.kv.auth.pwdfile.file=/home/nosql/login.passwd
oracle.kv.transport=ssl
oracle.kv.ssl.trustStore=/home/nosql/client.trust
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

Example security file content for wallet based authentication:

oracle.kv.password.noPrompt=true
oracle.kv.auth.username=admin
oracle.kv.auth.wallet.dir=/home/nosql/login.wallet
oracle.kv.transport=ssl
oracle.kv.ssl.trustStore=/home/nosql/client.trust
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)

requestTimeoutMs

Purpose: Specifies the time to wait for each read operation from the store to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer.
Data Type: integer
Mandatory (Y/N): N
Example: "requestTimeoutMs" : 5000

Oracle NoSQL Database Cloud Service

The configuration file format for Oracle NoSQL Database Cloud Service as a source of NoSQL Database Migrator is shown below.

Configuration Template

"source" : {
  "type" : "nosqldb_cloud",
  "endpoint" : "<Oracle NoSQL Cloud Service Endpoint. You can either specify  the complete URL or the Region ID alone>",
  "table" : "<table name>",
  "compartment" : "<OCI compartment name or id>",
  "credentials" : "</path/to/oci/credential/file>",
  "credentialsProfile" : "<oci credentials profile name>",
  "readUnitsPercent" : <table readunits percent>,
  "requestTimeoutMs" : <timeout in milli seconds>,
  "useInstancePrincipal" : <true|false>
}

Source Parameters

type

Purpose: Identifies the source type.
Data Type: string
Mandatory (Y/N): Y
Example: "type" : "nosqldb_cloud"

endpoint

Purpose: Specifies the Service Endpoint of the Oracle NoSQL Database Cloud Service.

You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service.
Data Type: string
Mandatory (Y/N): Y
Example:
- Region ID: "endpoint" : "us-ashburn-1"
- URL format: "endpoint" : "https://nosql.us-ashburn-1.oci.oraclecloud.com/"

table

Purpose: Name of the table from which to migrate the data.
Data Type: string
Mandatory (Y/N): Y
Example: "table" :"myTable"

compartment

Purpose: Specifies the name or OCID of the compartment in which the table resides.

If you do not provide any value, it defaults to the root compartment.

You can find your compartment's OCID from the Compartment Explorer window under Governance in the OCI Cloud Console.
Data Type: string
Mandatory (Y/N): Yes, if the table is not in the root compartment of the tenancy OR when the useInstancePrincipal parameter is set to true.

Note:
If the useInstancePrincipal parameter is set to true, the compartment must specify the compartment OCID and not the name.
Example:
- Compartment name
  
  "compartment" : "mycompartment"
- Compartment name qualified with its parent compartment
  
  "compartment" : "parent.childcompartment"
- No value provided. Defaults to the root compartment.
  
  "compartment": ""
- Compartment OCID
  
  "compartment" : "ocid1.tenancy.oc1...4ksd"

credentials

Purpose: Absolute path to a file containing OCI credentials.

If not specified, it defaults to $HOME/.oci/config

See Example Configuration for an example of the credentials file.

Note:
Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
Data Type: string
Mandatory (Y/N): N
Example:
1. "credentials" : "/home/user/.oci/config"
2. "credentials" : "/home/user/security/config"

credentialsProfile

Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'.

If you do not specify this value, it defaults to the DEFAULT profile.

Note:
This parameter is valid ONLY if the credentials parameter is specified.
Data Type: string
Mandatory (Y/N): N
Example:
1. "credentialsProfile" : "DEFAULT"
2. "credentialsProfile": "ADMIN_USER"

readUnitsPercent

Purpose: Percentage of table read units to be used while migrating the NoSQL table.

The default value is 90. The valid range is any integer between 1 to 100. Please note that amount of time required to migrate data is directly proportional to this attribute. It's better to increase the read throughput of the table for the migration activity. You can reduce the read throughput after the migration process completes.

To learn the daily limits on throughput changes, see Cloud Limits in Using Oracle NoSQL Database Cloud Service.

The default value is 90. The valid range is any integer between 1 to 100.

Note:
The time required for the data migration is directly proportional to the writeUnitsPercent value.

See Troubleshooting the Oracle NoSQL Database Migrator to learn how to use this attribute to improve the data migration speed.
Data Type: integer
Mandatory (Y/N): N
Example: "readUnitsPercent" : 90

requestTimeoutMs

Purpose: Specifies the time to wait for each read operation in the sink to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer.
Data Type: integer
Mandatory (Y/N): N
Example: "requestTimeoutMs" : 5000

useInstancePrincipal

Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security

If not specified, it defaults to false.
Note:
- It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI.
- Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.
Data Type: boolean
Mandatory (Y/N): N
Example: "useInstancePrincipal" : true