1. Migrate data from Amazon DynamoDB to Oracle NoSQL Database Service
  2. Understand the Files Used During Migration and Sample Outputs

Understand the Files Used During Migration and Sample Outputs

Review the following Amazon DynamoDB example files and command outputs used during data migration.

About Amazon DynamoDB Formatted JSON Files

When importing data from Amazon DynamoDB to Oracle NoSQL Database Cloud Service using formatted JSON files, a JSON file will be generating with the migrated records. Here's an example Amazon DynamoDB formatted JSON file with 4,609 records: 

[opc@jc-bastion-phx nosql-migrator-1.5.0]$ head -74 ~/dynamo/moviesnb.json 
{
  "Item": {
    "title": {
      "S": "Fantasia"
    },
    "year": {
      "N": "1940"
    },
    "info": {
      "M": {
        "actors": {
          "L": [
            {
              "S": "Leopold Stokowski"
            },
            {
              "S": "Deems Taylor"
            },
            {
              "S": "Corey Burton"
            }
          ]
        },
        "release_date": {
          "S": "1940-11-13T00:00:00Z"
        },
        "plot": {
          "S": "A collection of animated interpretations of great works of Western classical music."
        },
        "genres": {
          "L": [
            {
              "S": "Animation"
            },
            {
              "S": "Family"
            },
            {
              "S": "Fantasy"
            },
            {
              "S": "Music"
            }
          ]
        },
        "image_url": {
          "S": "http://ia.media.imdb.com/images/M/MV5BMTM2Nzk5ODU5NF5BMl5BanBnXkFtZTcwMjQ1ODYxMQ@@._V1_SX400_.jpg"
        },
        "directors": {
          "L": [
            {
              "S": "James Algar"
            },
            {
              "S": "Samuel Armstrong"
            },
            {
              "S": "Ford Beebe Jr."
            }
          ]
        },
        "rating": {
          "N": "7.8"
        },
        "rank": {
          "N": "3865"
        },
        "running_time_secs": {
          "N": "7500"
        }
      }
    }
  }
}
[opc@oci-compute nosql-migrator-1.5.0]$ grep Item ~/dynamo/moviesnb.json | wc -l4609
[opc@oci-compute nosql-migrator-1.5.0]$

About Oracle NoSQL Database Cloud Service Data Definition Language Files

When migrating data using the Oracle NoSQL Database Cloud Service custom schema method, the new Oracle NoSQL Database Cloud Service table schema has to be defined similar to the schema of the source table by using a Data Definition Language (DDL). The attribute names and types for the new table should be the same as the attribute names and types on the source table.

Here's an example of an Oracle NoSQL Database Cloud Service sample schema data definition language file. 

[opc@oci-compute nosql-migrator-1.5.0]$ cat /home/opc/movies.ddl CREATE TABLE IF NOT EXISTS movies (title String,year Number,info JSON, PRIMARY KEY(SHARD(title),year))
[opc@oci-compute nosql-migrator-1.5.0]$

About OCI Identity Files

The OCI Identity File contains information about the user performing a migration to Oracle NoSQL Database Cloud Service. Here's a sample OCI Identity File: 

[opc@oci-compute nosql-migrator-1.5.0]$ cat /home/opc/.oci/config 
[DEFAULT]
user=ocid1.user.oc1..user_OCID
fingerprint=user_fingerprint
tenancy=ocid1.tenancy.oc1..tenancy_OCID
region=region_ID
key_file=PEM_file_path
[opc@oci-compute nosql-migrator-1.5.0]$

About Amazon DynamoDB Configuration Files

The Amazon DynamoDB configuration file contains information about the Amazon DynamoDB account. Here's an example: 

[opc@oci-compute nosql-migrator-1.5.0]$ cat /home/opc/.aws/config 
[default]
region=region_ID
[opc@oci-compute nosql-migrator-1.5.0]$

About Amazon DynamoDB Credential Files

Amazon DynamoDB credential files contains information about the Amazon DynamoDB. Here's an example Amazon DynamoDB credential file: 

[opc@oci-compute nosql-migrator-1.5.0]$ cat /home/opc/.aws/credentials 
[default]
aws_access_key_id = AWS_access_key_ID
aws_secret_access_key = AWS_secret_access_key
[opc@oci-compute nosql-migrator-1.5.0]$

About Default Schema Oracle NoSQL Database Migrations Output

Migrating to Amazon DynamoDB generates an output file with the JSON schema of the configuration. Here's an example output from a default schema Oracle NoSQL Database Migration: 

[opc@oci-compute nosql-migrator-1.5.0]$ ./runMigrator --config ./migrator-config-dynamodb-bucket_def_schema.json 
2023-12-06 05:53:31.389 [INFO] Configuration for migration:
{
 "source" : {
 "type" : "aws_s3",
 "format" : "dynamodb_json",
 "s3URL" : "https://bucket_URL.s3.us-west-2.amazonaws.com/AWSDynamoDB/xyz/data",
 "credentials" : "/home/opc/.aws/credentials",
 "credentialsProfile" : "default"
 },
 "sink" : {
 "type" : "nosqldb_cloud",
 "endpoint" : "https://endpoint_URL.us-phoenix-1.oci.oraclecloud.com",
 "table" : "movies",
 "schemaInfo" : {
 "defaultSchema" : true,
 "DDBPartitionKey" : "title:String",
 "DDBSortKey" : "year:Number",
 "readUnits" : 50,
 "writeUnits" : 50,
 "storageSize" : 25
 },
 "compartment" : "compartment_name",
 "includeTTL" : false,
 "credentials" : "/home/opc/.oci/config",
 "credentialsProfile" : "DEFAULT",
 "writeUnitsPercent" : 90,
 "overwrite" : true,
 "requestTimeoutMs" : 5000
 },
 "abortOnError" : false,
 "migratorVersion" : "1.5.0"
}
2023-12-06 05:53:31.392 [INFO] creating source from given configuration:
2023-12-06 05:53:36.487 [INFO] source creation completed
2023-12-06 05:53:36.487 [INFO] creating sink from given configuration:
2023-12-06 05:53:40.280 [INFO] sink creation completed
2023-12-06 05:53:40.281 [INFO] creating migrator pipeline
2023-12-06 05:53:40.281 [INFO] migration started
2023-12-06 05:53:40.281 [INFO] [cloud sink] : start loading DDLs
2023-12-06 05:53:40.288 [INFO] [cloud sink] : executing DDL: CREATE TABLE IF NOT EXISTS movies (title String,year Number,document JSON, PRIMARY KEY(SHARD(title),year)),limits: [50, 50, 25]
2023-12-06 05:53:45.494 [INFO] [cloud sink] : completed loading DDLs
2023-12-06 05:53:45.800 [INFO] [cloud sink] : start loading records
2023-12-06 05:53:45.803 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/abc.json.gz
2023-12-06 05:53:51.085 [WARN] failed to write record to the sink table {"title":"Borat: Cultural Learnings of America for Make Benefit Glorious Nation .… Primary key of 88 exceeded the limit of 64
2023-12-06 05:53:55.313 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/cde.json.gz
2023-12-06 05:54:10.201 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/efg.json.gz
2023-12-06 05:54:24.480 [WARN] failed to write record to the sink table {"title":"Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb",.… Primary key of 73 exceeded the limit of 64
2023-12-06 05:54:24.934 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/ghi.json.gz
2023-12-06 05:54:26.479 [WARN] failed to write record to the sink table {"title":"Don't Be a Menace to South Central While Drinking Your Juice in the Ho.… Primary key of 77 exceeded the limit of 64
2023-12-06 05:54:30.403 [WARN] failed to write record to the sink table {"title":"The Chronicles of Narnia: The Lion, the Witch and the Wardrobe","year".… Primary key of 67 exceeded the limit of 64
2023-12-06 05:54:39.401 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/123.json.gz
2023-12-06 05:54:53.579 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/456.json.gz
2023-12-06 05:55:08.177 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/789.json.gz
2023-12-06 05:55:22.603 [WARN] failed to write record to the sink table {"title":"Those Magnificent Men in Their Flying Machines or How I Flew from Lond.… Primary key of 109 exceeded the limit of 64
2023-12-06 05:55:22.688 [INFO] migration completed. Records provided by source=4,609, Records written to sink=4,604, Records failed=5, Records skipped=0. Elapsed time: 1min 42sec 400ms
Migration completed.
[opc@oci-compute nosql-migrator-1.5.0]$

About Custom Schema Oracle NoSQL Database Migration Output

Migrating to Amazon DynamoDB generates an output file with the JSON schema of the configuration. Here's an example output from a custom schema Oracle NoSQL Database Migration: 

opc@oci-compute nosql-migrator-1.5.0]$ ./runMigrator --config ./migrator-config-dynamodb-bucket_custom-schema.json 
2023-12-06 17:38:12.604 [INFO] Configuration for migration:
{
 "source" : {
 "type" : "aws_s3",
 "format" : "dynamodb_json",
 "s3URL" : "https://bucket_URL.amazonaws.com/AWSDynamoDB/xyz/data",
 "credentials" : "/home/opc/.aws/credentials",
 "credentialsProfile" : "default"
 },
 "sink" : {
 "type" : "nosqldb_cloud",
 "endpoint" : "https://endpoint_URL.oci.oraclecloud.com",
 "table" : "movies",
 "schemaInfo" : {
 "schemaPath" : "/home/opc/movies.ddl",
 "readUnits" : 50,
 "writeUnits" : 50,
 "storageSize" : 25
 },
 "compartment" : "compartment_name",
 "includeTTL" : false,
 "credentials" : "/home/opc/.oci/config",
 "credentialsProfile" : "DEFAULT",
 "writeUnitsPercent" : 90,
 "overwrite" : true,
 "requestTimeoutMs" : 5000
 },
 "abortOnError" : false,
 "migratorVersion" : "1.5.0"
}
2023-12-06 17:38:12.680 [INFO] creating source from given configuration:
2023-12-06 17:38:17.488 [INFO] source creation completed
2023-12-06 17:38:17.489 [INFO] creating sink from given configuration:
2023-12-06 17:38:21.587 [INFO] sink creation completed
2023-12-06 17:38:21.588 [INFO] creating migrator pipeline
2023-12-06 17:38:21.588 [INFO] migration started
2023-12-06 17:38:21.588 [INFO] [cloud sink] : start loading DDLs
2023-12-06 17:38:21.595 [INFO] [cloud sink] : executing DDL: CREATE TABLE IF NOT EXISTS movies (title String,year Number,info JSON, PRIMARY KEY(SHARD(title),year)),limits: [50, 50, 25]
2023-12-06 17:38:26.793 [INFO] [cloud sink] : completed loading DDLs
2023-12-06 17:38:27.091 [INFO] [cloud sink] : start loading records
2023-12-06 17:38:27.093 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/abc.json.gz
2023-12-06 17:38:32.385 [WARN] failed to write record to the sink table{"year":2006,"title":"Borat: Cultural Learnings of America for Make Benefit Glor....Primary key of 88 exceeded the limit of 64
2023-12-06 17:38:36.613 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/cde.json.gz
2023-12-06 17:38:51.501 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/efg.json.gz
2023-12-06 17:39:05.777 [WARN] failed to write record to the sink table{"year":1964,"title":"Dr. Strangelove or: How I Learned to Stop Worrying and Lov....Primary key of 73 exceeded the limit of 64
2023-12-06 17:39:06.235 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/ghi.json.gz
2023-12-06 17:39:07.703 [WARN] failed to write record to the sink table{"year":1996,"title":"Don't Be a Menace to South Central While Drinking Your Jui....Primary key of 77 exceeded the limit of 64
2023-12-06 17:39:11.703 [WARN] failed to write record to the sink table{"year":2005,"title":"The Chronicles of Narnia: The Lion, the Witch and the Ward....Primary key of 67 exceeded the limit of 64
2023-12-06 17:39:20.701 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/123.json.gz
2023-12-06 17:39:34.881 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/456.json.gz
2023-12-06 17:39:49.477 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/678.json.gz
2023-12-06 17:40:03.903 [WARN] failed to write record to the sink table{"year":1965,"title":"Those Magnificent Men in Their Flying Machines or How I Fl....Primary key of 109 exceeded the limit of 64
2023-12-06 17:40:03.988 [INFO] migration completed.Records provided by source=4,609, Records written to sink=4,604, Records failed=5, Records skipped=0.
Elapsed time: 1min 42sec 393ms
Migration completed.
[opc@oci-compute nosql-migrator-1.5.0]$