1. Migrate data from Amazon DynamoDB to Oracle NoSQL Database Service
  2. Migrate Data from Amazon DynamoDB to Oracle NoSQL Database Cloud Service

Migrate Data from Amazon DynamoDB to Oracle NoSQL Database Cloud Service

There are two options to migrate data from Amazon DynamoDB to Oracle NoSQL Database Cloud Service using the Oracle NoSQL Database Migrator:

  • Using the Oracle NoSQL default schema
  • Using a custom schema

Both options require a configuration JSON file that includes information about the source files and the target database (sink). In both scenarios the source is Amazon DynamoDB formatted JSON files stored in Amazon S3 Object Storage, while the target is the Oracle NoSQL Database Cloud Service database. Since Amazon DynamoDB is a schema-less database, we recommend migrating using an Oracle NoSQL default schema.

Migrate Data Using the Oracle NoSQL Default Schema

When migrating using the default schema, a new Oracle NoSQL Database Cloud Service table is created with either two or three columns: partition key, sort key (optional), and document. The partition key and sort key are migrated from the Amazon DynamoDB table to the Oracle NoSQL Database Cloud Service table. Any remaining Amazon DynamoDB attributes are aggregated in JSON format in the document column.

The following steps describe how to migrate data using the Oracle NoSQL default schema.

  1. In the AWS console, create an access key (if one does not already exist) for the user who will access the Amazon S3 Object Storage where the Amazon DynamoDB data is being exported.
    We recommend creating a new user and creating a new access key for the user. Do not create an access key for the root Amazon account user. When creating an access key, a use case must be provided. Ensure to use the use case "Application running outside of AWS: You plan to use this access key to authenticate workloads running in your data center or other infrastructure outside of AWS that needs to access your AWS resources". For more information on creating a user or access keys, see the Review Documentation section.
    Note the access key and secret access key. These will not be displayed again.
  2. Create an Amazon S3 bucket if one does not already exist. Note the bucket name, region, and bucket URL.
    See "Creating a bucket" in the Review Documentation section for more information.
  3. Select the Amazon DynamoDB service and then select the Amazon DynamoDB table to be migrated to Oracle NoSQL Database Cloud Service.
    Under General Information, note the name and type of the partition key and sort key attributes.
  4. Click Export to S3 from the Amazon DynamoDB service menu, then click Export to S3.
  5. Select the name of the table to be exported, then select the name of the bucket where the data will be exported (the name of the bucket created in step 2).
  6. Configure Export Settings:
    • Select Full Export.
    • Select DynamoDB JSON for the exported file format.
    • Select the default Amazon S3 key for encryption.
    • Enable PITR (point in time recovery). Required for the table to be exported.
  7. Click Export.
    Note the Export ID while the export progresses. See "DynamoDB data export to Amazon S3: how it works" for more information about exporting.
  8. After the export is complete, navigate to the AWS console and select the Amazon S3 service.
  9. Select the bucket where the data was exported.
    A new directory was created within the bucket named AWSDynamoDB/ExportID. A subdirectory named data is created in this directory that contains a series of GZIP compressed JSON files with the exported Amazon DynamoDB table data. See "DynamoDB table export output format" in the Review Documentation section for more information about the formatted JSON files.
  10. In the OCI console, create an API key for the OCI user who will own the target Oracle NoSQL Database Cloud Service table. This user must have permissions to create and manage the Oracle NoSQL Database Cloud Service table.
    Note the user OCID, tenancy OCID, fingerprint, and OCI region ID. Save the private and public keys on PEM files. This API key is used by Oracle NoSQL Database Migrator to upload the Amazon DynamoDB formatted JSON files to the table.
  11. Create a compartment for the Oracle NoSQL Database Cloud Service table if one does not already exist.
    Note the compartment name.
  12. Create a compute instance.
    This compute instance will run the Oracle NoSQL Database Migrator software. An Always-Free Compute instance (VM.Standard.E4.Flex with 1 OCPU and 16GB memory) is sufficient and cost effective.
  13. Configure the compute instance with Oracle Linux 8.
  14. Use SSH to log in as the opc user to the new compute instance to install Java 11 as follows:
    [opc@oci-compute ~]$ sudo yum install java-11
  15. Download the Oracle NoSQL Database Migrator 1.5.0 binaries if you haven't already. See the Review Software Requirements section for details.
  16. Use SCP to upload the Oracle NoSQL Database Migrator ZIP file to the opc user's home directory in the new compute instance.
  17. Use SSH to log in as the opc user to the compute instance to unzip the Oracle NoSQL Database Migrator software downloaded in the previous step. The contents of the file is unzipped to a new directory named /home/opc/nosql-migrator-1.5.0. Unzip the file as follows:
    [opc@oci-compute ~]$ unzip V1033765-1.zip
[opc@oci-compute ~]$ ls -l
total 61964
drwxr-xr-x. 3 opc opc 4096 Feb 23 20:32 nosql-migrator-1.5.0
-rw-r--r--. 1 opc opc 63446080 Feb 14 19:26 V1033765-01.zip
[opc@jc-bastion-phx migrator]$
  18. Use SCP to upload the private key PEM file from step 10 to the home directory of the opc user of the compute instance where Oracle NoSQL Database Migrator was unzipped in the previous step.
  19. Use SSH to log in as the opc user to the compute instance to create an OCI identity file. Create the OCI identity file in the /home/opc/.oci directory with the following contents:
    [opc@oci-compute ~]$ cat .oci/config 
[DEFAULT]
user=user_OCID_from_step_10
fingerprint=fingerprint_from_step_10
tenancy=tenancy_OCID_from_step_10
region=region_ID_from_step_10
key_file=full_path_to_private_key_PEM_file_uploaded_step_18
  20. As the opc user in the compute instance, create AWS config and credentials files. Add these files in the /home/opc/.aws directory with the following contents:
    [opc@oci-compute .aws]$ cat config
[default]
region=bucket_region_from_step_2
[opc@oci-compute .aws]$ cat credentials
[default]
aws_access_key_id=aws_access_key_id_from_step_1
aws_secret_access_key=aws_access_key_secret_from_step_1
  21. In the compute instance, change the directory to the nosql-migrator-1.5.0 subdirectory. Create a migration configuration JSON file that includes information about the source and target (sink) as follows: 
    [opc@oci-compute nosql-migrator-1.5.0]$ cat migrator-config-dynamodb-bucket_custom-schema.json 
{
 "source" : {
 "type" : "aws_s3",
 "format" : "dynamodb_json",
 "s3URL" : "https://bucket_name_from_step_2.s3.bucket_region_from_step_2.amazonaws.c
om/AWSDynamoDB/export_ID/data",
 "credentials" : "/home/opc/.aws/credentials",
 "credentialsProfile" : "default"
 },
 "sink" : {
 "type" : "nosqldb_cloud",
 "endpoint" : "https://nosql.oci_region_ID_from_step_11.oci.oraclecloud.com",
 "table" : "new_Oracle_NoSQL_Database_Cloud_Service_table_name",
 "schemaInfo" : {
 "defaultSchema" : true,
 "DDBPartitionKey" : "partition_key_name_from_step_3:type_from_step_3",
 "DDBSortKey" : "sort_key_name_from_step_3:type_from_step_3",
 "readUnits" : 50,
 "writeUnits" : 50,
 "storageSize" : 25
 },
 "compartment" : "compartment_name_from_step_11",
 "includeTTL" : false,
 "credentials" : "/home/opc/.oci/config",
 "credentialsProfile" : "DEFAULT",
 "writeUnitsPercent" : 90,
 "overwrite" : true,
 "requestTimeoutMs" : 5000
 },
 "abortOnError" : false,
 "migratorVersion" : "1.5.0"
}
[opc@oci-compute nosql-migrator-1.5.0]$
  22. Run the data migration using Oracle NoSQL Database Migrator and the migration configuration JSON file created in the previous step as follows:
    [opc@oci-compute nosql-migrator-1.5.0]$ ./runMigrator --config ./migrator-config-dynamo-bucket_def_schema.json
    The migration job will create a new Oracle NoSQL Database Migrator table using the default schema, then read the Amazon DynamoDB formatted JSON file(s) from the Amazon S3 Object Storage Bucket, to then import the data from those files to the newly created table.

Migrate Data Using a Custom Schema

When using a custom schema to migrate, the new Oracle NoSQL Database Cloud Service table has to be defined similarly to the schema of the Amazon DynamoDB table, by using an Oracle NoSQL Database Cloud Service Table Data Definition Language (DDL). The attribute names and types for the new table should be the same as the attribute names and types on the Amazon DynamoDB table. The partition key and sort key have a limit of 2048 and 1024 bytes respectively, while in Oracle NoSQL Database Cloud Service these keys are limited to 64 bytes.

  1. In the AWS console, create an access key (if one does not already exist) for the user who will access the Amazon S3 Object Storage where the Amazon DynamoDB data is being exported.
    We recommend creating a new user and creating a new access key for the user. Do not create an access key for the root Amazon account user. When creating an access key, a use case must be provided. Ensure to use the use case "Application running outside of AWS: You plan to use this access key to authenticate workloads running in your data center or other infrastructure outside of AWS that needs to access your AWS resources". For more information on creating a user or access keys, refer to the Review Documentation section.
    Note the generated access key and secret access key, as these will not be displayed again.
  2. Create an Amazon S3 bucket if one does not already exist. Note the bucket name, region, and bucket URL.
    See "Creating a bucket" in the Review Documentation section for more information.
  3. Select the Amazon DynamoDB service and then select the Amazon DynamoDB table to be migrated to Oracle NoSQL Database Cloud Service.
    Under General Information, note the name and type of the partition key and sort key attributes. Note the other attributes in the table and their types.
  4. Click Export to S3 from the Amazon DynamoDB service menu, then click Export to S3.
  5. Select the name of the table to be exported, then select the name of the bucket where the data will be exported (the name of the bucket created in step 2).
  6. Configure Export Settings:
    • Select Full Export.
    • Select DynamoDB JSON for the exported file format.
    • Select the default Amazon S3 key for encryption.
    • Enable PITR (point in time recovery). Required for the table to be exported.
  7. Click Export.
    Note the Export ID while the export progresses. See "DynamoDB data export to Amazon S3: how it works" for more information about exporting.
  8. After the export is complete, navigate to the AWS console and select the Amazon S3 service.
  9. Select the bucket where the data was exported.
    A new directory was created within the bucket named AWSDynamoDB/ExportID. A subdirectory named data is created in this directory that contains a series of GZIP compressed JSON files with the exported Amazon DynamoDB table data. See "DynamoDB table export output format" in the Review Documentation section for more information about the formatted JSON files.
  10. In the OCI console, create an API key for the OCI user who will own the target Oracle NoSQL Database Cloud Service table. This user must have permissions to create and manage the Oracle NoSQL Database Cloud Service table.
    Note the user OCID, tenancy OCID, fingerprint, and OCI region ID. Save the private and public keys on PEM files. This API key is used by Oracle NoSQL Database Migrator to upload the Amazon DynamoDB formatted JSON files to the table.
  11. Create a compartment for the Oracle NoSQL Database Cloud Service table if one does not already exist.
    Note the compartment name.
  12. Create a compute instance.
    This compute instance will run the Oracle NoSQL Database Migrator software. An Always-Free Compute instance (VM.Standard.E4.Flex with 1 OCPU and 16GB memory) is sufficient and cost effective.
  13. Configure the compute instance with Oracle Linux 8.
  14. Use SSH to log in as the opc user to the new compute instance to install Java 11 as follows:
    [opc@oci-compute ~]$ sudo yum install java-11
  15. Download the Oracle NoSQL Database Migrator 1.5.0 binaries if you haven't already. See the Review Software Requirements section for details.
  16. Use SCP to upload the Oracle NoSQL Database Migrator ZIP file to the opc user's home directory in the new compute instance.
  17. Use SSH to log in as the opc user to the compute instance to unzip the Oracle NoSQL Database Migrator software downloaded in the previous step. The contents of the file is unzipped to a new directory named /home/opc/nosql-migrator-1.5.0. Unzip the file as follows:
    [opc@oci-compute ~]$ unzip V1033765-1.zip
[opc@oci-compute ~]$ ls -l
total 61964
drwxr-xr-x. 3 opc opc 4096 Feb 23 20:32 nosql-migrator-1.5.0
-rw-r--r--. 1 opc opc 63446080 Feb 14 19:26 V1033765-01.zip
[opc@jc-bastion-phx migrator]$
  18. Use SCP to upload the private key PEM file from step 10 to the home directory of the opc user of the compute instance where Oracle NoSQL Database Migrator was unzipped in the previous step.
  19. Use SSH to log in as the opc user to the compute instance to create an OCI identity file. Create the OCI identity file in the /home/opc/.oci directory with the following contents:
    [opc@oci-compute ~]$ cat .oci/config 
[DEFAULT]
user=user_OCID_from_step_10
fingerprint=fingerprint_from_step_10
tenancy=tenancy_OCID_from_step_10
region=region_ID_from_step_10
key_file=full_path_to_private_key_PEM_file_uploaded_step_18
  20. As the opc user in the compute instance, create AWS config and credentials files. Add these files in the /home/opc/.aws directory with the following contents:
    [opc@oci-compute .aws]$ cat config
[default]
region=bucket_region_from_step_2
[opc@oci-compute .aws]$ cat credentials
[default]
aws_access_key_id=aws_access_key_id_from_step_1
aws_secret_access_key=aws_access_key_secret_from_step_1
  21. Change the directory to the nosql-migrator-1.5.0 subdirectory, then create an Oracle NoSQL Database Cloud Service Data Definition Language (DDL) statement to create a new Oracle NoSQL Database Cloud Service table with similar attributes as those collected from the Amazon DynamoDB table in step 3. Here's an example of a create new Oracle NoSQL Database Cloud Service table DDL statement:
    [opc@oci-compute nosql-migrator-1.5.0]$ cat /home/opc/table_name.ddl 
CREATE TABLE IF NOT EXISTS table_name (partition_key_name_from_step_3 
type,sort_key_name_from_step_3 type, attribute_1 type, attribute_2 
type, PRIMARY 
KEY(SHARD(partition_key_name_from_step_3),sort_key_name_from_step_3))
  22. In the compute instance, change the directory to the nosql-migrator-1.5.0 subdirectory. Create a migration configuration JSON file that includes information about the source and target (sink) as follows: 
    [opc@oci-compute nosql-migrator-1.5.0]$ cat migrator-config-dynamodb-bucket_custom-schema.json 
{
  "source": {
    "type": "aws_s3",
    "format": "dynamodb_json",
    "s3URL": "https://bucket_name_from_step_2.s3.bucket_region_from_step_2.amazonaws.com/AWSDynamoDB/export_ID_from_step_7/data",
    "credentials": "/home/opc/.aws/credentials",
    "credentialsProfile": "default"
  },
  "sink": {
    "type": "nosqldb_cloud",
    "endpoint": "https://nosql.oci_region_ID_from_step_10.oci.oraclecloud.com",
    "table": "new_Oracle_NoSQL_Database_Cloud_Service_table_name",
    "schemaInfo": {
      "schemaPath": "/home/opc/nosql-migrator-1.5.0/table_name.ddl_from_step_21",
      "readUnits": 50,
      "writeUnits": 50,
      "storageSize": 25
    },
    "compartment": "compartment_name_from_step_11",
    "includeTTL": false,
    "credentials": "/home/opc/.oci/config",
    "credentialsProfile": "DEFAULT",
    "writeUnitsPercent": 90,
    "overwrite": true,
    "requestTimeoutMs": 5000
  },
  "abortOnError": false,
  "migratorVersion": "1.5.0"
}
  23. Run the data migration using Oracle NoSQL Database Migrator and the migration configuration JSON file created in the previous step as follows:
    [opc@oci-compute nosql-migrator-1.5.0]$ ./runMigrator --config ./migrator-config-dynamo-bucket_custom_schema.json
    The migration job will create a new Oracle NoSQL Database Migrator table using the custom schema DDL defined in step 21, then it will read the Amazon DynamoDB formatted JSON file(s) from the Amazon S3 Object Storage Bucket, to then import the data from those files to the newly created table.