1. Amazon DynamoDBからOracle NoSQL Database Serviceへのデータの移行
  2. 移行時およびサンプル出力時に使用されるファイルの理解

移行時およびサンプル出力時に使用されるファイルの理解

データ移行中に使用される次のAmazon DynamoDBサンプル・ファイルおよびコマンド出力を確認します。

Amazon DynamoDBフォーマット済JSONファイルについて

フォーマットされたJSONファイルを使用してAmazon DynamoDBからOracle NoSQL Database Cloud Serviceにデータをインポートすると、移行されたレコードを使用してJSONファイルが生成されます。次に、4,609レコードを含むAmazon DynamoDB形式のJSONファイルの例を示します: 

[opc@jc-bastion-phx nosql-migrator-1.5.0]$ head -74 ~/dynamo/moviesnb.json 
{
  "Item": {
    "title": {
      "S": "Fantasia"
    },
    "year": {
      "N": "1940"
    },
    "info": {
      "M": {
        "actors": {
          "L": [
            {
              "S": "Leopold Stokowski"
            },
            {
              "S": "Deems Taylor"
            },
            {
              "S": "Corey Burton"
            }
          ]
        },
        "release_date": {
          "S": "1940-11-13T00:00:00Z"
        },
        "plot": {
          "S": "A collection of animated interpretations of great works of Western classical music."
        },
        "genres": {
          "L": [
            {
              "S": "Animation"
            },
            {
              "S": "Family"
            },
            {
              "S": "Fantasy"
            },
            {
              "S": "Music"
            }
          ]
        },
        "image_url": {
          "S": "http://ia.media.imdb.com/images/M/MV5BMTM2Nzk5ODU5NF5BMl5BanBnXkFtZTcwMjQ1ODYxMQ@@._V1_SX400_.jpg"
        },
        "directors": {
          "L": [
            {
              "S": "James Algar"
            },
            {
              "S": "Samuel Armstrong"
            },
            {
              "S": "Ford Beebe Jr."
            }
          ]
        },
        "rating": {
          "N": "7.8"
        },
        "rank": {
          "N": "3865"
        },
        "running_time_secs": {
          "N": "7500"
        }
      }
    }
  }
}
[opc@oci-compute nosql-migrator-1.5.0]$ grep Item ~/dynamo/moviesnb.json | wc -l4609
[opc@oci-compute nosql-migrator-1.5.0]$

Oracle NoSQL Database Cloud Serviceのデータ定義言語ファイルについて

Oracle NoSQL Database Cloud Serviceカスタム・スキーマ・メソッドを使用してデータを移行する場合、新しいOracle NoSQL Database Cloud Service表スキーマは、データ定義言語(DDL)を使用してソース表のスキーマと同様に定義する必要があります。新しい表の属性名およびタイプは、ソース表の属性名およびタイプと同じである必要があります。

次に、Oracle NoSQL Database Cloud Serviceのサンプル・スキーマ・データ定義言語ファイルの例を示します。 

[opc@oci-compute nosql-migrator-1.5.0]$ cat /home/opc/movies.ddl CREATE TABLE IF NOT EXISTS movies (title String,year Number,info JSON, PRIMARY KEY(SHARD(title),year))
[opc@oci-compute nosql-migrator-1.5.0]$

OCIアイデンティティ・ファイルについて

OCIアイデンティティ・ファイルには、Oracle NoSQL Database Cloud Serviceへの移行を実行するユーザーに関する情報が含まれます。次に、OCIアイデンティティ・ファイルの例を示します。 

[opc@oci-compute nosql-migrator-1.5.0]$ cat /home/opc/.oci/config 
[DEFAULT]
user=ocid1.user.oc1..user_OCID
fingerprint=user_fingerprint
tenancy=ocid1.tenancy.oc1..tenancy_OCID
region=region_ID
key_file=PEM_file_path
[opc@oci-compute nosql-migrator-1.5.0]$

Amazon DynamoDB構成ファイルについて

Amazon DynamoDB構成ファイルには、Amazon DynamoDBアカウントに関する情報が含まれています。次に例を示します: 

[opc@oci-compute nosql-migrator-1.5.0]$ cat /home/opc/.aws/config 
[default]
region=region_ID
[opc@oci-compute nosql-migrator-1.5.0]$

Amazon DynamoDB資格証明ファイルについて

Amazon DynamoDB資格証明ファイルには、Amazon DynamoDBに関する情報が含まれます。次に、Amazon DynamoDB資格証明ファイルの例を示します: 

[opc@oci-compute nosql-migrator-1.5.0]$ cat /home/opc/.aws/credentials 
[default]
aws_access_key_id = AWS_access_key_ID
aws_secret_access_key = AWS_secret_access_key
[opc@oci-compute nosql-migrator-1.5.0]$

デフォルト・スキーマのOracle NoSQL Database移行の出力について

Amazon DynamoDBに移行すると、構成のJSONスキーマを含む出力ファイルが生成されます。デフォルト・スキーマのOracle NoSQL Database移行からの出力例を次に示します: 

[opc@oci-compute nosql-migrator-1.5.0]$ ./runMigrator --config ./migrator-config-dynamodb-bucket_def_schema.json 
2023-12-06 05:53:31.389 [INFO] Configuration for migration:
{
 "source" : {
 "type" : "aws_s3",
 "format" : "dynamodb_json",
 "s3URL" : "https://bucket_URL.s3.us-west-2.amazonaws.com/AWSDynamoDB/xyz/data",
 "credentials" : "/home/opc/.aws/credentials",
 "credentialsProfile" : "default"
 },
 "sink" : {
 "type" : "nosqldb_cloud",
 "endpoint" : "https://endpoint_URL.us-phoenix-1.oci.oraclecloud.com",
 "table" : "movies",
 "schemaInfo" : {
 "defaultSchema" : true,
 "DDBPartitionKey" : "title:String",
 "DDBSortKey" : "year:Number",
 "readUnits" : 50,
 "writeUnits" : 50,
 "storageSize" : 25
 },
 "compartment" : "compartment_name",
 "includeTTL" : false,
 "credentials" : "/home/opc/.oci/config",
 "credentialsProfile" : "DEFAULT",
 "writeUnitsPercent" : 90,
 "overwrite" : true,
 "requestTimeoutMs" : 5000
 },
 "abortOnError" : false,
 "migratorVersion" : "1.5.0"
}
2023-12-06 05:53:31.392 [INFO] creating source from given configuration:
2023-12-06 05:53:36.487 [INFO] source creation completed
2023-12-06 05:53:36.487 [INFO] creating sink from given configuration:
2023-12-06 05:53:40.280 [INFO] sink creation completed
2023-12-06 05:53:40.281 [INFO] creating migrator pipeline
2023-12-06 05:53:40.281 [INFO] migration started
2023-12-06 05:53:40.281 [INFO] [cloud sink] : start loading DDLs
2023-12-06 05:53:40.288 [INFO] [cloud sink] : executing DDL: CREATE TABLE IF NOT EXISTS movies (title String,year Number,document JSON, PRIMARY KEY(SHARD(title),year)),limits: [50, 50, 25]
2023-12-06 05:53:45.494 [INFO] [cloud sink] : completed loading DDLs
2023-12-06 05:53:45.800 [INFO] [cloud sink] : start loading records
2023-12-06 05:53:45.803 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/abc.json.gz
2023-12-06 05:53:51.085 [WARN] failed to write record to the sink table {"title":"Borat: Cultural Learnings of America for Make Benefit Glorious Nation .… Primary key of 88 exceeded the limit of 64
2023-12-06 05:53:55.313 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/cde.json.gz
2023-12-06 05:54:10.201 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/efg.json.gz
2023-12-06 05:54:24.480 [WARN] failed to write record to the sink table {"title":"Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb",.… Primary key of 73 exceeded the limit of 64
2023-12-06 05:54:24.934 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/ghi.json.gz
2023-12-06 05:54:26.479 [WARN] failed to write record to the sink table {"title":"Don't Be a Menace to South Central While Drinking Your Juice in the Ho.… Primary key of 77 exceeded the limit of 64
2023-12-06 05:54:30.403 [WARN] failed to write record to the sink table {"title":"The Chronicles of Narnia: The Lion, the Witch and the Wardrobe","year".… Primary key of 67 exceeded the limit of 64
2023-12-06 05:54:39.401 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/123.json.gz
2023-12-06 05:54:53.579 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/456.json.gz
2023-12-06 05:55:08.177 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/789.json.gz
2023-12-06 05:55:22.603 [WARN] failed to write record to the sink table {"title":"Those Magnificent Men in Their Flying Machines or How I Flew from Lond.… Primary key of 109 exceeded the limit of 64
2023-12-06 05:55:22.688 [INFO] migration completed. Records provided by source=4,609, Records written to sink=4,604, Records failed=5, Records skipped=0. Elapsed time: 1min 42sec 400ms
Migration completed.
[opc@oci-compute nosql-migrator-1.5.0]$

カスタム・スキーマのOracle NoSQL Database移行出力について

Amazon DynamoDBに移行すると、構成のJSONスキーマを含む出力ファイルが生成されます。カスタム・スキーマのOracle NoSQL Database移行からの出力例を次に示します: 

opc@oci-compute nosql-migrator-1.5.0]$ ./runMigrator --config ./migrator-config-dynamodb-bucket_custom-schema.json 
2023-12-06 17:38:12.604 [INFO] Configuration for migration:
{
 "source" : {
 "type" : "aws_s3",
 "format" : "dynamodb_json",
 "s3URL" : "https://bucket_URL.amazonaws.com/AWSDynamoDB/xyz/data",
 "credentials" : "/home/opc/.aws/credentials",
 "credentialsProfile" : "default"
 },
 "sink" : {
 "type" : "nosqldb_cloud",
 "endpoint" : "https://endpoint_URL.oci.oraclecloud.com",
 "table" : "movies",
 "schemaInfo" : {
 "schemaPath" : "/home/opc/movies.ddl",
 "readUnits" : 50,
 "writeUnits" : 50,
 "storageSize" : 25
 },
 "compartment" : "compartment_name",
 "includeTTL" : false,
 "credentials" : "/home/opc/.oci/config",
 "credentialsProfile" : "DEFAULT",
 "writeUnitsPercent" : 90,
 "overwrite" : true,
 "requestTimeoutMs" : 5000
 },
 "abortOnError" : false,
 "migratorVersion" : "1.5.0"
}
2023-12-06 17:38:12.680 [INFO] creating source from given configuration:
2023-12-06 17:38:17.488 [INFO] source creation completed
2023-12-06 17:38:17.489 [INFO] creating sink from given configuration:
2023-12-06 17:38:21.587 [INFO] sink creation completed
2023-12-06 17:38:21.588 [INFO] creating migrator pipeline
2023-12-06 17:38:21.588 [INFO] migration started
2023-12-06 17:38:21.588 [INFO] [cloud sink] : start loading DDLs
2023-12-06 17:38:21.595 [INFO] [cloud sink] : executing DDL: CREATE TABLE IF NOT EXISTS movies (title String,year Number,info JSON, PRIMARY KEY(SHARD(title),year)),limits: [50, 50, 25]
2023-12-06 17:38:26.793 [INFO] [cloud sink] : completed loading DDLs
2023-12-06 17:38:27.091 [INFO] [cloud sink] : start loading records
2023-12-06 17:38:27.093 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/abc.json.gz
2023-12-06 17:38:32.385 [WARN] failed to write record to the sink table{"year":2006,"title":"Borat: Cultural Learnings of America for Make Benefit Glor....Primary key of 88 exceeded the limit of 64
2023-12-06 17:38:36.613 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/cde.json.gz
2023-12-06 17:38:51.501 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/efg.json.gz
2023-12-06 17:39:05.777 [WARN] failed to write record to the sink table{"year":1964,"title":"Dr. Strangelove or: How I Learned to Stop Worrying and Lov....Primary key of 73 exceeded the limit of 64
2023-12-06 17:39:06.235 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/ghi.json.gz
2023-12-06 17:39:07.703 [WARN] failed to write record to the sink table{"year":1996,"title":"Don't Be a Menace to South Central While Drinking Your Jui....Primary key of 77 exceeded the limit of 64
2023-12-06 17:39:11.703 [WARN] failed to write record to the sink table{"year":2005,"title":"The Chronicles of Narnia: The Lion, the Witch and the Ward....Primary key of 67 exceeded the limit of 64
2023-12-06 17:39:20.701 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/123.json.gz
2023-12-06 17:39:34.881 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/456.json.gz
2023-12-06 17:39:49.477 [INFO] [DDB S3 source] : start parsing JSON records from object: AWSDynamoDB/xyz/data/678.json.gz
2023-12-06 17:40:03.903 [WARN] failed to write record to the sink table{"year":1965,"title":"Those Magnificent Men in Their Flying Machines or How I Fl....Primary key of 109 exceeded the limit of 64
2023-12-06 17:40:03.988 [INFO] migration completed.Records provided by source=4,609, Records written to sink=4,604, Records failed=5, Records skipped=0.
Elapsed time: 1min 42sec 393ms
Migration completed.
[opc@oci-compute nosql-migrator-1.5.0]$