Migrate Data from MongoDB to Oracle NoSQL Database Cloud Service

There are two options to migrate data from MongoDB to Oracle NoSQL Database Cloud Service using the Oracle NoSQL Database Migrator:

Using the Oracle NoSQL default schema
Using a custom schema

Both options require a configuration JSON file that includes information about the source files and the target database (sink). In both scenarios the source is MongoDB formatted JSON files stored in OCI Object Storage, while the target is the Oracle NoSQL Database Cloud Service database. The configuration JSON file must include the transforms section when using a custom schema.

Migrate Data Using the Oracle NoSQL Default Schema

When migrating using the default schema, a new Oracle NoSQL Database Cloud Service table is created with two columns: id, and document. For each entry in the MongoDB JSON export file, the _id column is migrated to id in the Oracle NoSQL Database Cloud Service table, while the other columns of the MongoDB JSON export file are aggregated in the new document column.

The following steps describe how to migrate data using the Oracle NoSQL default schema.

Log in to MongoDB and run mongoexport to export the data collection to MongoDB formatted JSON files. For example:
```
[opc@mongodb-server ~] mongoexport --collection collection_name --out collection_name.json
```
Split MongoDB JSON files with more than 13,500 records using the Oracle Linux split utility to divide the MongoDB JSON file into multiple JSON files of 13,500 records each. For example:
```
$split -l 13500 –additional-suffix .json collection_name.json collection_name
```
Log in to the OCI console to create an OCI Object Storage Bucket if one does not already exist.
Note the bucket name and namespace
Use the OCI console or REST API to upload the file(s) from step 2 to the bucket from step 3.
In the OCI console, create an API key for the OCI user who will own the target Oracle NoSQL Database Cloud Service table. This user must have permissions to create and manage the table.
Note the user OCID, tenancy OCID, fingerprint, and OCI region ID. Save the private and public keys on PEM files. This API key is used by Oracle NoSQL Database Migrator to retrieve the MongoDB formatted JSON files, create a new table, and then upload the data from the MongoDB JSON files to the table.
Create a compartment for the Oracle NoSQL Database Cloud Service table if one does not already exist.
Note the compartment name.
Create a compute instance.
This compute instance will run the Oracle NoSQL Database Migrator software. An Always-Free Compute instance (VM.Standard.E4.Flex with 1 OCPU and 16GB memory) is sufficient and cost effective.
Use SSH to log in as the opc user to the new compute instance to install Java 11 as follows:
```
[opc@oci-compute ~]$ sudo yum install java-11
```
Download the Oracle NoSQL Database Migrator 1.5.0 binaries if you haven't already. See the Review Software Requirements section for details.
Use SCP to upload the Oracle NoSQL Database Migrator ZIP file to the opc user's home directory in the new compute instance.
Use SSH to log in as the opc user to the compute instance to unzip the Oracle NoSQL Database Migrator software downloaded in the previous step. The contents of the file is unzipped to a new directory named /home/opc/nosql-migrator-1.5.0. Unzip the file as follows:
```
[opc@oci-compute ~]$ unzip V1033765-1.zip
[opc@oci-compute ~]$ ls -l
total 61964
drwxr-xr-x. 3 opc opc 4096 Feb 23 20:32 nosql-migrator-1.5.0
-rw-r--r--. 1 opc opc 63446080 Feb 14 19:26 V1033765-01.zip
[opc@jc-bastion-phx migrator]$
```
Use SCP to upload the private key PEM file from step 5 to the home directory of the opc user of the compute instance where Oracle NoSQL Database Migrator was unzipped in the previous step.

Use SSH to log in as the opc user to the compute instance to create an OCI identity file. Create the OCI identity file in the /home/opc/.oci directory with the following contents:

[opc@oci-compute ~]$ cat .oci/config 
[DEFAULT]
user=user_OCID_from_step_5
fingerprint=fingerprint_from_step_5
tenancy=tenancy_OCID_from_step_5
region=region_ID_from_step_5
key_file=full_path_to_private_key_PEM_file_uploaded_step_12

In the compute instance, change the directory to the nosql-migrator-1.5.0 subdirectory. Create a migration configuration JSON file that includes information about the source and target (sink) as follows:

[opc@oci-compute nosql-migrator-1.5.0]$ cat migrator-config-mongodbjson-bucket_def-schema.json 
{
   "source":{
      "type":"object_storage_oci",
      "format":"mongodb_json",
      "endpoint":"https://objectstorage.region_ID_from_step_5.oraclecloud.com",
      "namespace":"namespace_from_step_3",
      "bucket":"bucket_name_from_step_3",
      "prefix":"source_json_filename_or_prefix_from_step_2",
      "credentials":"/home/opc/.oci/config",
      "credentialsProfile":"default"
   },
   "sink":{
      "type":"nosqldb_cloud",
      "endpoint":"https://nosql.oci_region_ID_from_step_5.oci.oraclecloud.com",
      "table":"new_Oracle_NoSQL_Database_Cloud_Service_table_name",
      "schemaInfo":{
         "defaultSchema":true,
         "readUnits":50,
         "writeUnits":50,
         "storageSize":25
      },
      "compartment":"compartment_name_from_step_6",
      "includeTTL":false,
      "credentials":"/home/opc/.oci/config",
      "credentialsProfile":"DEFAULT",
      "writeUnitsPercent":90,
      "overwrite":true,
      "requestTimeoutMs":5000
   },
   "abortOnError":false,
   "migratorVersion":"1.5.0"
}
[opc@oci-compute nosql-migrator-1.5.0]$

Run the data migration using Oracle NoSQL Database Migrator and the migration configuration JSON file created in the previous step as follows:
```
[opc@oci-compute nosql-migrator-1.5.0]$ ./runMigrator --config ./migrator-config-mongodbjson-bucket_def_schema.json
```
The migration job will create a new Oracle NoSQL Database Migrator table using the default schema, then read the MongoDB formatted JSON file(s) from OCI Object Storage, to then import the data from those files to the newly created table.

Migrate Data Using a Custom Schema

When using a custom schema to migrate, the new Oracle NoSQL Database Cloud Service table must be defined similarly to the schema of the MongoDB table. The NoSQL Database Migrator configuration file must include a transform entry instructing NoSQL Database Migrator how to convert the column _id to id in the new table.

The following steps describe how to migrate data using an Oracle NoSQL custom schema.

Log in to MongoDB and run mongoexport to export the data collection to MongoDB formatted JSON files. For example:
```
[opc@mongodb-server ~] mongoexport --collection collection_name --out collection_name.json
```
Split MongoDB JSON files with more than 13,500 records using the Oracle Linux split utility to divide the MongoDB JSON file into multiple JSON files of 13,500 records each. For example:
```
$split -l 13500 –additional-suffix .json collection_name.json collection_name
```
Use the Oracle Linux cat or vi command to examine the generated MongoDB formatted JSON files. Note the following:
- Note the names and data types for each column. Use this information to create a Oracle NoSQL Database Cloud Service table schema data definition language (DDL).
- The first column must be named id. The remaining columns should have the same name and type as in the MongoDB JSON file.

Save the schema DDL to a file. The schema DDL should resemble the following:

$ cat collection_name.ddl
CREATE TABLE table_name ( id string, column_2_name column_2_type, column_n_name column_n_type, PRIMARY KEY ( id ) )

Log in to the OCI console to create an OCI Object Storage Bucket if one does not already exist.
Note the bucket name and namespace.
Use the OCI console or REST API to upload the file(s) from steps 2 and 3 to the new bucket created in step 5.
In the OCI console, create an API key for the OCI user who will own the target Oracle NoSQL Database Cloud Service table. This user must have permissions to create and manage the table.
Note the user OCID, tenancy OCID, fingerprint, and OCI region ID. Save the private and public keys on PEM files. This API key is used by Oracle NoSQL Database Migrator to retrieve the MongoDB formatted JSON files, create a new table, and then upload the data from the MongoDB JSON files to the table.
Create a compartment for the Oracle NoSQL Database Cloud Service table if one does not already exist.
Note the compartment name.
Create a compute instance.
This compute instance will run the Oracle NoSQL Database Migrator software. An Always-Free Compute instance (VM.Standard.E4.Flex with 1 OCPU and 16GB memory) is sufficient and cost effective.
Use SSH to log in as the opc user to the new compute instance to install Java 11 as follows:
```
[opc@oci-compute ~]$ sudo yum install java-11
```
Download the Oracle NoSQL Database Migrator 1.5.0 binaries if you haven't already. See the Review Software Requirements section for details.
Use SCP to upload the Oracle NoSQL Database Migrator ZIP file to the opc user's home directory in the new compute instance.
Use SSH to log in as the opc user to the compute instance to unzip the Oracle NoSQL Database Migrator software downloaded in the previous step. The contents of the file is unzipped to a new directory named /home/opc/nosql-migrator-1.5.0. Unzip the file as follows:
```
[opc@oci-compute ~]$ unzip V1033765-1.zip
[opc@oci-compute ~]$ ls -l
total 61964
drwxr-xr-x. 3 opc opc 4096 Feb 23 20:32 nosql-migrator-1.5.0
-rw-r--r--. 1 opc opc 63446080 Feb 14 19:26 V1033765-01.zip
[opc@jc-bastion-phx migrator]$
```
Use SCP to upload the private key PEM file from step 7 to the home directory of the opc user of the compute instance where Oracle NoSQL Database Migrator was unzipped in the previous step.

Use SSH to log in as the opc user to the compute instance to create an OCI identity file. Create the OCI identity file in the /home/opc/.oci directory with the following contents:

[opc@oci-compute ~]$ cat .oci/config 
[DEFAULT]
user=user_OCID_from_step_7
fingerprint=fingerprint_from_step_7
tenancy=tenancy_OCID_from_step_7
region=region_ID_from_step_7
key_file=full_path_to_private_key_PEM_file_uploaded_step_14

[opc@oci-compute nosql-migrator-1.5.0]$ cat migrator-config-mongodbjson-bucket-src_schema.json 
{
  "source": {
    "type": "object_storage_oci",
    "format": "mongodb_json",
    "endpoint": "https://objectstorage.region_ID_from_step_7.oraclecloud.com",
    "namespace": "namespace_from_step_5",
    "bucket": "bucket_name_from_step_5",
    "prefix": "source_json_filename_or_prefix_from_step_2",
    "schemaInfo": {
      "schemaObject": "collection_name_ddl_from_step_3"
    },
    "credentials": "/home/opc/.oci/config",
    "credentialsProfile": "DEFAULT"
  },
  "sink": {
    "type": "nosqldb_cloud",
    "endpoint": "https://nosql.oci_region_ID_from_step_7.oci.oraclecloud.com",
    "table": "new_Oracle_NoSQL_Database_Cloud_Service_table_name_from_step_3",
    "schemaInfo": {
      "useSourceSchema": true,
      "readUnits": 50,
      "writeUnits": 50,
      "storageSize": 25
    },
    "compartment": "compartment_name_from_step_8",
    "includeTTL": false,
    "credentials": "/home/opc/.oci/config",
    "credentialsProfile": "DEFAULT",
    "writeUnitsPercent": 90,
    "overwrite": true,
    "requestTimeoutMs": 5000
  },
  "transforms": {
    "renameFields": {
      "_id": "id"
    }
  },
  "abortOnError": true,
  "migratorVersion": "1.5.0"
}
[opc@oci-compute nosql-migrator-1.5.0]$

Run the data migration using Oracle NoSQL Database Migrator and the migration configuration JSON file created in the previous step as follows:
```
[opc@oci-compute nosql-migrator-1.5.0]$ ./runMigrator --config ./migrator-config-mongodbjson-bucket-src_schema.json
```
The migration job will create a new Oracle NoSQL Database Migrator table using the custom schema, then read the MongoDB formatted JSON file(s) from OCI Object Storage, to then import the data from those files to the newly created table.