Note:

Migrate a MongoDB Database Running on MongoDB Atlas or On-Premises to Oracle Autonomous JSON Database

Introduction

MongoDB is one of the popular open-source databases used to create and store data in flexible, JSON-like documents, meaning fields can vary from document to document and data structure can be changed over time.

Oracle, which is one of world famous leading relational, multi-model database called as Converged Database, has good capability of handling JSON. Oracle introduced support for JSON from v12c, stored JSON data as varchar2 or Large Object (LOB) (Character Large Object (CLOB)/Binary Large Object (BLOB)).

Challenges with MongoDB:

Advantage of using JSON Data Type in Oracle:

Note: Make sure connectivity is established between Oracle Cloud Infrastructure (OCI) GoldenGate Big Data deployment, source and target databases.

Objectives

Prerequisites

Task 1: Install MongoDB Binary and Start MongoDB Services

  1. Create three OCI Compute instances for replication setup and attach block volume for data directory.

    image

  2. Update yum repos to download MongoDB binary using the following command.

    vi /etc/yum.repos.d/mongodb-enterprise-8.0.repo
    [mongodb-enterprise-8.0]
    name=MongoDB Enterprise Repository
    baseurl=https://repo.mongodb.com/yum/redhat/8/mongodb-enterprise/8.0/$basearch/
    gpgcheck=1
    enabled=1
    gpgkey=https://pgp.mongodb.com/server-8.0.asc
    sudo yum install -y mongodb-enterprise
    

    image

  3. Create data directory and grant required permission using the following command.

    cd /data
    ls
    chmod 755 mongo
    cd
    chmod 755 -R /data/mongo
    chown mongod:mongod -R /data/mongo
    chmod 400 /etc/mongod.Keyfile
    chown mongod:mongod -R /etc/mongod.Keyfile
    

    image

    image

    Note: Keyfile is needed for secure authentication. For testing purpose, keyfile can be generated using the open-ssl command.

  4. Create config file using the following command.

    cat /etc/mongod.conf
    

    image

  5. Disable selinux and start mongod services using the following commands.

    setenforce 0
    getenforce
    systemctl start mongod
    systemctl status mongod
    

    image

  6. Perform the same operations on remaining two nodes which will be acting as the secondary node.

Task 2: Create MongoDB Replica Set

  1. Connect to first node and connect to admin database.

  2. Run the following commands to initiate replication and create first admin user. This replication user will be used for replication configuration and management.

    mongosh
    use admin
    

    image

    rs.initiate()
    db.createUser({
    ...  user: "admin",
    ...  pwd: "password",
    ...  roles: [
    ...         { role: "root", db: "admin" }
    ...   ]
    ...  })
    

    image

    db.auth("admin","password")
    rs.conf()
    

    image

  3. Add remaining two nodes using the following commands. Those will be added as secondary nodes.

    rs.add("10.0.1.163;27017")
    

    image

    rs.add("10.0.1.51:27017")
    

    image

  4. Check configuration and status of replica set using the following commands.

    db.auth("admin","password")
    rs.conf()
    

    image

    rs.status()
    

    image

    image

    image

    image

    rs.printReplicationInfo()
    

    image

Task 3: Load Data into MongoDB Replica Set

  1. Restore the dump into MongoDB replica set using the mongorestore command.

    mongorestore --username admin --password password dump/
    

    image

    image

    image

    Note: The above dump was obtained from another MongoDB replica set using mongodump command for all databases running under that replica set.

  2. Run the following command to check if all databases and its respective collections have been restored into MongoDB replica set.

    mongosh
    use admin
    db.auth("admin","password")
    show dbs
    

    image

Task 4: Configure Oracle Autonomous JSON Database in OCI

  1. Log in to the OCI Console, navigate to Oracle Database, Autonomous Database and enter the following information as shown in the images to create autonomous database.

    image

    image

    image

    Note: For MongoDB compatibility, network access needs to be set to either secure access from allowed IPs and VCNs only or private endpoint access only.

  2. Click Edit tool configuration, enable MongoDB API and copy the URL.

    image

    image

    image

    Note: You need to change the username and password in above connection string.

Task 5: Create Database Client Machine to Access Oracle Autonomous JSON Database and Install Required MongoDB Tools

  1. Download wallet file and configure connectivity for Oracle Autonomous JSON Database using the following commands.

    cat tnsnames.Oracle
    

    image

    cat sqlnet.Ora
    

    image

    image

  2. Download MongoDB tools for connectivity using the following commands.

    wget https://fastdl.mongodb.org/tools/db/mongodb-database-tools-rhel70-x86_64-100.5.2.tgz
    

    image

    tar -xvf mongodb-database-toosl-rhel70-x86_64-100.5.2.tgz
    

    image

    wget https://downloads.mongodb.com/compass/mongosh-1.3.1-linux-x64.tgz
    

    image

    tar -xvf mongosh-1.3.1-linux-x64.tgz
    

    image

  3. Set Path and URI (Oracle Autonomous Transaction Processing (ATP) endpoint) as environment variable using the following commands.

    cd
    export PATH=/home/oracle/mongosh-1.3.1-linux-x64/bin/:$PATH
    export PATH=$PATH:/home/oracle/mongodb-database-tools-rhel70-x86_64-100.5.2/bin
    echo $path
    

    image

    image

    image

  4. Check if you are able to connect to Oracle Autonomous JSON Database using the following command.

    export URI='mongodb://admin:xxxx@xxxxxx-ADBJ.adb.us-ashburn-1.oraclecloudapps.com:27017/admin?authMechanism=PLAIN&authSource=$external&ssl=true&retryWrites=false&loadBalanced=true'
    mongosh $URI
    

    image

Task 6: Upload Dump to OCI Object Storage Bucket and Import into ATP

  1. Take dump of a collection using the mongoexport command.

    image

    mongoexport --host=mrep1 --port 27017 --username=admin --authicationDatabase=admin --collection=data --db=sample_weatherdata --out=wheather_data.Json
    

    image

  2. Click Upload to upload dump of a single collection of database which was taken using the mongoexport command on MongoDB replica set.

    image

  3. Click View Object Details to get object endpoint detail.

    image

    image

  4. Run the curl command to read the data from that object and the mongoimport command to ATP using endpoint.

    curl -X GET https://objectstorage.us-ashburn-1.oraclecloud.com/p/ujXv8Pmhxt9EyLlA9ogm_X524pDRGEymJwbDVyUExAACra1VpAql32U0XaRnaxLt/n/orasenatdoracledigital01/b/jsonbucket/o/wheather_data.json | mongoimport --collection data --uri $URI
    

    image

  5. Check if collection has been imported into the ATP database.

    image

  6. Crosscheck if the number of documents is same as the source.

    image

    Test is Successful. Now, we are good to move all data running under multiple databases in source MongoDB replica set to target Oracle Autonomous JSON Database and deploy OCI GoldenGate services for near zero downtime migration.

  7. For data movement you can use the mongodump command to copy data from source to target database as a one time load.

    export URI="mongodb://mrep1:27017 --username admin --authenticationDatabase admin"
    mongodump --uri=$URI
    

    image

    image

  8. Copy this dump to the target system from where you can run the mongorestore command (already explained in Task 3 to load into target Oracle Autonomous JSON Database).

    Note: Make sure to create the same schema name in Oracle Autonomous JSON Database for the database collections you want to migrate.

    create user sample_airbnb identified by <password>;
    grant connect, resource to sample_airbnb;
    grant unlimited tablespace to sample_airbnb;
    BEGIN
    ORDS_ADMIN.ENABLE_SCHEMA(p_schema => 'SAMPLE_AIRBNB');
    commit;
    END;
    /
    

Task 7: Deploy OCI GoldenGate

  1. Go to the OCI Console, navigate to Oracle Database, GoldenGate, Deployments and click Create deployment.

    For MongoDB to Oracle Autonomous JSON Database, we need to select Technology as Big Data and Deployment Type as Data Replication.

    image

    image

  2. Create secret in advance from the OCI Console for admin user log in and management.

    image

    image

    image

Task 8: Create Connection for both MongoDB and Oracle Autonomous JSON Database

  1. Go to the OCI Console, navigate to Oracle Database, GoldenGate, Connections and click Create connection.

    image

  2. Enter the following information as shown in the image to create source database connection.

    image

  3. Repeat step 1 and enter the following information as shown in the image to create target database connection.

    image

Task 9: Assign and Test Connection to OCI GoldenGate Deployment

  1. Go to the OCI Console, navigate to Oracle Database, GoldenGate, Connections and click the connection created in Task 8.

  2. Click Assigned deployments and Assign deployment.

    image

  3. Select the deployment and click Assign deployment.

    image

  4. Click the three dots and Test connection to make sure it is successful.

    image

    image

Task 10: Create Extract and Replicat Processes

  1. Go to the OCI Console, navigate to Oracle Database, GoldenGate, Deployments and click the deployment created in Task 7.

    Click Launch Console and this will open the OCI GoldenGate configuration console.

    image

  2. Enter the Username and Password which was created during deployment.

    image

    Once you are logged in to the console page, you can see all required options for Extract and Replicat configuration. You can create extract and replicat services as shown in the later steps.

  3. Configure extract process.

    Select Connection Alias which has been created for the source MongoDB database.

    image

    image

    In Parameter File, sample_airbnb is a database in MongoDB and * represents all the collections in that database.

    image

  4. Configure replicat process.

    1. Select Classic Replicat as Replicat Type.

      image

    2. Enter the following required information.

      You need to select the same trail filename, which has been named in extract process creation. Select Target as Oracle Autonomous JSON Database and enter Connection Alias which has been created in Task 8.

      image

      Replicat process will replicat all the transactions from sample_airbnb database in MongoDB to sample_airbnb schemas in Oracle Autonomous JSON Database.

      image

    Note: You need to change the source and target database name and schema name respectively in the configuration.

  5. Once you are done with extract and replicat process configuration, start process accordingly.

    Note: Oracle Autonomous JSON Database should be configured with a private endpoint. If certificate is not used, you can change authentication as without mutual TLS (mTLS).

    In case the correct certification is not available, you may get the following error in the replication logfile:

    Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    

    image

Task 11: Start Extract and Replicate Processes and Validate

Click Administrative Service to check both processes status and its respective reports and log files for any issue.

image

Task 12: Test Data Synchronization

  1. In source and target, check document count in the collection detail before inserting some data using the following commands.

    • Source database:

      show dbs
      use sample_airbnb
      show collections
      db.emp.countDocuments()
      

      image

    • Target database:

      show dbs
      use Sample_airbnb
      show collections
      db.emp.countDocuments()
      

      image

  2. Check data on source and target database using the following commands.

    • Source database:

      db.emp.find({"name":"joe"})
      db.emp.find({"name":"John"})
      

      image

    • Target database:

      db.emp.find({"name":"joe"})
      db.emp.find({"name":"John"})
      

      image

  3. Insert data on source and check if it has been replicated to target using the following commands.

    • Source database:

      dm.emp.insertOne(
      { "name":"John","job":"Data Analyst","Salary":7000})
      db.emp.countDocuments()
      db.emp.find({"name":"John"})
      

      image

    • Target database:

      show DBs
      show collections
      db.emp.find({"name":"John"})
      

      image

      image

      image

Task 13: Connect To MongoDB Atlas

Connecting to MongoDB Atlas is different than connecting to an On-premises MongoDB database.

Note: Make sure you create a NAT gateway and attach with private subnet which is being used by OCI GoldenGate deployment as well if connection needs to be established in a public network.

image

Common Errors Due to Connectivity Issues

Connectivity issue to target Oracle Autonomous JSON Database can lead to miscellaneous errors while starting the replicate process:

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.