Overview

Oracle NoSQL Database Migrator lets you move Oracle NoSQL tables from one data source to another, such as Oracle NoSQL Database on-premises or cloud or even a simple JSON file.

There can be many situations that require you to migrate NoSQL tables from or to an Oracle NoSQL Database. For instance, a team of developers enhancing a NoSQL Database application may want to test their updated code in the local Oracle NoSQL Database Cloud Service (NDCS) instance using cloudsim. To verify all the possible test cases, they must set up the test data similar to the actual data. To do this, they must copy the NoSQL tables from the production environment to their local NDCS instance, the cloudsim environment. In another situation, NoSQL developers may need to move their application data from on-premise to the cloud and vice-versa, either for development or testing.

In all such cases and many more, you can use Oracle NoSQL Database Migrator to move your NoSQL tables from one data source to another, such as Oracle NoSQL Database on-premise or cloud or even a simple JSON file. You can also copy NoSQL tables from a MongoDB-formatted JSON input file or a DynamoDB-formatted JSON input file (either stored in AWS S3 source or from files) into your NoSQL Database on-premises or cloud.

As depicted in the following figure, the NoSQL Database Migrator utility acts as a connector or pipe between the data source and the target (referred to as the sink). In essence, this utility exports data from the selected source and imports that data into the sink. This tool is table-oriented, that is, you can move the data only at the table level. A single migration task operates on a single table and supports migration of table data from source to sink in various data formats.

Oracle NoSQL Database Migrator is designed such that it can support additional sources and sinks in the future. For a list of sources and sinks supported by Oracle NoSQL Database Migrator as of the current release, see Supported Sources and Sinks.

Terminology used with Oracle NoSQL Database Migrator

Learn about the different terms used in the above diagram, in detail.

  • Source: An entity from where the NoSQL tables are exported for migration. Some examples of sources are Oracle NoSQL Database on-premise or cloud, JSON file, MongoDB-formatted JSON file and DynamoDB-formatted JSON file.
  • Sink: An entity that imports the NoSQL tables from NoSQL Database Migrator. Some examples for sinks are Oracle NoSQL Database on-premise or cloud and JSON file.

The NoSQL Database Migrator tool supports different types of sources and sinks (that is physical media or repositories of data) and data formats (that is how the data is represented in the source or sink). Supported data formats are JSON, Parquet, MongoDB-formatted JSON and DynamoDB-formatted JSON. Supported source and sink types are files, OCI Object Storage, Oracle NoSQL Database on-premise, and Oracle NoSQL Database Cloud Service.

  • Migration Pipe: The data from a source will be transferred to the sink by NoSQL Database Migrator. This can be visualized as a Migration Pipe.
  • Transformations: You can add rules to modify the NoSQL table data in the migration pipe. These rules are called Transformations. Oracle NoSQL Database Migrator allows data transformations at the top-level fields or columns only. It does not let you transform the data in the nested fields. Some examples of permitted transformations are:
    • Drop or ignore one or more columns,
    • Rename one or more columns, or
    • Aggregate several columns into a single field, typically a JSON field.
  • Configuration File : A configuration file is a JSON file where you define all the parameters required for the migration activity. Later, you pass this JSON file as a single parameter to the runMigrator command from the CLI. A typical configuration file format looks like as shown below.
    {
     "source": {
       "type" : <source type>,
       //source-configuration for type. See Source Configuration Templates.
     },
     "sink": {
       "type" : <sink type>,
       //sink-configuration for type. See Sink Configuration Templates.
     },
     "transforms" : {
       //transforms configuration. See Transformation Configuration Templates.
     },
     "migratorVersion" : "<migrator version>",
     "abortOnError" : <true|false>
    }
    Group Parameters Mandatory (Y/N) Purpose Supported Values
    source type Y Represents the source from which to migrate the data. The source provides data and metadata (if any) for migration. To know the type value for each source, see Supported Sources and Sinks.
    source source-configuration for type Y Defines the configuration for the source. These configuration parameters are specific to the type of source selected above. See Source Configuration Templates for the complete list of configuration parameters for each source type.
    sink type Y Represents the sink to which to migrate the data. The sink is the target or destination for the migration. To know the type value for each source, see Supported Sources and Sinks.
    sink sink-configuration for type Y Defines the configuration for the sink. These configuration parameters are specific to the type of sink selected above. See Sink Configuration Templates for the complete list of configuration parameters for each sink type.
    transforms transforms configuration N Defines the transformations to be applied to the data in the migration pipe. See Transformation Configuration Templates for the complete list of transformations supported by the NoSQL Data Migrator.
    - migratorVersion N Version of the NoSQL Data Migrator -
    - abortOnError N

    Specifies whether to stop the migration activity in case of any error or not.

    The default value is true indicating that the migration stops whenever it encounters a migration error.

    If you set this value to false, the migration continues even in case of failed records or other migration errors. The failed records and migration errors will be logged as WARNINGs on the CLI terminal.

    true, false

    Note:

    As JSON is case-sensitive, all the parameters defined in the configuration file are case-sensitive unless specified otherwise.

Supported Sources and Sinks

This topic provides the list of the sources and sinks supported by the Oracle NoSQL Database Migrator.

You can use any combination of a valid source and sink from this table for the migration activity. However, you must ensure that at least one of the ends, that is, source or sink must be an Oracle NoSQL product. You can not use the NoSQL Database Migrator to move the NoSQL table data from one file to another.

Type
(value)

Format
(value)

Valid Source Valid Sink

Oracle NoSQL Database
(nosqldb)

NA Y Y

Oracle NoSQL Database Cloud Service
(nosqldb_cloud)

NA Y Y

File system
(file)

JSON
(json)

Y Y

MongoDB JSON
(mongodb_json)

Y N

DynamoDB JSON
(dynamodb_json)

Y N

Parquet(parquet)

N Y

OCI Object Storage
(object_storage_oci)

JSON
(json)

Y Y

MongoDB JSON
(mongodb_json)

Y N

Parquet(parquet)

N Y
AWS S3

DynamoDB JSON
(dynamodb_json)

Y N

Note:

Many configuration parameters are common across the source and sink configuration. For ease of reference, the description for such parameters is repeated for each source and sink in the documentation sections, which explain configuration file formats for various types of sources and sinks. In all the cases, the syntax and semantics of the parameters with the same name are identical.

Source and Sink Security

Some of the source and sink types have optional or mandatory security information for authentication purposes.

All sources and sinks that use services in the Oracle Cloud Infrastructure (OCI) can use certain parameters for providing optional security information. This information can be provided using an OCI configuration file or Instance Principal.

Oracle NoSQL Database sources and sinks require mandatory security information if the installation is secure and uses an Oracle Wallet-based authentication. This information can be provided by adding a jar file to the <MIGRATOR_HOME>/lib directory.

Wallet-based Authentication

If an Oracle NoSQL Database installation uses Oracle Wallet-based authentication, you need an additional jar file that is part of the EE installation. For more information, see Oracle Wallet.

Without this jar file, you will get the following error message:

Could not find kvstore-ee.jar in lib directory. Copy kvstore-ee.jar to lib directory.

To prevent the exception shown above, you must copy the kvstore-ee.jar file from your EE server package to the <MIGRATOR_HOME>/lib directory. <MIGRATOR_HOME> is the nosql-migrator-M.N.O/ directory created by extracting the Oracle NoSQL Database Migrator package and M.N.O represent the software release.major.minor numbers. For example, nosql-migrator-1.1.0/lib.

Note:

The wallet-based authentication is supported ONLY in the Enterprise Edition (EE) of Oracle NoSQL Database.

Authenticating with Instance Principals

Instance principals is an IAM service feature that enables instances to be authorized actors (or principals) that can perform actions on service resources. Each compute instance has its own identity, and it authenticates using the certificates added to it.

Oracle NoSQL Database Migrator provides an option to connect to a NoSQL cloud and OCI Object Storage sources and sinks using instance principal authentication. It is only supported when the NoSQL Database Migrator tool is used within an OCI compute instance, for example, the NoSQL Database Migrator tool running in a VM hosted on OCI. To enable this feature use the useInstancePrincipal attribute of the NoSQL cloud source and sink configuration file. For more information on configuration parameters for different types of sources and sinks, see Source Configuration Templates and Sink Configuration Templates.

For more information on instance principals, see Calling Services from an Instance.