14 Migrate New Data in Real Time from Oracle Database Classic to Oracle Cloud Infrastructure
You can migrate a single-instance database, created on Oracle Cloud Infrastructure Classic, to an Oracle Cloud Infrastructure virtual machine database system, by using Data Integration Platform Cloud. You can use this procedure as part of an overall migration of your Oracle Cloud environment to Oracle Cloud Infrastructure.
About Using Data Integration Platform Cloud to Migrate Oracle Database Cloud Deployment to Oracle Cloud Infrastructure
Using Data Integration Platform Cloud (DIPC), you can perform real-time data migration. The DIPC migration process watches the source database and updates the target with every change that happens on the source.
When to Use Data Integration Platform for Migrating an Oracle Database Cloud Deployment to Oracle Cloud Infrastructure
Use Data Integration Platform Cloud to migrate new data in real time from Oracle Database Classic to Oracle Cloud Infrastructure.
To use Data Integration Platform Cloud for migration, your source database should be an Oracle Database Classic instance and the target should be an Oracle database system on a virtual machine on OCI. This migration is application for Oracle database 11g to 11g, 11g to 12c, and 12c to 12c.
This migration method watches the source database and updates the target with every change that happens in the source. Every new insert, delete, or update that happens on the source will be reflected in the target database in real-time. Before you start this real-time migration, you must create the target database with the same schemas and tables that you want to replicate.
Use DIPC for migration in the following scenarios:
- When you don’t want to shut down your source production database.
- When you want to perform real-time migration of Oracle databases 11g to 12c, which is more complicated with other migration methods.
- When your new database only requires a copy of your source database from a certain point of time that you choose. This point of time has to be after you start the migration. The target database can be used for data analytics, to take the query load off your source database. It can also be used as a more current database, which doesn't include historical data from the source database.
Limitations:
- Replicate Data does not perform an initial load. For a complete migration of the source database, including the historical data, use Oracle Data Pump or Oracle Recovery Manager (RMAN) to copy the initial load from source to the target first and perform this real-time migration after that.
- The change in the source database isn't available for processing until after the Data Manipulation Language (DML) commands such as insert, update and delete have been committed. Therefore, only transactions are replicated in Replicate Data.
- Data Definition Language (DDL) operations such as drop or create are not part of the replication process.
- Replicate Data is one to one and uni-directional, which means that it accepts one source and one target for each task. If you want to include several sources to deliver data to one target, you can create a Replicate data task for each source and have them all deliver to the same target.
Architecture
You can migrate your data from an Oracle Database Classic instance hosted on Oracle Cloud Infrastructure Classic to a virtual machine database system on Oracle Cloud Infrastructure (OCI). Oracle Database versions 11.2.0.4, 12.1.02 and 12.2.0.1 are all supported.
Here is how data flows when Data Integration Platform Cloud replicates your data:
Your source database is on OCI Classic. You create a target database on OCI. An application on OCI called Oracle Data Integration Platform Cloud (DIPC) orchestrates data replication from the source database to the target database. To perform the replication, you set up two DIPC remote agents on two compute instances: one for source and one for target. The agents have some binaries called Oracle GoldenGate (OGG) that can send data information from source to target.
The data flows from source database directly to the source agent and from there to the target agent and finally to the target database. It doesn't go to the DIPC host. Only monitoring information is exchanged between the agents and the DIPC host.
The data that flows from source agent to the target agent has to pass through a firewall. You open up some ports on the target compute instance, so that the data can come in. You don't need to open any ports on the source side, because it's a one-way replication and data only goes out.
The connections between each database and its agent is through a private IP address. The compute instance on OCI Classic should be on the same network as the Database Classic instance. Similarly, the Compute instance on the OCI system should be on the same VCN as the virtual machine database system. With this setup, you don't need VPN to have the agents access the databases.
The connection between the agents and their DIPC host is through an outgoing https connection. Information only goes out to the DIPC host for monitoring the data flow.
The following diagram shows the migration process:
What You Need to Perform the Migration
Before you begin the database migration, in addition to knowledge of Oracle Cloud Infrastructure, you must have knowledge in several areas of Oracle Database tools.
You will need the following components or tools:
- An Oracle Cloud Infrastructure account with a role to create compartments, subnets, target database, and Data Integration Platform instances.
- An account on Oracle Support to create a service request.
- Tools to create SSH keys, such as PuTTY or SSH
- Tool to connect to VMs, such as PuTTY
- Tool to connect to databases, such as SQL developer or you can SSH to the database directly
You should be familiar with the following tasks:
- Setting up compartments, subnets, and databases in Oracle Cloud Infrastructure (OCI)
- Creating SSH keys using either PuTTY or SSH
- Using SQL*Plus
- Performing Oracle database management tasks, such as creating schemas, users, and tables.
Note:
You must create a service request through Oracle Support. Ask them to reserve a Data Integration Platform Cloud instance for you. Otherwise, you will get an error message that you don't have enough resources.How to Migrate an Oracle Database Cloud Deployment to Oracle Cloud Infrastructure by Using Data Integration Platform
You perform the following tasks to migrate an Oracle Database Classic cloud deployment to Oracle Cloud Infrastructure by using Data Integration Platform Cloud.
- Prepare for Migration
- Create a Data Integration Platform Cloud instance
- Set up the agents
- Prepare source database
- Prepare target database
- Migrate the Database
- Create Connections
- Create a Replicate Data Task
- Run the Replicate Data Task
Prepare for Migration
To prepare for migration with Data Integration Platform Cloud perform the following tasks:
Create a Data Integration Platform Cloud Instance
Data Integration Platform Cloud (DIPC) is an all-in-one platform with graphical menu options for data preparation, integration, replication and quality management.
- Log in to Oracle Cloud Infrastructure home page.
- Click the menu option for Oracle Cloud.
- Click My Services Dashboard.
- If you land on the Guided Journey page, then click Dashboard.
- Click Customize Dashboard.
- Under Integration, click Show for Data Integration Platform.
- Close the Customize Dashboard window.
- Confirm that Data Integration Platform appears as a tile on your dashboard.
- In the Data Integration Platform tile, click Open Service Console from the Action Menu.
- If you encounter a welcome page, click Go to Console.
- Don't click the prerequisite link on the page, because it is an old link and gives an error message. Currently the prerequisites are created for your when you create the DIPC instance. So there are no prerequisites, if you can already create an instance.
- Click Create Instance.
- Complete the Instance details. For example:
- Instance Name:
ABCMigration
- Description:
Data Migration from the ABC database to OCI
- Email: an email address to receive a notification when your instance is ready
- Region: No Preference
- Tags: Click the Click to Create a Tag button and create a tag called Migration and then click OK. (optional)
- License Type: Subscribe to a new Data Integration Platform Cloud software license and the Data Integration Platform Cloud.
- Instance Name:
- Click Next.
- Complete the Service Details. For example:
- Service Edition:
Enterprise Edition
Note:
You must either choose Enterprise or Governance Edition. Enterprise Edition offers choices for selective copy or real time replication of your data sources. This edition also offers extract, load and transform (ELT) options for your data entities. Governance Edition includes Enterprise features plus additional data quality management options. Standard Edition is for ELT only and does not offer you a choice to perform a migration. - Data Volume - GB: Select an option to be billed based upon. You're charged a monthly flat rate based on the volume of data processed per hour for data movement from/to Data Integration Platform Cloud. For more information, see Pricing.
- Instance Reserved: Enable this checkbox if an instance has already been reserved for you. If you have not reserved an item, you can't create an instance. Submit a request to reserve a DIPC instance for you through Oracle Support.
- Service Edition:
- Click Next.
- If you need to make changes, use the Previous button to navigate back to the Instance or Details pages, and then come back to this page and confirm and create the instance.
- Confirm that the name of your instance is in the list of Instances and the field Created On displays the creation date.
Set Up the Agents
Agents are applications that you allow to securely connect to data sources and orchestrate data integration activities. An agent's components depends on the activities that it orchestrates. For example, data replication and adding data to data lake have different components.
For data migration between two Oracle databases, you set up two agents:
- A source agent to capture data activities from the source and send it to the target agent
- A target agent to receive information from the source agent and apply the data changes to the target database
The best location for the source agent is on OCI-Classic where the Oracle Database Classic is. Similarly, the best place for the target agent is on OCI, in the same region as the target database system. A Compute instance is a physical location that you can create on OCI or OCI classic and then copy your agent binaries there.
Here are the steps to set up your agents:
Download Agent Binaries
To set up an agent, you must first download the proper agent binaries from the Data Integration Platform console to your machine.
If both source and target databases have the same version, then you can download the binaries once and use it twice.
Here is how you can get your agent binaries:
- On My Services Dashboard, click the Action Menu for the Data Integration Platform tile and then click Open Service Console.
- In the Instances page, locate your Data Integration Platform Cloud instance. For example,
ABCMigration
- From the instance's Manage this instance menu, select Data Integration Platform Console.
- Click Download in the Agents tile.
- Select the operating system and components:
- Operating System: This option is not your database's operating system. It is the operating system of the location that the agent will be installed. Select Linux 64-bit, because both the OCI and OCI Classic Compute instances have Linux operating systems.
- Connectors/Components: For this option, either choose Oracle 11g (OGG) or Oracle 12c (OGG). This option provides the replication binaries that match your Oracle database version.
- Click Download.
- Save the binaries so that later you can transfer it to a Compute instance.
Create a Location for Agents
Each Data Integration Platform Cloud agent must have access to a data source to orchestrate its data integration activities. Compute instances are ideal locations for agents that manage cloud database activities.
For an Oracle Database Classic, the best place to put the agent binaries is on a Compute Classic instance. Similarly, for a database on OCI, the best place for the agent is on a Compute instance on OCI. This way, you don't need to open any ports between each database and its agent and all that communication is done through private IP addresses. The only port you open, is on the target Compute instance to allow the source agent to send information to the target agent.
Here are the steps to create a location for your source and target agents.
Create a Compute Instance for Source Agent
If you have an Oracle Database Classic, then, place its agent binaries on a Compute Classic instance located in the same region as the database.
Here are the steps to create a Compute instance for your source agent:
- Go to My Services Dashboard.
- Click Customize Dashboard.
- Click Show for Compute Classic.
- Close the Customize Dashboard window.
- In the Compute Classic tile, click Open Service Console from the Action Menu.
- Click Create Instance.
- Click Customize.
- For Image, select the latest one. For example, OL_7.2_UEKR4_x86_64 which is an Oracle Linux 7.2 UEKR4 and proceed to the Shape page.
- For Shape, select oc5 which gives you 4 OCPUs and 30GB of memory.
- Proceed to the Instance page.
- For Placement, select the same domain or location as the source database.
- Add your public Key to SSH Keys.
- Proceed to the Network page.
- For Network options, select IP Network.
- Click the Action Menu for the listed default network and then select Update.
- Set the Public IP Address to Auto Generated and then click Update.
- Name your DNS Hostname Prefix and then proceed to the next page.
- Click the Action Menu for the listed default storage and then select Update.
- Update the storage Size to 120 GB.
- Confirm that the new storage size is 120 GB and then proceed to the Review page.
- Review all your options and either go back to modify the fields or if you agree with your options, then click Create.
Add Java to Your Compute Instance
Data Integration Platform agents refer to $JAVA_HOME to perform their activities. You must set up $JAVA_HOME to point to JDK with a Java version of 1.8 or higher, before you set up your agents.
Compute instances don't come with JDK, so you must install JDK on your Compute instances and then have $JAVA_HOME point to JDK.
- Download JDK with Java version 1.8 or higher from Java SE Downloads
- Use an application that has Secure Copy Protocol (SCP) such as WinSCP or SSH to transfer the JDK execution file to your Compute instance. Provide the IP address and the private key that matches the public key that you gave to your Compute instance to connect to it. The username is
opc
. - Use PuTTY or SSH to connect to the Compute instance. For instructions on how to connect to instances with PuTTY, see Connect to an Oracle Cloud Instance From Windows Using PuTTY.
- Install JDK on the Compute instance.
- Ensure that your environment is pointing to the correct Java.
- Ensure that
JAVA_HOME
points to your JDK. If not, then set it to point to the right location.
Example:
[xxx]$ java - version
java version "1.8.0_91"
[xxx]$ which java
~/Public/JDK/bin/java
[xxx]$ echo $JAVA_HOME
/home/oracle/Public/JDK
Create a Compute Instance for the Target Agent
The Compute instance that you create for the target agent is similar to the source Compute, except that it's on Oracle Cloud Infrastructure and it must be much larger than the source Compute. Ensure that the storage size that you allocate for the target is 2TB or larger.
The target agent receives and stores all the information about the changes in the source database in the target Compute storage. Because these files are constantly growing, you must have a large storage to ensure a great performance. Here are some guidelines to create a Compute instance on OCI.
- Log in to Oracle Cloud Infrastructure.
- Select a region on the top navigation bar that matches your target database region.
- From the Action Menu, go to Compute and then Instances.
- Click Create Instance.
- Select Virtual Machine for instance type.
- For Shape select VM.Standard2.1
- Select Custom boot volume size.
- Enter
2000
, because the size you enter is in GBs and 2TB is equivalent to 2000 GB. - Enter an SSH public key with the .pub extension. Ensure that you have the matching private key on your machine.
- Refer to Launching Your First Linux Instance for additional guidance.
- Click Create.
- Install JDK for Java 1.8 or higher on your Compute instance.
- Have $JAVA_HOME point to your JDK.
Related Topics
Open Ports on Target Compute Instance
Data Integration Platform Cloud source agents send data replication information to target agents. The only way for the target agents to receive this information is to open ports on the Oracle Cloud Infrastructure (OCI) Compute instance where they are located.
You must open port 7809 for the agent application that manages the replication and 7819-7829 for the data collector application. Here are steps on opening ports for a Compute instance on OCI:
- Log in to OCI.
- Go to the region that your target database is located. For example,
us-phoenix-1
. - From the Actions Menu, go to Compute and then Instances.
- Click the Compute instance hat you have allocated for your agent.
- Click the link in the Subnet field. For example,
Public_Subnet ilMx:PHX-AD-3
- Click Security Lists from the list of Resources.
- Click the default security list for your VCN. For example,
Default Security List for DIPC-VCN
. - In the Resources section, go to Ingress Rules.
- Click Add Ingress Rules.
- Enter the following information for your ingress rule:
- SOURCE TYPE: CIDR
- SOURCE CIDR:
0.0.0.0/0
- IP PROTOCOL: TCP
- SOURCE PORT RANGE: All
- DESTINATION PORT RANGE:
7809
- Click + Additional Ingress Rule.
- Enter the following information for your ingress rule:
- SOURCE TYPE: CIDR
- SOURCE CIDR:
0.0.0.0/0
- IP PROTOCOL: TCP
- SOURCE PORT RANGE: All
- DESTINATION PORT RANGE:
7819-7829
- Click Add Ingress Rules.
Register the Agents
After you download your Data Integration Platform agent components, you perform certain steps and send special parameters to the Data Integration Platform host, so that it can discover the agent. This procedure is called registering the agent.
Here are steps to register an agent. Perform these steps twice. Once for the source agent and once for the target agent.
Create a Confidential Application for Agents
A confidential application is a server-side application that uses OAUTH 2.0 to keep the confidentiality of the client secret. You can create a confidential application for your Data Integration Platform Cloud host. Then you can provide the client secret for the application to your agent so it can securely communicate with the DIPC host.
Prepare Source Database
To prepare a source database for migration with Data Integration Platform Cloud, you must either create a user on the source that is specific to the migration process or give an existing user specific data integration rights.
Perform the following steps to create a user at the CDB level:
Prepare Target Database
Before you migrate your data, you must create a database system with a virtual machine shape on Oracle Cloud Infrastructure. Then you must set up the target database with the same schema and structure as the source database.
Note:
Data Integration Platform Cloud's Replicate Data task doesn't perform an initial copy of the source, so you'll get all the changes from the point of time that you start your job. If you want the target database to be exactly like the source database, then you must perform an initial load first. For initial load, you can use theopcmigrate
tool to create a copy of the source database. This method creates the database for you. See Migrate Databases Using the Migration Tools.
If you don't want an initial load, then create a database system with a virtual machine shape type. Then create the the schema that you want to replicate from the source database. Then create the tables and the structure for this schema. Replicate Data task doesn't perform any DDL commands.
After you define the schema and structure for the target database, then perform the following steps:
Migrate the Database
To perform a real-time migration you can use the Replicate Data Task of Oracle Data Integration Platform Cloud. The Replicate Data Task captures changes in your data source and updates the target in real time with that change.
Here is a summary of steps to use the Replicate Data Task:
Create Connections
A Connection in Data Integration Platform Cloud is a set up to establish a connection to a data source. The first step for a migration or replication is to define the source and target Connections.
After you create your Connections, they are ready to be used in your data integration tasks including the Replicate Data Task.
If a source database is a PDB database, the Replicate Data Task captures the data from the source at the Container Database (CDB) level. Even though you select a schema within a PDB to be replicated, the task happens at the CDB level. Therefore, if your source is an Oracle 12c with PDB, you must create an additional CDB Connection for the Replicate Data Task. Replicate Data Task doesn't use a CDB connection for data delivery, so you don't need to set it up for the target connection.
Here are the type of Connections that you must create for your real-time migration:
Table 14-1 Source Connection Types
Source Database | Connection Type |
---|---|
Oracle 12c |
Oracle CDB and then Oracle |
Oracle 11g |
Oracle |
Table 14-2 Target Connection Types
Target Database | Connection Type |
---|---|
Oracle 12c |
Oracle |
Oracle 11g |
Oracle |
Find Source Database Information
When you create an Oracle Connection in Data Integration Platform Cloud, you must provide host name, service name and the database port number.
Here are steps to find Oracle Database Classic connection information:
Find Target Database Information
When you create an Oracle Connection in Data Integration Platform Cloud, you must provide host name, service name and the database port number.
Here are steps to find connection information for an Oracle database system on Oracle Cloud Infrastructure:
Create a CDB Connection
For PDB database migration, you must first create an Oracle CDB Connection. Then you create a PDB Connection for your source database and associate this CDB Connection within your PDB connection.
Note:
A CDB Connection for a Replicate Data Task only works if the source agent is set up with Oracle 12c (OGG) binaries. You can download these binaries from the Agents page. See Download Agent BinariesTo create a CDB Connection:
Create a Source Connection
Before you can perform a migration or replication in Data Integration Platform Cloud, you must create a Connection to the source database.
Note:
An Oracle 12c Connection for a Replicate Data task only works if its agent is set up with the Oracle 12c (OGG) binaries. Similarly, an Oracle 11g Connection only works for a Replicate Data Task if its agent is set up with the Oracle 11g (OGG) binaries. You can download these binaries from the Agents page. See Download Agent BinariesTo create a source Connection:
Create a Target Connection
Before you can perform a migration or replication in Data Integration Platform Cloud, you must create a Connection to the target database.
Note:
An Oracle 12c Connection for a Replicate Data task only works if its agent is set up with the Oracle 12c (OGG) binaries. Similarly, an Oracle 11g Connection only works for a Replicate Data Task if its agent is set up with the Oracle 11g (OGG) binaries. You can download these binaries from the Agents page. See Download Agent BinariesTo create a target Connection:
Create a Replicate Data Task
Here’s how you can set up a Replicate Data Task in Data Integration Platform Cloud.
Run the Replicate Data Task
Here are the steps to run a Replicate Data Task:
- Go to the Catalog page of the Data Integration Platform Console.
- Change the filter option from All to Tasks.
- Click the name of your Replicate Data Task.
- If you want to change the schema, or view the details of this task, click Edit.
- If you are happy with the setup, then click Run.
- Wait until the Start Delivery option in the Monitor page shows a Running status.
- Connect to the target database with SQL Developer. See Connect to a Database Cloud Service with Oracle SQL Developer. .
- Go to the schema you chose for the target and perform some queries to monitor the replication.
Related Topics