Oracle Zero Downtime Migration – Logical Online Migration to ExaDB-D on Oracle Database@Google Cloud

Purpose

Oracle customers are rapidly increasing their workload migration into the Oracle Cloud, Engineered Systems, and Oracle Database@Google Cloud. However, migrating workloads has been a source of challenges for many years. Migrating database workloads from one system to another or into the Cloud is easier said than done. 

Based on years of experience migrating Oracle workloads, Oracle has developed Zero Downtime Migration (ZDM). ZDM is Oracle’s premier solution for a simplified and automated migration experience, providing zero to negligible downtime for the production system depending on the migration scenario. ZDM allows you to migrate your on-premises Oracle Databases directly and seamlessly to and between Oracle Database@Azure, Oracle Database@ Google Cloud, Oracle Database@AWS, and any Oracle-owned infrastructure, including Exadata Database Machine On-Premises, Exadata Cloud at Customer, and Oracle Cloud Infrastructure. Oracle ZDM supports a wide range of Oracle Database versions and, as the name implies, ensures minimal to no production database impact during the migration.

ZDM follows Oracle Maximum Availability Architecture (MAA) principles and incorporates products such as GoldenGate and Data Guard to ensure High Availability and an online migration workflow that leverages technologies such as the Recovery Manager, Data Pump, and Database Links.

This technical brief is a step-by-step guide for migrating your on-premises Oracle Databases to Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D) on Oracle Database@Google Cloud, with ZDM’s Logical Online workflow.

Oracle ZDM will run on a separate node and connect to Source and Target to perform the migration. This guide will cover all requirements for installing the Oracle ZDM Service Host, the Source Database, the Target Database recipient of the migration process, and the networking used. The migration process will be dissected and done in a step-by-step fashion. This guide will answer the most frequently asked questions regarding the product and the overall migration process.

For more information on Oracle Zero Downtime Migration, please visit ZDM’s product website and Oracle Database@Google Cloud product website.

Zero Downtime Migration

Oracle Zero Downtime Migration (ZDM) is the Oracle Maximum Availability Architecture (MAA)-recommended solution to migrate Oracle Databases to the Oracle Cloud. ZDM's inherent design keeps in mind the migration process as straightforward as possible to ensure the most negligible impact on production workloads. The Source Database to be migrated can be on-premises, deployed on Oracle Cloud Infrastructure, or a 3^rd Party Cloud. The Target Database deployment can be on Oracle Database@Azure, Oracle Database@Google Cloud, Oracle Database@AWS, Database Cloud Service on Oracle Cloud Infrastructure (OCI) Virtual Machine, Exadata Cloud Service, Exadata Cloud at Customer, and Autonomous Database. ZDM automates the entire migration process, reducing the chance of human errors. ZDM leverages Oracle Database-integrated high availability (HA) technologies such as Oracle Data Guard and GoldenGate and follows all MAA best practices that ensure no significant downtime of production environments. Oracle ZDM supports both Physical and Logical Migration workflows. This technical brief covers a step-by-step guide for the Logical Online Migration Workflow.

A standard Logical Online migration with Oracle GoldenGate and Data Pump Export and Import will take the following steps:

1. Download and Configure ZDM.
2. ZDM Starts Database Migration.
3. ZDM Configures an Oracle GoldenGate Extract Microservice.
4. ZDM Starts a Data Pump Export Job.
5. ZDM Starts a Data Pump Import Job.
6. ZDM Configures an Oracle GoldenGate Replicat Microservice.
7. ZDM Monitors Oracle GoldenGate Replication.
8. ZDM Switches Over.
9. ZDM Validates, Cleans Up, and Finalizes.

Supported Configurations

Oracle ZDM supports Oracle Database versions 11.2.0.4, 12.1.0.2, 12.2.0.1, 18c, 19c, 21c, and 23ai. ZDM’s physical migration workflow requires the Source and Target Databases to be in the same database release.

Oracle ZDM supports Source Oracle Databases hosted on Linux, Solaris, and AIX operating systems. Oracle ZDM supports single-instance databases, Oracle RAC One Node databases, or Oracle RAC databases as sources. Oracle ZDM supports Oracle Database Enterprise & Standard Edition as Source and Target Databases.  ZDM’s physical migration workflow supports only Source Databases hosted on Linux platforms.  

Architecture

An architectural overview of the ZDM server, the source database on-premises, the target database on Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D) on Oracle Database@Google Cloud, and all networks and components required are described in the diagram below:

Figure 1. This is a High-Level Architectural overview showcasing the customer data center where the source database and ZDM’s server reside. It also shows all connectivity to the target Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D) on Oracle Database@Google Cloud.

Zero Downtime Migration Service Host

Zero Downtime Migration Service Host Requirements

Oracle Zero Downtime Migration installation must take place on a separate host, which must fulfill the following requirements:

• Linux host running on Oracle 7, 8, or RHEL 8 (only these OS platforms/versions are supported).
• 100 GB of free storage space. This space is required for all the logs that ZDM will generate.
• A zdm group and a zdmuser as part of this group.
• The following packages must be installed:
  • glibc-devel
  • expect
  • unzip
  • libaio
  • oraclelinux-developer-release-el7
• All hostnames and IP addresses to be used must be present as entries in the /etc/hosts file.

For more information on the ZDM Service Host requirements and setting up ZDM on RHEL platforms, please refer to Oracle ZDM’s product documentation, specifically the “Setting Up Zero Downtime Migration Software” section.

For this step-by-step guide, the ZDM Service Host runs on-premises on an Oracle Linux Server 8.9. The host private IP is masked for this guide, but as an example we will use the fictional zz.dd.mm.hh and the hostname is zdmhost.

Network and Connectivity

Google Cloud Region
A Google Cloud region is a geographical area that contains data centers and infrastructure for hosting resources. It is made up of zones that are isolated from each other within the region.

Google Cloud Project
A Google Cloud Project is required to use Google Workspace APIs and build Google Workspace add-ons or apps. A Cloud project forms the basis for creating, enabling, and using all Google Cloud services, including managing APIs, enabling billing, adding and removing collaborators, and managing permissions.

Google Virtual Private Cloud
Google Cloud Virtual Private Cloud (VPC) provides networking functionality to Compute Engine virtual machine (VM) instances, Google Kubernetes Engine (GKE) containers, database services, and serverless workloads. VPC provides global, scalable, and flexible networking for your cloud-based service.

Google Cloud Interconnect 
Cloud Interconnect extends your on-premises network to the Google network through a highly available, low-latency connection. You can use Dedicated Interconnect to connect directly to Google or use Partner Interconnect to connect to Google through a supported service provider.

Oracle Exadata Database Service
Oracle Exadata Database Service enables you to leverage the power of Exadata in the cloud. Oracle Exadata Database Service delivers proven Oracle Database capabilities on purpose-built, optimized Oracle Exadata infrastructure in the public cloud and on Cloud@Customer. Built-in cloud automation, elastic resource scaling, security, and fast performance for all Oracle Database workloads helps you simplify management and reduce costs.

 Source Database

The source database runs on-premises on an Oracle Linux Server 7.7 for this step-by-step guide. The host's private IP is masked for this guide, but as an example, we will use the fictional aa.bb.sr.db address, and the hostname is onphost. The source Oracle database is a single-instance Enterprise Edition database version 19.22 with multitenant architecture. The database name is oradb, and its unique name is oradb_onp.

The HR schema to be migrated resides in the source PDB pdbsrc.      

Target Database

Oracle Database@Google Cloud offers the following products:

• Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D)
  • You can provision flexible Exadata systems that allow you to add database compute servers and storage servers to your system anytime after provisioning.
    
    
• Oracle Autonomous Database Serverless (ADB-S)
  • Autonomous Database provides an easy-to-use, fully autonomous database that scales elastically, delivers fast query performance, and requires no database administration.

Oracle Database@Google Cloud integrates Oracle Exadata Database Service, Oracle Real Application Clusters (Oracle RAC), and Oracle Data Guard technologies into the Google Cloud platform. The Oracle Database service runs on Oracle Cloud Infrastructure (OCI) and is co-located in Google’s data centers. The service offers features and price parity with OCI.

Oracle Database@Google Cloud service offers the same low latency as other Google-native services and meets mission-critical workloads and cloud-native development needs. Users manage the service on the Google Cloud console and with Google Cloud automation tools. The service is deployed in Google Virtual Private Cloud (VPC). The service requires that users have a Google Cloud Project and an OCI tenancy.

For this step-by-step guide, the target platform is Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D) on Oracle Database@Google Cloud. The infrastructure contains a 2-node VM cluster. The VM cluster host private IPs are masked for this guide, but as an example, we will use the fictional ta.db.og.1 and ta.db.og.2, and the host names are exadbgoogle1 and exadbgoogle2.

ZDM requires configuring a placeholder database target environment before beginning the migration process.

The target Oracle database is a 2-node Oracle RAC version 19.24 with multitenant architecture created using Oracle Cloud Console. The database name is oradb, and the database's unique name is oradb_exa. 

The HR schema is to be migrated to the target PDB pdbtgt.

Source and Target Database Pre-Requisites

• Oracle GoldenGate requires a unique row identifier on the source and target tables to locate the correct target rows for replicated updates and deletes.
• The character set on the source database must be the same as the target database.
• If the source is Oracle Database 11.2, apply mandatory 11.2.0.4 RDBMS patches on the source database.  See My Oracle Support note Oracle GoldenGate -- Oracle RDBMS Server Recommended Patches (Doc ID 1557031.1)
• If the source database is Oracle Database 12.1.0.2 or a later release, apply mandatory RDBMS patches to it.
• If the source is Oracle Database Standard Edition 2, available with Oracle Database 18c or 19c, and lower than DBRU 19.11, apply the RDBMS patch for bug 29374604—Integrated Extract not starting against Oracle RDBMS Standard Edition.
• The DATAPUMP_IMP_FULL_DATABASE role is required for the import operation at the specified target database for the specified target database user.
• Disable auto-purge jobs on the target database immediately after instantiation (Data Pump Import). Otherwise, the purge jobs will cause data inconsistency conditions on the target database, causing GoldenGate Replicat to fail.

Additional Configuration

SSH Key

Check the key format:

[zdmuser@zdmhost ~]$ head -n1 id_rsa

Create an SSH Key in RSA format (if not already created):

[zdmuser@zdmhost ~]$ ssh-keygen -m PEM -t rsa

Change an existing SSH key into RSA format (if already created and need to reformat):

[zdmuser@zdmhost ~]$ ssh-keygen -p -m PEM -f id_rsa

NFS File Share via Google Cloud Managed NFS File Server

ZDM Logical Online migration workflow uses Oracle Data Pump export and import to migrate the data from the source to the target database. An NFS file share is provided through the Google Cloud-managed NFS File Server to store the backup files. 

The IP address of the NFS server is masked for this guide, but as an example, we will use the fictional aa.an.fs.pe address. The NFS path is aa.an.fs.pe:/zdm_share

The NFS share must be mounted on both the source and target database hosts.

To mount the NFS Share on the source and target database hosts:

As root:

mkdir -p /mnt/nfs3zdm
mount -o rw aa.an.fs.pe:/zdm_share /mnt/nfs3zdm

Make sure the Oracle user has access to the NFS mount

chown oracle:oinstall /mnt/nfs3zdm

As oracle user:

touch /mnt/nfs3zdm/test.txt

On the source and target PDBs:

SQL> create directory DATA_PUMP_DIR_NFS as '/mnt/nfs3zdm';

Oracle GoldenGate on Docker

For ZDM Logical Online migrations to Oracle Database@Google Cloud, you will run Oracle GoldenGate on a Docker VM on-premises or on a Google Cloud Compute VM. Oracle GoldenGate keeps your source and target databases in sync and enables you to achieve zero to negligible downtime for your Oracle database migrations across database versions and platforms. Please contact your account manager, Multi Cloud Product Manager, and the Zero Downtime Migration Product Manager for more details regarding GoldenGate usage for this use case.

For this step-by-step guide, the docker container runs on Google Cloud Compute VM on RHEL version 8.10. The host's private IP
is masked for this guide, but as an example, we will use the fictional aa.bb.gg.do address, and the hostname
is ggdockervm.

Install Oracle GoldenGate on Docker

Step 1: Download the GoldenGate Docker Image

On Oracle Cloud, create a VM using the Oracle GoldenGate – Database Migrations image from Marketplace. Search for “Goldengate migrations” in the Marketplace and choose the Database Migrations image. Choose the latest “Oracle DB – Microservices Edition – Promotional” version, and launch the stack.

Once the stack is completed, a VM in OCI Compute Instances will be created with the details you provided during the stack launch. Log in to that VM via SSH and issue the shell list command:

-bash-4.2$ ls -l
lrwxrwxrwx. 1 opc opc 36 Oct 3 11:21 ora21c-2115001.tar -> /opt/dockerimages/ora21c-2115001.tar

The file /opt/dockerimages/ora21c-2115001.tar is the GoldenGate Docker image. Copy it to your Google Cloud Compute VM. Once done, the VM in OCI can be terminated. Its only purpose was to download the ora21c-2115001.tar Docker image file.

Step 2: Set Up the Docker Engine

Set up the Docker engine on the Google Cloud Compute VM to host the GoldenGate Docker image. In this case, following the Install Docker engine on Oracle RHEL 8 available here:

# add docker repository
[gcpuser@ggdockervm ~]$ sudo yum install -y yum-utils
[gcpuser@ggdockervm ~]$ sudo yum-config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo

# install docker engine
[gcpuser@ggdockervm ~]$ sudo yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

# start and enable docker service
[gcpuser@ggdockervm ~]$ sudo systemctl start docker
[gcpuser@ggdockervm ~]$ sudo systemctl enable docker

# verify docker installation
[gcpuser@ggdockervm ~]$ docker –version
[gcpuser@ggdockervm ~]$ docker compose version

  
#Add the local user to docker group to run the docker commands with sudo
[gcpuser@ggdockervm ~]$ sudo usermod -aG docker <your_user_name>
[gcpuser@ggdockervm ~]$ newgrp docker
[gcpuser@ggdockervm ~]$ id

Step 3: Load the Docker Image

Load the Docker image to the Docker engine. The ora21c-2115001.tar file is the one you copied to this VM in Step 0:

[gcpuser@ggdockervm ~]$ sudo docker load < ./ora21c-2115001.tar
67d008ba80bc: Loading layer [==================================================>]  253.7MB/253.7MB
45904b12df07: Loading layer [==================================================>]  6.144kB/6.144kB
30d4b06f2cea: Loading layer [==================================================>]  376.3MB/376.3MB
d43b06bdfe01: Loading layer [==================================================>]  27.14kB/27.14kB
369165b4aa0e: Loading layer [==================================================>]  1.787GB/1.787GB
82e31cbe7ea8: Loading layer [==================================================>]  18.43kB/18.43kB
Loaded image: oracle/goldengate:21.15.0.0.1

List the images:

[gcpuser@ggdockervm ~]$ sudo docker image list
REPOSITORY          TAG           IMAGE ID       CREATED         SIZE
oracle/goldengate   21.15.0.0.1   062a307e99c7   2 months ago    2.4GB

Step 4: Run the Docker Image

Run the Oracle GoldenGate Docker image as a container:

[gcpuser@ggdockervm ~]$ sudo docker run --name ogg2115 -p 443:443 docker.io/oracle/goldengate:21.15.0.0.1

For more information about the run parameters, visit Running Oracle GoldenGate in a Container.

The run output will display the ggadmin user password. You will need this later when running the ZDM migration command:

----------------------------------------------------------------------------------

--  Password for OGG administrative user 'oggadmin' is 'SamplePassword1234*&=+'

----------------------------------------------------------------------------------
Check the status of the Docker container:
[gcpuser@ggdockervm ~]$ sudo docker ps -a

CONTAINER ID   IMAGE                           COMMAND                  CREATED              STATUS                        PORTS                                           NAMES

135ed264e10c   oracle/goldengate:21.15.0.0.1   "/usr/local/bin/depl…"   About a minute ago   Up About a minute (healthy)   80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   ogg2115
To start and stop the Docker container:
[gcpuser@ggdockervm ~]$ sudo docker stop 135ed264e10c
[gcpuser@ggdockervm ~]$ sudo docker start 135ed264e10c

Step 5: Ensure Connectivity to Source and Target Databases

Ensure that the source and target database hostnames are resolvable and that you can connect to the source and target databases from within the Docker container.

Download Oracle Instant Client locally, and proceed to copy it to the docker container.

[gcpuser@ggdockervm ~]$ sudo docker cp instantclient_21_3.zip 135ed264e10c:/ instantclient_21_3.zip

Connect to the docker container:

[gcpuser@ggdockervm ~]$ sudo docker exec -it ogg2115 /bin/bash
[root@135ed264e10c /]# mv instantclient_21_3.zip /home/ogg/
[root@135ed264e10c /]# chown ogg:ogg /home/ogg/instantclient_21_3.zip
[root@135ed264e10c /]# su - ogg
[ogg@135ed264e10c /]# unzip instantclient_21_3.zip
[ogg@135ed264e10c /]# chmod 744 instantclient_21_3/sqlplus

Edit the etc hosts file on the docker container:

[root@ggdockervm ~]$ vi /etc/hosts
aa.bb.sr.db onphost
ta.db.og.1 exadbgoogle1
ta.db.og.2 exadbgoogle2

Check database connectivity to the source and target using the hostnames in the connection string:

[ogg@135ed264e10c ]# sqlplus system@onphost:1521/oradb_onp
[ogg@135ed264e10c ]# sqlplus system@ exadbgoogle1:1521/oradb_exa

Database Migration Step by Step with ZDM

Step 1: Prepare the Source Database Host On-Premises

Copy the SSH public key of the zdmuser from the ZDM host to the .ssh/authorized_keys file on the source database host for the user you want to use for login, in this case, onpuser:

On ZDM host as zdmuser

[zdmuser@zdmhost ~]$ cat .ssh/id_rsa.pub

On the source database host as user onpuser

[onpuser@onphost ~]$ vi .ssh/authorized_keys

Insert the public key and save the changes.
Add the target database hostname, IP address, and SCAN name to the /etc/hosts file. As root user:

[root@onphost ~]# vi /etc/hosts
#add the following entries
ta.db.og.1 exadbgoogle1
ta.db.og.2 exadbgoogle2
ta.db.og.3 demo-scan-sample.oravcn.sample.com target-scan

Step 2: Prepare the Source Database On-Premises

For online logical migrations, set STREAMS_POOL_SIZE to at least 2GB. See MOS Note 2078459.1 for the recommendation 1GB STREAMS_POOL_SIZE per integrated extract + additional 25 percent.

As SYS user:

-- Set streams_pool_size to 2G
SQL> alter system set streams_pool_size=2G scope=both;

-- Set global_names to false
SQL> alter system set global_names=false; 

-- Enable ARCHIVELOG mode:
SQL> select log_mode from v$database;
LOG_MODE

------------

NOARCHIVELOG
SQL> shutdown immediate;
SQL> startup mount
SQL> alter database archivelog;
SQL> alter database open;
SQL> select log_mode from v$database;
LOG_MODE

------------

ARCHIVELOG
-- Enable FORCE LOGGING to ensure that all changes are found in the redo by the Oracle GoldenGate Extract process:
SQL> select force_logging from v$database;
FORCE_LOGGING

---------------------------------------
NO
SQL> alter database force logging;
SQL> select force_logging from v$database;
FORCE_LOGGING

---------------------------------------
YES

-- Enable database minimal supplemental logging:
SQL> select minimal from dba_supplemental_logging;

MINIMAL

----------
NO
SQL> alter database add supplemental log data;
SQL> select minimal from dba_supplemental_logging;
MINIMAL

----------
YES

-- Enable initialization parameter ENABLE_GOLDENGATE_REPLICATION:
SQL> alter system set ENABLE_GOLDENGATE_REPLICATION=TRUE scope=both;
System altered.

-- In case of Multitenant, create the user c##ggadmin in CDB$ROOT:
SQL> create user c##ggadmin identified by VerySecretPw_22 default tablespace users temporary tablespace temp;
grant connect, resource to c##ggadmin;
grant unlimited tablespace to c##ggadmin;
alter user c##ggadmin quota 10G on users;
grant select any dictionary to c##ggadmin;
grant create view to c##ggadmin;
grant execute on dbms_lock to c##ggadmin;
grant set container to c##ggadmin container=all;

exec dbms_goldengate_auth.GRANT_ADMIN_PRIVILEGE('c##ggadmin',container=>'all');

  

-- Create a GoldenGate administration user, ggadmin (in the PDB in case of Multitenant):
SQL> alter session set container=pdbsrc;
create user ggadmin identified by VerySecretPw_22 default tablespace users temporary tablespace temp;
grant connect, resource to ggadmin;
grant unlimited tablespace to ggadmin;
alter user ggadmin quota 10G on users;

grant select any dictionary to ggadmin;

grant create view to ggadmin;

grant execute on dbms_lock to ggadmin;

exec dbms_goldengate_auth.GRANT_ADMIN_PRIVILEGE('ggadmin');

Step 3: Prepare the target database host on ExaDB-D on Oracle Database@Google Cloud

Copy the SSH public key of the zdmuser from the ZDM host to the .ssh/authorized_keys file on the target database host for the user you want to use for login; in this case, OPC:

On ZDM host as zdmuser

[zdmuser@zdmhost ~]$ cat .ssh/id_rsa.pub

On the target database hosts as user opc (on all VMs of the VM cluster)

[opc@exadbgoogle1 ~]$ vi .ssh/authorized_keys

Insert the public key and save the changes

[opc@exadbgoogle2 ~]$ vi .ssh/authorized_keys

Insert the public key and save the changes

Add the source database hostname and IP information into the /etc/hosts file. As root user (on all VMs of the VM cluster)

[root@exadbgoogle1 ~]# vi /etc/hosts
Add the following entries
aa.bb.sr.db onphost

[root@exadbgoogle2 ~]# vi /etc/hosts
#add the following entries
aa.bb.sr.db onphost

Step 4: Prepare the target database on ExaDB-D on Oracle Database@Google Cloud

Prepare the Target Database, as SYS user:

-- on target CDB:
alter system set ENABLE_GOLDENGATE_REPLICATION=TRUE scope=both;

-- on target PDB: create ggadmin on target PDB and grant privileges
alter session set container=pdbtgt;
create user ggadmin identified by <password> default tablespace users temporary tablespace temp;
alter user ggadmin quota 100M ON USERS;
grant connect, resource to ggadmin;
grant unlimited tablespace to ggadmin;
grant select any dictionary to ggadmin;
grant create view to ggadmin;
grant select any table to ggadmin;
grant insert any table to ggadmin;
grant update any table to ggadmin;
grant delete any table to ggadmin;
grant comment any table to ggadmin;
grant execute on dbms_lock to ggadmin;
exec dbms_goldengate_auth.GRANT_ADMIN_PRIVILEGE('ggadmin');

Step 5: Prepare the ZDM Service Host On-Premises

Add the source and target hostnames and IP addresses into the /etc/hosts file. As root user:

[root@zdmhost ~]# vi /etc/hosts
#add the following entries
ta.db.og.1 exadbgoogle1
ta.db.og.2 exadbgoogle2
aa.bb.sr.db onphost

Test the SSH connectivity to the source and target database hosts:

[zdmuser@zdmhost ~]$ ssh -i /home/zdmuser/.ssh/id_rsa onpuser@onphost
[zdmuser@zdmhost ~]$ ssh -i /home/zdmuser/.ssh/id_rsa opc@exadbgoogle1
[zdmuser@zdmhost ~]$ ssh -i /home/zdmuser/.ssh/id_rsa opc@exadbgoogle2

Verify that TTY is disabled for the SSH-privileged user. If TTY is disabled, the following command returns the date from the remote host without any errors:

[zdmuser@zdmhost ~]$ ssh -oStrictHostKeyChecking=no -i /home/zdmuser/.ssh/id_rsa onpuser@onphost "/usr/bin/sudo /bin/sh -c date"
[zdmuser@zdmhost ~]$ ssh -oStrictHostKeyChecking=no -i /home/zdmuser/.ssh/id_rsa opc@exadbgoogle1 "/usr/bin/sudo /bin/sh -c date"
[zdmuser@zdmhost ~]$ ssh -oStrictHostKeyChecking=no -i /home/zdmuser/.ssh/id_rsa opc@exadbgoogle2 "/usr/bin/sudo /bin/sh -c date"

These commands should execute without any prompting and return the date from the remote host. 

Step 6: Create the Logical Online Migration Response File on the ZDM host

You’ll find a template on the ZDM host at $ZDMHOME/rhp/zdm/template/zdm_template.rsp, briefly describing the parameters and their possible values. Here, we will create a new response file with the minimal parameters required. As zdmuser:

[zdmuser@zdmhost ~]$ vi /home/zdmuser/logical_online/logical_online.rsp
#add the following parameters and save the changes
# migration method
MIGRATION_METHOD=ONLINE_LOGICAL
DATA_TRANSFER_MEDIUM=NFS

# data pump
DATAPUMPSETTINGS_JOBMODE=SCHEMA
INCLUDEOBJECTS-1=owner:HR
DATAPUMPSETTINGS_METADATAREMAPS-1=type:REMAP_TABLESPACE,oldValue:USERS,newValue:DATA
DATAPUMPSETTINGS_DATAPUMPPARAMETERS_EXPORTPARALLELISMDEGREE=2
DATAPUMPSETTINGS_DATAPUMPPARAMETERS_IMPORTPARALLELISMDEGREE=2
DATAPUMPSETTINGS_EXPORTDIRECTORYOBJECT_NAME=DATA_PUMP_DIR_NFS
DATAPUMPSETTINGS_IMPORTDIRECTORYOBJECT_NAME=DATA_PUMP_DIR_NFS


# on source and target db: select directory_name, directory_path from dba_directories;
DATAPUMPSETTINGS_EXPORTDIRECTORYOBJECT_PATH=/mnt/nfs3zdm
DATAPUMPSETTINGS_IMPORTDIRECTORYOBJECT_PATH=/mnt/nfs3zdm

# source db (pdb)

SOURCEDATABASE_CONNECTIONDETAILS_HOST=onphost
SOURCEDATABASE_CONNECTIONDETAILS_PORT=1521
SOURCEDATABASE_CONNECTIONDETAILS_SERVICENAME=pdbsrc
SOURCEDATABASE_ADMINUSERNAME=SYSTEM
SOURCEDATABASE_GGADMINUSERNAME=ggadmin


# source db (cdb)
SOURCECONTAINERDATABASE_CONNECTIONDETAILS_HOST=onphost
SOURCECONTAINERDATABASE_CONNECTIONDETAILS_PORT=1521
SOURCECONTAINERDATABASE_CONNECTIONDETAILS_SERVICENAME=oradb
SOURCECONTAINERDATABASE_ADMINUSERNAME=SYSTEM
SOURCECONTAINERDATABASE_GGADMINUSERNAME=c##ggadmin


# target db (pdb)
TARGETDATABASE_CONNECTIONDETAILS_HOST=exadbgoogle1
TARGETDATABASE_CONNECTIONDETAILS_PORT=1521
TARGETDATABASE_CONNECTIONDETAILS_SERVICENAME=test.test.oraclevcn.com
TARGETDATABASE_ADMINUSERNAME=SYSTEM
TARGETDATABASE_GGADMINUSERNAME=ggadmin


# oci cli
OCIAUTHENTICATIONDETAILS_USERPRINCIPAL_USERID=ocid1.user.oc1..aaaaaaaa
OCIAUTHENTICATIONDETAILS_USERPRINCIPAL_TENANTID=ocid1.tenancy.oc1..aaaaaaaa
OCIAUTHENTICATIONDETAILS_USERPRINCIPAL_FINGERPRINT=<fingerpring>
OCIAUTHENTICATIONDETAILS_USERPRINCIPAL_PRIVATEKEYFILE=/home/zdmuser/.oci/oci_api_key.pem
OCIAUTHENTICATIONDETAILS_REGIONID=us-ashburn-1


# GoldenGate
GOLDENGATEHUB_ADMINUSERNAME=oggadmin
GOLDENGATEHUB_SOURCEDEPLOYMENTNAME=Local
GOLDENGATEHUB_TARGETDEPLOYMENTNAME=Local

# Private IP of the VM where Docker is running
GOLDENGATEHUB_URL=https://aa.bb.gg.do
GOLDENGATEHUB_ALLOWSELFSIGNEDCERTIFICATE=TRUE

Step 7: Evaluate the Configuration

Execute the following command on the ZDM host as zdmuser to evaluate the migration. ZDM will check the source and target database configurations. The actual migration will not be started. On the ZDM host as zdmuser:

[zdmuser@zdmhost ~]$ $ZDMHOME/bin/zdmcli migrate database \
-rsp /home/zdmuser/logical_online/logical_online.rsp \
-sourcenode onphost \
-sourcesid oradb \
-srcauth zdmauth \
-srcarg1 user:onpuser \
-srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa \
-srcarg3 sudo_location:/usr/bin/sudo \
-targetnode exadbgoogle1 \
-tgtauth zdmauth \
-tgtarg1 user:opc \
-tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa \
-tgtarg3 sudo_location:/usr/bin/sudo \
-eval

Enter source database administrative user "SYSTEM" password:
Enter source database administrative user "ggadmin" password:
Enter source container database administrative user "SYSTEM" password:
Enter source container database administrative user "c##ggadmin" password:
Enter target database administrative user "ADMIN" password:
Enter target database administrative user "ggadmin" password:
Enter Oracle GoldenGate hub administrative user "oggadmin" password:
Enter Data Pump encryption password:
Operation "zdmcli migrate database" scheduled with the job ID "1".

If the source database uses ASM for storage management, use -sourcedb <db_unique_name> instead of -sourcesid <SID> in the zdmcli command.

Check the job status. On the ZDM host as zdmuser:

[zdmuser@zdmhost ~]$ $ZDMHOME/bin/zdmcli query job -jobid 1 
zdmhost.zdm: Audit ID: 334
Job ID: 1
User: zdmuser
Client: zdmhost
Job Type: "EVAL"

Scheduled job command: "zdmcli migrate database -rsp /home/zdmuser/logical_online_exa/logical_online_exa.rsp -sourcenode onphost -sourcesid oradb -srcauth zdmauth -srcarg1 user:sinan_petrus_toma_oracle_com -srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa -srcarg3 sudo_location:/usr/bin/sudo -targetnode targethost -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -eval"

Scheduled job execution start time: 2024-10-22T13:09:30Z. Equivalent local time: 2024-10-22 13:09:30
Current status: SUCCEEDED
Result file path: "/home/zdmuser/zdm/zdmbase/chkbase/scheduled/job-1-2024-10-22-13:09:54.log"
Metrics file path: "/home/zdmuser/zdm/zdmbase/chkbase/scheduled/job-1-2024-10-22-13:09:54.json"
Excluded objects file path: "/home/zdmuser/zdm/zdmbase/chkbase/scheduled/job-1-filtered-objects-2024-10-22T13:10:15.698.json"
Job execution start time: 2024-10-22 13:09:54
Job execution end time: 2024-10-22 13:11:57
Job execution elapsed time: 2 minutes 2 seconds

ZDM_VALIDATE_TGT ...................... COMPLETED
ZDM_VALIDATE_SRC ...................... COMPLETED
ZDM_SETUP_SRC ......................... COMPLETED
ZDM_PRE_MIGRATION_ADVISOR ............. COMPLETED
ZDM_VALIDATE_GG_HUB ................... COMPLETED
ZDM_VALIDATE_DATAPUMP_SETTINGS_SRC .... COMPLETED
ZDM_VALIDATE_DATAPUMP_SETTINGS_TGT .... COMPLETED
ZDM_PREPARE_DATAPUMP_SRC .............. COMPLETED
ZDM_DATAPUMP_ESTIMATE_SRC ............. COMPLETED
ZDM_CLEANUP_SRC ....................... COMPLETED

Detailed information about the migration process can be found by monitoring the log file: 

[zdmuser@zdmhost ~]$ tail -f /home/zdmuser/zdm/zdmbase/chkbase/scheduled/job-1-2024-10-22-13:09:54.log

Step 8: Initiate the Migration

To initiate the actual migration, execute the same command for evaluation, but this time without the -eval parameter.
Oracle ZDM allows you to pause the migration process at any given phase. For example, the migration process can be
paused after Oracle GoldenGate is keeping the target database in sync with the source. Upon executing the zdm migrate database command, the- pauseafter flag must be entered with the desired pausing stage, which in this case is ZDM_MONITOR_GG_LAG.
On the ZDM host as zdmuser:

[zdmuser@zdmhost ~]$ $ZDMHOME/bin/zdmcli migrate database \
-rsp /home/zdmuser/logical_online/logical_online.rsp \
-sourcenode onphost \
-sourcesid oradb \
-srcauth zdmauth \
-srcarg1 user:onpuser \
-srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa \
-srcarg3 sudo_location:/usr/bin/sudo \
-targetnode exadbgoogle1 \
-tgtauth zdmauth \
-tgtarg1 user:opc \
-tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa \
-tgtarg3 sudo_location:/usr/bin/sudo \
-pauseafter ZDM_MONITOR_GG_LAG


Enter source database administrative user "SYSTEM" password:
Enter source database administrative user "ggadmin" password:
Enter source container database administrative user "SYSTEM" password:
Enter source container database administrative user "c##ggadmin" password:
Enter target database administrative user "ADMIN" password:
Enter target database administrative user "ggadmin" password:
Enter Oracle GoldenGate hub administrative user "oggadmin" password:
Enter Data Pump encryption password:
Operation "zdmcli migrate database" scheduled with the job ID "2".

Check the job status. On the ZDM host as zdmuser:

[zdmuser@zdmhost ~]$ $ZDMHOME/bin/zdmcli query job -jobid 2

zdmhost.zdm: Audit ID: 340
Job ID: 2
User: zdmuser
Client: zdmhost
Job Type: "MIGRATE"
Scheduled job command: "zdmcli migrate database -rsp /home/zdmuser/logical_online_exa/logical_online_exa.rsp -sourcenode onphost -sourcesid oradb -srcauth zdmauth -srcarg1 user:sinan_petrus_toma_oracle_com -srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa -srcarg3 sudo_location:/usr/bin/sudo -targetnode targethost -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -pauseafter ZDM_MONITOR_GG_LAG"
Scheduled job execution start time: 2024-10-22T13:13:22Z. Equivalent local time: 2024-10-22 13:13:22
Current status: PAUSED
Current Phase: "ZDM_MONITOR_GG_LAG"
Result file path: "/home/zdmuser/zdm/zdmbase/chkbase/scheduled/job-2-2024-10-22-13:13:24.log"
Metrics file path: "/home/zdmuser/zdm/zdmbase/chkbase/scheduled/job-2-2024-10-22-13:13:24.json"
Excluded objects file path: "/home/zdmuser/zdm/zdmbase/chkbase/scheduled/job-2-filtered-objects-2024-10-22T13:13:43.708.json"
Job execution start time: 2024-10-22 13:13:24
Job execution end time: 2024-10-22 13:21:47
Job execution elapsed time: 8 minutes 22 seconds
Oracle GoldenGate replication metrics:
Extract "EXT3N8VI" status: RUNNING
Extract "EXT3N8VI" trail files generated: 1
Replicat "RMGZ3" status: RUNNING
Replicat "RMGZ3" trail files applied: 1
End-to-end heartbeat lag 1.8 seconds
Replication throughput: 0.0 GBph

ZDM_VALIDATE_TGT ...................... COMPLETED
ZDM_VALIDATE_SRC ...................... COMPLETED
ZDM_SETUP_SRC ......................... COMPLETED
ZDM_PRE_MIGRATION_ADVISOR ............. COMPLETED
ZDM_VALIDATE_GG_HUB ................... COMPLETED
ZDM_VALIDATE_DATAPUMP_SETTINGS_SRC .... COMPLETED
ZDM_VALIDATE_DATAPUMP_SETTINGS_TGT .... COMPLETED
ZDM_PREPARE_DATAPUMP_SRC .............. COMPLETED
ZDM_DATAPUMP_ESTIMATE_SRC ............. COMPLETED
ZDM_PREPARE_GG_HUB .................... COMPLETED
ZDM_ADD_HEARTBEAT_SRC ................. COMPLETED
ZDM_ADD_SCHEMA_TRANDATA_SRC ........... COMPLETED
ZDM_CREATE_GG_EXTRACT_SRC ............. COMPLETED
ZDM_PREPARE_DATAPUMP_TGT .............. COMPLETED
ZDM_DATAPUMP_EXPORT_SRC ............... COMPLETED
ZDM_TRANSFER_DUMPS_SRC ................ COMPLETED
ZDM_DATAPUMP_IMPORT_TGT ............... COMPLETED
ZDM_POST_DATAPUMP_SRC ................. COMPLETED
ZDM_POST_DATAPUMP_TGT ................. COMPLETED
ZDM_ADD_HEARTBEAT_TGT ................. COMPLETED
ZDM_ADD_CHECKPOINT_TGT ................ COMPLETED
ZDM_CREATE_GG_REPLICAT_TGT ............ COMPLETED
ZDM_START_GG_REPLICAT_TGT ............. COMPLETED
ZDM_MONITOR_GG_LAG .................... COMPLETED
ZDM_PREPARE_SWITCHOVER_APP ............ PENDING
ZDM_ADVANCE_SEQUENCES ................. PENDING
ZDM_SWITCHOVER_APP .................... PENDING
ZDM_POST_SWITCHOVER_TGT ............... PENDING
ZDM_RM_GG_EXTRACT_SRC ................. PENDING
ZDM_RM_GG_REPLICAT_TGT ................ PENDING
ZDM_DELETE_SCHEMA_TRANDATA_SRC ........ PENDING
ZDM_RM_HEARTBEAT_SRC .................. PENDING
ZDM_RM_CHECKPOINT_TGT ................. PENDING
ZDM_RM_HEARTBEAT_TGT .................. PENDING
ZDM_CLEAN_GG_HUB ...................... PENDING
ZDM_POST_ACTIONS ...................... PENDING
ZDM_CLEANUP_SRC ....................... PENDING


Pause After Phase: "ZDM_MONITOR_GG_LAG"

Pay attention to the current job status. It is in PAUSED status now. Also, the progress stopped after phase ZDM_MONITOR_GG_LAG was COMPLETED. At this stage, every change in the source database is immediately synchronized with the target database. Resume the job when your application is ready for migration.

Step 9: Complete the Migration

Resume the job from the previous step. On the ZDM host as zdmuser, resume the job and also query the status until all phases are completed:

[zdmuser@zdmhost ~]$ $ZDMHOME/bin/zdmcli resume job -jobid 2

[zdmuser@zdmhost ~]$ $ZDMHOME/bin/zdmcli query job -jobid 2

...

ZDM_VALIDATE_TGT ...................... COMPLETED
ZDM_VALIDATE_SRC ...................... COMPLETED
ZDM_SETUP_SRC ......................... COMPLETED
ZDM_PRE_MIGRATION_ADVISOR ............. COMPLETED
ZDM_VALIDATE_GG_HUB ................... COMPLETED
ZDM_VALIDATE_DATAPUMP_SETTINGS_SRC .... COMPLETED
ZDM_VALIDATE_DATAPUMP_SETTINGS_TGT .... COMPLETED
ZDM_PREPARE_DATAPUMP_SRC .............. COMPLETED
ZDM_DATAPUMP_ESTIMATE_SRC ............. COMPLETED
ZDM_PREPARE_GG_HUB .................... COMPLETED
ZDM_ADD_HEARTBEAT_SRC ................. COMPLETED
ZDM_ADD_SCHEMA_TRANDATA_SRC ........... COMPLETED
ZDM_CREATE_GG_EXTRACT_SRC ............. COMPLETED
ZDM_PREPARE_DATAPUMP_TGT .............. COMPLETED
ZDM_DATAPUMP_EXPORT_SRC ............... COMPLETED
ZDM_TRANSFER_DUMPS_SRC ................ COMPLETED
ZDM_DATAPUMP_IMPORT_TGT ............... COMPLETED
ZDM_POST_DATAPUMP_SRC ................. COMPLETED
ZDM_POST_DATAPUMP_TGT ................. COMPLETED
ZDM_ADD_HEARTBEAT_TGT ................. COMPLETED
ZDM_ADD_CHECKPOINT_TGT ................ COMPLETED
ZDM_CREATE_GG_REPLICAT_TGT ............ COMPLETED
ZDM_START_GG_REPLICAT_TGT ............. COMPLETED
ZDM_MONITOR_GG_LAG .................... COMPLETED
ZDM_PREPARE_SWITCHOVER_APP ............ COMPLETED
ZDM_ADVANCE_SEQUENCES ................. COMPLETED
ZDM_SWITCHOVER_APP .................... COMPLETED
ZDM_POST_SWITCHOVER_TGT ............... COMPLETED
ZDM_RM_GG_EXTRACT_SRC ................. COMPLETED
ZDM_RM_GG_REPLICAT_TGT ................ COMPLETED
ZDM_DELETE_SCHEMA_TRANDATA_SRC ........ COMPLETED
ZDM_RM_HEARTBEAT_SRC .................. COMPLETED
ZDM_RM_CHECKPOINT_TGT ................. COMPLETED
ZDM_RM_HEARTBEAT_TGT .................. COMPLETED
ZDM_CLEAN_GG_HUB ...................... COMPLETED
ZDM_POST_ACTIONS ...................... COMPLETED
ZDM_CLEANUP_SRC ....................... COMPLETED

Known Issues

All common issues are documented and updated periodically in Oracle Zero Downtime Migration’s documentation, specifically on the product release note, Known Issues section:                      

https://docs.oracle.com/en/database/oracle/zero-downtime-migration/

Troubleshooting Oracle GoldenGate Replication

During your migration, you can pause the ZDM migration job after the ZDM_MONITOR_GG_LAG phase.

At this stage, every transaction on the source database is replicated via GoldenGate to the target database. If this is not the case, log in to the Docker container and check the EXTRACT and REPLICATs deployments and log files:

Connect to the docker container and switch to ogg user

[gcpuser@ggdockervm ~]$ sudo docker exec -it ogg2115 /bin/bash
[root@ 135ed264e10c /]# su - ogg

Check the deployment files

[ogg@ 135ed264e10c ~]$ ls -l /u02/Deployment/etc/conf/ogg
total 16
-rw-r-----. 1 ogg ogg 257 Jun 10 19:05 EXT4BL3P.prm
-rw-r--r--. 1 ogg ogg  17 Jun 10 13:05 GLOBALS
-rw-r-----. 1 ogg ogg 260 Jun 10 19:34 RN22M.prm
-rw-r-----. 1 ogg ogg 260 Jun 10 19:57 RN22M.prm.backup

Check the deployments log files

[ogg@ 135ed264e10c ~]$ view /u02/Deployment/var/log/extract.log
[ogg@ 135ed264e10c ~]$ view /u02/Deployment/var/log/replicat.log

Check the Deployment Status:

[ogg@ 135ed264e10c ~]$ /u01/ogg/bin/adminclient
Oracle GoldenGate Administration Client for Oracle
Version 21.15.0.0.1 OGGCORE_21.15.0.0.1OGGRU_PLATFORMS_240108.2205
  
Copyright (C) 1995, 2024, Oracle and/or its affiliates. All rights reserved.
  
Oracle Linux 7, x64, 64bit (optimized) on Jan  9 2024 09:04:36

Operating system character set identified as US-ASCII. 

OGG (not connected) 1>
OGG (not connected) 1> connect http://127.0.0.1 as oggadmin password /Qlq1UTGMK+N-EksH

Using default deployment 'Local'
 

OGG (http://127.0.0.1 Local) 2> info all
Program     Status      Group       Type             Lag at Chkpt  Time Since Chkpt
  

ADMINSRVR   RUNNING
DISTSRVR    RUNNING
PMSRVR      RUNNING
RECVSRVR    RUNNING
EXTRACT     RUNNING     EXT4BL3P    INTEGRATED       00:00:01      00:00:05
REPLICAT    ABENDED     RN22M       PARALLEL NONINT  00:04:21      00:00:21

In this case, the REPLICAT status was ABENDED. The replicat.log indicated insufficient privileges for the ggadmin user on the target database. After fixing the issue, start the deployment:

OGG (http://127.0.0.1 Local) 3> start RN22M
2024-10-12T20:35:54Z  INFO    OGG-00975  Replicat group RN22M starting.
2024-10-12T20:35:54Z  INFO    OGG-15445  Replicat group RN22M started.

Troubleshooting Oracle ZDM & Other Resources

For Oracle ZDM log review:

• ZDM Server Logs:
  • $ZDM_BASE/crsdata/<zdm_service_node>/rhp/rhpserver.log.0
    
    
• Check source node logs
  • <oracle_base>/zdm/zdm_<src_db_name>_<job_id>/zdm/log
    
    
• Check target node logs.
  • <oracle_base>/zdm/zdm_<tgt_db_name>_<job_id>/zdm/log

For all Oracle Support Service Requests related to Zero Downtime Migration, please be sure to follow the instructions in My Oracle Support Document:

• SRDC – Data Collection for Database Migration Using Zero Downtime Migration (ZDM) (DOC ID 2595205.1)
• https://support.oracle.com/epmos/faces/DocContentDisplay?id=2595205.1

---

Title and Copyright Information

Oracle® Database, Oracle Zero Downtime Migration, Release 21.5

G23897-01

February 01, 2025

Copyright ©2023, 2025,

Oracle and/or its affiliates.