1. Oracle Zero Downtime Migration
  2. ZDM – Logical Online Migration to ADB-S on Oracle Database@Azure

ZDM – Logical Online Migration to ADB-S on Oracle Database@Azure

Purpose statement

Oracle customers are rapidly increasing their migration of workloads into the Oracle Cloud, Engineered Systems, and Oracle Database@Azure. However, migrating workloads has been a source of challenges for many years. Migrating database workloads from one system to another or into the Cloud is easier said than done. 

Based on years of experience migrating Oracle workloads, Oracle has developed Zero Downtime Migration (ZDM). ZDM is Oracle’s premier solution for a simplified and automated migration experience, providing zero to negligible downtime for the production system depending on the migration scenario. ZDM allows you to migrate your on-premises Oracle Databases directly and seamlessly to and between Oracle Database@Azure and any Oracle-owned infrastructure, including Exadata Database Machine On-Premises, Exadata Cloud at Customer (ExaDB-C@C), and Oracle Cloud Infrastructure. Oracle ZDM supports a wide range of Oracle Database versions and, as the name implies, ensures minimal to no production database impact during the migration.

ZDM follows Oracle Maximum Availability Architecture (MAA) principles and incorporates products such as GoldenGate and Data Guard to ensure High Availability and an online migration workflow that leverages technologies such as the Recovery Manager, Data Pump, and Database Links.

This technical brief is a step-by-step guide for migrating your on-premises Oracle Databases to Oracle Autonomous Database Serverless (ADB-S) on Oracle Database@Azure with ZDM’s Logical Online workflow.

Oracle ZDM will run on a separate node and connect to Source and Target to perform the migration. This guide will cover all requirements for installing the Oracle ZDM Service Host, the Source Database, the Target Database recipient of the migration process, and the networking used. The migration process will be dissected and done in a step-by-step fashion. This guide will answer the most frequently asked questions regarding the product and the overall migration process.

For more information on Oracle Zero Downtime Migration, please visit ZDM’s product website and Oracle Database@Azure product website.

Zero Downtime Migration

Oracle Zero Downtime Migration (ZDM) is the Oracle Maximum Availability Architecture (MAA)-recommended solution to migrate Oracle Databases to the Oracle Cloud. ZDM's inherent design keeps in mind the migration process as straightforward as possible to ensure the most negligible impact on production workloads. The Source Database to be migrated can be on-premises, deployed on Oracle Cloud Infrastructure, or a 3rd Party Cloud. The Target Database deployment can be in Oracle Autonomous Database or Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D) on Oracle Database@Azure, Database Cloud Service on Oracle Cloud Infrastructure (OCI) Virtual Machine, Exadata Cloud Service, Exadata Cloud at Customer, and Autonomous Database. ZDM automates the entire migration process, reducing the chance of human errors. ZDM leverages Oracle Database-integrated high availability (HA) technologies such as Oracle Data Guard and GoldenGate and follows all MAA best practices that ensure no significant downtime of production environments. Oracle ZDM supports both Physical and Logical Migration workflows. This technical brief covers a step-by-step guide for the Logical Online Migration Workflow.

A standard Logical Online migration with Oracle GoldenGate and Data Pump Export and Import will take the following steps:

  1. Download and Configure ZDM.
  2. ZDM Starts Database Migration.
  3. ZDM Configures an Oracle GoldenGate Extract Microservice.
  4. ZDM Starts a Data Pump Export Job.
  5. ZDM Starts a Data Pump Import Job.
  6. ZDM Configures an Oracle GoldenGate Replicat Microservice.
  7. ZDM Monitors Oracle GoldenGate Replication.
  8. ZDM Switches Over.
  9. ZDM Validates, Cleans Up, and Finalizes.

Supported Configurations

Oracle ZDM supports Oracle Database versions 11.2.0.4, 12.1.0.2, 12.2.0.1, 18c, 19c and 21c. ZDM’s physical migration workflow requires the Source and Target Databases to be in the same database release.

Oracle ZDM supports Source Oracle Databases hosted on Linux, Solaris, and AIX operating systems. Oracle ZDM supports single-instance databases, Oracle RAC One Node databases, or Oracle RAC databases as sources. Oracle ZDM supports Oracle Database Enterprise & Standard Edition as Source and Target Databases.  

Architecture

An architectural overview of the ZDM server, the source database on-premises, the target database on Oracle Autonomous Database Serverless (ADB-S) on Oracle Database@Azure, the Oracle GoldenGate on Docker, and all networks and components required are described in the diagram below:

High-Level Architectural overview showcasing the customer data center where the source database and ZDM’s server reside. It also shows all connectivity to the target Oracle Autonomous Database Serverless on Oracle Database@Azure

Figure 1. This is a High-Level Architectural overview showcasing the customer data center where the source database and ZDM’s server reside. It also shows all connectivity to the target Oracle Autonomous Database Serverless on Oracle Database@Azure.

Zero Downtime Migration Service Host

Zero Downtime Migration Service Host Requirements

Oracle Zero Downtime Migration installation must take place on a separate host, which must fulfill the following requirements:

  • Linux host running on Oracle 7, 8, or RHEL 8 (only these OS platforms/versions are supported).
  • 100 GB of free storage space. This space is required for all the logs that ZDM will generate.
  • A zdm group and a zdmuser as part of this group.
  • The following packages must be installed:
    • glibc-devel
    • expect
    • unzip
    • libaio
    • oraclelinux-developer-release-el7
  • All hostnames and IP addresses to be used must be present as entries in the /etc/hosts

For more information on the ZDM Service Host requirements and setting up ZDM on RHEL platforms, please refer to Oracle ZDM’s product documentation, specifically “Setting Up Zero Downtime Migration Software” section.

For this step-by-step guide, the ZDM Service Host runs on-premises on an Oracle Linux Server 8.9. The host private IP is masked for this guide, but as an example, we will use the fictional zz.dd.mm.hh, and the hostname is zdmhost.

On the ZDM host, as root, add the ADB Private Endpoint URL, for example, xxy.adb.region-1.oraclecloud.com, to the /etc/hosts file to be resolved to the ADB Private Endpoint IP, for example, aa.bb.cc.dd. If you do not have access to this information yet, please configure it once it is available after provisioning the target Autonomous Database.

Network and Connectivity

Region
An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other areas, and vast distances can separate them (across countries or continents).

Virtual Cloud Network (VCN) and subnet
A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.

OCI Network Security Group (NSG) 
A network security group (NSG) provides a virtual firewall for cloud resources with the same security posture. For example, a group of compute instances performs the same tasks and thus needs to use the same set of ports.

Azure VNet
Azure Virtual Network (VNet) is the fundamental building block for your private network in Azure. VNet enables many Azure resources, such as Azure virtual machines (VM), to securely communicate with each other, the internet, and on-premises networks.

Azure Delegated Subnet
Subnet delegation is Microsoft's ability to inject a managed service, specifically a platform-as-a-service service, directly into your virtual network. This means you can designate or delegate a subnet to be a home for an externally managed service inside your virtual network. In other words, that external service will act as a virtual network resource, even though technically it is an external platform-as-a-service service.

Virtual network interface card (VNIC)
The services in Azure data centers have physical network interface cards (NICs). Virtual machine instances communicate using virtual NICs (VNICs) associated with the physical NICs. Each instance has a primary VNIC that's automatically created and attached during launch and is available during the instance's lifetime.

Azure Route table (User Defined Route – UDR)
Virtual route tables contain rules to route traffic from subnets to destinations outside a VNet, typically through gateways. Route tables are associated with subnets in a VNet.

Local Network Virtual Appliance (NVA)

For routing purposes, deploy a Network Virtual Appliance (NVA) within the Oracle Database@Azure VNet following the Microsoft documentation links for NVA and database.

Source Database

For this step-by-step guide, the source database runs on-premises on an Oracle Linux Server 7.7. The host's private IP is masked for this guide, but as an example, we will use the fictional aa.bb.sr.db address, and the hostname is onphost.

The source Oracle database is a single-instance Enterprise Edition database version 19.21 with multitenant architecture. The Oracle SID is zdmdb. The PDB name is orclpdb.

Target Database

Oracle Database@Azure offers the following products:

  • Oracle Exadata Database Service on Dedicated Infrastructure
    • You can provision flexible Exadata systems that allow you to add database compute servers and storage servers to your system anytime after provisioning.

  • Oracle Autonomous Database Serverless
    • Autonomous Database provides an easy-to-use, fully autonomous database that scales elastically, delivers fast query performance, and requires no database administration.

Oracle Database@Azure integrates Oracle Exadata Database Service, Oracle Real Application Clusters (Oracle RAC), and Oracle Data Guard technologies into the Azure platform. The Oracle Database service runs on Oracle Cloud Infrastructure (OCI) and is co-located in Microsoft Azure data centers. The service offers features and price parity with OCI. Users purchase the service on Azure Marketplace.

Oracle Database@Azure service offers the same low latency as other Azure-native services and meets mission-critical workloads and cloud-native development needs. Users manage the service on the Azure console and with Azure automation tools. The service is deployed in Azure Virtual Network (VNet) and integrated with the Azure identity and access management system. The OCI and Oracle Database metrics and audit logs are natively available in Azure. The service requires that users have an Azure tenancy and an OCI tenancy.

For this step-by-step guide, the target platform is Oracle Autonomous Database Serverless (ADB-S) on Oracle Database@Azure. ZDM requires configuring a placeholder database target environment before beginning the migration process.

Enhanced Security for Outbound Connections with Private Endpoints

Setting the ROUTE_OUTBOUND_CONNECTIONS database property to the value PRIVATE_ENDPOINT enforces that all outgoing connections to a target host are subject to and limited by the private endpoint's egress rules.

ALTER DATABASE PROPERTY SET ROUTE_OUTBOUND_CONNECTIONS = 'PRIVATE_ENDPOINT';

NFS File Share

An NFS file share can be provided via Oracle Advanced Cluster File System (Oracle ACFS), NFS Server, Azure Files, or Azure NetApp. Depending on the solution, an Azure Firewall or a Network Virtual Appliance (NVA) might be required to route the traffic from on-premises and Oracle delegated subnet for Oracle Database@Azure to the NFS file share, please see the following links for NFS and for connectivity design.

ZDM Logical Online migration workflow uses Oracle Data Pump export and import to migrate the data from the source to the target database and Oracle GoldenGate to synchronize both databases for minimal downtime migration. An NFS file share is provided through the Azure Files service to store the Data Pump dump files. 

For this step-by-step guide, the file share path is odaamigration.file.core.windows.net:/odaamigration/testmigration. The NFS private endpoint IP is masked for this guide, but as an example, we will use the fictional aa.an.fs.pe address.
The NFS share must be mounted on both the source database host and the target Autonomous Database.

To mount the NFS Share on the source database server:

As root:

mkdir /nfstest

mount -t nfs odaamigration.file.core.windows.net:/odaamigration/testmigration /nfstest -o vers=4,minorversion=1,sec=sys

or using IP:

mount -t nfs aa.an.fs.pe:/odaamigration/testmigration /nfstest -o vers=4,minorversion=1,sec=sys

Make sure the oracle user has access to the NFS mount

chown oracle:oinstall /nfstest

As oracle user:

touch /nfstest/test.txt

On the source PDB:

SQL> create directory DATA_PUMP_DIR_NFS as '/nfstest';

Prerequisites

Source Database Prerequisites

  • Oracle GoldenGate requires a unique row identifier on the source and target tables to locate the correct target rows for replicated updates and deletes.
  • The character set on the source database must be the same as the target database.
  • If the source is Oracle Database 11.2, apply mandatory 11.2.0.4 RDBMS patches on the source database. See My Oracle Support note Oracle GoldenGate -- Oracle RDBMS Server Recommended Patches (Doc ID 1557031.1)
  • If the source database is Oracle Database 12.1.0.2 or a later release, apply mandatory RDBMS patches.
  • If the source is Oracle Database Standard Edition 2, available with Oracle Database 18c or 19c, and lower than DBRU 19.11, apply the RDBMS patch for bug 29374604 -- Integrated Extract not starting against Oracle RDBMS Standard Edition.

Source Database Preparation

Prepare the source database with the following instructions.
As SYS user:

Prepare the source database. As SYS user:

-- Set streams_pool_size to 2G

SQL> alter system set streams_pool_size=2G scope=both;

-- Set global_names to false

SQL> alter system set global_names=false;

-- Enable ARCHIVELOG mode:

SQL> select log_mode from v$database;

LOG_MODE
------------
NOARCHIVELOG

SQL> shutdown immediate;
SQL> startup mount
SQL> alter database archivelog;
SQL> alter database open;
SQL> select log_mode from v$database;

LOG_MODE
------------
ARCHIVELOG

-- Enable FORCE LOGGING to ensure that all changes are found in the redo by the Oracle GoldenGate Extract process:

SQL> select force_logging from v$database;

FORCE_LOGGING
---------------------------------------
NO

SQL> alter database force logging;
SQL> select force_logging from v$database;

FORCE_LOGGING
---------------------------------------
YES

-- Enable database minimal supplemental logging:

SQL> select minimal from dba_supplemental_logging;

MINIMAL
----------
NO

SQL> alter database add supplemental log data;
SQL> select minimal from dba_supplemental_logging;

MINIMAL
----------
YES
-- Enable initialization parameter ENABLE_GOLDENGATE_REPLICATION:

SQL> alter system set ENABLE_GOLDENGATE_REPLICATION=TRUE scope=both;

System altered.

-- In case of Multitenant, create the user c##ggadmin in CDB$ROOT:

SQL> create user c##ggadmin identified by VerySecretPw_22 default tablespace users temporary tablespace temp;
grant connect, resource to c##ggadmin;
grant unlimited tablespace to c##ggadmin;
alter user c##ggadmin quota 10G on users;
grant select any dictionary to c##ggadmin;
grant create view to c##ggadmin;
grant execute on dbms_lock to c##ggadmin;
grant set container to c##ggadmin container=all;
exec dbms_goldengate_auth.GRANT_ADMIN_PRIVILEGE('c##ggadmin',container=>'all');


-- Create a GoldenGate administration user, ggadmin (in the PDB in case of Multitenant):

SQL> alter session set container=pdbsrc;
create user ggadmin identified by VerySecretPw_22 default tablespace users temporary tablespace temp;
grant connect, resource to ggadmin;
grant unlimited tablespace to ggadmin;
alter user ggadmin quota 10G on users;
grant select any dictionary to ggadmin;
grant create view to ggadmin;
grant execute on dbms_lock to ggadmin;
exec dbms_goldengate_auth.GRANT_ADMIN_PRIVILEGE('ggadmin');

 

ZDM Service Host

On the ZDM host, as root, add the Autonomous Database Private Endpoint URL sample.adb.us-region-1.oraclecloud.com to the /etc/hosts file to be resolved to the Autonomous Database Private Endpoint IP aa.dd.bb.ss.

Access Network File System from Autonomous Database

You can attach a Network File System to a directory location in your Autonomous Database. This allows you to load data from Oracle Cloud Infrastructure File Storage in your Virtual Cloud Network (VCN), Azure Files NFS, or any other Network File System in on-premises data centers. Depending on the Network File System version you want to access, both NFSv3 and NFSv4 are supported.

Step 1: Add NFS mount point FQDN to OCI DNS VCN Resolver

Bear in mind that if the OCI tenancy is a new tenancy created within the Oracle Database@Azure provisioning process, the limits for OCI private DNS and A records might need to be increased. To increase the limits, open a Service Request with Oracle Support. A limit of at least three records is needed.

Follow these steps to create an A-record in OCI DNS to resolve the FQDN of the Azure NFS mount point:

  1. From the Oracle Autonomous Database details page in OCI, click on the virtual cloud network in the Network section.
  2. On the network details page, click on the DNS Resolver.
  3. On the private resolver details page, click on the default private view.
  4. Click the create zone button and create a new zone using the name file.core.windows.net.
  5. Click on the newly created zone, manage records, and add a record. Replace aa.an.fs.pe by the actual IP address.
  6. Publish the changes.
  7. Update the Network Security Group (NSG) in OCI to allow network traffic flow from the VNet where the NFS private endpoint resides

On the target database:

Step 2:  Add the NFS Mount FQDN to the Access Control List (ACL)

SQL> exec DBMS_NETWORK_ACL_ADMIN.APPEND_HOST_ACE(host => 'odaamigration.file.core.windows.net', ace => xs$ace_type(privilege_list => xs$name_list('connect', 'resolve'), principal_name => 'ADMIN', principal_type => xs_acl.ptype_db));

PL/SQL procedure successfully completed.

Step 3: Create a Directory on the Autonomous Database

SQL> CREATE or replace DIRECTORY FSS_DIR AS 'fss';

Directory created.

Step 4: Attach NFS to Autonomous Database

Use "params => JSON_OBJECT('nfs_version' value 4)"; otherwise, NFS cannot be mounted.

SQL> BEGIN
DBMS_CLOUD_ADMIN.ATTACH_FILE_SYSTEM(
file_system_name => 'AZUREFILES',
file_system_location => 'odaamigration.file.core.windows.net:/odaamigration/testmigration',
directory_name => 'FSS_DIR',
description => 'Attach Azure Files',
params => JSON_OBJECT('nfs_version' value 4)
);
END;
/

PL/SQL procedure successfully completed.

SQL> SELECT object_name FROM DBMS_CLOUD.LIST_FILES('FSS_DIR');

OBJECT_NAME--------------------------------------------------------------------------------
test.txt

Additional Configuration

SSH Key

ZDM connects via SSH to the Source Database servers; hence an SSH key pair for the zdmuser is required. As zdmuser, run the following:

[zdmuser@zdmhost ~]$ mkdir ~/.ssh
[zdmuser@zdmhost ~]$ chmod 700 ~/.ssh
[zdmuser@zdmhost ~]$ /usr/bin/ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/zdmuser/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/zdmuser/.ssh/id_rsa.
Your public key has been saved in /home/zdmuser/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:keyfingerprintsample zdmuser@zdmhost

[zdmuser@zdmhost ~]$ cd ~/.ssh
[zdmuser@zdmhost .ssh]$ cat id_rsa.pub

Insert the public to the authorized_keys file on the source database host.

You can find more information on ZDM Product’s documentation section, “Generating a Private SSH Key Without a Passphrase.”

API Signing Public Key and Configuration File

ZDM uses an API Signing Public Key to call REST APIs. First, you need to create the API Keys. Do so by accessing the terminal on the ZDM Service Host, and as the zdmuser, run the following:

[zdmuser@zdmhost ~]$ mkdir .oci
[zdmuser@zdmhost ~]$ cd .oci
[zdmuser@zdmhost ~]$ openssl genrsa -out /u01/app/zdmhome/.oci/oci_api_key.pem 2048
[zdmuser@zdmhost ~]$ openssl rsa -pubout -in /u01/app/zdmhome/.oci/oci_api_key.pem -out /u01/app/zdmhome/.oci/oci_api_key_public.pem
[zdmuser@zdmhost ~]$ cat oci_api_key_public.pem

Copy the catted 'oci_api_key_public.pem' file and save it; you will need it in the next step. Include the "Begin Public Key" and "End Public Key" lines during the copy. Go to your Oracle Cloud OCI Dashboard, navigate to the top right, click on your user profile icon, and select the top option representing your user. Select API Keys and Add API Key. Paste the public OCI API key file you copied above and click Add Key.

You will see a configuration file preview. Copy its contents; you will use them to populate your configuration file in the following step.

As the zdmuser in the ZDM Service Host, create a configuration file in the command prompt; you can use vi/vim or any editor you prefer. In the empty file, paste the configuration file contents copied from above. Replace < path to your private key file > # TODO with the line above; once done, save the file and quit the editor:

/u01/app/zdmhome/.oci/oci_api_key.pem

Oracle GoldenGate on Docker

For ZDM Logical Online migrations to Oracle Database@Azure, you will run Oracle GoldenGate on a Docker VM on-premises or on an Azure VM. Oracle GoldenGate keeps your source and target databases in sync and enables you to achieve zero to negligible downtime for your Oracle database migrations across database versions and platforms.

For this step-by-step guide, the docker container runs on Azure IaaS VM on Oracle Linux 7.9. The host's private IP
is masked for this guide, but as an example, we will use the fictional aa.bb.gg.do address, and the hostname
is ggdockervm.

Install Oracle GoldenGate on Docker

Step 1: Download the GoldenGate Docker Image

On Oracle Cloud, create a VM using the Oracle GoldenGate – Database Migrations image from Marketplace. Search for “Goldengate migrations” in the Marketplace and choose the Database Migrations image. Choose the latest “Oracle DB – Microservices Edition – Promotional” version, and launch the stack. Once the stack is completed, a VM in OCI Compute Instances will be created with the details you provided during the stack launch. Log in to that VM via SSH and issue the shell list command:

-bash-4.2$ ls -l
total 0
lrwxrwxrwx. 1 opc opc 37 Jul 3 14:41 ogg-credentials.json -> /u02/deployments/ogg-credentials.json
lrwxrwxrwx. 1 opc opc 36 Jul 3 14:41 ora21c-2113000.tar -> /opt/dockerimages/ora21c-2113000.tar
drwxr-x---. 3 opc opc 18 Jul 3 15:55 oradiag_opc

The file /opt/dockerimages/ora21c-2113000.tar is the GoldenGate Docker image. Copy it to your Azure IaaS VM. Once done, the VM in OCI can be terminated. Its only purpose was to download the ora21c-2113000.tar Docker image file.

Step 2: Set Up the Docker Engine

Set up the Docker engine on the Azure IaaS VM to host the GoldenGate Docker image. In this case, following the Install Docker engine on Oracle Linux 7:

# install docker engine

[azureuser@ggdockervm ~]$ sudo yum install docker-engine docker-cli

# start docker service and configure it to start at boot time:

[azureuser@ggdockervm ~]$ sudo systemctl enable --now docker

# ensure docker service is running:

[azureuser@ggdockervm ~]$ systemctl status docker

# display configuration and version of docker engine

[azureuser@ggdockervm ~]$ sudo docker info
Step 3: Load the Docker Image

Load the Docker image to the Docker engine. The ora21c-2113000.tar file is the one you copied to this VM in Step 0:

[azureuser@ggdockervm ~]$ sudo docker load < ./ora21c-2113000.tar

67d008ba80bc: Loading layer [==================================================>]  253.7MB/253.7MB
69e9cf1483ea: Loading layer [==================================================>]  6.144kB/6.144kB
b8c7ceef0daa: Loading layer [==================================================>]  372.8MB/372.8MB
5e5793be3604: Loading layer [==================================================>]  26.62kB/26.62kB
377155359816: Loading layer [==================================================>]  1.739GB/1.739GB
4411981a6090: Loading layer [==================================================>]  18.43kB/18.43kB
Loaded image: oracle/goldengate:21.13.0.0.0

List the images:

[azureuser@ggdockervm ~]$ sudo docker image list

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
oracle/goldengate   21.13.0.0.0         f9d2aea5f6a7        4 months ago        2.34GB

Step 4: Run the Docker Image

Run the Oracle GoldenGate Docker image as a container:

[azureuser@ggdockervm ~]$ sudo docker run --name ogg2113 -p 443:443 docker.io/oracle/goldengate:21.13.0.0.0

For more information about the run parameters, visit Running Oracle GoldenGate in a Container.

The run output will display the ggadmin user password. You will need this later when running the ZDM migration command:

----------------------------------------------------------------------------------

--  Password for OGG administrative user 'oggadmin' is 'SamplePassword1234*&=+'

----------------------------------------------------------------------------------

Check the status of the Docker container:

[azureuser@ggdockervm ~]$ sudo docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d9156f6223c3 oracle/goldengate:21.13.0.0.0 "/usr/local/bin/depl…" 2 hours ago Up 2 hours (healthy) 80/tcp, 0.0.0.0:443->443/tcp   ogg2113

To start and stop the Docker container:

[azureuser@ggdockervm ~]$ sudo docker stop d9156f6223c3
[azureuser@ggdockervm ~]$ sudo docker start d9156f6223c3

Step 5: Docker, Instant Client & Wallet Further Configuration Steps

Download Oracle Instant Client locally, and proceed to copy it and the ADB-S wallet to the docker container, and check connectivity to source and target databases from the docker container:

sudo docker cp walletXYX.zip dockercontainerhost:/walletXYX.zip
sudo docker cp instant_client_21_3.zip dockercontainerhost:/instant_client_21_3.zip
sudo docker exec -it ogg2113 /bin/bash

Ensure the wallet containing certificates for TLS authentication is in the correct location in the GoldenGate Hub as per ZDM’s documentation:

- For an Autonomous Database, the wallet file should be in the following directory:

/u02/deployments/deployment_name/etc/adb


- For a co-managed database, the wallet file should be in directory:

/u02/deployments/deployment_name/etc

Move the Instant Client and Wallet files to the GoldenGate home:

[root@dockercontainerhost /]# mv instantclient_21_3.zip /home/ogg/
[root@dockercontainerhost /]# mv Wallet_zdm2adb2.zip /home/ogg/
Change File Ownership:
[root@dockercontainerhost /]# chown ogg:ogg /home/ogg/instantclient_21_3.zip
[root@dockercontainerhost /]# chown ogg:ogg /home/ogg/Wallet_zdm2adb2.zip

Log in as the Oracle GoldenGate user, create the required wallet directory, copy the wallet and instant client, and unzip them.

[root@dockercontainerhost /]# su - ogg
[ogg@dockercontainerhost ~]$ mkdir /u02/Deployment/etc/adb
[ogg@dockercontainerhost ~]$ mv Wallet_zdm2adb2.zip /u02/Deployment/etc/adb
[ogg@dockercontainerhost ~]$ cd /u02/Deployment/etc/adb
[ogg@dockercontainerhost etc]$ unzip Wallet_zdm2adb2.zip

[ogg@dockercontainerhost ~]$ unzip instantclient_21_3.zip
[ogg@dockercontainerhost ~]$ chmod 744 instantclient_21_3/sqlplus

Open and Edit the sqlnet.ora file:

$ vi sqlnet.ora

Copy the following line:

DIRECTORY="/u02/Deployment/etc/adb"

Export the following environment variables:

export LD_LIBRARY_PATH=/home/ogg/instantclient_21_3
export TNS_ADMIN=/u02/Deployment/etc/adb

Edit the etc hosts file on the docker container:

[ogg@dockercontainerhost ~]$ vi /etc/hosts
aa.bb.sr.db onphost
aa.dd.bb.ss sample.adb.us-region-1.oraclecloud.com

Database Migration Step by Step with ZDM

Step 1: Fill the response file

vi /home/zdmuser/logical_online_adb_nfs/logical_online_adb_nfs.rsp

# migration method
MIGRATION_METHOD=ONLINE_LOGICAL
DATA_TRANSFER_MEDIUM=NFS

# data pump
DATAPUMPSETTINGS_JOBMODE=SCHEMA
DATAPUMPSETTINGS_METADATAREMAPS-1=type:REMAP_TABLESPACE,oldValue:USERS,newValue:DATA
INCLUDEOBJECTS-1=owner:HR
DATAPUMPSETTINGS_EXPORTDIRECTORYOBJECT_NAME=DATA_PUMP_DIR_NFS
DATAPUMPSETTINGS_IMPORTDIRECTORYOBJECT_NAME=FSS_DIR6

# source db (pdb)
SOURCEDATABASE_CONNECTIONDETAILS_HOST=source2
SOURCEDATABASE_CONNECTIONDETAILS_PORT=1521
SOURCEDATABASE_CONNECTIONDETAILS_SERVICENAME=orclpdb
SOURCEDATABASE_ADMINUSERNAME=SYSTEM
SOURCEDATABASE_GGADMINUSERNAME=ggadmin

# source cdb
SOURCECONTAINERDATABASE_CONNECTIONDETAILS_HOST=source2
SOURCECONTAINERDATABASE_CONNECTIONDETAILS_PORT=1521
SOURCECONTAINERDATABASE_CONNECTIONDETAILS_SERVICENAME=zdmdb
SOURCECONTAINERDATABASE_ADMINUSERNAME=SYSTEM
SOURCECONTAINERDATABASE_GGADMINUSERNAME=c##ggadmin


# target db
TARGETDATABASE_OCID=ocid1.autonomousdatabase.oc1.iad.aaaa.bbb.cccc.ddddd
TARGETDATABASE_ADMINUSERNAME=ADMIN
TARGETDATABASE_GGADMINUSERNAME=ggadmin

# oci cli
OCIAUTHENTICATIONDETAILS_USERPRINCIPAL_USERID=ocid1.user.oc1..aaaa.bbb.ccccc.ddddd
OCIAUTHENTICATIONDETAILS_USERPRINCIPAL_TENANTID=ocid1.tenancy.oc1.aaa.bbbbb
OCIAUTHENTICATIONDETAILS_USERPRINCIPAL_FINGERPRINT=12:ac:34:cc:aa
OCIAUTHENTICATIONDETAILS_USERPRINCIPAL_PRIVATEKEYFILE=/home/zdmuser/.oci/oci_api_key.pem
OCIAUTHENTICATIONDETAILS_REGIONID=us-ashburn-1

# GoldenGate
GOLDENGATEHUB_ADMINUSERNAME=oggadmin
GOLDENGATEHUB_SOURCEDEPLOYMENTNAME=Local
GOLDENGATEHUB_TARGETDEPLOYMENTNAME=Local

# Private IP of the VM where Docker is running
GOLDENGATEHUB_URL=https://aa.bb.gg.do (SAMPLE URL)
GOLDENGATEHUB_ALLOWSELFSIGNEDCERTIFICATE=TRUE


#The Zero Downtime Migration server should be allowed to make HTTPS over port 443 calls to an OCI REST endpoint. If there is no connectivity to OCI endpoints, then skip the following parameters:
#TARGETDATABASE_OCID
#OCIAUTHENTICATIONDETAILS_*
#and add the following parameters:

TARGETDATABASE_DBTYPE=ADBCC
TARGETDATABASE_CONNECTIONDETAILS_HOST=example.adb.us-ashburn-1.oraclecloud.com
TARGETDATABASE_CONNECTIONDETAILS_PORT=1522
TARGETDATABASE_CONNECTIONDETAILS_SERVICENAME=adbzdm_high
TARGETDATABASE_CONNECTIONDETAILS_TLSDETAILS_CREDENTIALSLOCATION=/home/zdmuser/adbwallet
TABLESPACEDETAILS_EXCLUDE=UNDOTBS1,UNDO_2

In this example, /home/zdmuser/adbwallet is the directory where you have downloaded and unzipped the Autonomous Database Credentials Wallet file, and adbzdm_high is the TNS alias in the tnsname.ora file is this directory.

Step 2: Evaluate the Configuration

Execute the following command on the ZDM host as zdmuser to evaluate the migration. ZDM will check the source and target database configurations. The actual migration will not be started. On the ZDM host as zdmuser:

[zdmuser@zdmhost ~]$ $ZDMHOME/bin/zdmcli migrate database \
-rsp /logical_online_adb_nfs/logical_online_adb_nfs.rsp \
-sourcenode onphost \
-sourcesid orclpdb \
-srcauth zdmauth \
-srcarg1 user:azureuser \
-srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa \
-srcarg3 sudo_location:/usr/bin/sudo \
-eval

Enter source database administrative user "SYSTEM" password:
Enter source database administrative user "ggadmin" password:
Enter source container database administrative user "SYSTEM" password:
Enter source container database administrative user "c##ggadmin" password:
Enter target database administrative user "ADMIN" password:
Enter target database administrative user "ggadmin" password:
Enter Oracle GoldenGate hub administrative user "oggadmin" password:
Enter Data Pump encryption password:
Operation "zdmcli migrate database" scheduled with the job ID "1".

If the source database uses ASM for storage management, use -sourcedb <db_unique_name> instead of -sourcesid <SID> in the zdmcli command.

Check the job status. On the ZDM host as zdmuser:

[zdmuser@zdmhost ~]$ $ZDMHOME/bin/zdmcli query job -jobid 1

...
Job ID: 1
User: zdmuser
Client: zdmhost
Job Type: "EVAL"
...
Current status: SUCCEEDED
Result file path: "/home/zdmuser/zdm/zdmbase/chkbase/scheduled/job-1.log"
Metrics file path: "/home/zdmuser/zdm/zdmbase/chkbase/scheduled/job-1.json"
...
ZDM_VALIDATE_TGT ...................... COMPLETED
ZDM_VALIDATE_SRC ...................... COMPLETED
ZDM_SETUP_SRC ......................... COMPLETED
ZDM_PRE_MIGRATION_ADVISOR ............. COMPLETED
ZDM_VALIDATE_GG_HUB ................... COMPLETED
ZDM_VALIDATE_DATAPUMP_SETTINGS_SRC .... COMPLETED
ZDM_VALIDATE_DATAPUMP_SETTINGS_TGT .... COMPLETED
ZDM_PREPARE_DATAPUMP_SRC .............. COMPLETED
ZDM_DATAPUMP_ESTIMATE_SRC ............. COMPLETED
ZDM_CLEANUP_SRC ....................... COMPLETED


Detailed information about the migration process can be found by monitoring the log file:

[zdmuser@zdmhost ~]$ tail -f /home/zdmuser/zdm/zdmbase/chkbase/scheduled/job-1.log

In case troubleshooting is required, please check the ZDM server log on the ZDM Service Host under the following location:

$ZDM_BASE/crsdata/<zdm_service_host>/rhp/zdmserver.log.0

Step 3: Initiate the Migration

To initiate the actual migration, execute the same command for evaluation, but this time without the -eval parameter.
Oracle ZDM allows you to pause the migration process at any given phase. For example, the migration process can be paused after Oracle GoldenGate keeps the target database in sync with the source. Upon executing the zdm migrate database command, the -pauseafter flag must be entered with the desired pausing stage, ZDM_MONITOR_GG_LAG.
On the ZDM host as zdmuser:

[zdmuser@zdmhost ~]$ $ZDMHOME/bin/zdmcli migrate database \
-rsp /logical_online_adb_nfs/logical_online_adb_nfs.rsp \
-sourcenode onphost \
-sourcesid orclpdb \
-srcauth zdmauth \
-srcarg1 user:azureuser \
-srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa \
-srcarg3 sudo_location:/usr/bin/sudo \
-pauseafter ZDM_MONITOR_GG_LAG

Enter source database administrative user "SYSTEM" password:
Enter source database administrative user "ggadmin" password:
Enter source container database administrative user "SYSTEM" password:
Enter source container database administrative user "c##ggadmin" password:
Enter target database administrative user "ADMIN" password:
Enter target database administrative user "ggadmin" password:
Enter Oracle GoldenGate hub administrative user "oggadmin" password:
Enter Data Pump encryption password:
Operation "zdmcli migrate database" scheduled with the job ID "2".

Check the job status. On the ZDM host as zdmuser:

[zdmuser@zdmhost ~]$ $ZDMHOME/bin/zdmcli query job -jobid 2

...
Job ID: 2
User: zdmuser
Client: zdmhost
Job Type: "MIGRATE"
...
Current status: PAUSED
Current Phase: "ZDM_MONITOR_GG_LAG"
Result file path: "/home/zdmuser/zdm/zdmbase/chkbase/scheduled/job-2.log"
Metrics file path: "/home/zdmuser/zdm/zdmbase/chkbase/scheduled/job-2.json"
...

ZDM_VALIDATE_TGT ...................... COMPLETED
ZDM_VALIDATE_SRC ...................... COMPLETED
ZDM_SETUP_SRC ......................... COMPLETED
ZDM_PRE_MIGRATION_ADVISOR ............. COMPLETED
ZDM_VALIDATE_GG_HUB ................... COMPLETED
ZDM_VALIDATE_DATAPUMP_SETTINGS_SRC .... COMPLETED
ZDM_VALIDATE_DATAPUMP_SETTINGS_TGT .... COMPLETED
ZDM_PREPARE_DATAPUMP_SRC .............. COMPLETED
ZDM_DATAPUMP_ESTIMATE_SRC ............. COMPLETED
ZDM_PREPARE_GG_HUB .................... COMPLETED
ZDM_ADD_HEARTBEAT_SRC ................. COMPLETED
ZDM_ADD_SCHEMA_TRANDATA_SRC ........... COMPLETED
ZDM_CREATE_GG_EXTRACT_SRC ............. COMPLETED
ZDM_PREPARE_DATAPUMP_TGT .............. COMPLETED
ZDM_DATAPUMP_EXPORT_SRC ............... COMPLETED
ZDM_TRANSFER_DUMPS_SRC ................ COMPLETED
ZDM_DATAPUMP_IMPORT_TGT ............... COMPLETED
ZDM_POST_DATAPUMP_SRC ................. COMPLETED
ZDM_POST_DATAPUMP_TGT ................. COMPLETED
ZDM_ADD_HEARTBEAT_TGT ................. COMPLETED
ZDM_ADD_CHECKPOINT_TGT ................ COMPLETED
ZDM_CREATE_GG_REPLICAT_TGT ............ COMPLETED
ZDM_START_GG_REPLICAT_TGT ............. COMPLETED
ZDM_MONITOR_GG_LAG .................... COMPLETED
ZDM_PREPARE_SWITCHOVER_APP ............ PENDING
ZDM_ADVANCE_SEQUENCES ................. PENDING
ZDM_SWITCHOVER_APP .................... PENDING
ZDM_POST_SWITCHOVER_TGT ............... PENDING
ZDM_RM_GG_EXTRACT_SRC ................. PENDING
ZDM_RM_GG_REPLICAT_TGT ................ PENDING
ZDM_DELETE_SCHEMA_TRANDATA_SRC ........ PENDING
ZDM_RM_HEARTBEAT_SRC .................. PENDING
ZDM_RM_CHECKPOINT_TGT ................. PENDING
ZDM_RM_HEARTBEAT_TGT .................. PENDING
ZDM_CLEAN_GG_HUB ...................... PENDING
ZDM_POST_ACTIONS ...................... PENDING
ZDM_CLEANUP_SRC ....................... PENDING
Pause After Phase: "ZDM_MONITOR_GG_LAG"

Pay attention to the current job status. It is in PAUSED status now. Also, the progress stopped after phase ZDM_MONITOR_GG_LAG was COMPLETED. At this stage, every change in the source database is immediately synchronized with the target database. Resume the job when your application is ready for migration.

Step 4: Complete the Migration

Resume the job from the previous step. On the ZDM host as zdmuser, resume the job and query the status until all phases are completed:

[zdmuser@zdmhost ~]$ $ZDMHOME/bin/zdmcli resume job -jobid 2
[zdmuser@zdmhost ~]$ $ZDMHOME/bin/zdmcli query job -jobid 2
...
ZDM_VALIDATE_TGT ...................... COMPLETED
ZDM_VALIDATE_SRC ...................... COMPLETED
ZDM_SETUP_SRC ......................... COMPLETED
ZDM_PRE_MIGRATION_ADVISOR ............. COMPLETED
ZDM_VALIDATE_GG_HUB ................... COMPLETED
ZDM_VALIDATE_DATAPUMP_SETTINGS_SRC .... COMPLETED
ZDM_VALIDATE_DATAPUMP_SETTINGS_TGT .... COMPLETED
ZDM_PREPARE_DATAPUMP_SRC .............. COMPLETED
ZDM_DATAPUMP_ESTIMATE_SRC ............. COMPLETED
ZDM_PREPARE_GG_HUB .................... COMPLETED
ZDM_ADD_HEARTBEAT_SRC ................. COMPLETED
ZDM_ADD_SCHEMA_TRANDATA_SRC ........... COMPLETED
ZDM_CREATE_GG_EXTRACT_SRC ............. COMPLETED
ZDM_PREPARE_DATAPUMP_TGT .............. COMPLETED
ZDM_DATAPUMP_EXPORT_SRC ............... COMPLETED
ZDM_TRANSFER_DUMPS_SRC ................ COMPLETED
ZDM_DATAPUMP_IMPORT_TGT ............... COMPLETED
ZDM_POST_DATAPUMP_SRC ................. COMPLETED
ZDM_POST_DATAPUMP_TGT ................. COMPLETED
ZDM_ADD_HEARTBEAT_TGT ................. COMPLETED
ZDM_ADD_CHECKPOINT_TGT ................ COMPLETED
ZDM_CREATE_GG_REPLICAT_TGT ............ COMPLETED
ZDM_START_GG_REPLICAT_TGT ............. COMPLETED
ZDM_MONITOR_GG_LAG .................... COMPLETED
ZDM_PREPARE_SWITCHOVER_APP ............ COMPLETED
ZDM_ADVANCE_SEQUENCES ................. COMPLETED
ZDM_SWITCHOVER_APP .................... COMPLETED
ZDM_POST_SWITCHOVER_TGT ............... COMPLETED
ZDM_RM_GG_EXTRACT_SRC ................. COMPLETED
ZDM_RM_GG_REPLICAT_TGT ................ COMPLETED
ZDM_DELETE_SCHEMA_TRANDATA_SRC ........ COMPLETED
ZDM_RM_HEARTBEAT_SRC .................. COMPLETED
ZDM_RM_CHECKPOINT_TGT ................. COMPLETED
ZDM_RM_HEARTBEAT_TGT .................. COMPLETED
ZDM_CLEAN_GG_HUB ...................... COMPLETED
ZDM_POST_ACTIONS ...................... COMPLETED
ZDM_CLEANUP_SRC ....................... COMPLETED

Known Issues

All common issues are documented and updated periodically in Oracle Zero Downtime Migration’s documentation, specifically on the product release note, Known Issues section: https://docs.oracle.com/en/database/oracle/zero-downtime-migration/.

Troubleshooting Oracle GoldenGate Replication

During your migration, you can pause the ZDM migration job after the ZDM_MONITOR_GG_LAG phase.

At this stage, every transaction on the source database is replicated via GoldenGate to the target database. If this is not the case, log in to the Docker container and check the EXTRACT and REPLICATs deployments and log files:

# connect to the docker container and switch to ogg user
[azureuser@ggdockervm ~]$ sudo docker exec -it ogg2113 /bin/bash
[root@d9156f6223c3 /]# su - ogg

# check the deployment files
[ogg@d9156f6223c3 ~]$ ls -l /u02/Deployment/etc/conf/ogg
total 16
-rw-r-----. 1 ogg ogg 257 Jun 10 19:05 EXT4BL3P.prm
-rw-r--r--. 1 ogg ogg  17 Jun 10 13:05 GLOBALS
-rw-r-----. 1 ogg ogg 260 Jun 10 19:34 RN22M.prm
-rw-r-----. 1 ogg ogg 260 Jun 10 19:57 RN22M.prm.backup


# check the deployments log files
[ogg@d9156f6223c3 ~]$ view /u02/Deployment/var/log/extract.log
[ogg@d9156f6223c3 ~]$ view /u02/Deployment/var/log/replicat.log


Check the Deployment Status:
[ogg@d9156f6223c3 ~]$ /u01/ogg/bin/adminclient
Oracle GoldenGate Administration Client for Oracle
Version 21.13.0.0.0 OGGCORE_21.13.0.0.0OGGRU_PLATFORMS_240108.2205
Copyright (C) 1995, 2024, Oracle and/or its affiliates. All rights reserved.
Oracle Linux 7, x64, 64bit (optimized) on Jan  9 2024 09:04:36
Operating system character set identified as US-ASCII.

OGG (not connected) 1>
OGG (not connected) 1> connect http://127.0.0.1 as oggadmin password /Qlq1UTGMK+N-EksH
Using default deployment 'Local'

OGG (http://127.0.0.1 Local) 2> info all
Program     Status      Group       Type             Lag at Chkpt  Time Since Chkpt
ADMINSRVR   RUNNING
DISTSRVR    RUNNING
PMSRVR      RUNNING
RECVSRVR    RUNNING
EXTRACT     RUNNING     EXT4BL3P    INTEGRATED       00:00:01      00:00:05
REPLICAT    ABENDED     RN22M       PARALLEL NONINT  00:04:21      00:00:21

In this case, the REPLICAT status was ABENDED. The replicat.log indicated insufficient privileges for the ggadmin user on the target database. After fixing the issue, start the deployment:


OGG (http://127.0.0.1 Local) 3> start RN22M

2024-06-10T20:35:54Z  INFO    OGG-00975  Replicat group RN22M starting.

2024-06-10T20:35:54Z  INFO    OGG-15445  Replicat group RN22M started.

Troubleshooting Oracle ZDM & Other Resources

For Oracle ZDM log review:

  • ZDM Server Logs:
    • Check - $ZDM_BASE/crsdata/<zdm_service_node>/rhp/rhpserver.log.0

  • Check source node logs:
    • <oracle_base>/zdm/zdm_<src_db_name>_<job_id>/zdm/log

  • Check target node logs:
    • <oracle_base>/zdm/zdm_<tgt_db_name>_<job_id>/zdm/log

For all Oracle Support Service Requests related to Zero Downtime Migration, please be sure to follow the instructions in My Oracle Support Document: