Oracle® Zero Downtime Migration
Release Notes
Release 19c
F19479-03
March 2020
These release notes provide downloading instructions for the latest product software and documentation, and describe known issues and troubleshooting information.
- Downloading the Zero Downtime Migration Installation Software
- Downloading the Zero Downtime Migration Documentation
- Zero Downtime Migration Runbooks
- General Information
- Known Issues
- Additional Information for Migrating to Oracle Cloud Infrastructure
- Additional Information for Migrating to Exadata Cloud Service
- Additional Information for Migrating to Exadata Cloud at Customer
- Documentation Addenda
- Documentation Accessibility
1.1 Downloading the Zero Downtime Migration Installation Software
For a fresh installation of the latest Zero Downtime Migration software version, go to https://www.oracle.com/database/technologies/rac/zdm-downloads.html.
1.2 Downloading the Zero Downtime Migration Documentation
1.3 Zero Downtime Migration Runbooks
The Zero Downtime Migration runbooks can be found at https://docs.oracle.com/en/database/oracle/zero-downtime-migration/19.2/books.html.
The runbooks are a shorter form of Move to Oracle Cloud Using Zero Downtime Migration. See Move to Oracle Cloud Using Zero Downtime Migration for concepts and more details about Zero Downtime Migration.
1.4 General Information
At the time of this release, there are some details and considerations about Zero Downtime Migration behavior that you should take note of.
1.4.1 Patch Level Differences Between Source and Target Database
The major database release numbers of the source and target database must match for the migration job to succeed; however, patch level differences between the source and target databases might exist where the target database patch level is equal to or higher than the source database.
For example, if the source database is Oracle Database 12c Release 1 (12.1.0.2), the target database must also be at release 12.1.0.2. However, the patch level on the target database might be higher than the patch level on the source database. For example, if the target database is at Jan 2019 PSU/BP and the source database is at Oct 2018 PSU/BP, then you must run datapatch after database migration.
To avoid patch checking as part of the migration procedure, you can use the
-ignore PATCH_CHECK
option in the ZDMCLI MIGRATE
DATABASE
command.
1.4.2 ZDMSERVICE Script Can Only Be Run By Installed User
If you are prevented from running the zdmservice script, note that it is required that the zdmservice script only be run by the installed user. For security reasons the zdmservice script should not be run by any user other than the zdmservice user. To avoid running zdmservice by a non-zdmservice installed user, change the zdmservice binary permissions to 700.
1.4.3 UNDO Tablespaces Added to the Source Database
Zero Downtime Migration adds UNDO tablespaces to the production database to match the target instance count if the production database has fewer instances.
To prevent Zero Downtime Migration from adding UNDO tablespaces to the source database, you can match the target database nodes count to that of the source database until the switchover, then you can add additional nodes to the target database after the switchover.
1.4.4 Cross-Edition Migration Is Not Supported
Zero Downtime Migration cannot be used to migrate an Enterprise edition database to a Standard edition database, and vice versa.
1.5 Known Issues
At the time of this release, there are several issues with Zero Downtime Migration that could occur in rare circumstances. For each issue, a workaround is provided.
1.5.1.1 General Connectivity Issues
Issue: If connectivity issues occur between the Zero Downtime Migration service node and the source or target environments, or between source and target environments, check the following areas.
Solution: Verify that the SSH configuration file (/root/.ssh/config) has the appropriate entries:
Host *
ServerAliveInterval 10
ServerAliveCountMax 2
Host rptest
HostName 192.0.2.9
IdentityFile ~/.ssh/rptest.ppk
User opc
ProxyCommand /usr/bin/nc -X connect -x www-proxy.example.com:80 %h %p
Note that the proxy setup might not be required when you are not using a
proxy server for connectivity. For example, when the source database server is on Oracle
Cloud Infrastructure Classic, you can remove or comment the line starting with
ProxyCommand
.
If the source is an Oracle RAC database, then make sure you copy the ~/.ssh/config file to all of the source Oracle RAC servers. The SSH configuration file refers to the first Oracle RAC server host name, public IP address, and private key attributes.
1.5.1.2 Evaluation Fails in Phase ZDM_GET_TGT_INFO
Issue: During the evaluation (-eval
) phase of the migration
process, the evaluation fails in the ZDM_GET_TGT_INFO
phase with the
following error for the Oracle RAC instance migration.
Executing phase ZDM_GET_TGT_INFO
Retrieving information from target node "trac11" ...
PRGZ-3130 : failed to establish connection to target listener from nodes [srac11, srac12]
PRCC-1021 : One or more of the submitted commands did not execute successfully.
PRCC-1025 : Command submitted on node srac11 timed out after 15 seconds.
PRCC-1025 : Command submitted on node srac12 timed out after 15 seconds.
Solution:
- Get the SCAN name of source database and add it to the
/etc/hosts
file on both target database servers, with the public IP address of the source database server and the source database SCAN name. For example:192.0.2.1 source-scan
- Get the SCAN name of the target database and add it to the
/etc/hosts
file on both source database servers, with the public IP address of the target database server and target database SCAN name. For example:192.0.2.3 target-scan
Note:
This issue, where the SCAN IP address is not added to/etc/hosts
file, might occur because in some cases the
SCAN IP address is assgined as a private IP address, so it might not be
resolvable.
1.5.1.3 Evaluation Fails in Phase ZDM_GET_SRC_INFO
Issue: During the evaluation (-eval
) phase of the migration
process, the evaluation fails in the ZDM_GET_SRC_INFO
phase with the
following error for the source single instance deployed without Grid infrastructure.
Executing phase ZDM_GET_SRC_INFO
retrieving information about database "zdmsidb" ...
PRCF-2056 : The copy operation failed on node: "zdmsidb".
Details: {1}
PRCZ-4002 : failed to execute command "/bin/cp" using the privileged
execution plugin "zdmauth" on nodes "zdmsidb"
scp: /etc/oratab: No such file or directory
Solution: Make an ORACLE_HOME
value entry in file
/etc/oratab
with value
db_name:$ORACLE_HOME:N
,
as shown in this example.
zdmsidb:/u01/app/oracle/product/12.2.0.1/dbhome_1:N
1.5.1.4 Object Storage Is Not Accessible
About to connect() to swiftobjectstorage.xx-region-1.oraclecloud.com port 443 (#0)
Trying 192.0.2.1... No route to host
Trying 192.0.2.2... No route to host
Trying 192.0.2.3... No route to host
couldn't connect to host
Closing connection #0
curl: (7) couldn't connect to host
Solution: On the Zero Downtime Migration service node, in the response file
template ($ZDM_HOME/rhp/zdm/template/zdm_template.rsp
), set the Object
Storage Service proxy host and port parameters listed below, if a proxy is required to
connect to Object Storage from the source database server. For example:
SRC_OSS_PROXY_HOST=www-proxy-source.example.com
SRC_OSS_PROXY_PORT=80
In the response file template
($ZDM_HOME/rhp/zdm/template/zdm_template.rsp
), set the Object
Storage Service proxy host and port parameters listed below, if a proxy is required to
connect to Object Storage from the target database server. For example:
TGT_OSS_PROXY_HOST=www-proxy-target.example.com
TGT_OSS_PROXY_PORT=80
1.5.2.1 Backup Fails with ORA-19836
Issue: Source database full backup fails with one of the following errors.
</ERRLINE><ERRLINE>ORA-19836: cannot use passphrase encryption for this backup
</ERRLINE><ERRLINE>RMAN-03009: failure of backup command on C8 channel at 04/29/2019
20:42:16
</ERRLINE><ERRLINE>ORA-19836: cannot use passphrase encryption for this backup
</ERRLINE><ERRLINE>RMAN-03009: continuing other job steps, job failed will not be
re-run
Solution 1: This issue can occur if you specify the
-sourcedb
value in the wrong case. For example, if the value
obtained from SQL command SHOW PARAMETER DB_UNIQUE_NAME
is zdmsdb, then
you need to specify it as zdmsdb in lower case, and not as ZDMSDB in upper case, as
shown in the following example.
zdmuser> ./zdmcli migrate database -sourcedb zdmsdb -sourcenode srcnode -srcroot
-targetnode rptest -targethome /u01/app/oracle/product/12.1.0.2/dbhome_1
-backupuser backup_user@example.com -rsp /scratch/zdm/zdm_template_zdmsdb.rsp
-tgtauth zdmauth -tgtarg1 user:opc
-tgtarg2 identity_file:/home/oracle/.ssh/zdm_service_node.ppk
-tgtarg3 sudo_location:/usr/bin/sudo
Solution 2: For Oracle Database 12c Release 1 and later, ensure that
$ORACLE_HOME/network/admin/sqlnet.ora
points to the correct
location of the TDE wallet, as shown here.
ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))
For Oracle Database 11g Release 2 (11.2.0.4) only, ensure that
$ORACLE_HOME/network/admin/sqlnet.ora
points to the correct
location of the TDE wallet as shown below, and replace the variable
$ORACLE_UNQNAME
with the value obtained with the SQL statement
SHOW PARAMETER DB_UNIQUE_NAME
.
ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))
For example:
SQL> show parameter db_unique_name
db_unique_name string oci112_region
ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/oci112_region)))
Solution 3: Run the following query and make sure that the wallet status is
OPEN
.
SQL> select * from v$encryption_wallet
WRL_TYPE
-------------
WRL_PARAMETER
-------------
STATUS
-------------
file
/opt/oracle/dcs/commonstore/wallets/tde/abc_test
OPEN
1.5.2.2 Backup Fails with ORA-19914 and ORA-28365
Issue: Source database full backup fails with the following errors.
channel ORA_SBT_TAPE_3: backup set complete, elapsed time: 00:00:15
channel ORA_SBT_TAPE_3: starting compressed full datafile backup set
channel ORA_SBT_TAPE_3: specifying datafile(s) in backup set
input datafile file number=00005 name=+DATA/ODA122/7312FA75F2B202E5E053050011AC5977/DATAFILE/system.382.1003858429
channel ORA_SBT_TAPE_3: starting piece 1 at 25-MAR-19
RMAN-03009: failure of backup command on ORA_SBT_TAPE_3 channel at 03/25/2019 19:09:30
ORA-19914: unable to encrypt backup
ORA-28365: wallet is not open
continuing other job steps, job failed will not be re-run
channel ORA_SBT_TAPE_3: starting compressed full datafile backup set
channel ORA_SBT_TAPE_3: specifying datafile(s) in backup set
Solution: Ensure that the wallet is opened in the database, and in case of CDB, ensure that the wallet is opened in the CDB, all PDBs, and PDB$SEED. See Setting Up the Transparent Data Encryption Wallet in the Zero Downtime Migration documentation for information about setting up TDE.
1.5.2.3 Either the Bucket Named Object Storage Bucket Name Does Not Exist in the Namespace Namespace or You Are Not Authorized to Access It
See Oracle Support Knowledge Base article "Either the Bucket Named '<Object Storage Bucket Name>' Does not Exist in the Namespace '<Namespace>' or You are not Authorized to Access it (Doc ID 2605518.1)" for the desciption and workarounds for this issue.
1.5.3.1 Transparent Data Encryption General Information
Depending on your source database release, Transparent Data Encryption (TDE) configuration may be required.
- Oracle Database 12c Release 2 and later
For Oracle Database 12c Release 2 and later releases, TDE configuration is mandatory and must be enabled on the source database.
If TDE is not enabled, the database migration will fail.
Upon restore, the database tablespaces are encrypted using the wallet.
- Oracle Database 12c Release 1 and earlier
On Oracle Database 12c Release 1 and Oracle Database 11g Release 2 (11.2.0.4), TDE configuration is not mandatory.
For information about the behavior of TDE in an Oracle Cloud environment, see My Oracle Support document Oracle Database Tablespace Encryption Behavior in Oracle Cloud (Doc ID 2359020.1).
1.5.3.2 Job Fails in Phase ZDM_SETUP_TDE_TGT
Issue: The phase ZDM_SETUP_TDE_TGT
fails with one of the
following errors.
Executing phase ZDM_SETUP_TDE_TGT
Setting up Oracle Transparent Data Encryption (TDE) keystore on the target node oci1121 ...
oci1121: <ERR_FILE><Facility>PRGZ</Facility><ID>ZDM_KEYSTORE_NOT_SETUP_ERR</ID><ARGS><ARG>oci112_phx1z3</ARG></ARGS></ERR_FILE>
PRGO-3007 : failed to migrate database "db11204" with zero downtime
PRCZ-4002 : failed to execute command "/u01/app/18.0.0.0/grid/perl/bin/perl" using the privileged execution plugin "zdmauth" on nodes "oci1121"
PRCZ-2103 : Failed to execute command "/u01/app/18.0.0.0/grid/perl/bin/perl" on node "oci1121" as user "root". Detailed error:
<ERR_FILE><Facility>PRGZ</Facility><ID>ZDM_KEYSTORE_NOT_SETUP_ERR</ID><ARGS><ARG>oci112_phx1z3</ARG></ARGS></ERR_FILE>
Error at target server in /tmp/zdm749527725/zdm/log/mZDM_oss_standby_setup_tde_tgt_71939.log
2019-06-13 10:00:20: Keystore location /opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME does not exists for database 'oci112_region'
2019-06-13 10:00:20: Reporting error:
<ERR_FILE><Facility>PRGZ</Facility><ID>ZDM_KEYSTORE_NOT_SETUP_ERR</ID><ARGS><ARG>oci112_region</ARG></ARGS></ERR_FILE>
Solution:
- Oracle Database 12c Release 1 and later
On the target database, make sure that
$ORACLE_HOME/network/admin/sqlnet.ora
points to the correct location of the TDE wallet. For exmaple:ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)
- Oracle Database 11g Release 2 (11.2.0.4) only
On the target database, make sure that
$ORACLE_HOME/network/admin/sqlnet.ora
points to the correct location of the TDE wallet, and replace the$ORACLE_UNQNAME
variable with the value obtained from theSHOW PARAMETER DB_UNIQUE_NAME
SQL command.For example, run
SQL> show parameter db_unique_name db_unique_name string oci112_region
and replace
ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))
with
ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/oci112_region)))
1.5.4.1 Troubleshooting Post Migration Automatic Backup Failures
Issue: Post migration, on the target database, Automatic Backup might fail.
You can verify the failure using the console in Bare Metal, VM and Exadata > DB Systems > DB System Details > Database Details > Backups.
Solution: Get the RMAN configuration settings from one of the following places.
- Zero Downtime Migration documentation in Target Database Prerequisites, if captured
- The log files at
/opt/oracle/dcs/log/hostname/rman/bkup/db_unique_name/
/tmp/zdmXXX/zdm/zdm_TDBNAME_rman.dat
For example, using the second option, you can get the RMAN configuration
settings from
/opt/oracle/dcs/log/ocidb1/rman/bkup/ocidb1_abc127/rman_configure*.log
,
then reset any changed RMAN configuration settings for the target database to ensure
that automatic backup works without any issues.
If this workaround does not help, then debug further by getting the RMAN job ID by
running the DBCLI command, list-jobs
, and describe the job details for
more error details by running the DBCLI command describe-job -i JOB
ID
from the database server as the root user.
For example, during the test, the following highlighted settings were modified to make Automatic Backup work.
rman target /
Recovery Manager: Release 12.2.0.1.0 - Production on Mon Jul 8 11:00:18 2019
Copyright (c) 1982, 2017, Oracle and/or its affiliates. All rights reserved.
connected to target database: ORCL (DBID=1540292788)
RMAN> show all;
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name OCIDB1_ABC127 are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF;
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO '%F'; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4 BACKUP TYPE TO COMPRESSED BACKUPSET;
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 2 G;
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' MAXPIECESIZE 2 G FORMAT '%d_%I_%U_%T_%t' PARMS
'SBT_LIBRARY=/opt/oracle/dcs/commonstore/pkgrepos/oss/odbcs/libopc.so ENV=(OPC_PFILE=/opt/oracle/dcs/commonstore/objectstore/opc_pfile/1245080042/opc_OCIDB1_ABC127.ora)';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE ON;
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'MEDIUM' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE;
CONFIGURE RMAN OUTPUT TO KEEP FOR 7 DAYS; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE';
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+RECO/ OCIDB1_ABC127/controlfile/snapcf_ocidb1_abc127.f';
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK clear;
RMAN>
1.5.4.2 Post Migration Automatic Backup Fails With DCS-10045
Issue: Post migration, Automatic Backup fails with the following error for non-TDE enabled migrated Oracle Database releases 11.2.0.4 and 12.1.0.2.
DCS-10045: Validation error encountered: Backup password is mandatory to take OSS backup for non-tde enabled database...
You can verify this error by getting the RMAN job ID by running DBCLI command
list-jobs
, and describe the job details to get the error details by
running DBCLI command describe-job -i JOB ID
from
the database server as the root user.
Solution:
- Find the TDE wallet location.
The Oracle Cloud Infrastructure provisioned database instance will have following entry in
sqlnet.ora
.ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))
- Remove the
cwallet.sso
file from the wallet location.For example,
/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME
. - For Oracle Database 11g Release 2, do the folowing steps.
- Connect to database using SQL*Plus as sysdba and verify the current wallet
location.
SQL> select * from v$encryption_wallet; WRL_TYPE WRL_PARAMETER STATUS file /opt/oracle/dcs/commonstore/wallets/tde/ocise112_region OPEN
- Close the wallet in the
database.
SQL> alter system set wallet close;
- Open the wallet using the wallet
password.
SQL> alter system SET WALLET open IDENTIFIED BY "walletpassword"
- Set the master encryption
key.
SQL> alter system set encryption key identified by "walletpassword"
- Recreate the autologin SSO
file.
/home/oracle>orapki wallet create -wallet /opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME -auto_login Oracle PKI Tool : Version 11.2.0.4.0 - Production Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. Enter wallet password: #
- Retry Automatic Backup.
- Connect to database using SQL*Plus as sysdba and verify the current wallet
location.
- For Oracle Database 12c, do the folowing steps.
- Connect to database using SQL*Plus as sysdba and verify the current wallet
location and
status.
SQL> SELECT wrl_parameter, status, wallet_type FROM v$encryption_wallet; WRL_PARAMETER STATUS WALLET_TYPE /opt/oracle/dcs/commonstore/wallets/tde/ocise112_region OPEN_NO_MASTER_KEY OPEN
If the STATUS column contains a value of OPEN_NO_MASTER_KEY, you must create and activate the master encryption key.
- Close the wallet in the
database.
SQL> alter system set wallet close;
- Open the wallet-using
password.
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE open IDENTIFIED BY "walletpassword" CONTAINER=all;
- Set the master encryption
key.
SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY "walletpassword" with backup;
Log in to each PDB and run
SQL> ALTER SESSION SET CONTAINER = PDB_NAME; SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY "walletpassword" with backup;
- Create the auto login
keystore.
SQL> ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE FROM KEYSTORE 'path to wallet directory' IDENTIFIED BY "walletpassword";
- Retry Automatic Backup.
- Connect to database using SQL*Plus as sysdba and verify the current wallet
location and
status.
1.5.4.3 Post Migration Automatic Backup Fails With DCS-10045
Issue: Post migration, Automatic Backup fails with the following error.
DCS-10096:RMAN configuration 'Retention policy' must be configured as 'configure retentio n
policy to recovery window of 30 days'
You can verify this error by getting the RMAN job ID by running DBCLI
command list-jobs
, and describe the job details for more error details
by running DBCLI command describe-job -i JOB
ID
from the database server as the root user.
Solution: Log in into RMAN prompt and configure the retention policy.
[oracle@racoci1 ~]$ rman target /
Recovery Manager: Release 12.2.0.1.0 - Production on Wed Jul 17 11:04:35 2019
Copyright (c) 1982, 2017, Oracle and/or its affiliates. All rights reserved.
connected to target database: SIODA (DBID=2489657199)
RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;
old RMAN configuration parameters:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
new RMAN configuration parameters:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;
new RMAN configuration parameters are successfully stored
Retry Automatic Backup.
1.5.5.1 Unable to Rerun MIGRATE DATABASE Command
Issue:
Zero Downtime Migration blocks attempts to rerun the MIGRATE DATABASE
command for a specified database if that database is already part of an ongoing
migration job.
EXECUTING
or
PAUSED
state using the ZDMCLI ABORT JOB
command as
follows.-bash-4.2$ ./zdmcli abort job -jobid 70
server.example.com: Audit ID: 189
1.5.5.2 Oracle RAC Migration Job Fails at ZDM_PREPARE_TGT
Issue: An Oracle RAC migration job fails at phase
ZDM_PREPARE_TGT
with the following error.
Executing phase ZDM_PREPARE_TGT
Setting up standby on the target node oci1121 ...
oci1121: 2019-06-13 09:54:20: Copy '/u01/app/oracle/admin/oci112_region' to remote node 'oci1122' failed
oci1121: 2019-06-13 09:54:27: Copy '/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/orapwoci1121' to remote node 'oci1122:/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/orapwoci1122' failed
PRCZ-4002 : failed to execute command "/u01/app/18.0.0.0/grid/perl/bin/perl" using the privileged execution plugin "zdmauth" on nodes "oci1121"
PRCZ-2103 : Failed to execute command "/u01/app/18.0.0.0/grid/perl/bin/perl" on node "oci1121" as user "root". Detailed error:
2019-06-13 09:54:20: Copy '/u01/app/oracle/admin/oci112_phx1z3' to remote node 'oci1122' failed
2019-06-13 09:54:27: Copy '/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/orapwoci1121' to remote node 'oci1122:/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/orapwoci1122' failed
Solution: You must set up SSH connectivity without a passphrase between the target Oracle RAC servers for the oracle user.
1.5.5.3 INS-42505 Warning Shown During Installation
/stage/user/ZDM_KIT_relnumber>./zdminstall.sh setup
oraclehome=/stage/user/grid oraclebase=/stage/user/base
ziploc=/stage/user/ZDM_KIT_relnumber/rhp_home.zip -zdm
---------------------------------------
Unzipping shiphome to gridhome
---------------------------------------
Unzipping shiphome...
Shiphome unzipped successfully..
---------------------------------------
##### Starting GridHome Software Only Installation #####
---------------------------------------
Launching Oracle Grid Infrastructure Setup Wizard...
[WARNING] [INS-42505] The installer has detected that the Oracle Grid
Infrastructure home software at (/stage/user/grid) is not complete.
CAUSE: Following files are missing:
...
Solution: This warning message can be ignored. It does not affect the installation or cause any issues for migration.
1.5.5.4 Migration Evaluation Failure with Java Exception Invalid Key Format
Issue: The following conditions are seen:
-
Zero Downtime Migration
migration -eval
command fails with the following error.Result file path contents: "/u01/app/zdmbase/chkbase/scheduled/job-19-2019-12-02-03:46:19.log" zdm-server.ocitoolingsn.ocitooling.oraclevcn.com: Processing response file ... null
-
The file
$ZDM_BASE/<zdm service host>/rhp/rhpserver.log.0
contains the following entry.Verify below error message observed in file $ZDM_BASE/<zdm service host>/rhp/rhpserver.log.0 rhpserver.log.7:[pool-58-thread-1] [ 2019-12-02 02:08:15.178 GMT ] [JSChChannel.getKeyPair:1603] Exception : java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: invalid key format
-
The Zero Downtime Migration installed user (For example: zdmuser) private key (id_rsa) file has the following entries.
-----BEGIN OPENSSH PRIVATE KEY---------- MIIEogIBAAKCAQEAuPcjftR6vC98fAbU4FhYVKPqc0CSgibtMSouo1DtQ06ROPN0 XpIEL4r8nGp+c5GSDONyhf0hiltBzg0fyqyurSw3XfGJq2Q6EQ61aL95Rt9CZh6b JSUwc69T4rHjvRnK824k4UpfUIqafOXb2mRgGVUkldo4yy+pLoGq1GwbsIYbS4tk uaYPKZ3A3H9ZA7MtZ5M0sNqnk/4Qy0d8VONWozxOLFC2A8zbbe7GdQw9khVqDb/x END OPENSSH PRIVATE KEY-----
Solution: Authentication key pairs (private and public key) are not
generated using the ssh-keygen
utility, so you must generate
authentication key pairs using steps in Generating a Private SSH Key Without a Passphrase.
After generating authentication key pairs, the private key file content looks like the following.
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAuPcjftR6vC98fAbU4FhYVKPqc0CSgibtMSouo1DtQ06ROPN0
XpIEL4r8nGp+c5GSDONyhf0hiltBzg0fyqyurSw3XfGJq2Q6EQ61aL95Rt9CZh6b
JSUwc69T4rHjvRnK824k4UpfUIqafOXb2mRgGVUkldo4yy+pLoGq1GwbsIYbS4tk
uaYPKZ3A3H9ZA7MtZ5M0sNqnk/4Qy0d8VONWozxOLFC2A8zbbe7GdQw9khVqDb/x
-----END RSA PRIVATE KEY-----
Set up connectivity with the newly generated authentication key pairs and resume the migration job.
1.5.5.5 Migration Evaluation Fails with Error PRCG-1022
Issue: The following conditions are seen:
[zdmuser@host bin]$ ./zdmcli migrate database -sourcedb ORCL -sourcenode source_node
-srcauth zdmauth -srcarg1 user:opc -srcarg2 identity_file:/home/zdmuser/key/pvt.txt
-srcarg3 sudo_location:/bin/sudo -targetnode target_node
-targethome /u01/app/oracle/product/18.0.0.0/dbhome_1
-backupuser backupuser -rsp /home/zdmuser/respfile/zdm_template_ocic_oci.rsp
-tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/key/pvt.txt
-tgtarg3 sudo_location:/bin/sudo -eval
PRCG-1238 : failed to execute the Rapid Home Provisioning action for command 'migrate database'
PRCG-1022 : failed to connect to the Rapid Home Provisioning daemon for cluster anandutest
Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException
[Root exception is java.rmi.ConnectException: Connection refused to host:
anandutest; nested exception is: java.net.ConnectException: Connection refused (Connection refused)]
Solution: Start the Zero Downtime Migration service using the
$ZDM_HOME/bin/zdmservice start
command, then run any
ZDMCLI
commands.
1.5.5.6 Unable to Resume a Migration Job
Issue: Zero Downtime Migration writes the
source and target log files to the /tmp/zdm-unique id
directory in the
respective source and target database servers.
If you pause a migration job and and then resume the job after several (sometimes 15-20
days), the /tmp/zdm-unique id
directory might be deleted or purged as
part of a clean up or server reboot that also cleans up /tmp.
Solution: After pausing a migration job, back up the
/tmp/zdm-unique id
directory. Before resuming the migration job,
check the /tmp
directory for /zdm-unique id
, and if it
is missing, restore the directory and its contents with your backup.
1.6 Additional Information for Migrating to Oracle Cloud Infrastructure
Read the following for general information, considerations, and links to more information about using Zero Downtime Migration to migrate your database to Oracle Cloud Infrastructure.
1.7 Additional Information for Migrating to Exadata Cloud Service
Read the following for general information, considerations, and links to more information about using Zero Downtime Migration to migrate your database to Exadata Cloud Service.
1.7.1 Considerations for Migrating to Exadata Cloud Service
For this release of Zero Downtime Migration be aware of the following considerations.
- If the source database is release 18c, then the target home should be at release 18.6 or later to avoid issues such as Bug 29445548 Opening Database In Cloud Environment Fails With ORA-600.
- PDB conversion related phases are listed in
-listphases
and can be ignored. Those are no-op phases. - Non-CDB to PDB related input parameters in the response file are place holders that
should not be set. Setting
NONCDBTOPDB_*
inputs to true will break the migration. - If a backup was performed when one of the configured instances is down, you will encounter Bug 29863717 - DUPLICATING SOURCE DATABASE FAILED BECAUSE INSTANCE 1 WAS DOWN.
- The TDE keystore password must be set in the credential wallet. To set
the password as part of the Zero Downtime Migration workflow, specify the
-tdekeystorepasswd
argument irrespective of whether the wallet usesAUTOLOGIN
orPASSWORD
. In either case the password is stored in the credential wallet. If the-tdekeystorepasswd
argument is not supplied, then Zero Downtime Migration skips the settingtde_ks_passwd
key in the credential wallet, and no error is thrown. - The target environment must be installed with latest DBaaS Tooling RPM
with
db_unique_name
change support to be installed. - Provision a target database from the console without enabling auto-backups. In the Configure database backups section do not select the Enable automatic backups option.
1.7.2 Exadata Cloud Service Database Registration
Post migration, register the Exadata Cloud Service database, and make sure its meets all of the requirements.
Run the following commands on the Exadata Cloud Service database server as the root user.
/root>dbaascli registerdb prereqs --dbname db_name --db_unique_name db_unique_name
/root>dbaascli registerdb begin --dbname db_name --db_unique_name db_unique_name
For example
/root>dbaascli registerdb prereqs --dbname ZDM122 --db_unique_name ZDM122_phx16n
DBAAS CLI version 18.2.3.2.0
Executing command registerdb prereqs --db_unique_name ZDM122_phx16n
INFO: Logfile Location: /var/opt/oracle/log/ZDM122/registerdb/registerdb_2019-08-14_05:35:31.157978280334.log
INFO: Prereqs completed successfully
/root>
/root>dbaascli registerdb begin --dbname ZDM122 --db_unique_name ZDM122_phx16n
DBAAS CLI version 18.2.3.2.0
Executing command registerdb begin --db_unique_name ZDM122_phx16n
Logfile Location: /var/opt/oracle/log/ZDM122/registerdb/registerdb_2019-08-14_05:45:27.264851309165.log
Running prereqs
DBAAS CLI version 18.2.3.2.0
Executing command registerdb prereqs --db_unique_name ZDM122_phx16n
INFO: Logfile Location: /var/opt/oracle/log/ZDM122/registerdb/registerdb_2019-08-14_05:45:29.000432309894.log
INFO: Prereqs completed successfully
Prereqs completed
Running OCDE .. will take time ..
OCDE Completed successfully.
INFO: Database ZDM122 registered as Cloud database
/root>
1.7.3 Exadata Cloud Service Automatic Backup Issues
Check the backup configuration before you enable automatic backup from the console. You
can use the get config
command as shown in the first step below. You
should see bkup_oss=no
before you enable automatic backup.
You might see the error message in the console, "A backup configuration exists for this database. You must remove the existing configuration to use Oracle Cloud Infrastructure's managed backup feature."
To fix this error, remove the existing configuration.
First, make sure the automatic backup is disabled from the UI, then follow these steps to remove the existing backup configuration.
- Generate a backup configuration
file.
/var/opt/oracle/bkup_api/bkup_api get config --file=/tmp/db_name.bk --dbname=db_name
For example:
/var/opt/oracle/bkup_api/bkup_api get config --file=/tmp/zdmdb.bk --dbname=zdmdb
- Open the /tmp/db_name.bk file you created in the previous
step.
For example: Open /tmp/zdmdb.bk
change bkup_oss=yes from bkup_oss=no
- Disable OSS backup by setting
bkup_oss=no
./var/opt/oracle/bkup_api/bkup_api set config --file=/tmp/db_name.bk --dbname=db_name
For example:
/var/opt/oracle/bkup_api/bkup_api set config --file=/tmp/zdmdb.bk --dbname=zdmdb
- Check reconfigure
status.
/var/opt/oracle/bkup_api/bkup_api configure_status --dbname=db_name
For example:
/var/opt/oracle/bkup_api/bkup_api configure_status --dbname=zdmdb
Now enable automatic backup from console.
Verify the backups from the console. Click Create Backup to create a manual backup, and a backup should be created without any issues. and also Automatic Backup should be successful.
1.8 Additional Information for Migrating to Exadata Cloud at Customer
Read the following for general information, considerations, and links to more information about using Zero Downtime Migration to migrate your database to Exadata Cloud at Customer.
1.8.1 Considerations for Migrating to Exadata Cloud at Customer
For this release of Zero Downtime Migration be aware of the following considerations.
- You must apply the regDB patch for Bug 29715950 - "modify regdb to
handle db_unique_name not same as db_name" on all Exadata Cloud at Customer nodes.
This is required for the
ZDM_MANIFEST_TO_CLOUD
phase. Please note that the regDB tool is part of DBaaS Tooling. - If the source database is release 18c, then the target home should be at release 18.6 or later to avoid issues such as Bug 29445548 Opening Database In Cloud Environment Fails With ORA-600.
- PDB conversion related phases are listed in
-listphases
and can be ignored. Those are no-op phases. - Non-CDB to PDB related input parameters in the response file are place holders that
should not be set. Setting
NONCDBTOPDB_*
inputs to true will break the migration. - If the backup medium is Zero Data Loss Recovery Appliance, then all configured
instances should be up at the source when a
FULL
orINCREMENTAL
backup is performed. - If a backup was performed when one of the configured instances is down, you will encounter Bug 29863717 - DUPLICATING SOURCE DATABASE FAILED BECAUSE INSTANCE 1 WAS DOWN.
- The TDE keystore password must be set in the credential wallet. To set
the password as part of the Zero Downtime Migration workflow, specify the
-tdekeystorepasswd
argument irrespective of whether the wallet usesAUTOLOGIN
orPASSWORD
. In either case the password is stored in the credential wallet. If the-tdekeystorepasswd
argument is not supplied, then Zero Downtime Migration skips the settingtde_ks_passwd
key in the credential wallet, and no error is thrown. - The target environment must be installed with latest DBaaS Tooling RPM
with
db_unique_name
change support to be installed.
1.9 Documentation Addenda
The following are additional documentation sections to support moving to oracle Cloud with Zero Downtime Migration.
1.9.1 Installing Zero Downtime Migration Software
The following is an update to Move to Oracle Cloud Using Zero Downtime Migration, Chapter 2, Installing Zero Downtime Migration Software.
Step 2a is updated to:
Change to the directory to where Zero Downtime Migration software is downloaded and unzip the software.
zdmuser> cd zdm_download_directory
zdmuser> unzip zdm_home.zip
1.9.2 Terminiate a Running Migration Job
The following is an addendum to Move to Oracle Cloud Using Zero Downtime Migration, Chapter 4, Migrating Your Database with Zero Downtime Migration .
Zero Downtime Migration blocks attempts to rerun the MIGRATE DATABASE
command for a specified database if that database is already part of an ongoing
migration job.
If you want to resubmit a database migration job for a specified database,
you must first terminiate the running migration job in either EXECUTING
or PAUSED
state using the ZDMCLI ABORT JOB
command.
zdmuser> ./zdmcli abort job -jobid job-id
1.10 Documentation Accessibility
For information about Oracle's commitment to accessibility, visit the Oracle Accessibility Program website at http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc.
Access to Oracle Support
Oracle customers that have purchased support have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.
Oracle Zero Downtime Migration Release Notes, Release 19c
F19479-03
Copyright © 2019, 2020, Oracle and/or its affiliates.
This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.
The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.
If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable:
U.S. GOVERNMENT END USERS: Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs) and Oracle computer documentation or other Oracle data delivered to or accessed by U.S. Government end users are "commercial computer software" or “commercial computer software documentation” pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, reproduction, duplication, release, display, disclosure, modification, preparation of derivative works, and/or adaptation of i) Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs), ii) Oracle computer documentation and/or iii) other Oracle data, is subject to the rights and limitations specified in the license contained in the applicable contract. The terms governing the U.S. Government’s use of Oracle cloud services are defined by the applicable contract for such services. No other rights are granted to the U.S. Government.
This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Intel and Intel Inside are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Epyc, and the AMD logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.
This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle.