These release notes provide downloading instructions for the latest product software and documentation, and describe new features, fixed bugs, known issues, and troubleshooting information for Zero Downtime Migration Release 19.7.

1.1 What's New in This Release

Zero Downtime Migration Release 19.7 includes the following changes to features and behavior.

  • The following new parameters are included in the migration response file.

    • The following parameters let you set intervals, in minutes, at which to monitor the progress of backup and restore operations. To disable the monitoring interval parameters, set them to 0 (zero). The default value is 10 (minutes).
      • ZDM_BACKUP_FULL_SRC_MONITORING_INTERVAL specifies the time interval, in minutes, at which to monitor and report the progress of the ZDM_BACKUP_FULL_SRC migration job phase.
      • ZDM_BACKUP_INCREMENTAL_SRC_MONITORING_INTERVAL specifies the time interval, in minutes, at which to monitor and report the progress of the ZDM_BACKUP_INCREMENTAL_SRC migration job phase.
      • ZDM_BACKUP_DIFFERENTIAL_SRC_MONITORING_INTERVAL specifies the time interval, in minutes, at which to monitor and report the progress of the ZDM_BACKUP_DIFFERENTIAL_SRC migration job phase.
      • ZDM_CLONE_TGT_MONITORING_INTERVAL specifies the time interval, in minutes, at which to monitor and report the progress of the ZDM_CLONE_TGT migration job phase.
      • ZDM_OSS_RECOVER_TGT_MONITORING_INTERVAL specifies the time interval, in minutes, at which to monitor and report the progress of the ZDM_OSS_RECOVER_TGT migration job phase.
      • ZDM_OSS_RESTORE_TGT_MONITORING_INTERVAL specifies the time interval, in minutes, at which to monitor and report the progress of the ZDM_OSS_RESTORE_TGT migration job phase.

    • SKIP_SRC_SERVICE_RETENTION=TRUE lets you skip source database service retention. Set to TRUE if you do not wish to retain the services during database migration from source to target.

    • ZDM_BACKUP_RETENTION_WINDOW lets you specify the number of days after which database backups created by Zero Downtime Migration become obsolete. The default is 60 (days).

    • ZDM_OPC_RETRY_WAIT_TIME lets you specify the Object Store retry wait time in seconds. The default value is 529 (seconds).

      ZDM_OPC_RETRY_COUNT lets you specify how many times to retry the Object Store. The default value is 0 (zero).

    • ZDM_CURL_LOCATION lets you specify a custom location for CURL binary on the source. The default value is /usr/bin/curl.

    • The following parameters let you specify how long, in minutes, to reattempt SSH connection failures to the source and target servers. Retries stop once the timeout value has been exceeded.

      SRC_SSH_RETRY_TIMEOUT

      TGT_SSH_RETRY_TIMEOUT

    • The following parameters automate aspects of post-migration Datapatch execution.

      TGT_SKIP_DATAPATCH lets you specify whether or not Zero Downtime Migration runs the datapatch utility on the target database as part of the post-migration tasks. The default value is FALSE, which allows Zero Downtime Migration to run datapatch. Set to TRUE if you want to skip this phase.

      MAX_DATAPATCH_DURATION_MINS lets you configure a timeout value, in minutes, after which, if the datapatch utility has failed to complete, then the operation is stopped. There is no default value. By deafult, Zero Downtime Migration waits until datapatch completes.

      DATAPATCH_WITH_ONE_INSTANCE_RUNNING, when set to TRUE, lets Zero Downtime Migration stop all instances except one running on the target database server when the datapatch utility runs. When datapatch completes, all of the stopped instances are started. The default value is FALSE.

    • ZDM_LOG_OSS_PAR_URL lets you specify a pre-authenticated URL to which to upload log files. The logs also capture the current phase as well as the execution status of the phase.

    • ZDM_SRC_TNS_ADMIN lets you specify a custom location for TNS_ADMIN on the source database when there is no Grid Infrastructure. If a Grid Infrastructure exists, then the TNS_ADMIN property must be set in the CRS resource attribute environment of the database resource.

    • Zero Downtime Migration automatically discovers the location of the target database for Oracle-managed datafiles, control files, and online redo logs. Parameters previously required to set these locations will remain in the response file as optional parameters that you can use to override the automatically discovered locations.

  • New parameters in the ZDMCLI MIGRATE DATABASE command facilitate the automated use of wallet-based credentials.

    • -sourcesyswallet sys_wallet_path lets you specify the full path for the auto-login wallet file on the Zero Downtime Migration host containing the SYS password of the source database
    • -osswallet oss_wallet_path lets you specify the full path for the auto-login wallet file on the Zero Downtime Migration host containing the credential for the OSS backup user
    • -tdekeystorewallet tde_wallet_path lets you specify the full path for the auto-login wallet file on the Zero Downtime Migration host containing the TDE keystore password

  • The ZDMCLI QUERY JOB command has several new parameters which can filter the list of jobs for which you receive output and provide more information about the jobs.

    • -job_type returns the type of the scheduled job
    • -sourcenode lets you specify the source database server
    • -sourcedb lets you specify the source database name
    • -sourcesid lets you specify the source database ORACLE_SID
    • -targetnode lets you specify the target database server
    • -latest returns the most recent job that matches the given criteria
    • -eval returns evaluation jobs only
    • -migrate returns migration jobs only
    • -statusonly returns only the status and phase name of the job

    See Zero Downtime Migration ZDMCLI Command Reference for more information about the QUERY JOB command.

  • The -targethome option no longer needs to be specified when you run the ZDMCLI MIGRATE DATABASE command, because the target database home is automatically discovered.

Parent topic: Zero Downtime Migration Release Notes

1.2 Downloading the Zero Downtime Migration Installation Software

For a fresh installation of the latest Zero Downtime Migration software version, go to https://www.oracle.com/database/technologies/rac/zdm-downloads.html.

Parent topic: Zero Downtime Migration Release Notes

1.3 Downloading the Zero Downtime Migration Documentation

You can browse and download Zero Downtime Migration documentation at https://docs.oracle.com/en/database/oracle/zero-downtime-migration/

Parent topic: Zero Downtime Migration Release Notes

1.4 General Information

At the time of this release, there are some details and considerations about Zero Downtime Migration behavior that you should take note of.

Parent topic: Zero Downtime Migration Release Notes

1.4.1 Patch Level Differences Between Source and Target Database

The major database release numbers of the source and target database must match for the migration job to succeed; however, patch level differences between the source and target databases might exist where the target database patch level is equal to or higher than the source database.

For example, if the source database is Oracle Database 12c Release 1 (12.1.0.2), the target database must also be at release 12.1.0.2. However, the patch level on the target database might be higher than the patch level on the source database. For example, if the target database is at Jan 2020 PSU/BP and the source database is at April 2020 PSU/BP, then you must run datapatch after database migration.

To avoid patch checking as part of the migration procedure, you can use the -ignore PATCH_CHECK option in the ZDMCLI MIGRATE DATABASE command.

Parent topic: General Information

1.4.2 ZDMSERVICE Script Can Only Be Run By Installed User

If you are prevented from running the zdmservice script, note that it is required that the zdmservice script only be run by the installed user. For security reasons the zdmservice script should not be run by any user other than the zdmservice user. To avoid running zdmservice by a non-zdmservice installed user, change the zdmservice binary permissions to 700.

Parent topic: General Information

1.4.3 UNDO Tablespaces Added to the Source Database

Zero Downtime Migration adds UNDO tablespaces to the production database to match the target instance count if the production database has fewer instances.

To prevent Zero Downtime Migration from adding UNDO tablespaces to the source database, you can match the target database nodes count to that of the source database until the switchover, then you can add additional nodes to the target database after the switchover.

Parent topic: General Information

1.4.4 Cross-Edition Migration Is Not Supported

Zero Downtime Migration cannot be used to migrate an Enterprise edition database to a Standard edition database, and vice versa.

Parent topic: General Information

1.5 Bugs Fixed

Zero Downtime Migration Release 19.7 incudes the bug fixes listed in the following table.

Table 1-1 Bugs Fixed In Zero Downtime Migration Release 19.7

Bug Number Description

29151628

ZDM:SI:CDB+PDB:NON CLUSTER DATABASE MIGRATION IS FAILING WHILE RETRIEVE ORACLE OBJECT STORAGE SERVICE CONTAINER

29383436

ZDM:VERIFY EXECUTING DATAPATCH STEP NECESSITY POST MIGRATION

29704996

ZDM: ZDM_NONCDBTOPDB_CONVERSION LISTED IN ALL MIGRATION WORKFLOW

29750170

OCI MIGRATION - ZDM ER: EXTRACT LOGS FROM EACH STAGE FOR ZDM INTO OSS

30294364

ZDM CONFIG CHECK - ZDM INTROSPECT ASM VALUES INSTEAD OF HARDCODING.

30397301

ZDM RESILIENCY : SSH RETRIES TO BE IMPLEMENTED FOR SSH CONNECTION ERRORS

30408193

NEED ZDM TO VALIDATE AND ENCRYPT SYS , TDE PASSWORD

30422373

OMCS:ZDM: RMAN BACKUP FAILURE FOR COMPUTE - IAAS - NON RAC : ORA-19836: CANNOT USE PASSPHRASE ENCRYPTION FOR THIS BACKUP

30528926

LOG_ARCHIVE_CONFIG NOT CLEANED AFTER REMOVING DATA GUARD

30603943

OCI MIGRATION: ZDM NOT REPORTING ERROR WHEN DATABASE IS DOWN AT SOURCE

30606660

ZDMCLI QUERY JOB WITH EVAL OPTION REPORTS INCORRECT JOB DETAILS WHEN THERE IS NO "ZDMCLI MIGRATE DATABASE" JOBS WITH -EVAL OPTION

30617388

BACKUP OF ARCHIVELOGS NOT INCLUDED IN CONTROLFILE

30741086

ZDM: ASSOCIATE /TMP/ZDM-UNIQUE ID DIRECTORY WITH JOB ID

30824963

ZDM: DATAPATCH STOP ALL INSTANCES BUT ONE INCORRECTLY DEPENDS ON BASTION_IP

30857772

ZDM: MIGRATION TO ACFS FILE SYSTEM IS FAILING

30863623

OFFLINE MIGRATE IS FAILING DUE TO MISSING FULL BACKUP

31087739

ZDM: SINGLE INSTANCE WITHOUT GI AND BACKUP MEDIUM AS ZDLRA MIGRATION TO EXACC IS FAILING

Parent topic: Zero Downtime Migration Release Notes

1.6 Known Issues

At the time of this release, there are several issues with Zero Downtime Migration that could occur in rare circumstances. For each issue, a workaround is provided.

Parent topic: Zero Downtime Migration Release Notes

1.6.1 Connectivity Issues

Parent topic: Known Issues

1.6.1.1 General Connectivity Issues

Issue: If connectivity issues occur between the Zero Downtime Migration service host and the source or target environments, or between source and target environments, check the following areas.

Solution: Verify that the SSH configuration file (/root/.ssh/config) has the appropriate entries: 

Host *
  ServerAliveInterval 10
  ServerAliveCountMax 2

Host ocidb1
  HostName 192.0.2.1
  IdentityFile ~/.ssh/ocidb1.ppk
  User opc
  ProxyCommand /usr/bin/nc -X connect -x www-proxy.example.com:80 %h %p

Note that the proxy setup might not be required when you are not using a proxy server for connectivity. For example, when the source database server is on Oracle Cloud Infrastructure Classic, you can remove or comment the line starting with ProxyCommand.

If the source is an Oracle RAC database, then make sure you copy the ~/.ssh/config file to all of the source Oracle RAC servers. The SSH configuration file refers to the first Oracle RAC server host name, public IP address, and private key attributes.

Parent topic: Connectivity Issues

1.6.1.2 Evaluation Fails in Phase ZDM_GET_TGT_INFO

Issue: During the evaluation (-eval) phase of the migration process, the evaluation fails in the ZDM_GET_TGT_INFO phase with the following error for the Oracle RAC instance migration. 

Executing phase ZDM_GET_TGT_INFO
Retrieving information from target node "trac11" ...
PRGZ-3130 : failed to establish connection to target listener from nodes [srac11, srac12]
PRCC-1021 : One or more of the submitted commands did not execute successfully.
PRCC-1025 : Command submitted on node srac11 timed out after 15 seconds.
PRCC-1025 : Command submitted on node srac12 timed out after 15 seconds.

Solution:

  1. Get the SCAN name of source database and add it to the /etc/hosts file on both target database servers, with the public IP address of the source database server and the source database SCAN name. For example: 
    192.0.2.3 source-scan
  2. Get the SCAN name of the target database and add it to the /etc/hosts file on both source database servers, with the public IP address of the target database server and target database SCAN name. For example:
    192.0.2.1  target-scan

Note:

This issue, where the SCAN IP address is not added to /etc/hosts file, might occur because in some cases the SCAN IP address is assigned as a private IP address, so it might not be resolvable.

Parent topic: Connectivity Issues

1.6.1.3 Evaluation Fails in Phase ZDM_GET_SRC_INFO

Issue: During the evaluation (-eval) phase of the migration process, the evaluation fails in the ZDM_GET_SRC_INFO phase with the following error for the source single instance deployed without Grid infrastructure. 

Executing phase ZDM_GET_SRC_INFO
retrieving information about database "zdmsidb" ...
PRCF-2056 : The copy operation failed on node: "zdmsidb".
Details: {1}
PRCZ-4002 : failed to execute command "/bin/cp" using the privileged
execution plugin "zdmauth" on nodes "zdmsidb"
scp: /etc/oratab: No such file or directory

Solution: Make an ORACLE_HOME value entry in file /etc/oratab with value db_name:$ORACLE_HOME:N, as shown in this example. 

zdmsidb:/u01/app/oracle/product/12.2.0.1/dbhome_1:N

Parent topic: Connectivity Issues

1.6.1.4 Object Storage Is Not Accessible

Issue: When Object Storage is accessed from the source or target database server, it may fail with the following error.
About to connect() to swiftobjectstorage.xx-region-1.oraclecloud.com port 443 (#0)
Trying 192.0.2.1... No route to host
Trying 192.0.2.2... No route to host
Trying 192.0.2.3... No route to host
couldn't connect to host
Closing connection #0
curl: (7) couldn't connect to host

Solution: On the Zero Downtime Migration service host, in the response file template ($ZDM_HOME/rhp/zdm/template/zdm_template.rsp), set the Object Storage Service proxy host and port parameters listed below, if a proxy is required to connect to Object Storage from the source database server. For example: 

SRC_OSS_PROXY_HOST=www-proxy-source.example.com
SRC_OSS_PROXY_PORT=80

In the response file template ($ZDM_HOME/rhp/zdm/template/zdm_template.rsp), set the Object Storage Service proxy host and port parameters listed below, if a proxy is required to connect to Object Storage from the target database server. For example: 

TGT_OSS_PROXY_HOST=www-proxy-target.example.com
TGT_OSS_PROXY_PORT=80

Parent topic: Connectivity Issues

1.6.1.5 Error Messages That Can Be Ignored

Ignore the following error messages in $ZDM_BASE/crsdata/zdm service hostname/rhp/rhpserver.log.0. 

[sshd-SshClient[3051eb49]-nio2-thread-1] [ 2020-04-04 00:26:24.142 GMT ]
 [JSChChannel$LogOutputStream.flush:1520]  2020-04-04: WARNING: org.apache.sshd.client.session.C:
 globalRequest(ClientConnectionService[ClientSessionImpl[opc@samidb-db/140.238.254.80:22]])[hostkeys-00@openssh.com,
 want-reply=false] failed (SshException) to process: EdDSA provider not supported

[sshd-SshClient[3051eb49]-nio2-thread-1] [ 2020-04-04 00:26:24.142 GMT ]
 [JSChChannel$LogOutputStream.flush:1520]  2020-04-04: FINE   : org.apache.sshd.client.session.C:
 globalRequest(ClientConnectionService[ClientSessionImpl[opc@samidb-db/140.238.254.80:22]])[hostkeys-00@openssh.com,
 want-reply=false] failure details
org.apache.sshd.common.SshException: EdDSA provider not supported
    at org.apache.sshd.common.util.buffer.Buffer.getRawPublicKey(Buffer.java:446)
    at org.apache.sshd.common.util.buffer.Buffer.getPublicKey(Buffer.java:420)
    at org.apache.sshd.common.global.AbstractOpenSshHostKeysHandler.process(AbstractOpenSshHostKeysHandler.java:71)
    at org.apache.sshd.common.global.AbstractOpenSshHostKeysHandler.process(AbstractOpenSshHostKeysHandler.java:38)
    at org.apache.sshd.common.session.helpers.AbstractConnectionService.globalRequest(AbstractConnectionService.java:723)
    at org.apache.sshd.common.session.helpers.AbstractConnectionService.process(AbstractConnectionService.java:363)
    at org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:400)
    at org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:333)
    at org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1097)
    at org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:294)
    at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:63)
    at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:357)
    at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:335)
    at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:332)
    at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
    at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
    at sun.nio.ch.Invoker$2.run(Invoker.java:218)
    at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.security.NoSuchAlgorithmException: EdDSA provider not supported
    at org.apache.sshd.common.util.security.SecurityUtils.generateEDDSAPublicKey(SecurityUtils.java:596)
    at org.apache.sshd.common.util.buffer.keys.ED25519BufferPublicKeyParser.getRawPublicKey(ED25519BufferPublicKeyParser.java:45)
    at org.apache.sshd.common.util.buffer.keys.BufferPublicKeyParser$2.getRawPublicKey(BufferPublicKeyParser.java:98)
    at org.apache.sshd.common.util.buffer.Buffer.getRawPublicKey(Buffer.java:444)
    ... 22 more
[sshd-SshClient[3051eb49]-nio2-thread-1] [ 2020-04-04 00:26:24.142 GMT ]
 [JSChChannel$LogOutputStream.flush:1520]  2020-04-04: FINE   : org.apache.sshd.client.session.C:
 sendGlobalResponse(ClientConnectionService[ClientSessionImpl[opc@samidb-db/140.238.254.80:22]])[hostkeys-00@openssh.com]
 result=ReplyFailure, want-reply=false

[sshd-SshClient[3051eb49]-nio2-thread-2] [ 2020-04-04 00:26:24.182 GMT ]
 [JSChChannel$LogOutputStream.flush:1520]  2020-04-04: FINE   : org.apache.sshd.common.io.nio2.N:
 handleReadCycleCompletion(Nio2Session[local=/192.168.0.2:41198, remote=samidb-db/140.238.254.80:22])
 read 52 bytes

Parent topic: Connectivity Issues

1.6.2 Full Backup Phase (ZDM_BACKUP_FULL_SRC) Issues

Parent topic: Known Issues

1.6.2.1 Backup Fails with ORA-19836

Issue: Source database full backup fails with one of the following errors. 

</ERRLINE><ERRLINE>ORA-19836: cannot use passphrase encryption for this backup
</ERRLINE><ERRLINE>RMAN-03009: failure of backup command on C8 channel at 04/29/2019
      20:42:16
</ERRLINE><ERRLINE>ORA-19836: cannot use passphrase encryption for this backup
</ERRLINE><ERRLINE>RMAN-03009: continuing other job steps, job failed will not be
      re-run

Solution 1: This issue can occur if you specify the -sourcedb value in the wrong case. For example, if the value obtained from SQL command SHOW PARAMETER DB_UNIQUE_NAME is zdmsdb, then you need to specify it as zdmsdb in lower case, and not as ZDMSDB in upper case, as shown in the following example. 

zdmuser> $ZDM_HOME/bin/zdmcli migrate database -sourcedb zdmsdb -sourcenode ocicdb1 -srcroot
-targetnode ocidb1 -targethome /u01/app/oracle/product/12.1.0.2/dbhome_1
-backupuser backup_user@example.com -rsp /u01/app/zdmhome/rhp/zdm/template/zdm_template_zdmsdb.rsp
-tgtauth zdmauth -tgtarg1 user:opc
-tgtarg2 identity_file:/home/zdmuser/.ssh/zdm_service_host.ppk
-tgtarg3 sudo_location:/usr/bin/sudo

Solution 2: For Oracle Database 12c Release 1 and later, ensure that $ORACLE_HOME/network/admin/sqlnet.ora points to the correct location of the TDE wallet, as shown here. 

ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))

For Oracle Database 11g Release 2 (11.2.0.4) only, ensure that $ORACLE_HOME/network/admin/sqlnet.ora points to the correct location of the TDE wallet as shown below, and replace the variable $ORACLE_UNQNAME with the value obtained with the SQL statement SHOW PARAMETER DB_UNIQUE_NAME. 

ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))

For example:

SQL> show parameter db_unique_name
db_unique_name    string      oci112_region

ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/oci112_region)))

Solution 3: Run the following query and make sure that the wallet status is OPEN. 

SQL> select * from v$encryption_wallet
WRL_TYPE
-------------
WRL_PARAMETER
-------------
STATUS
-------------
file
/opt/oracle/dcs/commonstore/wallets/tde/abc_test
OPEN

Parent topic: Full Backup Phase (ZDM_BACKUP_FULL_SRC) Issues

1.6.2.2 Backup Fails with ORA-19914 and ORA-28365

Issue: Source database full backup fails with the following errors. 

channel ORA_SBT_TAPE_3: backup set complete, elapsed time: 00:00:15
channel ORA_SBT_TAPE_3: starting compressed full datafile backup set
channel ORA_SBT_TAPE_3: specifying datafile(s) in backup set
input datafile file number=00005 name=+DATA/ODA122/7312FA75F2B202E5E053050011AC5977/DATAFILE/system.382.1003858429
channel ORA_SBT_TAPE_3: starting piece 1 at 25-MAR-19
RMAN-03009: failure of backup command on ORA_SBT_TAPE_3 channel at 03/25/2019 19:09:30
ORA-19914: unable to encrypt backup
ORA-28365: wallet is not open
continuing other job steps, job failed will not be re-run
channel ORA_SBT_TAPE_3: starting compressed full datafile backup set
channel ORA_SBT_TAPE_3: specifying datafile(s) in backup set

Solution: Ensure that the wallet is opened in the database, and in case of CDB, ensure that the wallet is opened in the CDB, all PDBs, and PDB$SEED. See Setting Up the Transparent Data Encryption Wallet in the Zero Downtime Migration documentation for information about setting up TDE.

Parent topic: Full Backup Phase (ZDM_BACKUP_FULL_SRC) Issues

1.6.2.3 Either the Bucket Named Object Storage Bucket Name Does Not Exist in the Namespace Namespace or You Are Not Authorized to Access It

See Oracle Support Knowledge Base article "Either the Bucket Named '<Object Storage Bucket Name>' Does not Exist in the Namespace '<Namespace>' or You are not Authorized to Access it (Doc ID 2605518.1)" for the desciption and workarounds for this issue.

https://support.oracle.com/rs?type=doc&id=2605518.1

Parent topic: Full Backup Phase (ZDM_BACKUP_FULL_SRC) Issues

1.6.3 Restore Phase (ZDT_CLONE_TGT) Issues

Parent topic: Known Issues

1.6.3.1 Restore Database Fails With AUTOBACKUP does not contain an SPFILE

Issue: During the execution of phase ZDT_CLONE_TGT, restore database fails with the following error. 

channel C1: looking for AUTOBACKUP on day: 20200427
channel C1: AUTOBACKUP found: c-1482198272-20200427-12
channel C1: restoring spfile from AUTOBACKUP c-1482198272-20200427-12
channel C1: the AUTOBACKUP does not contain an SPFILE

The source database is running using init.ora file, but during the restore target phase, the database is trying to restore the server parameter file (SPFILE) from autobackup, therefore it fails.

Solution: Start the source database using an SPFILE and resubmit the migration job.

Parent topic: Restore Phase (ZDT_CLONE_TGT) Issues

1.6.3.2 Restore Database Fails With PRGO-4027

Issue: During the execution of phase ZDT_CLONE_TGT, restore database fails with the following error. 

Retrieving SPFILE from backup...
<ERR_FILE><Facility>PRGO</Facility><ID>ZDM_OSS_SPFILE_FROM_PFILE_FAIL</ID><ARGS><ARG><ERRLINES><ERRLINE>
</ERRLINE><ERRLINE>Recovery Manager: Release 11.2.0.4.0 - Production on Fri Apr 24 16:07:19 2020
</ERRLINE><ERRLINE>
</ERRLINE><ERRLINE>Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
</ERRLINE><ERRLINE>
</ERRLINE><ERRLINE>connected to target database: ORCL (not mounted)
</ERRLINE><ERRLINE>
</ERRLINE><ERRLINE>RMAN>
</ERRLINE><ERRLINE>executing command: SET encryption
</ERRLINE><ERRLINE>using target database control file instead of recovery catalog
</ERRLINE><ERRLINE>
</ERRLINE><ERRLINE>RMAN>
</ERRLINE><ERRLINE>RMAN> 2> 3> 4> 5>
</ERRLINE><ERRLINE>allocated channel: C1
</ERRLINE><ERRLINE>channel C1: SID=15 device type=SBT_TAPE
</ERRLINE><ERRLINE>channel C1: Oracle Database Backup Service Library VER=12.2.0.2
</ERRLINE><ERRLINE>
</ERRLINE><ERRLINE>executing command: SET DBID
</ERRLINE><ERRLINE>
</ERRLINE><ERRLINE>Starting restore at 24-APR-20 16:07:21
</ERRLINE><ERRLINE>
</ERRLINE><ERRLINE>channel C1: looking for AUTOBACKUP on day: 20200424
</ERRLINE><ERRLINE>channel C1: AUTOBACKUP found: c-3490470762-20200424-02
</ERRLINE><ERRLINE>channel C1: restoring spfile from AUTOBACKUP c-3490470762-20200424-02
</ERRLINE><ERRLINE>released channel: C1
</ERRLINE><ERRLINE>RMAN-00571: ===========================================================
</ERRLINE><ERRLINE>RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
</ERRLINE><ERRLINE>RMAN-00571: ===========================================================
</ERRLINE><ERRLINE>RMAN-03002: failure of restore command at 04/24/2020 16:07:32
</ERRLINE><ERRLINE>ORA-19870: error while restoring backup piece c-3490470762-20200424-02
</ERRLINE><ERRLINE>ORA-19913: unable to decrypt backup
</ERRLINE><ERRLINE>ORA-28365: wallet is not open
</ERRLINE><ERRLINE>
</ERRLINE><ERRLINE>RMAN>
</ERRLINE><ERRLINE>
</ERRLINE><ERRLINE>Recovery Manager complete.
</ERRLINE></ERRLINES></ARG></ARGS></ERR_FILE>
PRGO-4027 : SPFILE creation failed.

Solution 1: If a source database is configured for a PASSWORD based wallet, resubmit the migration job, by adding the -tdekeystorepasswd option to the command, and for the prompt, specify the source database TDE keystore password value.

Solution 2: For Oracle Database 12c Release 1 and later, ensure that $ORACLE_HOME/network/admin/sqlnet.ora points to the correct location of the TDE wallet, as shown here. 

ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))

For Oracle Database 11g Release 2 (11.2.0.4) only, ensure that $ORACLE_HOME/network/admin/sqlnet.ora points to the correct location of the TDE wallet as shown below, and replace the variable $ORACLE_UNQNAME with the value obtained with the SQL statement SHOW PARAMETER DB_UNIQUE_NAME. 

ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))

For example:

SQL> show parameter db_unique_name
db_unique_name    string      oci112_region

ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/oci112_region)))

Parent topic: Restore Phase (ZDT_CLONE_TGT) Issues

1.6.3.3 Restore Database Fails With ORA-01565

Issue: During the execution of phase ZDT_CLONE_TGT, restore database fails with the following error. 

</ERRLINE><ERRLINE>With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP
</ERRLINE><ERRLINE>and Real Application Testing options
</ERRLINE><ERRLINE>
</ERRLINE><ERRLINE>CREATE PFILE='/tmp/zdm833428275/zdm/PFILE/zdm_tgt_mclone_nrt139.pfile' FROM SPFILE
</ERRLINE><ERRLINE>*
</ERRLINE><ERRLINE>ERROR at line 1:
</ERRLINE><ERRLINE>ORA-01565: error in identifying file '?/dbs/spfile@.ora'
</ERRLINE><ERRLINE>ORA-27037: unable to obtain file status
</ERRLINE><ERRLINE>Linux-x86_64 Error: 2: No such file or directory
</ERRLINE><ERRLINE>Additional information: 3
</ERRLINE><ERRLINE>
</ERRLINE><ERRLINE>
</ERRLINE><ERRLINE>Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
</ERRLINE><ERRLINE>With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP

Solution: Start the target database using an SPFILE and resume the migration job.

Parent topic: Restore Phase (ZDT_CLONE_TGT) Issues

1.6.4 Transparent Data Encryption Related Issues

Parent topic: Known Issues

1.6.4.1 Transparent Data Encryption General Information

Depending on your source database release, Transparent Data Encryption (TDE) wallet configuration may be required.

  • Oracle Database 12c Release 2 and later

    For Oracle Database 12c Release 2 and later releases, TDE wallet configuration is mandatory and must be enabled on the source database before migration begins.

    If TDE is not enabled, the database migration will fail.

    Upon restore, the database tablespaces are encrypted using the wallet.

  • Oracle Database 12c Release 1 and earlier

    On Oracle Database 12c Release 1 and Oracle Database 11g Release 2 (11.2.0.4), TDE configuration is not required.

For information about the behavior of TDE in an Oracle Cloud environment, see My Oracle Support document Oracle Database Tablespace Encryption Behavior in Oracle Cloud (Doc ID 2359020.1).

Parent topic: Transparent Data Encryption Related Issues

1.6.4.2 Job Fails in Phase ZDM_SETUP_TDE_TGT

Issue: The phase ZDM_SETUP_TDE_TGT fails with one of the following errors. 

Executing phase ZDM_SETUP_TDE_TGT
Setting up Oracle Transparent Data Encryption (TDE) keystore on the target node oci1121 ...
oci1121: <ERR_FILE><Facility>PRGZ</Facility><ID>ZDM_KEYSTORE_NOT_SETUP_ERR</ID><ARGS><ARG>oci112_phx1z3</ARG></ARGS></ERR_FILE>
PRGO-3007 : failed to migrate database "db11204" with zero downtime
PRCZ-4002 : failed to execute command "/u01/app/18.0.0.0/grid/perl/bin/perl" using the privileged execution plugin "zdmauth" on nodes "oci1121"
PRCZ-2103 : Failed to execute command "/u01/app/18.0.0.0/grid/perl/bin/perl" on node "oci1121" as user "root". Detailed error:
<ERR_FILE><Facility>PRGZ</Facility><ID>ZDM_KEYSTORE_NOT_SETUP_ERR</ID><ARGS><ARG>oci112_phx1z3</ARG></ARGS></ERR_FILE>

Error at target server in /tmp/zdm749527725/zdm/log/mZDM_oss_standby_setup_tde_tgt_71939.log
2019-06-13 10:00:20: Keystore location /opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME does not exists for database 'oci112_region'
2019-06-13 10:00:20: Reporting error:
<ERR_FILE><Facility>PRGZ</Facility><ID>ZDM_KEYSTORE_NOT_SETUP_ERR</ID><ARGS><ARG>oci112_region</ARG></ARGS></ERR_FILE>

Solution:

  • Oracle Database 12c Release 1 and later

    On the target database, make sure that $ORACLE_HOME/network/admin/sqlnet.ora points to the correct location of the TDE wallet. For exmaple: 

    ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)
  • Oracle Database 11g Release 2 (11.2.0.4) only

    On the target database, make sure that $ORACLE_HOME/network/admin/sqlnet.ora points to the correct location of the TDE wallet, and replace the $ORACLE_UNQNAME variable with the value obtained from the SHOW PARAMETER DB_UNIQUE_NAME SQL command.

    For example, run

    SQL> show parameter db_unique_name
db_unique_name         string      oci112_region

    and replace

    ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))

    with

    ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/oci112_region)))

Parent topic: Transparent Data Encryption Related Issues

1.6.5 Post Migration Automatic Backup Issues

Parent topic: Known Issues

1.6.5.1 Troubleshooting Post Migration Automatic Backup Failures

Issue: Post migration, on the target database, Automatic Backup might fail.

You can verify the failure using the console in Bare Metal, VM and Exadata > DB Systems > DB System Details > Database Details > Backups.

Solution: Get the RMAN configuration settings from one of the following places.

  • Zero Downtime Migration documentation in Target Database Prerequisites, if captured
  • The log files at /opt/oracle/dcs/log/hostname/rman/bkup/db_unique_name/
  • /tmp/zdmXXX/zdm/zdm_TDBNAME_rman.dat

For example, using the second option, you can get the RMAN configuration settings from /opt/oracle/dcs/log/ocidb1/rman/bkup/ocidb1_abc127/rman_configure*.log, then reset any changed RMAN configuration settings for the target database to ensure that automatic backup works without any issues.

If this workaround does not help, then debug further by getting the RMAN job ID by running the DBCLI command, list-jobs, and describe the job details for more error details by running the DBCLI command describe-job -i JOB ID from the database server as the root user.

For example, during the test, the following highlighted settings were modified to make Automatic Backup work.

rman target /
Recovery Manager: Release 12.2.0.1.0 - Production on Mon Jul 8 11:00:18 2019
Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.
connected to target database: ORCL (DBID=1540292788)
RMAN> show all;
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name OCIDB1_ABC127 are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF;
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO '%F'; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4 BACKUP TYPE TO COMPRESSED BACKUPSET;
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 2 G;
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' MAXPIECESIZE 2 G FORMAT '%d_%I_%U_%T_%t' PARMS
 'SBT_LIBRARY=/opt/oracle/dcs/commonstore/pkgrepos/oss/odbcs/libopc.so ENV=(OPC_PFILE=/opt/oracle/dcs/commonstore/objectstore/opc_pfile/1245080042/opc_OCIDB1_ABC127.ora)';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE ON;
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'MEDIUM' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE;
CONFIGURE RMAN OUTPUT TO KEEP FOR 7 DAYS; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE';
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+RECO/ OCIDB1_ABC127/controlfile/snapcf_ocidb1_abc127.f';
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK clear;
RMAN>

Parent topic: Post Migration Automatic Backup Issues

1.6.5.2 Post Migration Automatic Backup Fails With DCS-10045

Issue: Post migration, Automatic Backup fails with the following error for non-TDE enabled migrated Oracle Database releases 11.2.0.4 and 12.1.0.2. 

DCS-10045: Validation error encountered: Backup password is mandatory to take OSS backup for non-tde enabled database...

You can verify this error by getting the RMAN job ID by running DBCLI command list-jobs, and describe the job details to get the error details by running DBCLI command describe-job -i JOB ID from the database server as the root user.

Solution:

  1. Find the TDE wallet location.

    The Oracle Cloud Infrastructure provisioned database instance will have following entry in sqlnet.ora. 

    ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))
  2. Remove the cwallet.sso file from the wallet location.

    For example, /opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME.

  3. For Oracle Database 11g Release 2, do the folowing steps.
    1. Connect to database using SQL*Plus as sysdba and verify the current wallet location.
      SQL> select * from v$encryption_wallet;
WRL_TYPE    WRL_PARAMETER                                            STATUS
file        /opt/oracle/dcs/commonstore/wallets/tde/ocise112_region  OPEN
    2. Close the wallet in the database.
      SQL> alter system set wallet close;
    3. Open the wallet using the wallet password.
      SQL> alter system SET WALLET open IDENTIFIED BY "walletpassword"
    4. Set the master encryption key.
      SQL> alter system set encryption key identified by "walletpassword"
    5. Recreate the autologin SSO file.
      /home/oracle>orapki wallet create -wallet /opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME -auto_login
Oracle PKI Tool : Version 11.2.0.4.0 - Production
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
Enter wallet password:            #
    6. Retry Automatic Backup.
  4. For Oracle Database 12c, do the folowing steps.
    1. Connect to database using SQL*Plus as sysdba and verify the current wallet location and status.
      SQL> SELECT wrl_parameter, status, wallet_type FROM v$encryption_wallet;
WRL_PARAMETER                                            STATUS              WALLET_TYPE
/opt/oracle/dcs/commonstore/wallets/tde/ocise112_region  OPEN_NO_MASTER_KEY  OPEN

      If the STATUS column contains a value of OPEN_NO_MASTER_KEY, you must create and activate the master encryption key.

    2. Close the wallet in the database.
      SQL> alter system set wallet close;
    3. Open the wallet-using password.
      SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE open IDENTIFIED BY "walletpassword" CONTAINER=all;
    4. Set the master encryption key.
      SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY "walletpassword" with backup;

      Log in to each PDB and run

      SQL> ALTER SESSION SET CONTAINER = PDB_NAME;
SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY "walletpassword" with backup;
    5. Create the auto login keystore.
      SQL> ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE FROM KEYSTORE 'path to wallet directory' IDENTIFIED BY "walletpassword";
    6. Retry Automatic Backup.

Parent topic: Post Migration Automatic Backup Issues

1.6.5.3 Post Migration Automatic Backup Fails With DCS-10096

Issue: Post migration, Automatic Backup fails with the following error. 

DCS-10096:RMAN configuration 'Retention policy' must be configured as 'configure retentio n
      policy to recovery window of 30 days'

You can verify this error by getting the RMAN job ID by running DBCLI command list-jobs, and describe the job details for more error details by running DBCLI command describe-job -i JOB ID from the database server as the root user.

Solution: Log in into RMAN prompt and configure the retention policy. 

[oracle@racoci1 ~]$ rman target /
Recovery Manager: Release 12.2.0.1.0 - Production on Wed Jul 17 11:04:35 2019
Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.
connected to target database: SIODA (DBID=2489657199)
RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;

old RMAN configuration parameters:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;

new RMAN configuration parameters:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;

new RMAN configuration parameters are successfully stored

Retry Automatic Backup.

Parent topic: Post Migration Automatic Backup Issues

1.6.6 Miscellaneous Issues

Parent topic: Known Issues

1.6.6.1 Unable to Rerun MIGRATE DATABASE Command

Issue: Zero Downtime Migration blocks attempts to rerun the MIGRATE DATABASE command for a specified database if that database is already part of an ongoing migration job.

Workaround: If you want to resubmit a database migration, you can stop the ongoing migration job in either EXECUTING or PAUSED state using the ZDMCLI ABORT JOB command as follows.
-bash-4.2$ ./zdmcli abort job -jobid 70
server.example.com: Audit ID: 189

Parent topic: Miscellaneous Issues

1.6.6.2 Oracle RAC Migration Job Fails at ZDM_PREPARE_TGT

Issue: An Oracle RAC migration job fails at phase ZDM_PREPARE_TGT with the following error. 

Executing phase ZDM_PREPARE_TGT
Setting up standby on the target node oci1121 ...
oci1121: 2019-06-13 09:54:20: Copy '/u01/app/oracle/admin/oci112_region' to remote node 'oci1122' failed
oci1121: 2019-06-13 09:54:27: Copy '/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/orapwoci1121' to remote node 'oci1122:/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/orapwoci1122' failed
PRCZ-4002 : failed to execute command "/u01/app/18.0.0.0/grid/perl/bin/perl" using the privileged execution plugin "zdmauth" on nodes "oci1121"
PRCZ-2103 : Failed to execute command "/u01/app/18.0.0.0/grid/perl/bin/perl" on node "oci1121" as user "root". Detailed error:
2019-06-13 09:54:20: Copy '/u01/app/oracle/admin/oci112_phx1z3' to remote node 'oci1122' failed
2019-06-13 09:54:27: Copy '/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/orapwoci1121' to remote node 'oci1122:/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/orapwoci1122' failed

Solution: You must set up SSH connectivity without a passphrase between the target Oracle RAC servers for the oracle user.

Parent topic: Miscellaneous Issues

1.6.6.3 INS-42505 Warning Shown During Installation

Issue: The following warning is shown during installation.
/stage/user/ZDM_KIT_relnumber>./zdminstall.sh setup
oraclehome=/stage/user/grid oraclebase=/stage/user/base
ziploc=/stage/user/ZDM_KIT_relnumber/rhp_home.zip -zdm
---------------------------------------
Unzipping shiphome to gridhome
---------------------------------------
Unzipping shiphome...
Shiphome unzipped successfully..
---------------------------------------
##### Starting GridHome Software Only Installation #####
---------------------------------------
Launching Oracle Grid Infrastructure Setup Wizard...

[WARNING] [INS-42505] The installer has detected that the Oracle Grid
Infrastructure home software at (/stage/user/grid) is not complete.
   CAUSE: Following files are missing:
...

Solution: This warning message can be ignored. It does not affect the installation or cause any issues for migration.

Parent topic: Miscellaneous Issues

1.6.6.4 Migration Evaluation Failure with Java Exception Invalid Key Format

Issue: The following conditions are seen:

  • Zero Downtime Migration migration -eval command fails with the following error. 

    Result file path contents:
"/u01/app/zdmbase/chkbase/scheduled/job-19-2019-12-02-03:46:19.log"
zdm-server.ocitoolingsn.ocitooling.oraclevcn.com: Processing response
file ...
null

  • The file $ZDM_BASE/<zdm service host>/rhp/rhpserver.log.0 contains the following entry. 

    Verify below error message observed in file $ZDM_BASE/<zdm service
host>/rhp/rhpserver.log.0
rhpserver.log.7:[pool-58-thread-1] [ 2019-12-02 02:08:15.178 GMT ]
[JSChChannel.getKeyPair:1603]  Exception :
java.security.spec.InvalidKeySpecException:
java.security.InvalidKeyException: invalid key format

  • The Zero Downtime Migration installed user (For example: zdmuser) private key (id_rsa) file has the following entries.

    -----BEGIN OPENSSH PRIVATE KEY----------
MIIEogIBAAKCAQEAuPcjftR6vC98fAbU4FhYVKPqc0CSgibtMSouo1DtQ06ROPN0
XpIEL4r8nGp+c5GSDONyhf0hiltBzg0fyqyurSw3XfGJq2Q6EQ61aL95Rt9CZh6b
JSUwc69T4rHjvRnK824k4UpfUIqafOXb2mRgGVUkldo4yy+pLoGq1GwbsIYbS4tk
uaYPKZ3A3H9ZA7MtZ5M0sNqnk/4Qy0d8VONWozxOLFC2A8zbbe7GdQw9khVqDb/x
END OPENSSH PRIVATE KEY-----

Solution: Authentication key pairs (private and public key) are not generated using the ssh-keygen utility, so you must generate authentication key pairs using steps in Generating a Private SSH Key Without a Passphrase.

After generating authentication key pairs, the private key file content looks like the following.

-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAuPcjftR6vC98fAbU4FhYVKPqc0CSgibtMSouo1DtQ06ROPN0
XpIEL4r8nGp+c5GSDONyhf0hiltBzg0fyqyurSw3XfGJq2Q6EQ61aL95Rt9CZh6b
JSUwc69T4rHjvRnK824k4UpfUIqafOXb2mRgGVUkldo4yy+pLoGq1GwbsIYbS4tk
uaYPKZ3A3H9ZA7MtZ5M0sNqnk/4Qy0d8VONWozxOLFC2A8zbbe7GdQw9khVqDb/x
-----END RSA PRIVATE KEY-----

Set up connectivity with the newly generated authentication key pairs and resume the migration job.

Parent topic: Miscellaneous Issues

1.6.6.5 Migration Evaluation Fails with Error PRCG-1022

Issue: The following conditions are seen: 

$ZDM_HOME/bin/zdmcli migrate database -sourcedb zdmsdb -sourcenode ocicdb1 
-srcauth zdmauth -srcarg1 user:opc 
-srcarg2 identity_file:/home/zdmuser/.ssh/zdm_service_host.ppk 
-srcarg3 sudo_location:/usr/bin/sudo -targetnode ocidb1 -backupuser backup_user@example.com 
-rsp /u01/app/zdmhome/rhp/zdm/template/zdm_template_zdmsdb.rsp -tgtauth zdmauth 
-tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/zdm_service_host.ppk 
-tgtarg3 sudo_location:/usr/bin/sudo -eval

PRCG-1238 : failed to execute the Rapid Home Provisioning action for command  'migrate database'
PRCG-1022 : failed to connect to the Rapid Home Provisioning daemon for cluster anandutest
Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException
[Root exception is java.rmi.ConnectException: Connection refused to host:
anandutest; nested exception is: java.net.ConnectException: Connection refused (Connection refused)]

Solution: Start the Zero Downtime Migration service using the $ZDM_HOME/bin/zdmservice START command, then run any ZDMCLI commands.

Parent topic: Miscellaneous Issues

1.6.6.6 Unable to Resume a Migration Job

Issue: Zero Downtime Migration writes the source and target log files to the /tmp/zdm-unique id directory in the respective source and target database servers.

If you pause a migration job and and then resume the job after several (sometimes 15-20 days), the /tmp/zdm-unique id directory might be deleted or purged as part of a clean up or server reboot that also cleans up /tmp.

Solution: After pausing a migration job, back up the /tmp/zdm-unique id directory. Before resuming the migration job, check the /tmp directory for /zdm-unique id, and if it is missing, restore the directory and its contents with your backup.

Parent topic: Miscellaneous Issues

1.6.6.7 Migration Job Fails at ZDM_SWITCHOVER_SRC

Issue: A migration job fails at ZDM_SWITCHOVER_SRC phase.

Solutions:

  1. Ensure that there is connectivity from PRIMARY database nodes to STANDBY database nodes so the redo log are shipped as expected.

  2. A job will fail at ZDM_SWITCHOVER_SRC if the recovery process (MRP0) is not running at the target. The recovery process reason for failure should be corrected if MRP0 is not running at Oracle Cloud Database Standby Instance, and then the process should be started manually at Oracle Cloud Database Standby Instance before the migration job can be resumed.

Parent topic: Miscellaneous Issues

1.7 Additional Information for Migrating to Oracle Cloud Infrastructure

Read the following for general information, considerations, and links to more information about using Zero Downtime Migration to migrate your database to Oracle Cloud Infrastructure.

Parent topic: Zero Downtime Migration Release Notes

1.8 Additional Information for Migrating to Exadata Cloud Service

Read the following for general information, considerations, and links to more information about using Zero Downtime Migration to migrate your database to Exadata Cloud Service.

Parent topic: Zero Downtime Migration Release Notes

1.8.1 Considerations for Migrating to Exadata Cloud Service

For this release of Zero Downtime Migration be aware of the following considerations.

  • If the source database is release 18c, then the target home should be at release 18.6 or later to avoid issues such as Bug 29445548 Opening Database In Cloud Environment Fails With ORA-600.
  • PDB conversion related phases are listed in -listphases and can be ignored. Those are no-op phases.
  • Non-CDB to PDB related input parameters in the response file are place holders that should not be set. Setting NONCDBTOPDB_* inputs to true will break the migration.
  • If a backup was performed when one of the configured instances is down, you will encounter Bug 29863717 - DUPLICATING SOURCE DATABASE FAILED BECAUSE INSTANCE 1 WAS DOWN.
  • The TDE keystore password must be set in the credential wallet. To set the password as part of the Zero Downtime Migration workflow, specify the -tdekeystorepasswd argument irrespective of whether the wallet uses AUTOLOGIN or PASSWORD. In either case the password is stored in the credential wallet. If the -tdekeystorepasswd argument is not supplied, then Zero Downtime Migration skips the setting tde_ks_passwd key in the credential wallet, and no error is thrown.
  • The target environment must be installed with latest DBaaS Tooling RPM with db_unique_name change support to be installed.
  • Provision a target database from the console without enabling auto-backups. In the Configure database backups section do not select the Enable automatic backups option.

Parent topic: Additional Information for Migrating to Exadata Cloud Service

1.8.2 Exadata Cloud Service Database Registration

Post migration, register the Exadata Cloud Service database, and make sure its meets all of the requirements.

Run the following commands on the Exadata Cloud Service database server as the root user.

/root>dbaascli registerdb prereqs --dbname db_name --db_unique_name db_unique_name

/root>dbaascli registerdb begin  --dbname db_name --db_unique_name db_unique_name

For example

/root>dbaascli registerdb prereqs --dbname ZDM122 --db_unique_name ZDM122_phx16n
DBAAS CLI version 18.2.3.2.0
Executing command registerdb prereqs --db_unique_name ZDM122_phx16n
INFO: Logfile Location: /var/opt/oracle/log/ZDM122/registerdb/registerdb_2019-08-14_05:35:31.157978280334.log
INFO: Prereqs completed successfully
/root>

/root>dbaascli registerdb begin --dbname ZDM122 --db_unique_name ZDM122_phx16n
DBAAS CLI version 18.2.3.2.0
Executing command registerdb begin --db_unique_name ZDM122_phx16n
Logfile Location: /var/opt/oracle/log/ZDM122/registerdb/registerdb_2019-08-14_05:45:27.264851309165.log
Running prereqs
DBAAS CLI version 18.2.3.2.0
Executing command registerdb prereqs --db_unique_name ZDM122_phx16n
INFO: Logfile Location: /var/opt/oracle/log/ZDM122/registerdb/registerdb_2019-08-14_05:45:29.000432309894.log
INFO: Prereqs completed successfully
Prereqs completed
Running OCDE .. will take time ..
OCDE Completed successfully.
INFO: Database ZDM122 registered as Cloud database
/root>

Parent topic: Additional Information for Migrating to Exadata Cloud Service

1.8.3 Exadata Cloud Service Automatic Backup Issues

Check the backup configuration before you enable automatic backup from the console. You can use the get config command as shown in the first step below. You should see bkup_oss=no before you enable automatic backup.

You might see the error message in the console, "A backup configuration exists for this database. You must remove the existing configuration to use Oracle Cloud Infrastructure's managed backup feature."

To fix this error, remove the existing configuration.

First, make sure the automatic backup is disabled from the UI, then follow these steps to remove the existing backup configuration.

  1. Generate a backup configuration file.
    /var/opt/oracle/bkup_api/bkup_api get config --file=/tmp/db_name.bk --dbname=db_name

    For example:

    /var/opt/oracle/bkup_api/bkup_api get config --file=/tmp/zdmdb.bk --dbname=zdmdb
  2. Open the /tmp/db_name.bk file you created in the previous step.

    For example: Open /tmp/zdmdb.bk

    change bkup_oss=yes from bkup_oss=no

  3. Disable OSS backup by setting bkup_oss=no.
    /var/opt/oracle/bkup_api/bkup_api set config --file=/tmp/db_name.bk --dbname=db_name

    For example:

    /var/opt/oracle/bkup_api/bkup_api set config --file=/tmp/zdmdb.bk --dbname=zdmdb
  4. Check reconfigure status.
    /var/opt/oracle/bkup_api/bkup_api configure_status --dbname=db_name

    For example:

    /var/opt/oracle/bkup_api/bkup_api configure_status --dbname=zdmdb

Now enable automatic backup from console.

Verify the backups from the console. Click Create Backup to create a manual backup, and a backup should be created without any issues. and also Automatic Backup should be successful.

Parent topic: Additional Information for Migrating to Exadata Cloud Service

1.9 Additional Information for Migrating to Exadata Cloud at Customer

Read the following for general information, considerations, and links to more information about using Zero Downtime Migration to migrate your database to Exadata Cloud at Customer.

Parent topic: Zero Downtime Migration Release Notes

1.9.1 Considerations for Migrating to Exadata Cloud at Customer

For this release of Zero Downtime Migration be aware of the following considerations.

  • You must apply the regDB patch for Bug 29715950 - "modify regdb to handle db_unique_name not same as db_name" on all Exadata Cloud at Customer nodes. This is required for the ZDM_MANIFEST_TO_CLOUD phase. Please note that the regDB tool is part of DBaaS Tooling.
  • If the source database is release 18c, then the target home should be at release 18.6 or later to avoid issues such as Bug 29445548 Opening Database In Cloud Environment Fails With ORA-600.
  • PDB conversion related phases are listed in -listphases and can be ignored. Those are no-op phases.
  • Non-CDB to PDB related input parameters in the response file are place holders that should not be set. Setting NONCDBTOPDB_* inputs to true will break the migration.
  • If the backup medium is Zero Data Loss Recovery Appliance, then all configured instances should be up at the source when a FULL or INCREMENTAL backup is performed.
  • If a backup was performed when one of the configured instances is down, you will encounter Bug 29863717 - DUPLICATING SOURCE DATABASE FAILED BECAUSE INSTANCE 1 WAS DOWN.
  • The TDE keystore password must be set in the credential wallet. To set the password as part of the Zero Downtime Migration workflow, specify the -tdekeystorepasswd argument irrespective of whether the wallet uses AUTOLOGIN or PASSWORD. In either case the password is stored in the credential wallet. If the -tdekeystorepasswd argument is not supplied, then Zero Downtime Migration skips the setting tde_ks_passwd key in the credential wallet, and no error is thrown.
  • The target environment must be installed with latest DBaaS Tooling RPM with db_unique_name change support to be installed.

Parent topic: Additional Information for Migrating to Exadata Cloud at Customer

1.10 Documentation Accessibility

For information about Oracle's commitment to accessibility, visit the Oracle Accessibility Program website at http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc.

Access to Oracle Support

Oracle customers that have purchased support have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.

Parent topic: Zero Downtime Migration Release Notes