4 High Availability, Backup and Restore Operations

Oracle Key Vault may be configured for high availability and automatic backups for continuous, reliable, and protected access to security objects with minimum downtime.

• Why High Availability?
  
• About High Availability for Oracle Key Vault
  
• Backup and Restore Oracle Key Vault Data
  

4.1 Why High Availability?

With data centers geographically dispersed around the globe there is an increased need for data to be reliably accessible, on-demand, at any time. Users carrying out business-critical operations need data to be accessible and recoverable with minimum downtime. These requirements are met in a high availability deployment.

You achieve high availability by adding redundancy in the form of a standby server, that can take over the functions of the primary server in case of failure. The standby server helps you eliminate single points of failure and reduce server downtime, two main reasons to deploy Oracle Key Vault in a high availability configuration.

4.2 About High Availability for Oracle Key Vault

A high availability Oracle Key Vault deployment consists of two Key Vault peer servers called primary and standby. The primary is the active server servicing endpoint requests. The standby takes over if the primary fails.

• How High Availability for Oracle Key Vault Works
  
• Configuring High Availability
  
• Switching Primary and Standby Servers
  
• Restoring High Availability After a Failover
  
• Disabling High Availability
  
• High Availability Read-Only Restricted Mode
  

4.2.1 How High Availability for Oracle Key Vault Works

You configure high availability by providing the primary and standby servers with each other's IP address and certificate, then pairing them. While pairing the primary and standby servers, you can select one as the primary server, and the other as the standby. A failover timeout that you set, determines when the standby starts to take over as primary.

Prerequisites:

It is highly recommended to keep the primary and standby systems as identical as possible, as their roles can be reversed in maintenance periods and failure situations. These include the following:

• Key Vault software versions

• Disk size

• RAM size

• System clocks on both systems must be synchronized

If your deployment requires high availability, we recommend configuring it before adding endpoints to Key Vault so that endpoints know about both primary and standby servers. An endpoint added before the standby server will not know about the standby server unless you re-enroll the endpoint. If you configure high availability after adding endpoints, you must re-enroll the endpoints that were previously enrolled with the primary and standby servers in standalone mode.

If you want to add SNMP support in a high availability environment, you must configure SNMP on both primary and standby servers before pairing them. This is because the standby server is no longer accessible from the management console as all requests are forwarded to the primary server.

If you want to use a third-party certificate in a high availability configuration, you must install it on the primary and standby servers first, and then pair them.

With persistent cache enabled, both the primary and the standby will cache the master keys from Oracle Key Vault independently. Ensure that a TDE operation is executed on primary and standby after they are started. The persistent cache feature also enables endpoints to be operational during high availability configuration, and switchover and failover operations.

If enabled, the Read-Only Restricted mode of Oracle Key Vault 12.2.0.5.0 and higher ensures endpoint operational continuity if either primary or standby server is not available. Read-Only Restricted mode also ensures uniform behavior of the surviving Oracle Key Vault server in the event that either the primary or standby servers go down.

A High Availability configuration is characterized by continuous synchronization between the primary and standby server. When synchronization is lost between the primary and standby servers, it is possible to encounter a split-brain scenario where two primary servers might be active simultaneously. In such a scenario, both servers record new data that diverges from the last synchronized state. When connectivity is restored between the primary and standby servers, it may not be possible to reconcile the changes on the two servers, and data loss may occur.

You can enable or disable restricted mode when configuring High Availability by setting the Allow Read-Only Restricted Mode radio button to Yes or No on the Configure High Availability page.

When Read-Only Restricted mode is enabled, the primary server enters Read-Only Restricted mode if the standby server is unavailable. In Read-Only Restricted mode, the primary server will allow keys to be retrieved, but will not allow keys to be modified or new keys to be added. This ensures that endpoints still have access to their keys, and key data or metadata is not lost due to a split-brain scenario. However, the primary server still writes audit records, which may be lost if a split-brain scenario occurs with the standby server.

When Read-Only Restricted mode is disabled, the primary server becomes unavailable, and stops accepting new requests if the standby server is unavailable. Endpoints connected to Oracle Key Vault will be unable to retrieve keys from the server until connectivity is restored between primary and standby servers. The Persistent Master Key Cache feature can be used to avoid endpoint downtime. Data integrity is ensured by allowing endpoints to communicate with one primary server at any given time. Split-brain situations, and the risk of data loss associated with such situations, are avoided.

4.2.2 Configuring High Availability

To configure high availability you must be a user with System Administrator privileges. You must access the primary and standby appliance separately from two browser instances (separate tabs in the same browser or different browsers), and copy the IP address and certificate from one appliance to the other, and vice versa. With persistent cache enabled, endpoints will continue to be operational while high availability is configured.

To configure High Availability:

1. Open a web browser and enter the IP address of the designated primary server. The Oracle Key Vault Management Console login screen is displayed.
2. Log in as the System Administrator.
3. Click the System tab, then click High Availability in the left pane. The Configure High Availability page is displayed.

   Figure 4-1 Configure High Availability Page

   
   Description of "Figure 4-1 Configure High Availability Page"

   The following are the fields on the Configure High Availability page:

   • Current status: Displays the IP address and status of the current server.

   • Fast Start Failover Threshold (in secs): Displays the duration (in seconds) that will elapse before the server takes over from a failed peer server. The default is 60 seconds.

     To avoid failover during brief or intermittent failures, increase the duration.

   • Configure this server as: Displays whether the server is configured as a Primary server or Standby server.

   • Allow Read-Only Restricted Mode: Displays the status of Read-Only Restricted mode. The default is Yes.

     When enabled, Read-Only Restricted mode ensures operational continuity of the endpoints if the primary or standby Oracle Key Vault server is affected by server, hardware, or network failures

   • Current Server Certificate: Displays the server certificate.

4. Copy the following information, and store it in a text file named primary.txt. You will require this information when you configure the standby server:

   • From the Current status field, copy the IP address and paste it in primary.txt.

   • From the Current Server Certificate field, copy the server certificate and paste it in primary.txt.

   Save primary.txt.

5. Open a web browser and enter the IP address of the designated standby server. The Oracle Key Vault Management Console login screen is displayed.
6. Log in as the System Administrator.
7. Click the System tab, then click High Availability in the left pane. The Configure High Availability page is displayed.
8. Copy the following information, and store it in a text file named standby.txt. You will require this information when you configure the primary server:

   • From the Current status field, copy the IP address and paste it in standby.txt.

   • From the Current Server Certificate field, copy the server certificate and paste it in standby.txt.

   Save standby.txt.

9. In the Configure this server as field, select Standby server. The Primary server IP address and Primary server certificate fields are displayed.

   Figure 4-2 Configure High Availability Page (Standby Server)

   
   Description of "Figure 4-2 Configure High Availability Page (Standby Server)"

   Ensure that Yes is selected in the Allow Read-Only Restricted Mode field.

   Note:

   Do not disable Read Only Restricted mode unless necessary. If High Availability is set up with Read Only Restricted mode disabled, you must enable it by reinstalling and configuring Oracle Key Vault again.
10. Copy the following information from primary.txt, and paste it in the Configure High Availability page of the standby server:

    • Copy the IP address and paste it in the Primary server IP address field.

    • Copy the server certificate and paste it in the Primary server certificate field.

11. Click Save. The Settings Saved page is displayed.

    Figure 4-3 Settings Saved Page

    
    Description of "Figure 4-3 Settings Saved Page"

    The Reset button allows you to delete the High Availability configuration, if required.

    High Availability is enabled on the designated standby server. The next step is to enable High Availability on the designated primary server.
12. On the Settings Saved page, click the IP address of the primary server displayed at the top of the page. The Oracle Key Vault Management Console login screen of the primary server is displayed.
13. Log in as the System Administrator.
14. Click the System tab, then click High Availability in the left pane. The Configure High Availability page is displayed.
15. In the Configure this server as field, select Primary server. The Standby server IP address and Standby server certificate fields are displayed.

    Ensure that Yes is selected in the Allow Read-Only Restricted Mode field.

    Note:

    Do not disable Read Only Restricted mode unless necessary. If High Availability is set up with Read Only Restricted mode disabled, you must enable it by reinstalling and configuring Oracle Key Vault again.
16. Copy the following information from standby.txt, and paste it in the Configure High Availability page of the primary server:

    • Copy the IP address and paste it in the Standby server IP address field.

    • Copy the server certificate and paste it in the Standby server certificate field.

    Figure 4-4 Configure High Availability Page (Primary Server)

    
    Description of "Figure 4-4 Configure High Availability Page (Primary Server)"
17. Click Initiate Pairing.
    High Availability is enabled on the designated primary server.
18. In the confirmation message that is displayed, click OK. The Operation in Progress page is displayed.

    Caution:

    Allow at least 10 minutes to elapse before performing the next operation.
19. After at least 10 minutes have elapsed, click Refresh.
    If the pairing of primary and standby servers is successful, the current session is terminated. The Oracle Key Vault Management Console login screen of the primary server is displayed.
20. Log in as the System Administrator.
21. Click the System tab, then click High Availability in the left pane. The High Availability Status page is displayed.

    Figure 4-5 High Availability Status Page

    
    Description of "Figure 4-5 High Availability Status Page"

    The Unpair button allows you to disconnect the primary server from the standby server, if required.

    The Switch Roles button allows you to switch the roles of the primary server and the standby server, if required. The primary server then assumes the role of the standby server, while the standby server assumes the role of the new primary server.

High Availability is successfully configured.

Note:

When High Availability is configured, you cannot log in to the standby server using a web browser.

To manage High Availability, log in to the primary server using a web browser.

Caution:

Ensure that you leave Read-Only Restricted Mode enabled while configuring High Availability. Enabling it later requires a reinstall of the Oracle Key Vault appliance software on the standby server.

After configuring High Availability, do not change the system time on the primary server. The changed system time causes the standby server to go down, thus disrupting the functioning of the High Availability configuration.

4.2.3 Switching Primary and Standby Servers

The high availability configuration allows you to switch roles of the primary and standby server. This is useful during maintenance periods, when you might want to bring a server down to upgrade software or install patches.

If you have persistent cache enabled and the persistent cache timeout is sufficiently tuned, then the endpoints will continue to be operational during the switchover, minimizing endpoint downtime.

You can switch roles as follows:

1. Log in to the Oracle Key Vault management console of the primary node as a user with the System Administrator role.

   Before switching the primary and standby servers, ensure that there are no HA-related alerts on the Alerts page. To access the Alerts page, click the Reports tab, and then click Alerts in the left pane.

   Ensure that all HA-related alerts on the Alerts page are addressed before switching the primary and standby servers.

2. Click the System tab, then High Availability from the left side bar.

   The High Availability Status page appears.

3. Click Switch Roles on the top right.

   The Switch Roles button allows you to switch the roles of the primary server and the standby server. The primary server then assumes the role of the standby server, while the standby server assumes the role of the new primary server.

   Select OK to the confirmation message.

   An operation initiated message is followed by the Operation in Progress page indicating that the switchover operation will take 10 minutes to complete successfully.

   Caution:

   You must wait for a minimum period of 10 minutes for the switchover operation to complete successfully. If you refresh the UI before the switchover operation is complete, an error message is displayed. The error message is displayed until the switchover is completed successfully.
4. Ensure that at least 10 minutes have elapsed, and only then, click Refresh.

   This will log you out of the current session, and open a login page to the switched primary server.

   Both the primary and standby are restarted, however, note that you will only be able to log in to the primary node's web console. The primary server is the active server, and all requests to the standby will be forwarded to the primary.

5. Log in to the primary server to see the IP address of the switched standby node.
6. Click the System tab, then High Availability from the left side bar.

   The High Availability Status page appears.

   The Standby server IP address field displays the IP address.

4.2.4 Restoring High Availability After a Failover

A failover takes place if the primary server fails. If the primary is not available, the standby server takes over the primary role. If the standby does not hear from the primary for a time exceeding the Fast Start Failover Threshold value, it will assume that the primary is down and start the failover process. You can configure the value in the Fast Start Failover Threshold field from the Key Vault user interface from the default of 60 seconds. Should the failed server (old primary) come back up, it will automatically become the new standby server, in most cases.

If the primary server fails permanently, the standby server will take over as primary. In this case high availability will have to be restored with manual intervention as follows:

1. Replace the failed server with a newly installed Oracle Key Vault appliance. Be sure to use the original IP address for the failed server.
2. Log on to the newly installed appliance and follow the steps to configure high availability. You may designate the new appliance as the standby (since the cluster has a functional primary) and pair it with the functioning primary.
3. If you want to restore the original configuration and set the new appliance as primary, you can use the Switch Roles option, after you successfully pair the two nodes and enable high availability.

   The Switch Roles button allows you to switch the roles of the primary server and the standby server. The primary server then assumes the role of the standby server, while the standby server assumes the role of the new primary server.

Note:

When Read-Only Restricted mode is disabled, the primary server's failover status goes into suspended state causing the standby server to wait indefinitely for the primary server to come back up. This is expected behavior to avoid a split-brain scenario where two primary servers are simultaneously active.

When Read-Only Restricted mode is enabled, a primary or standby server failure causes the operational peer to enter Read-Only Restricted mode, thus ensuring endpoint operational continuity.

See Also:

For more information about recovering from High Availability failover situations, see Failover Situations in High Availability Mode.

4.2.5 Disabling High Availability

You can disable high availability by un-pairing the primary and standby servers. After un-pairing, the primary and standby will operate in standalone mode. To prevent endpoints from connecting to the old standby (new standalone Oracle Key Vault server) you need to take the old standby off the network.

To disable high availability follow the steps below:

1. Log in to the primary server's management console as a user with System Administrator privileges.
2. Select the System tab on top, then select High Availability from the left side bar.

   The High Availability Status page appears with Unpair and Switch Roles on the top right. The Unpair and Switch Roles options do the following:

   • The Unpair button allows you to disconnect the primary server from the standby server, if required.

   • The Switch Roles button allows you to switch the roles of the primary server and the standby server, if required. The primary server then assumes the role of the standby server, while the standby server assumes the role of the new primary server.

3. Click Unpair.

   A brief message with a green check appears indicating that the operation has been successfully initiated.

   The Operation in Progress page appears indicating a wait time of at least 10 minutes for the un-pairing to complete.

   Wait 10 minutes.

4. Click the Refresh button to get logged out of the current session.
5. Log back in to the management console of the primary server. Select System, then High Availability from the left side bar.

   The Configure High Availability page appears.

   The Current status field shows the server in standalone mode.

   Caution:

   If you want to use the (now standalone) standby Key Vault server as a standby in a new HA deployment, you must reinstall the Oracle Key Vault software on the standby server.

4.2.6 High Availability Read-Only Restricted Mode

You can configure Oracle Key Vault for high availability restricted mode.

• About High Availability Read-Only Restricted Mode
  
• States of Read-Only Restricted Mode
  
• Recovering from Read-Only Restricted Mode
  
• Read-Only Restricted State Functionality
  
• Enabling Read-Only Restricted Mode
  
• Disabling Read-Only Restricted Mode
  
• Best Practices
  

4.2.6.1 About High Availability Read-Only Restricted Mode

High Availability Read-Only Restricted mode ensures endpoint operational continuity when primary or standby Oracle Key Vault servers are affected by server, hardware, or network failures.

• How High Availability Read-Only Restricted Mode Works
  
• High Availability without Read-Only Restricted Mode
  
• High Availability with Read-Only Restricted Mode
  

4.2.6.1.1 How High Availability Read-Only Restricted Mode Works

High Availability Read-Only Restricted mode is supported in Oracle Key Vault 12.2.0.5.0 and later.

In Oracle Key Vault 12.2.0.4.0, when an unplanned shutdown caused the standby server to go offline, the primary server was also unavailable to the endpoints. However, when a planned shutdown caused the standby server to go offline, the primary server was still available to the endpoints.

Oracle Key Vault 12.2.0.5.0 and later support High Availability Read-Only Restricted mode. High Availability Read-Only Restricted mode ensures endpoint operational continuity when the primary or standby Oracle Key Vault servers are affected by server, hardware, or network failures.

When an unplanned shutdown causes the primary or standby server to go offline, the endpoints can still connect to the surviving peer server in order to perform critical operations. High Availability Read-Only Restricted mode ensures that operations that replicate data are blocked. Operations that replicate data are allowed when both primary and standby servers are back online, thus ensuring that no critical data is lost.

In High Availability Oracle Key Vault deployments, the single point of failure is eliminated by replicating the primary server’s data to the standby server. In Read-Only Restricted mode, generation of non-critical data such as audit records is enabled. However, generation of critical data such as keys is disabled. When the primary server is down, operations that generate new critical data on the standby are disabled. The reverse is also true. When the standby server is down, operations that attempt to modify or create any data on the primary server are disabled.

In a High Availability deployment without Read-Only Restricted mode, most endpoint operations are blocked because endpoint operations generate audit records, which is data that needs replication, thus disrupting operational continuity.

The following are the benefits of using Read-Only Restricted mode:

• Allows endpoint operational continuity when the primary or standby server is offline

• Ensures symmetrical behavior when the primary or standby server is offline

4.2.6.1.2 High Availability without Read-Only Restricted Mode

When High Availability is configured without Read-Only Restricted mode, the impact on endpoint operations differs, depending on the type of failure encountered: primary failure, standby failure, or a network failure that prevents communication between the primary and standby servers. The following are the possible scenarios:

• Primary server failure: The standby server will failover and take over from the affected primary server. This allows the Oracle Key Vault service to remain operational. Data modifications are stored on the primary server until they can be replicated to the standby server. This ensures endpoint operational continuity when the primary server goes offline due to an unplanned shutdown.

• Standby server failure: The primary server is unavailable to the endpoints, as it is not possible to distinguish a standby server failure from a network failure that prevents communication between the primary and standby servers.

• Power loss or network connectivity failure: The primary and standby servers are unable to communicate. The standby server will failover and take over from the primary server. To avoid a split-brain scenario, only one of the servers is allowed to service the endpoints.

Note:

A split-brain scenario in Oracle Key Vault occurs when the primary server fails, causing the standby server to failover and take over from the primary server. This causes a situation where the primary and standby servers are available to service the endpoints, and create new data. A split-brain scenario causes data on the primary and standby servers to go out of sync. This can lead to data loss and corruption, as well as loss of operational continuity. To avoid a split-brain scenario, only one of the servers is allowed to service the endpoints after a failover occurs.

In High Availability without Read-Only Restricted mode, one of the following situations is triggered when a failure occurs:

• Endpoints suffer a temporary operational disruption to avoid a split-brain scenario.

• The standby server accepts new requests and generates new data without attempting to synchronize the data with the failed primary server. Replication of data is temporarily disabled until the primary server is online, thus ensuring operational continuity. 

4.2.6.1.3 High Availability with Read-Only Restricted Mode

Read-Only Restricted mode is the default High Availability mode in Oracle Key Vault 12.2.0.5.0 and later, and can be disabled during High Availability configuration, if required.

Note:

It is recommended that you configure High Availability with Read-Only Restricted mode enabled, which is the default mode. While configuring High Availability, ensure that Yes is selected in the Allow Read-Only Restricted Mode field on the Configure High Availability page.

Read-Only Restricted mode ensures endpoint operational continuity as well as symmetrical behavior when the primary or standby server is offline. Symmetrical behavior ensures that the online server seamlessly takes over from its failed peer, and continues to service the endpoints without any disruption. For more information about High Availability failover situations with Read-Only Restricted mode, see Failover Situations with Read-Only Restricted Mode.

In Read-Only Restricted mode, the surviving Oracle Key Vault server operates with limited functionality. Endpoint operations that add or modify critical data on the Oracle Key Vault server are blocked. However, endpoint operations that involve fetching of data are allowed. This ensures endpoint operational continuity and data integrity. For more information about blocked and allowed operations, see Read-Only Restricted State Functionality.

For more information about Read-Only Restricted state, see States of Read-Only Restricted Mode.

Note:

Read-Only Restricted Mode has no impact on a standalone server.

4.2.6.2 States of Read-Only Restricted Mode

A server using read-only restricted mode is affected by the failure in a primary server, a standby server, and the network.

4.2.6.3 Recovering from Read-Only Restricted Mode

To recover an instance from Read-Only Restricted mode after a network failure or standby server failure, manual intervention may be required. You will need to unpair and reset the surviving instance, reinstate a new Oracle Key Vault server, and pair it as the new standby to the surviving server. The following are the possible scenarios:

• Primary server failure: Depending on the operational state of the primary server at the time of failure, it could be restarted and some functionality may be available. However, due to possible corruption of the embedded key vault database, recovery may not be possible and the Oracle Key Vault instance would need to be reinstated because of the partial failure. If the failed server is unable to re-pair with the peer server within 20 minutes, the server needs to be re-instantiated.

  Even though the endpoint processes communicating with the Oracle Key Vault servers retain the IP address of the last known reachable server, they do have to determine the IP address of the new Key Vault server when spawned. The endpoint processes attempt to communicate with the Oracle Key Vault server configured as the primary server in the configuration scripts, and then waits for a response before trying to reach the server configured as the standby server in the configuration scripts. The wait for the response is an unnecessary delay that every new process incurs when communicating with the standby Oracle Key Vault server. To minimize downtime, it is recommended that you initiate a switchover after reinstating the failed primary server.

• Standby server failure: The primary server will run in the Read-Only Restricted mode if there is a standby server failure. Re-instate the standby server if it does not automatically re-pair with the primary server.

• Power loss or network connectivity failure: When a network failure occurs, the primary and standby servers are unable to communicate, and both servers enter Read-Only Restricted mode. The standby also attempts to failover to the primary server. Once communication is re-established between the primary and standby servers, the old primary server is automatically converted to the new standby. The data from the new primary server overwrites the old primary server’s data, resulting in the loss of audit records from the old primary server. It is recommended that you enable SYSLOG auditing to preserve the audit records that were overwritten on the old primary. Similar to recovering from primary server failure, it is recommended to do a switchover after recovery. It is also recommended that you do not enroll any new endpoints before the switchover.

For more information about restoring High Availability after a failover, see Restoring High Availability After a Failover.

4.2.6.4 Read-Only Restricted State Functionality

Read-Only Restricted mode puts the Oracle Key Vault instance into the Read-Only Restricted mode state, but does not put the embedded Oracle Key Vault database into the Read-Only Restricted mode state. Read-Only Restricted mode introduces the following deviations from normal functionality:

• All operations that generate new data are blocked. Operations that fetch existing data are allowed. Audit records for endpoint operations are generated as in normal operation. Internal system operations of the Oracle Key Vault database are not impacted. Functionality such as alerts continue to work normally.

• Endpoints are allowed to fetch keys from the Oracle Key Vault server. Endpoints cannot create new keys or modify existing keys.

• Administrators can log in to the Oracle Key Vault Management Console. Creation of an endpoint or a wallet, deletion of keys, and operations that modify or delete data are blocked.

• Unpairing of primary and standby Oracle Key Vault servers running in Read-Only Restricted mode are allowed.

• Backup operations are blocked to avoid data mismatches between backups.

Table 4-1 Allowed and Blocked Operations in Read-Only Restricted Mode

Operation	Allowed or Blocked
Log in to Oracle Key Vault	Allowed
Endpoint operations such as fetching keys from the cache	Allowed
Endpoint operations that add, modify, or delete data such as rotation of keys on the database	Blocked
System operations such as enabling SSH access	Allowed
System operations that write data such as setting up a REST server and creating virtual wallets	Blocked
Oracle Key Vault Management Console access	Allowed
All Administrator and endpoint operations that add new data or modify existing data	Blocked
Backup operations	Blocked

In Read-Only Restricted mode, if you attempt to execute operations that generate new data or modify existing data on the Oracle Key Vault server, the Key Vault Server in Read-Only Restricted Mode error is displayed. 

If you attempt to upload a wallet to the Java keystore, you are prompted for the source Java keystore password. After entering the password, the Key Vault Server in Read-Only Restricted Mode error is displayed.

4.2.6.5 Enabling Read-Only Restricted Mode

Read-Only Restricted mode is enabled by default when High Availability is configured.

Caution:

It is recommended that you configure High Availability with Read-Only Restricted mode enabled.

If Read-Only Restricted mode is disabled, you must perform the following steps to enable it:

1. Unpair the primary server from the standby server, and reinstall Oracle Key Vault on the standby server. For more information about installing the Oracle Key Vault appliance software, see Installing the Oracle Key Vault Appliance Software.
2. Perform post-installation tasks on the standby server. For more information about performing post-installation tasks, see Performing Post-Installation Tasks.
3. Configure High Availability on the standby server. On the Configure High Availability page, ensure that Yes is selected in the Allow Read-Only Restricted Mode field.
   For more information about configuring High Availability, see Configuring High Availability.
4. Log in to the primary server as the System Administrator, and on the Configure High Availability page, ensure that Yes is selected in the Allow Read-Only Restricted Mode field.
5. Click Initiate Pairing.
   Read-Only Restricted mode is enabled on the High Availability deployment.

Read-Only Restricted mode takes effect if connectivity is lost between the primary and standby servers.

Note:

Read-Only Restricted mode has no effect on a standalone server.

4.2.6.6 Disabling Read-Only Restricted Mode

Read-Only Restricted mode is enabled by default when High Availability is configured.

Caution:

It is recommended that you configure High Availability with Read-Only Restricted mode enabled. Do not disable Read-Only Restricted mode while configuring High Availability, unless it is necessary.

To disable Read-Only Restricted mode, you must perform the following steps:

1. Unpair the primary server from the standby server, and reinstall Oracle Key Vault on the standby server. For more information about installing the Oracle Key Vault appliance software, see Installing the Oracle Key Vault Appliance Software.
2. Perform post-installation tasks on the standby server. For more information about performing post-installation tasks, see Performing Post-Installation Tasks.
3. Configure High Availability on the standby server. On the Configure High Availability page, ensure that No is selected in the Allow Read-Only Restricted Mode field.
   For more information about configuring High Availability, see Configuring High Availability.
4. Log in to the primary server as the System Administrator, and on the Configure High Availability page, ensure that No is selected in the Allow Read-Only Restricted Mode field.
5. Click Initiate Pairing.
   Read-Only Restricted mode is disabled on the High Availability deployment.

Read-Only Restricted mode is disabled, and will not take effect if connectivity is lost between the primary and standby servers.

Note:

Read-Only Restricted mode has no effect on a standalone server.

4.2.6.7 Best Practices

The following are best practices to ensure operational continuity and minimal downtime of Oracle Key Vault:

• Configure auto-login for Hardware Security Module (HSM) on TDE-enabled endpoint databases. For more information about configuring auto-login for Hardware Security Module (HSM), see “Configuring Auto-Login Hardware Security Modules” in the Oracle Database Advanced Security Guide.

• Apply the database patch for Bug 22734547 to tune the Oracle Key Vault heartbeat.

• Ensure the Read-Only Restricted mode is enabled in High Availability Oracle Key Vault deployments.

• Set the duration in the Fast Start Failover Threshold field on the Configure High Availability page to a value that avoids unnecessary failover due to transient network interruptions.

• Configure Syslog auditing to capture audit records in Read-Only Restricted mode.

• Switch over to the original primary server in case the primary server is reinstated.

4.3 Backup and Restore Oracle Key Vault Data

Oracle Key Vault provides the facility to backup and restore Key Vault data for disaster recovery purposes.

It is highly recommended to back up data periodically to reduce down time and recover from unexpected data losses and system failures. You can restore a new or existing Oracle Key Vault appliance from a backup.

• About Backing Up and Restoring Data in Key Vault
  
• Backing Up Key Vault Data
  
• About Backup Schedule Types and States
  
• Scheduling and Managing Key Vault Backups
  
• Restoring Oracle Key Vault Data
  
• Backup and Restore Best Practices
  

4.3.1 About Backing Up and Restoring Data in Key Vault

Key Vault provides the facility to backup and restore security objects stored and managed in Key Vault for disaster recovery purposes.

Backup and restore operations may be performed from the Key Vault management console. You must be a user with system administrative privileges to back up and restore Key Vault data. Backups may be scheduled at periodic intervals to run automatically at designated times. They may also be run on-demand to save a current snapshot of the system.

It is highly recommended that you back up Key Vault data regularly on a schedule. This practice ensures that backups are current and hold the most recent data. The backup can be used to restore a new or existing Key Vault server and be fully operational with minimum downtime and data loss.

Key Vault encrypts all backed up data, which is copied to the backup destination using the secure copy protocol (SCP). You must therefore ensure that SCP is supported at the backup destination.

4.3.2 Backing Up Key Vault Data

The first step to backing up data is to create a backup destination, which is the location where Key Vault data will be copied to and stored.

The primary reason for adding a backup destination is to have the backup data available in a location other than the Key Vault server itself. This ensures that you have all the relevant data to recover in case of a catastrophic failure with the Key Vault server or hardware.

The backup destination is usually another server or computer system that you have access to. You can add, delete, and modify a backup destination.

• About Key Vault Backup Destinations
  
• Create a Remote Backup Destination
  
• Change Settings on a Remote Backup Destination
  
• Delete a Remote Backup Destination
  

4.3.2.1 About Key Vault Backup Destinations

The backup operation copies Oracle Key Vault data to a backup destination of your choice. The backup destination stores the data until it is needed.

Key Vault provides two types of backup destinations: local and remote. The local backup destination resides on the Key Vault server itself, the remote one resides externally in a different server or computer system. You can create more than one backup destination for greater availability.

Backup destinations may be:

• Local

  The local backup destination, LOCAL, is present out of the box and cannot be removed.

  Backups to LOCAL are useful to save a current state of Key Vault. Since these backups are stored in Key Vault, they will be lost in case of a failover or switchover in a high availability deployment. It is therefore highly recommended that you back up the data to a remote destination before you perform operations like failover and switchover.

  A LOCAL destination can store only the last full backup and the cumulative incremental backups after that full backup. After a new full backup of the periodic backup to LOCAL completes, the previous periodic full or cumulative incremental backups are deleted. For more information about backups, see Two Types of Key Vault Backups.

• Remote

  Remote backup destinations reside on external servers and can be dispersed geographically for disaster recovery purposes.

  Each backup destination on the external server is associated with a backup catalog file called okvbackup.mgr that Key Vault maintains at the backup destination. The file okvbackup.mgr catalogs the backups performed and is used to restore data.

  Note:

  You cannot use another Key Vault appliance as a remote backup destination.

  Caution:

  • Oracle Key Vault may not be able to find the backups if you delete or modify the backup catalog file. Therefore do not delete or modify this file.

  • Do not configure the same remote backup destination directory for different Key Vault servers as backup destinations, because backups that happen concurrently from different Key Vault servers will overwrite each other's catalog file, with the result that Key Vault may not be able to locate the backups correctly.

4.3.2.2 Create a Remote Backup Destination

To create a remote backup destination, you must provide a user account, a unique existing directory location on an external server, and an authentication method (password or key-based). Oracle Key Vault needs this information to make a secure connection with the remote server.

To create a remote backup destination:

1. Log in to the Oracle Key Vault management console as a user with the System Administrator role.
2. Select the System tab, then click System Backup from the left sidebar.

   The System Backup page appears. It lists the scheduled backups and details of the last 10 backups performed.

   Figure 4-6 System Backup Page

   
   Description of "Figure 4-6 System Backup Page"
3. Click Manage Backup Destinations.

   The Manage Backup Destinations page displays the local backup destination that comes with Key Vault, and any remote destinations, if configured.

   Figure 4-7 Manage Backup Destinations

   
   Description of "Figure 4-7 Manage Backup Destinations "
4. Click Create.

   The Create Backup Destination page appears.

   Figure 4-8 Create Backup Destination

   
   Description of "Figure 4-8 Create Backup Destination"
5. Enter the following information for the backup location:

   Caution:

   The username, hostname or the destination directory paths for remote backup destinations should not have a space, single quotes or double quotes in them.

   • Destination Name: Enter a descriptive name to identify the backup destination.

   • Transfer Method: This is automatically populated with the value scp for the SCP protocol that is used to copy files to the remote destination.

   • Hostname: Enter the hostname or IP address of the remote server for the backup. If you enter the host name, ensure that DNS is configured to translate the host name to its corresponding IP address.

   • Port: Enter the SCP port number on the external server. The default is 22.

   • Destination Path: Enter the path to an existing directory on the external server, where the backup file will be copied. This directory location is cannot be modified after the backup destination is created.This path should not be the destination for backups from another Oracle Key Vault appliance.

   • Username: Enter the username of the user account on the remote server. Ensure that write permissions are set on the directory specified in Destination Path for the user identity that establishes the SCP connection.

   • Authentication Method: Choose one of the following:

     • Password Authentication

       The password of the user account entered in the Username field.

     • Key-based Authentication

       Copy the public key that appears and paste it in the appropriate configuration file, such as authorized_keys, on the destination server. Check that the permissions of the configuration file are set to allow access only to the backup account owner and no other group or user.

6. Click Save.

   Oracle Key Vault validates the input supplied to create the backup destination. If the validation fails, the backup destination is not created. If this happens, recheck values for the user account on the remote server (username and password or key) and ensure that the directory has write permissions for the user. Finally, ensure that the remote server is up and running.

4.3.2.3 Change Settings on a Remote Backup Destination

Once the backup destination is created, you can only change the SCP port number and details of the user account. You may not change any other setting.

To change allowable settings of a backup destination:

1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
2. Select the System tab, then click System Backup.
3. Select Manage Backup Destinations.

   The Manage Backup Destinations page appears displaying LOCAL and remote backup destinations.

4. Click the backup destination name to edit it. The Edit Backup Destination page appears.
5. Modify the following information:

   Figure 4-9 Edit Backup Destination

   
   Description of "Figure 4-9 Edit Backup Destination"

   • Port: Change the default port number running SCP on the external server.

   • Username: Enter the username of the user account on the remote server. Ensure that the new user has write permissions on the directory specified in Destination Path since this path may not be changed.

   • Authentication Method: Choose one of the following:

     • Password Authentication

       The password of the user account entered in the Username field.

     • Key-based Authentication

       Copy the public key that appears and paste it in the appropriate configuration file, such as authorized_keys, on the destination server.

6. Click Save.

   Key Vault validates the input supplied to update the backup destination. If the validation fails, the backup destination is not created. If this happens, recheck values for the user account on the remote server (username and password) and ensure that the directory has write permissions for the user. Finally, ensure that the remote server is up and running.

4.3.2.4 Delete a Remote Backup Destination

You can delete a remote backup destination to stop future backups to that destination server. Backups already on the destination server will remain there.

To delete a remote backup destination:

1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
2. Select the System tab, and then click System Backup.
3. Select Manage Backup Destinations.

   The Manage Backup Destinations page appears displaying LOCAL and remote backup destinations.

   Figure 4-10 Manage Backup Destinations Page

   
   Description of "Figure 4-10 Manage Backup Destinations Page"
4. Check the box(es) for the backup destinations you want to delete.
5. Click Delete.

4.3.3 About Backup Schedule Types and States

You can schedule backups in Key Vault for specific times and backup destinations. The backup process starts at the scheduled time and generates a system backup, which is a file that is stored on the backup destination. There is one backup file for each completed backup.

No backup can start if another backup is in progress. You can change the schedule of backups as needs change. You can continue working with Key Vault while the backup is in progress.

A system reboot will terminate any ongoing backup. If you must reboot the system, you can cancel a backup that is scheduled to happen at the same time, and backup the system after the reboot.

• Two Types of Key Vault Backups
  
• Scheduled Backup States in Key Vault
  

4.3.3.1 Two Types of Key Vault Backups

Oracle Key Vault provides two types of backups that can be scheduled:

1. One-time backup

   A one-time backup makes a full backup of the Key Vault system. More than one such one-time backup can be scheduled together.

   One-time local backups should be taken before making significant configuration changes to Key Vault, in case you need to recover from configuration failures.

   LOCAL destinations can only store the last one-time backup. When a one-time backup to LOCAL completes, the previous backup is deleted.

2. Periodic backup

   A periodic backup makes a backup at regular intervals at a specified frequency. The process first makes a full backup of the Key Vault system and puts it in active state. For more information about backup states, see Scheduled Backup States in Key Vault. At the end of the subsequent periodic interval, a cumulative incremental backup starts. This cumulative incremental backup holds changes from the last full backup. Another full backup is made after 7 days have passed since the last full backup.

   For example, if the backup period is once a day, then every seventh one is a full backup. If the backup period is every 8 days, then all backups are full backups. If the backup period is 12 hours, then there are 13 cumulative backups before a full backup.

   Periodic backups should be scheduled with a period of at least one day to minimize data loss.

   A LOCAL destination can store only the last full backup and the cumulative incremental backups after that full backup. After a new full backup of the periodic backup to LOCAL completes, previous periodic full or cumulative incremental backups are deleted.

   Cumulative incremental backups are faster than full backups. Only one periodic backup can be scheduled at any time.

4.3.3.2 Scheduled Backup States in Key Vault

Scheduled backups have four states, which indicate whether the backup is scheduled, in progress, completed, or paused:

1. ACTIVE

   The backup is scheduled and will be processed at the specified start time or period.

2. ONGOING

   The backup is in progress.

3. DONE

   The backup is complete.

4. PAUSED

   All future backups are on hold and will not start even if the start time has passed. They will start when they are explicitly resumed.

   You can change the state from active to paused and back. Put a scheduled backup in the paused state for these situations:

   • When communication between Key Vault and the remote destination is broken

   • If the remote destination is down

   • If you want to defer the backup

   You can delete the scheduled backups that have not completed.

4.3.4 Scheduling and Managing Key Vault Backups

You can schedule Key Vault backup(s) to specific backup destination(s) and time(s). Note, that you must create the backup destinations that you will use prior to this step. Backup schedules may be modified or deleted to accommodate changes.

• Schedule a Backup on Key Vault
  
• Change Backup Schedule on Oracle Key Vault
  
• Delete a Backup Schedule from Key Vault
  
• How High Availability Affects Key Vault Backups
  
• Protecting the Backup Using the Recovery Passphrase
  

4.3.4.1 Schedule a Backup on Key Vault

You can also schedule a one-time or periodic backup to a local or remote backup destination. You can start a one-time backup to start immediately without setting a time.

To schedule a backup:

1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.

2. Select the System tab, and then System Backup from the left sidebar.

   The System Backup page appears.

3. Click Backup.

   The Backup page appears.

4. In the Name field, enter a name for the backup.

5. Click the Calendar icon and select the Start Time.

6. Select the Destination for the backup from the list.

7. Select the Type: ONE-TIME or PERIODIC.

   Figure 4-11 Backup Page

   
   Description of "Figure 4-11 Backup Page"

   The Backup page is the same for one-time or periodic backups. In case of a periodic backup additional fields for Days, Hours and Mins appear. You must enter values for these fields.

8. Click Schedule.

   The System Backup page appears listing the scheduled backup in Scheduled Backup(s).

To start an immediate one-time backup:

1. Enter a name in the Name field for the backup on the Backup page.
2. Select the type of backup as ONE-TIME.
3. Select the Destination for the backup from the list.
4. Click Now.
5. Click Schedule.

   The System Backup page appears listing the scheduled backup in Scheduled Backup(s).

4.3.4.2 Change Backup Schedule on Oracle Key Vault

You can not change the schedule of a backup in progress. To change the backup schedule the state must be active or paused.

To edit a scheduled Key Vault backup:

1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
2. Select the System tab, and then select System Backup.
3. Click the Name of the scheduled backup in Scheduled Backup(s).

   The Backup page appears.

   Figure 4-12 System Backup Page

   
   Description of "Figure 4-12 System Backup Page"
4. Enter the Start Time or click the calender icon for a one-time backup.

   Note, that for a one-time backup you can only change the start time if the backup has not already started. This means that the state cannot be ongoing or done. For a periodic backup you can change the start time if the scheduled start time has not passed.

5. Enter the Days, Hours, and Mins for a periodic backup.
6. Select Save to save the changes.

4.3.4.3 Delete a Backup Schedule from Key Vault

To delete an Oracle Key Vault scheduled backup:

1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
2. Select the System tab, and then System Backup from the left sidebar.
3. Check the box(es) of scheduled backup(s) listed in Scheduled Backup(s).
4. Click Delete to delete the scheduled backups.

4.3.4.4 How High Availability Affects Key Vault Backups

It is important to note that backups are performed on the primary server in a high availability deployment. Since the standby synchronizes its state with the primary, it is not necessary to backup the standby.

Points to consider during failover or switchover in a high availability deployment would be:

• Any backups in progress will terminate if there is a failover or a high availability switchover. Backups to LOCAL are private to the Oracle Key Vault appliance and therefore the local backup on the primary appliance is not available after a failover or switchover.

• Backups scheduled with password authentication start as usual after the failover or switchover.

• Remote backups using key-based authentication will need to update the public key on the destination to match the one shown on the new primary system.

4.3.4.5 Protecting the Backup Using the Recovery Passphrase

Oracle Key Vault uses the recovery passphrase to control who can restore user and system data.

To restore a backup, use the Oracle Key Vault recovery passphrase from the time when the backup was initiated. This is necessary even if the recovery passphrase was changed after the backup completed. Oracle recommends that you make a new backup every time the recovery passphrase is changed to ensure that there is always a copy of the backup that is protected by the most recent recovery passphrase.

See Also:

"Emergency System Recovery Process" for information on the recovery passphrase and how it is used

4.3.5 Restoring Oracle Key Vault Data

You can restore Key Vault data from a remote backup destination onto a new or existing Key Vault server to minimize downtime and data loss. The restore process replaces all the data on the new server except the root and support user passwords. You will not be able to restore data to a server if there is a scheduled backup in process on the server.

Note:

You must restore Key Vault data to a server only after ensuring that all scheduled backups on the server are completed.

• About the Key Vault Restore Process
  
• Restore Key Vault Data
  
• High Availability and the Restore Operation
  
• Third-Party Certificates and the Restore Operation
  
• Changes Resulting from a System State Restore
  

4.3.5.1 About the Key Vault Restore Process

Restoring data to a Key Vault server replaces the data in the server with that of the backup. Any changes made since the last backup will be lost.

The maximum life of a backup is 1 year. Any backup older than a year cannot be restored.

You must have the recovery passphrase that was in effect at the time of the backup in order to restore data from a backup. If you have not changed the recovery passphrase since installing Key Vault, then you must use the recovery passphrase that you created during the post-installation process.

There are two steps to restoring data in Key Vault: Setup and Restore.

1. Setup consists of:

   1. Installing the Oracle Key Vault appliance.

   2. Setting up the backup destinations.

2. Restore the Oracle Key Vault appliance by:

   1. Determining the backup to use from a remote or local backup destination.

   2. Providing the recovery passphrase to begin the restore process.

Note:

The recovery passphrase was created in Performing Post-Installation Tasks

4.3.5.2 Restore Key Vault Data

Before you restore ensure that you have the correct recovery passphrase. You will need to enter it during the restore process.

To restore data from a backup:

1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
2. Select the System tab, and then System Backup.
3. Click Restore.

   The Restore page appears.

4. Select Source from the drop-down list. Values are either LOCAL or Remote.
5. Select Restore next to the backup you want to restore from.

   Figure 4-13 Restore Page

   
   Description of "Figure 4-13 Restore Page"
6. Click Restore to initiate the restore or recovery process.

   You are prompted for the recovery passphrase.

7. Enter the recovery passphrase and then click Restore to begin.

   The system will restore from the backup and then reboot.

4.3.5.3 High Availability and the Restore Operation

In a high availability deployment you must consider the following points while restoring data to Key Vault:

1. Restore only if both primary and standby are lost.

2. You must restore the backup on a standalone Key Vault appliance only, even if the backup was taken from the primary.

3. The restore operation replaces the Key Vault appliance with the backup. This means that some data can be lost. You might need to restore the endpoint database.

4. If you restore a backup taken from the primary node, then you must discard (or reinstall) the standby server and configure a new standby.

5. If the standby server has taken over as primary, then there is no need to restore data from a backup to the new standby server. Just configure a new standby server and it automatically synchronizes with the functioning primary.

4.3.5.4 Third-Party Certificates and the Restore Operation

A third-party certificate installed at the time of a backup will not be copied when you restore another appliance from this backup. You will have to re-install the third-party certificate on the new appliance in order to use it.

See Also:

" Third-party certificates in Key Vault"

4.3.5.5 Changes Resulting from a System State Restore

Restoring an Oracle Key Vault appliance brings the system state back to the time when the backup was created.

Therefore, any changes made after the backup was made do not exist on the restored system. For example, if a user's password was changed after the backup operation, the new password will not be available in the restored system. The restored system will have the password that was in effect when the backup was made.

Note:

Restoring also changes the recovery passphrase to the one that was in effect during the backup.

You should change the user passwords, enroll the endpoints created after backup, and make other similar changes, if required. You should confirm that everything is configured correctly after restoring.

If you are not certain that you restored the correct backup, then you can restore a different one. To restore another backup, first configure the remote destination of this backup on the restored Oracle Key Vault itself, and then start the restore process. You do not need to reinstall the Oracle Key Vault appliance.

When the appliance has been restored and functional, you can continue to backup Key Vault data to new or previous remote destinations.

Depending on the age of your backup, the restored server may be missing endpoints, security objects, and other changes made after the restored backup was taken. You may need to enroll missing endpoints and upload missing security objects, or choose a more recent backup to restore. It is also recommended that you change user passwords after a restore operation.

4.3.6 Backup and Restore Best Practices

The following best practices will help you to keep backups current so that you can recover from catastrophic failures with minimum down time and data loss:

1. Ensure that the recovery passphrase at the time of backup is accessible because you will need it to restore data from a backup.

2. Backup data any time you change the recovery passphrase.

3. Ensure that you create at least one remote backup destination in a high availability deployment. Since the local backup resides on the Key Vault server itself, it will be lost in a failover or switchover situation.

4. Do not delete the backup catalog file associated with a remote backup destination, even if you stop using the backup destination. If you ever need to restore from a backup on this server, you will need the backup catalog file.

5. If you use the same remote server for multiple backup destinations, ensure that the directories are unique so that you have distinct backup catalog files associated with each backup destination. If you fail to do this, the backup catalog file will get overwritten during subsequent backups and become unusable.

6. Before you restore data ensure that all scheduled backups are complete.

7. To create remote backup destinations successfully:

   1. Ensure that the servers used as remote backup destinations are up.

   2. Ensure connectivity between Key Vault and remote server you plan to use as a backup destination.

   3. Ensure that the remote server designated as a backup destination supports the secure copy protocol (SCP).

   4. Validate user account credentials on remote server before you create the backup destination on Key Vault.

   5. Ensure that the destination directory has write permissions.

   6. Create more than one remote backup destination on multiple servers for redundancy.

   7. Ensure that destination directories are unique if you are using the same remote server for multiple backup destinations. You must do this to prevent later backups from overwriting previous ones.

8. Schedule a periodic backup with a period of one day. This ensures that you have a full backup once in seven days.

9. Perform a one-time backup once every seven days.

10. Perform a local one-time backup before system changes. You can use this backup as a restore point.

11. Backup before and after upgrading Key Vault server software.

12. Change the backup destination after each upgrade. If at all possible do not reuse the backup destination.