6 Managing an Oracle Key Vault Primary-Standby Configuration

You can deploy Oracle Key Vault in a primary-standby server configuration.

• Overview of the Oracle Key Vault Primary-Standby Configuration
  The Oracle Key Vault primary-standby configuration provides benefits based on the type of deployment your site needs.
• Configuring the Primary-Standby Environment
  To configure a primary-standby environment, you must have the System Administrator role and have access to the two servers (one primary and one standby).
• Switching the Primary and Standby Servers
  You can switch the roles of the primary and standby server for situations such as maintenance periods.
• Restoring Primary-Standby After a Failover
  A failover takes place if the primary server fails.
• Disabling (Unpairing) the Primary-Standby Configuration
  You can disable the primary-standby configuration by unpairing the primary and standby servers.
• Read-Only Restricted Mode in a Primary-Standby Configuration
  The read-only restricted mode is the default mode in a primary-standby configuration.
• Best Practices for Using Oracle Key Vault in a Primary-Standby Configuration
  Oracle provides guidelines for ensuring operational continuity and minimal downtime of Oracle Key Vault.

6.1 Overview of the Oracle Key Vault Primary-Standby Configuration

The Oracle Key Vault primary-standby configuration provides benefits based on the type of deployment your site needs.

• About the Oracle Key Vault Primary-Standby Configuration
  You configure a primary-standby environment by providing the primary and standby servers with each other's IP address and certificate, and then pairing them.
• Benefits of an Oracle Key Vault Primary-Standby Configuration
  The benefits of an Oracle Key Vault primary-standby configuration include high availability, necessary for business-critical operations.
• Difference Between Primary-Standby Configuration and Multi-Master Cluster
  One server will always operate in read-write mode.
• Primary Server Role in a Primary-Standby Configuration
  A primary-standby deployment consists of two Oracle Key Vault servers operating in a primary-standby configuration.
• Standby Server Role in a Primary-Standby Configuration
  In a primary-standby environment, one server runs in the standby server role.

6.1.1 About the Oracle Key Vault Primary-Standby Configuration

You configure a primary-standby environment by providing the primary and standby servers with each other's IP address and certificate, and then pairing them.

While pairing the primary and standby servers, you can select one as the primary server, and the other as the standby. A failover timeout that you set determines when the standby starts to take over as the primary server.

Note:

Oracle strongly recommends that you keep the primary and standby systems as identical as possible, because their roles can be reversed in maintenance periods and failure situations. These include the following:

• Oracle Key Vault software versions

• Disk size

• RAM size

• System clocks on both systems must be synchronized

If your deployment requires a primary-standby configuration, then Oracle recommends that you configure it before adding endpoints to Oracle Key Vault. This enables the endpoints to know about both the primary and standby servers. An endpoint that is added before the standby server configuration will not know about the standby server, unless you re-enroll the endpoint. If you configure the primary-standby environment after adding endpoints, then you must re-enroll the endpoints to ensure the endpoints recognize both servers that were previously enrolled with the primary and standby servers in standalone mode.

WARNING:

Configure primary-standby deployments before adding endpoints to ensure that the endpoints know about both nodes.

If you want to add SNMP support in a primary-standby environment, then ideally, configure SNMP on both the primary and the standby servers before pairing them. This is because the standby server is no longer accessible from the Oracle Key Vault management console, because all requests are forwarded to the primary server. However you can also add SNMP support to the standby after pairing the servers by accessing the standby using SSH.

If you want to use a third-party certificate in a primary-standby configuration, then you must install it on the primary and standby servers first, and then pair them.

If you want to enable FIPS mode in a primary-standby environment, then you must ensure that both the primary and standby servers use the same FIPS mode: either both are enabled, or both are disabled for FIPS mode. This is because the standby server is no longer accessible from the Oracle Key Vault management console, because all requests are forwarded to the primary server.

With persistent cache enabled, both the primary and the standby will cache the master encryption keys from Oracle Key Vault independently. Ensure that TDE operations have executed on the primary and standby servers after these servers have started to verify the persistent cache. The persistent cache feature also enables endpoints to be operational during primary-standby operations, such as configuration, switchovers, and failovers.

If enabled, read-only restricted mode ensures endpoint operational continuity (such as enabling the endpoints to fetch keys) if either the standby or primary server is not available. For example, if the standby shuts down, then the primary will go into read-only restricted mode and enable the endpoints to fetch keys and continue operations.

A primary-standby configuration is characterized by continuous synchronization between the primary server and the standby server. When synchronization is lost between the primary and standby servers, it is possible to encounter a split-brain scenario where two primary servers might be active simultaneously. In such a scenario, both servers record new data that diverges from the last synchronized state. When connectivity is restored between the primary and standby servers, it may not be possible to reconcile the changes on the two servers and data loss may occur.

You can enable or disable restricted mode when configuring the primary-standby environment by selecting the Allow Read-Only Restricted Mode option to Yes or No on the Configure Primary-Standby page.

When read-only restricted mode is enabled, the primary server enters read-only restricted mode if the standby server is unavailable. In read-only restricted mode, the primary server allows keys to be retrieved, but does not allow keys to be modified or new keys to be added. This ensures that endpoints still have access to their keys, and key data or metadata is not lost due to a split-brain scenario. However, the primary server still writes audit records, which may be lost if a split-brain scenario occurs with the standby server.

When read-only restricted mode is disabled, the primary server becomes unavailable and stops accepting new requests if the standby server is unavailable. Endpoints connected to Oracle Key Vault will be unable to retrieve keys from the server until connectivity is restored between primary and standby servers. You can use the persistent master encryption key cache feature to avoid endpoint downtime. With this feature, data integrity is ensured by allowing endpoints to communicate with one primary server at any given time. This avoids split-brain situations, and the risk of data loss associated with such situations.

6.1.2 Benefits of an Oracle Key Vault Primary-Standby Configuration

The benefits of an Oracle Key Vault primary-standby configuration include high availability, necessary for business-critical operations.

Users performing business-critical operations must have data to be accessible and recoverable with minimum downtime. These requirements are met in a primary-standby configuration.

You achieve high availability by adding redundancy in the form of a standby server that can take over the functions of the primary server in case of failure. The standby server helps you eliminate single points of failure and reduce server downtime. This is a significant reason to deploy Oracle Key Vault in a primary-standby configuration. In a classic primary-standby configuration, the emphasis is on key preservation. In a multi-master cluster, emphasis is on both key preservation and availability of the keys.

You can create a cluster of Oracle Key Vault server nodes for greater availability and redundancy. A primary-standby configuration is limited to two servers, whereas a multi-master cluster can have up to 16 geographically distributed nodes. The primary-standby configuration and the multi-master configuration are mutually exclusive.

6.1.3 Difference Between Primary-Standby Configuration and Multi-Master Cluster

One server will always operate in read-write mode.

In a primary-standby configuration, when both servers are available, one of the servers operates in read-write mode in the primary server role, and the other operates in the standby server role. The endpoints only connect to the server running in the primary server role. The roles can be switched manually to support maintenance operations, or automatically due to server or connectivity failure. If either the primary or standby server becomes unavailable, then the remaining server operates in a read-only restricted mode, limiting normal updates while allowing audits and other internal updates.

In a multi-master cluster, the endpoints can connect to any Oracle Key Vault server.  Some servers are configured as bi-directional read-write pairs in which information updated in either node must be successfully replicated to the other node immediately. If one of the nodes in a read-write pair becomes unavailable, the surviving node operates in read-only restricted mode until the other node is restored and synchronization resumes. A fully functional multi-master cluster must have at least one read-write pair.  

When a successful update occurs in a read-write pair, the update is propagated to all other nodes in the cluster.

A primary-standby configuration and a multi-master cluster configuration are mutually exclusive and incompatible configurations. The specific configuration of an Oracle Key Vault deployment has no ramification on the endpoint side configuration.

6.1.4 Primary Server Role in a Primary-Standby Configuration

A primary-standby deployment consists of two Oracle Key Vault servers operating in a primary-standby configuration.

By default, endpoints only connect to the primary server until is becomes unavailable. At any time, only one server operates in the primary server role and that server actively accepts client connections. The other server operates in the standby server role, which receives updates from the primary server. On failure of the server running in the primary server role, the standby assumes the primary role. There may be restrictions in operations if the primary-standby pair is not fully available and operational.

6.1.5 Standby Server Role in a Primary-Standby Configuration

In a primary-standby environment, one server runs in the standby server role.

This standby server does not accept client connections while in that role. The server receives updates only from the paired server running in primary server role. If the primary server is no longer available, including being available to the administrator, then the server running in the standby role switches to assume the primary server role. There may be restrictions in operations if the primary-standby pair is not fully available and operational.

6.2 Configuring the Primary-Standby Environment

To configure a primary-standby environment, you must have the System Administrator role and have access to the two servers (one primary and one standby).

After you complete the configuration, with persistent cache enabled, endpoints will continue to operate while the primary-standby configuration is enabled.

• Step 1: Configure the Primary Server
  To configure the primary server, you must enable it to connect to the standby server.
• Step 2: Configure the Standby Server
  To configure the standby server, you must enable it to connect to the primary server.
• Step 3: Complete the Configuration on the Primary Server
  After you configure the primary and standby servers, you can enable the primary-standby on the designated primary server.

6.2.1 Step 1: Configure the Primary Server

To configure the primary server, you must enable it to connect to the standby server.

If you plan to configure an HSM (such as SafeNet, nCipher, or Utimaco) with Oracle Key Vault, then you must first enable this HSM in Oracle Key Vault before configuring the primary server.

1. Open a web browser and enter the IP address of the designated primary server.
   The Oracle Key Vault Management Console login screen is displayed.
2. Log in as the System Administrator.
3. Check if the server has FIPS mode enabled, and if necessary, enable or disable it.
   You must ensure that both the primary and standby servers use the same FIPS mode setting: either both are enabled, or both are disabled, for FIPS mode. Changing the FIPS mode setting requires a restart of Oracle Key Vault.
   1. Select the System tab, and then select System Settings in the left pane.
   2. In the FIPS Mode section, either select or clear the Enable check box, depending on whether both servers will use FIPS mode.
   3. Click Save.
      In a moment, Oracle Key Vault will restart.
4. If you changed the FIPS mode, then log back into the designated primary server as a user who has the System Administrator role.
5. Click the System tab, then click Primary-Standby in the left pane.

   The Configure Primary-Standby page appears.

   
   
   Description of the illustration primary_standby_1.png
   

   The following are the fields on the Configure Primary-Standby page:

   • Current status: Displays the IP address and status of the current server.

   • Fast Start Failover Threshold (in secs): Displays the duration (in seconds) that will elapse before the server takes over from a failed peer server. The default is 60 seconds.

     To avoid failover during brief or intermittent failures, increase the duration.

   • Configure this server as: Displays whether the server is configured as a Primary server or Standby server.

   • Allow Read-Only Restricted Mode: Displays the status of read-only restricted mode. The default is Yes.

     When enabled, read-only restricted mode ensures operational continuity of the endpoints if the primary or standby Oracle Key Vault server is affected by server, hardware, or network failures

   • FIPS Mode: Displays the current FIPS mode status of the server.

   • Current Server Certificate: Displays the server certificate.

6. Copy the following information, and then store it in a text file named primary.txt.

   You will need this information when you configure the standby server.

   • From the Current status field, copy the IP address and paste it in primary.txt.

   • From the Current Server Certificate field, copy the server certificate and paste it on a new line in primary.txt after the IP address.

   Save primary.txt.

Next, you are ready to configure the standby server.

6.2.2 Step 2: Configure the Standby Server

To configure the standby server, you must enable it to connect to the primary server.

If you plan to configure an HSM (such as SafeNet, nCipher, or Utimaco) with Oracle Key Vault, then you must first enable this HSM in Oracle Key Vault before configuring the standby server.

1. Open a web browser and enter the IP address of the designated standby server.
   The Oracle Key Vault Management Console login screen is displayed.
2. Log in as the System Administrator.
3. Check if the server has FIPS mode enabled, and if necessary, enable or disable it.
   You must ensure that both the primary and standby servers use the same FIPS mode setting: either both are enabled, or both are disabled, for FIPS mode. Changing the FIPS mode setting requires a restart of Oracle Key Vault.
   1. Select the System tab, and then select System Settings in the left pane.
   2. In the FIPS Mode section, either select or clear the Enable check box, depending on whether both servers will use FIPS mode.
   3. Click Save.
      In a moment, Oracle Key Vault will restart.
4. If you changed the FIPS mode, then log back into the designated standby server as a user who has the System Administrator role.
5. Click the System tab, then click Primary-Standby in the left pane.
   The Configure Primary-Standby page is displayed.
6. Copy the following information, and store it in a text file named standby.txt.

   You will need this information when you configure the primary server.

   • From the Current status field, copy the IP address and paste it in standby.txt.

   • From the Current Server Certificate field, copy the server certificate and paste it on a new line in standby.txt after the IP address.

   Save standby.txt.

7. In the Configure this server as field, select Standby server.

   The Primary server IP address and Primary server certificate fields are displayed.

   
   
   Description of the illustration primary_standby_2.png
   

   Ensure that Yes is selected in the Allow Read-Only Restricted Mode field.

   Do not disable read only restricted mode unless necessary. If the primary-standby configuration is configured with read only restricted mode disabled, then you must enable it by reinstalling and configuring Oracle Key Vault again.

8. Copy the following information from primary.txt, and paste it in the Configure Primary-Standby page of the standby server:

   • Copy the IP address and paste it in the Primary server IP address field.

   • Copy the server certificate and paste it in the Primary server certificate field.

9. Click Save.

   The Settings Saved page is displayed.

   
   
   Description of the illustration primary_standby_3.png
   

   The Reset button enables you to delete the primary-standby configuration, if necessary.

10. Do not exit the management console.

At this stage, the primary-standby configuration is complete on the designated standby server. The next step is to enable primary-standby on the designated primary server.

6.2.3 Step 3: Complete the Configuration on the Primary Server

After you configure the primary and standby servers, you can enable the primary-standby on the designated primary server.

1. Ensure that you are logged in to the standby server as a user with the System Administrator Role and that the Oracle Key Vault management console Configure Primary-Standby page is displayed.
2. On the Settings Saved page, click the IP address of the primary server displayed at the top of the page.
   The Oracle Key Vault Management Console login screen of the primary server is displayed.
3. Log in as the System Administrator.
4. Click the System tab, then click Primary-Standby in the left pane.
   The Configure Primary-Standby page appears.
5. In the Configure this server as field, select Primary server.

   The Standby server IP address and Standby server certificate fields are displayed.

   Ensure that Yes is selected in the Allow Read-Only Restricted Mode field.

   Do not disable read only restricted mode unless necessary. If the primary-standby configuration is configured with read only restricted mode disabled, then you must enable it by reinstalling and configuring Oracle Key Vault again.

6. Copy the following information from standby.txt, and paste it in the Configure Primary-Standby page of the primary server:

   • Copy the IP address and paste it in the Standby server IP address field.

   • Copy the server certificate and paste it in the Standby server certificate field.

   
   
   Description of the illustration primary_standby_4.png
   

7. Click Initiate Pairing.
8. In the confirmation message that is displayed, click OK. The Operation in Progress page is displayed.

   Caution:

   Allow at least 10 minutes to elapse before performing the next step.
9. After at least 10 minutes have elapsed, click Refresh.
   If the pairing of primary and standby servers is successful, then the current session is terminated. The Oracle Key Vault Management Console login screen of the primary server is displayed. The primary-standby configuration is now complete.
10. Check that the configuration was successful.
    1. Log in as the System Administrator.
    2. Click the System tab, then click Primary-Standby in the left pane.

       The Primary-Standby Status page appears.

       
       
       Description of the illustration primary_standby_5.png
       

    3. Ensure that the Status label is set to Primary-Standby mode is enabled.
    4. Ensure that the Switchover Status is correct. In this example, the status is correctly set to TO STANDBY.

At this stage, the primary-standby configuration should be ready to use. Note the following:

• When the primary-standby configuration is complete, you cannot log in to the standby server using a web browser because all configuration is propagated from the primary.

• To manage the primary-standby deployment, log in to the primary server using a web browser.

Caution:

Ensure that you leave read-only restricted mode enabled while configuring primary-standby. Enabling it later requires a reinstall of the Oracle Key Vault server software on the standby server.

After configuring the primary-standby environment, do not change the system time on the primary server. The changed system time causes the standby server to go down, thus disrupting the functioning of the primary-standby configuration.

6.3 Switching the Primary and Standby Servers

You can switch the roles of the primary and standby server for situations such as maintenance periods.

During such maintenance periods, you might want to shut down a server to upgrade software or install patches. If you have persistent cache enabled and the persistent cache timeout is sufficiently tuned, then the endpoints will continue to be operational during the switchover, minimizing endpoint downtime.

1. Log in to the Oracle Key Vault management console of the primary node as a user with the System Administrator role.
2. Before switching the primary and standby servers, ensure that there are no primary-standby related alerts on the Alerts page.

   To access the Alerts page, click the Reports tab, and then click Alerts in the left pane. Ensure that all primary-standby related alerts on the Alerts page are addressed before switching the primary and standby servers.

3. Click the System tab, then Primary-Standby from the left side bar.

   The Primary-Standby Status page appears.

4. Click Switch Roles on the top right.

   The Switch Roles button allows you to switch the roles of the primary server and the standby server. The primary server then assumes the role of the standby server, while the standby server assumes the role of the new primary server.

   Click OK in the confirmation message.

   An operation-initiated message is followed by the Operation in Progress page indicating that the switchover operation will take 10 minutes to complete successfully.

   Caution:

   You must wait for a minimum period of 10 minutes for the switchover operation to complete successfully. If you refresh the UI before the switchover operation is complete, an error message is displayed. The error message is displayed until the switchover is completed successfully.
5. Ensure that at least 10 minutes have elapsed, and only then, click Refresh.

   This logs you out of the current session and then opens a login page to the switched primary server. Otherwise, try accessing the new primary server's URL directly.

   Both the primary and standby servers are restarted. However, you will only be able to log in to the new primary node's web console. The primary server is the active server, and all requests to the standby will be forwarded to the primary.

6. Log in to the primary server to see the IP address of the switched standby node.
7. Click the System tab, then Primary-Standby from the left side bar.

   The Primary-Standby Status page appears. The Standby server IP address field displays the IP address.

6.4 Restoring Primary-Standby After a Failover

A failover takes place if the primary server fails.

If the primary server is not available, then the standby server takes over the primary role. If the standby server does not hear from the primary server for a time exceeding the Fast Start Failover Threshold value, then it will assume that the primary is shut down and start the failover process. You can configure the value in the Fast Start Failover Threshold field from the Oracle Key Vault management console from the default of 60 seconds. If the failed server (the old primary) becomes available again, then in most cases it will automatically become the new standby server. If the primary server fails permanently, then the standby server will take over as the primary. In this case, you must restore the primary-standby configuration.

1. Reinstall the Oracle Key Vault image on the failed server.
   Ensure that you use the original IP address for the failed server.
2. Log on to the newly installed server and follow the steps to configure the primary-standby environment.
   You can designate the new server as the standby server (because the cluster has a functional primary) and then pair it with the functioning primary.
3. If you want to restore the original configuration and set the new server as the primary, then click the Switch Roles option after you successfully pair the two nodes and enable primary-standby.

   The Switch Roles button enables you to switch the roles of the primary server and the standby server. The primary server then assumes the role of the standby server, while the standby server assumes the role of the new primary server.

Note:

When read-only restricted mode is disabled, the primary server's failover status goes into suspended state causing the standby server to wait indefinitely for the primary server to come back up. This is expected behavior to avoid a split-brain scenario where two primary servers are simultaneously active.

When read-only restricted mode is enabled, a primary or standby server failure causes the operational peer to enter read-only restricted mode, thus ensuring endpoint operational continuity.

6.5 Disabling (Unpairing) the Primary-Standby Configuration

You can disable the primary-standby configuration by unpairing the primary and standby servers.

After the two servers are unpaired, the primary and standby servers will operate in standalone mode. To prevent endpoints from connecting to the old standby (now standalone) Oracle Key Vault server, you must take the old standby off the network. See Oracle Key Vault Release Notes for guidance about setting the permissions of the /var/lib/oracle/diag/rdbms/dbfwdb/dbfwdb/metadata_pv directory beforehand. Check the Release Notes for additional issues related to unpair operations.

1. Log in to the primary server's management console as a user with System Administrator privileges.
2. Select the System tab on top, then select Primary-Standby from the left side bar.

   The Primary-Standby Status page appears with Unpair and Switch Roles on the top right. The Unpair and Switch Roles options do the following:

   • The Unpair button allows you to disconnect the primary server from the standby server, if required.

   • The Switch Roles button allows you to switch the roles of the primary server and the standby server, if required. The primary server then assumes the role of the standby server, while the standby server assumes the role of the new primary server.

3. Click Unpair.

   A brief message with a green check appears indicating that the operation has been successfully initiated.

   The Operation in Progress page appears, indicating a wait time of at least 10 minutes for the un-pairing to complete.

   Wait 10 minutes.

4. After 10 minutes, click the Refresh button to be logged out of the current session.
5. Log back in to the management console of the primary server.
6. Select System, then Primary-Standby from the left side bar.

   The Configure Primary-Standby page appears. The Current status field shows the server in standalone mode.

   Caution:

   If you want to use the old standby (now standalone) Oracle Key Vault server as a standby in a new primary-standby deployment, or as part of a multi-master cluster, then you must re-install the Oracle Key Vault software

6.6 Read-Only Restricted Mode in a Primary-Standby Configuration

The read-only restricted mode is the default mode in a primary-standby configuration.

• About Read-Only Restricted Mode in a Primary-Standby Configuration
  Primary-standby read-only restricted mode ensures endpoint operational continuity.
• Primary-Standby with Read-Only Restricted Mode
  Read-only restricted mode is the default primary-standby mode in Oracle Key Vault.
• Primary-Standby without Read-Only Restricted Mode
  When a primary-standby environment is configured without read-only restricted mode, the impact on endpoint operations differs.
• States of Read-Only Restricted Mode
  A server using read-only restricted mode is affected by the failure in a primary server, a standby server, and the network.
• Enabling Read-Only Restricted Mode
  Read-only restricted mode is enabled by default when primary-standby is configured.
• Disabling Read-Only Restricted Mode
  Read-only restricted mode is enabled by default when primary-standby is configured.
• Recovering from Read-Only Restricted Mode
  To recover an instance from read-only restricted mode after a network failure or standby server failure, manual intervention may be required.
• Read-Only Restricted Mode Notifications
  When the primary or standby server enters read-only restricted mode, an alert is generated.

6.6.1 About Read-Only Restricted Mode in a Primary-Standby Configuration

Primary-standby read-only restricted mode ensures endpoint operational continuity.

This endpoint operational continuity is essential when the primary or standby Oracle Key Vault servers are affected by server, hardware, or network failures.

When an unplanned shutdown makes the primary or standby server offline, the endpoints can still connect to the surviving peer server to perform critical operations. Primary-standby read-only restricted mode ensures that operations that replicate data are blocked. Operations that replicate data are allowed when both primary and standby servers are back online, thus ensuring that no critical data is lost.

In a primary-standby Oracle Key Vault configuration, the single point of failure is eliminated when you replicate the primary server’s data to the standby server. Read-only restricted mode enables the generation of non-critical data such as audit records. However, generation of critical data such as keys is disabled. When the primary server is down, operations that generate new critical data on the standby are disabled. The reverse is also true. When the standby server is down, operations that attempt to modify or create any data on the primary server are disabled.

In a primary-standby deployment without read-only restricted mode, most endpoint operations are blocked because endpoint operations generate audit records, which is data that needs replication, thus disrupting operational continuity.

The following are the benefits of using read-only restricted mode:

• Enables endpoint operational continuity when the primary or standby server is offline

• Ensures symmetrical behavior when the primary or standby server is offline

The following sections describe the behavior of:

6.6.2 Primary-Standby with Read-Only Restricted Mode

Read-only restricted mode is the default primary-standby mode in Oracle Key Vault.

Note:

You can disable read-only restricted mode during the primary-standby configuration. Oracle recommends that you configure primary-standby with read-only restricted mode enabled, which is the default mode. While configuring primary-standby, ensure that Yes is selected in the Allow Read-Only Restricted Mode field on the Configure Primary-Standby page.

Read-only restricted mode ensures endpoint operational continuity as well as symmetrical behavior when the primary or standby server is offline. Symmetrical behavior ensures that the online server seamlessly takes over from its failed peer, and continues to service the endpoints without any disruption. For more information about primary-standby failover situations with read-only restricted mode, see Failover Situations with Read-Only Restricted Mode.

In read-only restricted mode, the surviving Oracle Key Vault server operates with limited functionality. Endpoint operations that add or modify critical data on the Oracle Key Vault server are blocked. However, endpoint operations that involve fetching of data are allowed. This ensures endpoint operational continuity and data integrity. For more information about blocked and allowed operations, see About the States of Read-Only Restricted Mode.

For more information about read-only restricted mode, see States of Read-Only Restricted Mode.

Note:

Read-only restricted mode has no impact on a standalone server.

6.6.3 Primary-Standby without Read-Only Restricted Mode

When a primary-standby environment is configured without read-only restricted mode, the impact on endpoint operations differs.

This impact depends on the type of failure encountered: primary failure, standby failure, or a network failure that prevents communication between the primary and standby servers. The following are the possible scenarios:

• Primary server failure: The standby server will failover and take over from the affected primary server. This allows the Oracle Key Vault service to remain operational. Data modifications are stored on the primary server until they can be replicated to the standby server. This ensures endpoint operational continuity when the primary server goes offline due to an unplanned shutdown.

• Standby server failure: The primary server is unavailable to the endpoints, because it is not possible to distinguish a standby server failure from a network failure that prevents communication between the primary and standby servers.

• Power loss or network connectivity failure: The primary and standby servers are unable to communicate. The standby server will failover and take over from the primary server. To avoid a split-brain scenario, only one of the servers is allowed to service the endpoints.

Note:

A split-brain scenario in Oracle Key Vault occurs when the primary server fails, causing the standby server to failover and take over from the primary server. This causes a situation where the primary and standby servers are available to service the endpoints, and create new data. A split-brain scenario causes data on the primary and standby servers to go out of sync. This can lead to data loss and corruption, as well as loss of operational continuity. To avoid a split-brain scenario, only one of the servers is allowed to service the endpoints after a failover occurs.

In primary-standby without read-only restricted mode, one of the following situations is triggered when a failure occurs:

• Endpoints suffer a temporary operational disruption to avoid a split-brain scenario.

• The standby server accepts new requests and generates new data without attempting to synchronize the data with the failed primary server. Replication of data is temporarily disabled until the primary server is online, thus ensuring operational continuity. 

6.6.4 States of Read-Only Restricted Mode

A server using read-only restricted mode is affected by the failure in a primary server, a standby server, and the network.

• About the States of Read-Only Restricted Mode
  Read-only restricted mode puts the Oracle Key Vault instance into the read-only restricted mode state.
• Read-Only Restricted State Functionality During a Primary Server Failure
  You can set a failover threshold value to determine when a standby server takes over for a failed primary server.
• Read-Only Restricted Mode Functionality During a Standby Server Failure
  If a standby fails, the primary server waits for the duration in the Fast Start Failover Threshold field on the Configure Primary-Standby page.
• Read-Only Restricted State Functionality During a Network Failure
  When a network failure affects communication between primary and standby servers, communication between certain endpoints and the primary server may also be affected.

6.6.4.1 About the States of Read-Only Restricted Mode

Read-only restricted mode puts the Oracle Key Vault instance into the read-only restricted mode state.

However, read-only restricted mode does not put the embedded Oracle Key Vault database into the read-only restricted mode state. In read-only restricted mode, the following behavior occurs when a primary or a standby server is unavailable:

• When the primary server is down, data cannot be replicated and so the standby server will failover and disable all operations that generate new data. However, the standby can fetch existing data.

• When the standby server is down, data cannot be replicated and so the primary server disables all operations that generate new data. However, the primary can fetch existing data.

Read-only restricted mode introduces the following deviations from normal functionality:

• All operations that generate new data are blocked. Operations that fetch existing data are allowed. Audit records for endpoint operations are generated as in normal operation. Internal system operations of the Oracle Key Vault database are not impacted. Functionality such as alerts continue to work normally.

• Endpoints are allowed to fetch keys from the Oracle Key Vault server. Endpoints cannot create new keys or modify existing keys.

• Administrators can log in to the Oracle Key Vault management console. Creation of an endpoint or a wallet, deletion of keys, and operations that modify or delete data are blocked.

• Unpairing of primary and standby Oracle Key Vault servers running in read-only restricted mode are allowed.

• Backup operations are blocked to avoid data mismatches between backups.

Table 6-1 Allowed and Blocked Operations in Read-Only Restricted Mode

Operation	Allowed or Blocked
Log in to Oracle Key Vault	Allowed
Endpoint operations such as fetching keys from the cache	Allowed
Endpoint operations that add, modify, or delete data such as rotation of keys on the database	Blocked
System operations such as enabling SSH access	Allowed
System operations that write data such as setting up a REST server and creating virtual wallets	Blocked
Oracle Key Vault management console access	Allowed
All Administrator and endpoint operations that add new data or modify existing data	Blocked
Backup operations	Blocked

In read-only restricted mode, if you attempt to execute operations that generate new data or modify existing data on the Oracle Key Vault server, the Key Vault Server in read-only restricted Mode error is displayed. 

If you attempt to upload a wallet to the Java keystore, then you are prompted for the source Java keystore password. After entering the password, the Key Vault Server in read-only restricted Mode error is displayed.

6.6.4.2 Read-Only Restricted State Functionality During a Primary Server Failure

You can set a failover threshold value to determine when a standby server takes over for a failed primary server.

In the event of a primary server failure, the standby server waits for the duration specified in the Fast Start Failover Threshold (in secs) field on the Configure Primary-Standby page. If the primary server is not reachable after the specified duration has elapsed, the standby server enters read-only restricted mode. In read-only restricted mode, only operations that fetch data are allowed. Endpoint operations that add new data or modify existing data on the Oracle Key Vault server are blocked.

6.6.4.3 Read-Only Restricted Mode Functionality During a Standby Server Failure

If a standby fails, the primary server waits for the duration in the Fast Start Failover Threshold field on the Configure Primary-Standby page.

If the standby server is not reachable after the specified duration has elapsed, the primary server enters read-only restricted mode. In read-only restricted mode, only operations that fetch data are allowed. Endpoint operations that add new data or modify existing data on the Oracle Key Vault server are blocked.

The primary server continues to provide limited service to the endpoints.

6.6.4.4 Read-Only Restricted State Functionality During a Network Failure

When a network failure affects communication between primary and standby servers, communication between certain endpoints and the primary server may also be affected.

The primary server waits for the duration specified in the Fast Start Failover Threshold field on the Configure Primary-Standby page. If the standby server is not reachable after the specified duration has elapsed, the primary server enters read-only restricted mode.

The standby server will also wait for the same duration. If the primary server is not reachable after the specified duration has elapsed, the standby server enters read-only restricted mode. The standby server takes over as the new primary server, and provides service to endpoints that cannot communicate with the affected primary server.

6.6.5 Enabling Read-Only Restricted Mode

Read-only restricted mode is enabled by default when primary-standby is configured.

Oracle recommends that you configure the primary-standby servers with read-only restricted mode enabled.

1. Unpair the primary server from the standby server, and then reinstall Oracle Key Vault on the standby server.
2. Perform post-installation tasks on the standby server.
3. Log in to the standby server as the System Administrator.
4. Select the System tab.
5. Select Primary-Standby and then configure primary-standby on the standby server.
   On the Configure Primary-Standby page, ensure that Yes is selected in the Allow Read-Only Restricted Mode field.
6. Log in to the primary server as the System Administrator.
7. Select Primary-Standby and then on the Configure Primary-Standby page, ensure that Yes is selected in the Allow Read-Only Restricted Mode field.
8. Click Initiate Pairing.

Read-only restricted mode takes effect if connectivity is lost between the primary and standby servers. Read-only restricted mode has no effect on a standalone server.

6.6.6 Disabling Read-Only Restricted Mode

Read-only restricted mode is enabled by default when primary-standby is configured.

Oracle recommends that you configure primary-standby with read-only restricted mode enabled. Follow these steps if an existing primary-standby deployment with read-only restricted mode that is enabled must be converted to a deployment that has read-only restricted mode disabled.

1. Unpair the primary server from the standby server, and reinstall Oracle Key Vault on the standby server.
2. Perform post-installation tasks on the standby server.
3. Log in to the standby server as the System Administrator.
4. Select the System tab.
5. Select Primary-Standby and then configure primary-standby on the standby server.
   On the Configure Primary-Standby page, ensure that No is selected in the Allow Read-Only Restricted Mode field.
6. Log in to the primary server as the System Administrator.
7. Select Primary-Standby and on the Configure Primary-Standby page, ensure that No is selected in the Allow Read-Only Restricted Mode field.
8. Click Initiate Pairing.

After read-only restricted mode is disabled, it does not take effect if connectivity is lost between the primary and standby servers. Read-only restricted mode has no effect on a standalone server.

6.6.7 Recovering from Read-Only Restricted Mode

To recover an instance from read-only restricted mode after a network failure or standby server failure, manual intervention may be required.

You will need to unpair and reset the surviving instance, reinstate a new Oracle Key Vault server, and pair it as the new standby to the surviving server. The following are the possible scenarios:

• Primary server failure: Depending on the operational state of the primary server at the time of failure, it could be restarted and some functionality may be available. However, due to possible corruption of the embedded Oracle Key Vault database, recovery may not be possible. You would then need to reinstate the Oracle Key Vault instance because of the partial failure. If the failed server is unable to again pair with the peer server within 20 minutes, then you must reinstantiate the server.

  Even though the endpoint processes communicating with the Oracle Key Vault servers retain the IP address of the last known reachable server, they must determine the IP address of the new Oracle Key Vault server when spawned. The endpoint processes attempt to communicate with the Oracle Key Vault server configured as the primary server in the configuration scripts, and then wait for a response before trying to reach the server configured as the standby server in the configuration scripts. To minimize downtime, Oracle recommends that you initiate a switchover after reinstating the failed primary server.

• Standby server failure: The primary server will run in the read-only restricted mode if there is a standby server failure. Reinstate the standby server if it does not automatically pair with the primary server.

• Power loss or network connectivity failure: When a network failure occurs, the primary and standby servers are unable to communicate, and both servers enter read-only restricted mode. The standby also attempts to failover to the primary server. Once communication is re-established between the primary and standby servers, the old primary server is automatically converted to the new standby. The data from the new primary server overwrites the old primary server’s data, resulting in the loss of audit records from the old primary server. It is recommended that you enable syslog auditing to preserve the audit records that were overwritten on the old primary. Similar to recovering from primary server failure, Oracle recommends that you perform a switchover after recovery. You should also not enroll any new endpoints before the switchover.

6.6.8 Read-Only Restricted Mode Notifications

When the primary or standby server enters read-only restricted mode, an alert is generated.

You can view these alerts on the Alerts page. If email notifications are configured, then an email notification is sent.

6.7 Best Practices for Using Oracle Key Vault in a Primary-Standby Configuration

Oracle provides guidelines for ensuring operational continuity and minimal downtime of Oracle Key Vault.

• Configure your Transparent Data Encryption (TDE)-enabled databases to have an auto-login connection into Oracle Key Vault. Oracle Database Advanced Security Guide describes how to configure auto-login keystores.

• Apply the database patch for Bug 22734547 to tune the Oracle Key Vault heartbeat.

• Ensure that read-only restricted mode is enabled in primary-standby Oracle Key Vault deployments.

• Set the duration in the Fast Start Failover Threshold field on the Configure Primary-Standby page to a value that avoids unnecessary failover due to transient network interruptions.

• Configure syslog auditing to capture audit records in read-only restricted mode.

• Switch over to the original primary server in case the primary server is reinstated.

• Before attempting any unpair operations, check Oracle Key Vault Release Notes for known issues.