Replication with Warm Standby

Maintain a continuously updated standby database system in a separate region using PostgreSQL Replication with Warm Standby.

Replication with Warm Standby is a disaster recovery tool that enables you to maintain a continuously updated standby database system in a separate region. By implementing this method, you can reduce recovery time, safeguard your data, and ensure business continuity when unexpected disruptions occur. Replication with Warm Standby supports disaster recovery with Recovery Point Objective (RPO) SLO both in the same region and cross region which enables data protection against unexpected events. Warm standby in the same region can enable further horizontal scaling as a single database system supports only 8 nodes.

Replication with Warm Standby enhances the following capabilities

  • Disaster recovery and business continuity: By regularly replicating data from primary to secondary region, you can rapidly rebuild applications in another region if a region-wide disaster occurs.
  • Migration and expansion: You can easily migrate and expand your applications to another region.
  • Simplifying Compliance: Organizations subject to regulatory requirements for high availability or geographic distribution, such as financial or government entities, can meet compliance standards more easily.

Database System Roles

If replication with warm standby isn't set up, then all existing database systems are treated as standalone, with no replication. Each standalone database system operates independently and provides both read and write access (Read/Write).

When you set up a warm standby database, that database acts in the warm standby role. The associated standalone database acts in the primary database system role. Following is more information about these roles.

  • Primary database system role:
    • A primary database system allows data updates through its primary endpoint and replication with warm standby setup would have one database system in this role.
    • Supports read and write operations (Read/Write access).
    • Continuously streams write-ahead logs (WAL) to the standby database system for replication.
  • Warm standby database system role:
    • The warm standby database system is configured under cross-region replication (CRR) as the replica/DR system.
    • Runs in read-only mode to ensure data consistency (read-only access).
    • Continuously receives WAL updates from the primary.
    • Can be promoted to primary system during a disaster, but otherwise is limited to read queries (such as reporting, analytics, or read-scale operations).

For information about configuring extensions, see Configurations and Extensions.

Prerequisites

Note and follow these constraints and best practices for setting up and configuring the primary and warm standby database systems.

Shape Requirements

You're allowed to select any heterogeneous shapes or input/output operations per second (IOPS) between the primary and warm standby databases. However, we recommend configuring them with the same shape and performance to ensure a seamless failover to the standby database system with a comparable performance profile in the event of a disaster.

IAM

To use Oracle Cloud Infrastructure services, you must have the appropriate security access through IAM policies. If you receive "permission denied" or "unauthorized" errors, ensure you have the required group membership and compartment access.

To gain access to use Replication with Warm Standby, create the following policy:

Allow group <group_name> to manage { POSTGRES_DB_SYSTEM_REPLICATE } in [ tenancy | compartment <compartment_name> | compartment id <compartment_ocid> ]

If you already have a policy granting management of postgres-db-systems at the database level, this policy isn't needed, because POSTGRES_DB_SYSTEM_REPLICATE is already included in the broader postgres-db-systems permissions.

Here is an example of the broader policy:

Allow group <group_name> to manage postgres-db-systems in [ tenancy | compartment <compartment_name> | compartment id <compartment_ocid> ]

Administrator Credentials

Note

Users and roles are replicated from the primary database to the warm standby database. The warm standby database can't have dedicated users or roles.

The admin username for the warm standby database is the same as the admin username defined during the creation of the primary database system.

Failover

Automatic failover isn't supported. You must perform failover manually by promoting the warm standby database system.

Recovery Point Objective (RPO)

Use Recovery Point Objective (RPO) to help limit potential data loss when using replication with warm standby. Enable or disable RPO enforcement depending on customer requirements, such as availability and data protection needs.

RPO is the amount of data, measured in time, that can be lost in the event of a disaster. RPO is calculated as the wall clock time difference between the timestamp of the most recent write on the database and the oldest WAL record that hasn't yet been replicated.

Following is more information about RPO.

  • Write-ahead logs (WAL) in the primary region are asynchronously replicated to the warm standby in the secondary region.

  • RPO tracks the replication lag, which represents how far the standby is behind the primary.

  • Supported and default values for the enforcement threshold:
    • Supported range: 300 seconds (5 minutes) to 10,800 seconds (3 hours).
    • Default value: 300 seconds (5 minutes).
  • When RPO enforcement is enabled:
    • OCI Database with PostgreSQL continuously evaluates RPO while committing a transaction.
    • When replication lag exceeds the configured RPO value:

      • The primary database system automatically switches to read-only mode.

      • This switch prevents further data changes and excessive data loss.

      • The warm standby is allowed to catch up, after which normal operations resume.

  • When RPO enforcement is disabled:

    • The database system continues normal operations even if replication lag exceeds the configured RPO.

    • The primary database doesn't switch to read-only mode.

Supported Database Versions for Warm Standby

The warm standby feature requires both the primary and warm standby database systems having database version of 14 or higher.

Configurations and Extensions

Note

Extensions installed on the primary database system must always be same or subset of those applied on the warm standby database system.

You can create configurations for the warm standby database system in the replica region. These configurations aren't automatically replicated from the primary to the warm standby database systems as part of Replication with Warm Standby setup.

Setting Up Replication with Warm Standby

Replication with Warm Standby provides resilience by maintaining a warm standby database in a secondary OCI region. This standby is continuously updated from the primary in the source region and can be promoted manually in the event of a failure.

This section describes the tasks required to set up this environment.

Creating the Primary Database System

  1. In the OCI Console, select the source region where you want the primary database system to reside in.
  2. Create a database system as described in Creating a Database System.
    Select Create New Database System as the creation type and continue with the creation steps.
  3. Wait until the Source Region Database System status shows Active.
At this stage, the database is a standalone database.

Creating the Warm Standby Database System

  1. In the OCI Console, select the secondary region where you want the warm standby database system to reside in.
  2. Create a database system as described in Creating a Database System.
  3. Select Create Replica Database System as the creation type.
    The Choose primary database system panel opens.
  4. Select source region under Region.
    All subscribed regions by the tenancy are listed.
  5. Select the primary database system you created under Database system in <compartment>.
    Check your compartment if you don't see the primary database.
  6. (Optional) Enable Enforce Recovery Point Objective (RPO) if your business needs require it.
    This feature isn't enabled by default.
  7. Complete the creation of the warm standby database system.
  8. Select Submit.
After you submit the warm standby database system for creation, it appears in the database systems list page as being the Creating state.

Verifying the Role Assignment

Following are Console instructions.
  1. After the warm standby database system you created is shown as Active in the database systems list page, select it.
    The warm standby database system's details page opens.
  2. Find the DB system role and confirm that its value is now Primary.
  3. Select Replica DB systems from the details page.
    The Replica DB system page opens. Confirm the warm standby database system you created is listed.
  4. Select the warm standby database system.
    Its details page opens.
  5. Find the DB system role and confirm that its value is Warm Standby.

List Replica Database Systems

In the Oracle Cloud Console, the associated replica database systems are displayed on the Primary Database System details page, where users can view the list of replicas linked to the primary database system along with their corresponding regions and identifiers.

Converting a Warm Standby Database System to Standalone

  1. Open the details page of the warm standby database system as described in Verifying the Role Assignment.
  2. Select Convert role to standalone from the Actions menu.
    The Convert role to standalone panel opens.
  3. Select a conversion option:
    • Replay Pending: The system applies all replicated write-ahead logs (WAL) into the warm standby database.
    • Immediate: Role change is performed instantly, without applying pending write-ahead logs (WAL).
  4. Select Submit.
Access the database system's details page to see the conversion status. Initially, the database system undergoing conversion displays the state as Updating. When the conversion completes successfully, the state displays Active. The DB system role shows Standalone.

The primary database system that had been associated with it changes automatically converts to standalone status as it's no longer part of a Replication with Warm Standby configuration.

Converting a Standalone Database System to Warm Standby

You can convert an existing PostgreSQL database system to warm standby and associate it with a primary database system in another region. The data in the existing database system is overwritten with a copy from the primary database system,
  1. In the OCI Console, select the source region where the primary database system you want to convert resides.
  2. Open the details page of the existing PostgreSQL database system you want to convert as described in Getting a Database System's Details.
  3. Select Convert role to replica from the Actions menu.
    The Convert role to replica panel opens.
  4. Select Create Replica Database System as the creation type.
    The Choose primary database system panel opens.
  5. Select source region under Region.
    All subscribed regions by the tenancy are listed.
  6. Select the primary database system you want under Database system in <compartment>.
    Check your compartment if you don't see the PostgreSQL database system you want to specify as the primary database.
  7. Select Submit.
After you submit the conversion, the database system appears in the database systems list page as being in the Updating state. After the conversion completes, it returns to the Active state in its new role as a warm standby database system.

Open the details page of the converted warm standby database system and confirm the DB system role value is Warm standby.

Open the details page for the database system you specified as the primary and confirm the DB system role value is Primary. Select Replica DB systems to open the Replica DB system panel and ensure the associated warm standby database you converted is listed.

Switching Roles Between Primary and Warm Standby Database Systems

A switchover allows you to safely switch roles between a primary database system and its warm standby database system without data loss. After the switchover, the warm standby database system becomes the new primary database system.
  1. Open the details page of your primary database system as described in Getting Database System Details.
  2. Select Switchover from the Actions menu.
    The Switchover panel opens.
  3. Select the warm standby database system that you want to promote to primary.
  4. Select Submit.
After you submit the switchover, the primary database system appears in the database systems list page as being the Updating state. After the switchover completes, it returns to the Active state in its new role as a warm standby database system. Similarly, the warm standby database system converts to primary.

Disaster Recovery

This section describes the actions to take when a primary region outage occurs and how to restore normal operations after the region is recovered.

In this disaster recovery scenario, the primary database system is in a source region such as PHX. The primary database system has Read/Write access. The warm standby database system is a replica region such as LHR. The warm standby database system has Read-only access, and is always synchronized with the primary database system using cross region replication.

Convert the Original Primary to Warm Standby

After the source region is restored, the original primary database (PHX) remains a standalone database system.

Update the role of the original primary database system (PHX) to a warm standby database system as described in Converting a Standalone Database System to Warm Standby. Specify the original warm standby database server (LHR, now converted to standalone) as the primary database system.

Now the primary database system is LHR with Read/Write access, and the warm standby database server is PHX with Read-only access.

Reestablish the Original Database Setup

Use the switchover function to return the current primary database system (LHR) back to the original primary (PHX) as described in Switching Roles Between Primary and Warm Standby Database Systems. After the switchover occurs, the updated primary is PHX and the updated warm standby is LHR. You can resume normal database operations with the primary in the source region and the warm standby in the replica region.

Applying RPO

You can enable RPO for a primary database system and specify the desired RPO value in the following ways.

  • When setting up Replication with Warm Standby

    Applying RPO at setup ensures that RPO enforcement is applied from the start and that any potential data loss remains within the configured threshold. If the configured RPO is exceeded, the primary database system might temporarily transition to read-only mode until the standby catches up.

    To enable RPO at setup, use the Choose primary database system panel. For more information, see Creating the Warm Standby Database System.

  • After setting up Replication with Warm Standby

    To enable (or disable) RPO after setup:

    1. Open the primary database's details page (as described in Getting a Database System's Details).
    2. Select Edit management policy from the Actions menu.
    3. In the Edit management policy panel, enable or disable RPO enforcement and configure the RPO value within the supported range.
    4. Save your changes.

    Your changes are immediately applied.

Metrics

Use metrics to monitor replication and RPO status for both your primary and warm standby database systems.

From the database system's details page, select Metrics to display the Metrics page. Following are included metrics.

Primary Database Metrics

  • Current RPO (seconds): Current RPO of the primary database system. Displays the current replication lag in seconds, indicating how far the standby is behind the primary.
  • Current transaction mode: Current transaction mode of the primary database system.
    • 1: Read/Write
    • 0: Read-only

      A value of 0 indicates that the database system has turned into read-only mode due to RPO enforcement.

Warm Standby Database Metrics

  • Last transaction replay epoch: Displays the time when last replayed transaction's write-ahead logs record was generated on the primary.
  • Last transaction replay lag: Displays the time lag in milliseconds between the epoch at which the last transaction was replayed on a warm standby database system and the epoch at which the same transaction was committed on the primary database system.

CLI Support

You can use CLI commands to perform Replication with Warm Standby tasks.

Create the Primary Database System

Switch to the region where you want your primary database system to reside and create the database using the CLI as described in Creating a Database System.

Create Warm Standby Database System

Switch to the region where you want the warm standby database system to reside. Use the following command to create the warm standby.

Update source database system details like sourceType and primaryDbSystemId with the appropriate values. For example:

oci raw-request --http-method POST --auth security_token --config-file ~/.oci/config --profile DEFAULT --target-uri 'https://postgresql.us-phoenix-1.oci.oraclecloud.com/20220915/dbSystems' \
  --request-body '{
  "displayName": "Warm Standby Database",
  "description": "Test db",
  "compartmentId": "ocid.compartment.oc1..exampleuniqueID",
  "dbVersion": "14",
  "storageDetails": {
    "systemType": "OCI_OPTIMIZED_STORAGE",
    "isRegionallyDurable": false,
    "availabilityDomain": "gXfg:UK-LONDON-1-AD-2"
  },
  "shape": "PostgreSQL.VM.Standard.E5.Flex",
  "instanceOcpuCount": 2,
  "instanceCount": 1,
  "instanceMemorySizeInGBs": 32,
  "networkDetails": {
    "subnetId": "ocid1.subnet..exampleuniqueID"
  },
  "source": {
    "sourceType": "DB_SYSTEM",
    "primaryDbSystemId": "ocid1.postgresqldbsystem..exampleuniqueID"
  }
}'

List Replica Database Systems

Use the following command to list warm standby database systems associated with the primary database system.

oci raw-request --http-method GET --target-uri 'https://postgresql.us-phoenix-1.oci.oraclecloud.com/20220915/dbSystems/<dbSystemId>/replicas' --region us-phoenix-1 --profile DEFAULT --config-file ~/.oci/config --auth security_token

where <dbSystemId> is the OCID of the primary database system.

Converting a Warm Standby Database System to Standalone

Use the following command to convert a warm standby database system to standalone.

oci raw-request --http-method POST --target-uri 'https://postgresql.us-phoenix-1.oci.oraclecloud.com/20220915/dbSystems/<warm_standby>/actions/changeRoleToStandalone' --region us-phoenix-1 --request-body '{"changeMode":"[immediately | "Replay Pending Updates"]"}'  --profile DEFAULT --config-file ~/.oci/config --auth security_token

where <warm_standby> is the warm standby database system.

Converting a Standalone Database System to Warm Standby

Use the following command to convert a standalone database system to warm standby.

oci raw-request --http-method POST --target-uri 'https://postgresql.us-phoenix-1.oci.oraclecloud.com/20220915/dbSystems/<warm_standby>/actions/changeRoleToReplica' --region us-phoenix-1 --request-body '{"primaryDbSystemId":"<primary>"}'  --profile DEFAULT --config-file ~/.oci/config --auth security_token

where <warm_standby> is the warm standby database system and <primary> is the primary database system.

Updating RPO

Use the following command to update the replication configuration (RPO settings) for a database system.

oci raw-request --http-method PUT --target-uri 'https://postgresql.us-phoenix-1.oci.oraclecloud.com/20220915/dbSystems/<dbSystemId>' --region us-phoenix-1 --request-body '{"replicationConfig": {"isRpoEnforced": true, "rpoInSeconds": 300}}' --profile DEFAULT --config-file ~/.oci/config --auth security_token

where <dbSystemId> is the OCID of the database system.

  • isRpoEnforced: Specifies whether RPO enforcement is enabled or disabled.

  • rpoInSeconds: Specifies the RPO value (in seconds) to enforce when enabled. The value must be between 300 and 10,800 seconds.

Switchover

Use the following command to perform a switchover between a standalone database system and a warm standby.

oci raw-request --http-method POST --target-uri 'https://postgresql.us-phoenix-1.oci.oraclecloud.com/20220915/dbSystems/<primary>/actions/switchover' --region us-phoenix-1 --request-body
'{"replicaDbSystemId":"<warm-standby>"}' --profile DEFAULT --config-file ~/.oci/config --auth security_token

where <primary> is the primary database system and <warm_standby> is the warm standby database system.

Restrictions

Restore Database Systems

Restore Database system isn't supported for Replication with Warm Standby database systems (primary & warm standby).