Managing Disaster Recovery Configurations

DR configurations specify the resources that play a critical role in protecting workloads against site-level incidents. These resources include compute instances with their associated block volumes, as well as the compartments they belong to and the network resources that provide their connectivity. Relevant network resources and compartment hierarchies must be set up on both the primary and standby rack, and associated with each other through site mappings.

Rules and Conditions

When populating DR configurations, respect the following rules regarding compute and storage resources.

A compute instance must be stopped before it can be added to a DR configuration. There is one exception: when all volumes attached to the instance are also attached to one or more instances already included in the same DR configuration.
A compute instance must be stopped before it can be removed from a DR configuration. There is one exception: when all volumes attached to the instance are also attached to one or more instances still included in the same DR configuration.
All compute instances in a DR configuration must be stopped before the DR configuration can be deleted.
A volume attached to a compute instance might be created from another source volume or volume backup. Such an instance (instance T) can be added to a DR configuration on condition that the source volume is not attached to any instance in any DR configuration. Note that source volume also refers to the volume used for the volume backup, and its direct or indirect source.

Alternatively, the instance with the source volume attached can be added to a DR configuration on condition that instance T is not added to any DR configuration. Due to the volume source/target relationship, only one of the instances involved can be part of a DR configuration, not both.
A DR configuration must be refreshed when configuration for any of its instances has changed, including changes for the instance and the attached storage or network resources. This is to ensure the instances on the standby rack after switchover or failover are started with needed attributes preserved.
Refreshing a DR configuration results in a failure in case a volume and the source from which it was created, are both attached to one or more compute instances in any DR configuration.
Instances in a DR configuration preserve the primary and secondary private IPs upon switchover or failover on condition that the subnet for the IP address is tagged appropriately. The subnet must be assigned a freeform tag, with the key preserve_private_ips and a value set to anything.

When this freeform tag is applied, any instance with a VNIC attached to this subnet preserves the associated primary and secondary IP addresses after switchover or failover. The precheck on the standby rack verifies whether the IP address is already in use, and fails if this is true.

Creating a DR Configuration

A DR configuration is the parent object to which you add cloud resources that you want to protect against system outages. You start by creating an empty DR configuration with an associated ZFS project on the ZFS Storage Appliance.

If your system contains optional high-performance disk shelves, two ZFS projects are created for each DR configuration: one in the standard pool and one in the high-performance pool. When you add instances to the DR configuration that have disks running on standard as well as high-performance storage, those storage resources are automatically added to the ZFS project in the appropriate pool.

Using the Service CLI

Gather the information that you need to run the command:
- a unique name for the DR configuration
- a unique name for the associated ZFS storage project

Create an empty DR configuration with the create DrConfig command.

Syntax (entered on a single line):

create DrConfig 
configName=<DR_configuration_name>
zfsProjectName=<ZFS_storage_project_name>

Example:

PCA-ADMIN> create DrConfig configName=mydrconf1 zfsProjectName=mydrconf1-project
JobId: 16047c38-1a7e-48ac-9588-297b915d49fc
Data: DrConfig id: d3cd87de-afd4-4718-a3e6-1105b56b42d8. Successfully started job to create config mydrconf1

Use the job ID to check the status of the operation you started.

PCA-ADMIN> show Job id=16047c38-1a7e-48ac-9588-297b915d49fc
Data:
  Id = 16047c38-1a7e-48ac-9588-297b915d49fc
  Type = Job
  Associated Work Request Id = 20d20925-39d1-4f9a-beef-5e9d3ca465d5
  Done = true
  Name = CREATE_TYPE
  Progress Message = DrConfig id: d3cd87de-afd4-4718-a3e6-1105b56b42d8. Successfully created DR config mydrconf1: d3cd87de-afd4-4718-a3e6-1105b56b42d8
  Run State = Succeeded
  Transcript = Created job CREATE_TYPE
  Username = admin
  WorkItemIds 1 = id:e82eeec2-7881-45c5-ae84-6a3894ef66c6  type:WorkItem  name:

When the DR configuration is created, the storage project for data replication is set up on the ZFS Storage Appliances. Note the DR configuration ID. You need it for all subsequent commands to modify the configuration.

To display a list of existing DR configurations, use the drGetConfigs command.

PCA-ADMIN> drGetConfigs
Data:
  id                                     Config Name
  --                                     -----------
  d3cd87de-afd4-4718-a3e6-1105b56b42d8   mydrconf1
  e8291afa-a413-4932-880a-abb8ac22c85d   mydrconf2
  7ad05d9f-731c-41b8-b477-35da4b999071   mydrconf3

To display the status and details of a DR configuration, use the show DrConfig command.

PCA-ADMIN> show DrConfig id=d3cd87de-afd4-4718-a3e6-1105b56b42d8
Data:
  Id = d3cd87de-afd4-4718-a3e6-1105b56b42d8
  Type = DrConfig
  Config State = Enabled
  Config Name = mydrconf1
  Config Id = d3cd87de-afd4-4718-a3e6-1105b56b42d8
  Zfs Project Name = mydrconf1-project
  Message = Successfully retrieved config data
  Message = Successfully retrieved config data
  Replica State = OK
  Replica Lag In Seconds = 3

Using the Service Web UI

Under Disaster Recovery Service, open the DR Configurations page. In the top-right corner, click Create Configuration.
In the Create Configuration window, enter the following information:
- a unique name for the DR configuration
- a unique name for the associated ZFS storage project
Click Submit. A new, empty DR configuration appears in the table.

Next, you add site mappings and instances to the DR configuration.

About the DR Configuration State

The normal working state of a DR configuration is "Enabled". However, as part of the process of DR plan execution, the state of the DR configurations changes. In general, an administrator is not expected to manually change the state, but it might be useful, and it is possible to temporarily disable a DR configuration.

The following configuration states occur:

Enabled: The DR configuration is in active working state. The mappings and instances can be modified, and DR plans can be modified and executed.
Disabled: The DR configuration is locked. The mappings and instances cannot be modified, and DR plans cannot be created, edited, or executed.
Frozen: The DR configuration is locked, like in the disabled state. It is no longer usable because it was migrated to the standby appliance.

Maintaining Site Mappings

Site mappings determine how and where on the standby system the instances should be brought back up in case the primary system experiences an outage. Each site mapping consists of a source object on the primary system and a corresponding target object on the standby system. Ensure that these resources exist on both systems before you add the site mappings to the DR configuration.

Adding Site Mappings

These are the site mapping types you can include in a DR configuration:

Compartment: specifies that, if a failover or switchover occurs, instances from the source compartment must be brought up in the target compartment on the standby system
Subnet: specifies that, if a failover or switchover occurs, instances connected to the source subnet must be connected to the target subnet on the standby system

Using the Service CLI

Gather the information that you need to run the command:
- DR configuration ID (drGetConfigs)
- Mapping source and target object OCIDs
  
  Use the Compute Enclave UI or CLI on the primary and standby system respectively. CLI commands:
  - oci iam compartment list
  - oci network subnet list --compartment-id "ocid1.compartment.....uniqueID"

Add a site mapping to the DR configuration with the drAddSiteMapping command.

Syntax (entered on a single line):

drAddSiteMapping 
drConfigId=<DR_configuration_id>
objType=[compartment | subnet]
sourceId=<source_object_OCID>
targetId=<target_object_OCID>

Examples:

PCA-ADMIN> drAddSiteMapping drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8 objType=compartment \
sourceId=ocid1.compartment....<region1>....uniqueID targetId=ocid1.compartment....<region1>....uniqueID
Data:
  Message = Successfully added site mapping to DR config [mydrconf1]
  Returned Object Id = 77233b2c-aa4a-4314-8196-4d8dd1d20721

PCA-ADMIN> drAddSiteMapping drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8 objType=subnet \
sourceId=ocid1.subnet....<region1>....uniqueID targetId=ocid1.subnet....<region2>....uniqueID
Data:
  Message = Successfully added site mapping to DR config [mydrconf1]
  Returned Object Id = 4467bb39-7256-4715-9a01-bedd25f73a82

Repeat the command with the OCIDs of all the source and target objects that you want to include in the site mappings of the DR configuration.

To display the list of site mappings included in the DR configuration, use the drGetSiteMappings command. The DR configuration ID is a required parameter.

Syntax:

PCA-ADMIN> drGetSiteMappings drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8
Data:
  id                                     Object Type
  --                                     -----------
  77233b2c-aa4a-4314-8196-4d8dd1d20721   compartment
  6ef9e3b6-6704-4260-b345-f55e364ba9e9   compartment
  4467bb39-7256-4715-9a01-bedd25f73a82   subnet
  096a5499-9720-4a4f-a8be-ce055f0efd5d   subnet

To display the status and details of a site mapping included in the DR configuration, use the drGetSiteMapping command. The DR configuration ID and site mapping ID are required parameters.

PCA-ADMIN> drGetSiteMapping drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8 mappingId=4467bb39-7256-4715-9a01-bedd25f73a82
Data:
  Type = DrSiteMapping
  Mapping Id = 4467bb39-7256-4715-9a01-bedd25f73a82
  Object Type = subnet
  Source Id = ocid1.subnet1....<region1>....uniqueID
  Target Id = ocid1.subnet1....<region2>....uniqueID
  Message = Successfully retrieved mapping data

Using the Service Web UI

Collect the OCIDs of the source and target objects you are going to map. Use the Compute Enclave UI or CLI on the primary and standby system respectively. CLI commands:
- oci iam compartment list
- oci network subnet list --compartment-id "ocid1.compartment.....uniqueID"
Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration to which you want to add site mappings. The DR Configuration detail page appears.
In the Resources section, click Site Mappings.

On the right hand side of the Site Mappings box, click Add Site Mapping.
In the Add Site Mapping window, enter the following information in the respective fields:
- Object Type: Specify which resource type you are mapping. Enter either compartment or subnet.
- Target ID: Enter the OCID of the target object on the remote system.
- Source ID: Enter the OCID of the source object on the local system.
Click Add Site Mapping. The new site mapping appears in the resources table.

Optionally, click the site mapping to display its detail page.
Repeat these steps to add all the required site mappings.

As cloud resources across the environment are added and removed over time, verify on a regular basis that the site mappings in the DR configuration are up-to-date.

Removing Site Mappings

You can remove a site mapping from the DR configuration if it is no longer required.

Using the Service CLI

Gather the information that you need to run the command:
- DR configuration ID (drGetConfigs)
- Site mapping ID (drGetSiteMappings)

Remove the selected site mapping from the DR configuration with the drRemoveSiteMapping command.

Syntax (entered on a single line):

drRemoveSiteMapping 
drConfigId=<DR_configuration_id>
mappingId=<site_mapping_id>

Example:

PCA-ADMIN> drRemoveSiteMapping drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8 mappingId=1ce736ef-13f9-40a9-aa6b-3e8960261224
Data:
  Message = Successfully removed mapping [1ce736ef-13f9-40a9-aa6b-3e8960261224] from DR config [mydrconf1]

Repeat the command with the IDs of all the site mappings that you want to remove from the DR configuration.

Using the Service Web UI

Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration for which you want to edit site mappings. The DR Configuration detail page appears.
In the Resources section, click Site Mappings. All existing site mappings are displayed in the table.
In the Actions column, open the quick menu (3 dots) and click Remove Site Mapping.

Alternatively, click the site mapping to display its detail page. In the top-right corner, click Remove Site Mapping.
Repeat until all obsolete site mappings have been removed.

As cloud resources across the environment are added and removed over time, verify on a regular basis that the site mappings in the DR configuration are up-to-date.

Adding and Removing Compute Instances

When the relevant site mappings have been set up in the DR configuration, you add the required compute instances. Their data and disks are stored in the ZFS storage project associated with the DR configuration, and replicated over the peer connection between the Private Cloud Appliance systems.

Adding Instances

Compute instances must be stopped before you add them to a DR configuration. Instances can be added one at a time, per compartment, or all at once across all compartments.

Note:

In the Service Web UI, compute instances must be added one by one.

Using the Service CLI

Gather the information that you need to run the command:
- DR configuration ID (drGetConfigs)
- Instance OCIDs from the Compute Enclave UI or CLI (oci compute instance list --compartment-id <compartment_OCID>)

Add compute instances to the DR configuration with the drAddComputeInstance command.

Syntax (entered on a single line):

drAddComputeInstance 
drConfigId=<DR_configuration_id> 
instanceId="<instance-1_OCID>","<instance-n_OCID>"
compartmentId="<compartment-1_OCID>","<compartment-1_OCID>"
all=True

Note:

When selecting "all", the instance and compartment parameters cannot be included in the command.

Examples:

individual instances: single instance or comma-separated list

PCA-ADMIN> drAddComputeInstance drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8 instanceId="ocid1.instance1....uniqueID","ocid1.instance2....uniqueID"
Data:
  Message = Successfully started job to add instance to config d3cd87de-afd4-4718-a3e6-1105b56b42d8
  Job Id = dc0787ef-c254-4d91-b7dc-ca4d2412f461

all instances in a compartment: single compartment or comma-separated list

PCA-ADMIN> drAddComputeInstance drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8 compartmentId="ocid1.compartment1....uniqueID","ocid1.compartment2....uniqueID"
Data:
  Message = Successfully started job to add instance to config d3cd87de-afd4-4718-a3e6-1105b56b42d8
  Job Id = 3801d0be-f2ab-48d5-ad12-cd8afd444bc2

all instances across all mapped compartments

PCA-ADMIN> drAddComputeInstance drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8 all=True
Data:
  Message = Successfully started job to add instance to config d3cd87de-afd4-4718-a3e6-1105b56b42d8
  Job Id = d0a11091-f202-4350-b59a-f0cddc9fb035

Use the job ID to check the status of the operation you started.

PCA-ADMIN> drGetJob jobId=3801d0be-f2ab-48d5-ad12-cd8afd444bc2
Data:
  Type = add_computeinstance
  Job Id = 3801d0be-f2ab-48d5-ad12-cd8afd444bc2
  Status = finished
  Result = success
  Message = job successfully retrieved
  Response = Successfully added the following instances to mydrconf1 ['ocid1.instance1....uniqueID','ocid1.instance2....uniqueID']

Repeat the drAddComputeInstance command to add all the required compute instances to the DR configuration.

To display the list of instances included in the DR configuration, use the drGetComputeInstances command. The DR configuration ID is a required parameter.

PCA-ADMIN> drGetComputeInstances drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8
Data:
  id
  --
  ocid1.instance1....uniqueID
  ocid1.instance2....uniqueID
  ocid1.instance3....uniqueID
  ocid1.instance4....uniqueID

To display the status and details of an instance included in the DR configuration, use the drGetComputeInstance command. The DR configuration ID and instance ID are required parameters.

PCA-ADMIN> drGetComputeInstance drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8 instanceId=ocid1.instance1....uniqueID
Data:
  Compartment Id = ocid1.compartment....uniqueID
  Boot Volume Id = ocid1.bootvolume....uniqueID
  Compute Instance Shape = VM.PCAStandard1.Flex
  Message = Successfully retrieved instance data

Using the Service Web UI

Collect the OCIDs of the compute instances that need to be added to the DR configuration. Use the Compute Enclave UI, or these CLI commands:
- oci iam compartment list
- oci compute instance list --compartment-id <compartment_OCID>
Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration to which you want to add compute instances. The DR Configuration detail page appears.
In the Resources section, click Compute Instances.

On the right hand side of the Compute Instances box, click Add Instance.
In the Add Compute Instances window, select compute instances using one of these methods:
- By instance OCID
  
  Add a single instance by entering its OCID. Add multiple instances by entering their OCIDs as a comma-separated list.
- By compartment OCID
  
  Add all instances in a compartment by entering the compartment OCID. Add instances from multiple compartments by entering the compartment OCIDs as a comma-separated list.
- All
  
  Add all instances from all mapped compartments.
Click Submit. A DR job is started. When it completes successfully, the instances appear in the resources table.

To track progress, under Disaster Recovery Service, select Jobs. The Jobs table reports the status of each job. Click a record in the table to display the job details.
Check regularly for instances requiring DR protection, and repeat these steps whenever necessary.

As cloud resources across the environment are added and removed over time, verify on a regular basis that the list of compute instances in the DR configuration is up-to-date.

Removing Instances

Instances can only be part of a single DR configuration. You can remove a compute instance from the DR configuration to which it was added.

Using the Service CLI

Gather the information that you need to run the command:
- DR configuration ID (drGetConfigs)
- Instance OCID (drGetComputeInstances)

Remove the selected compute instances from the DR configuration with the drRemoveComputeInstance command.

Syntax (entered on a single line):

drRemoveComputeInstance 
drConfigId=<DR_configuration_id> 
instanceId=<instance_OCID>
compartmentId=<compartment_OCID>
all=True

Note:

When selecting "all", the instance and compartment parameters cannot be included in the command.

Examples:

single instance

PCA-ADMIN> drRemoveComputeInstance drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8 instanceId=ocid1.instance....uniqueID
Data:
  Message = Successfully started job to remove instance(s) from DR config d3cd87de-afd4-4718-a3e6-1105b56b42d8
  Job Id = 20f6735e-7702-4d27-ba19-41cf08a86d90

all instances in a compartment

PCA-ADMIN> drRemoveComputeInstance drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8 compartmentId=ocid1.compartment....uniqueID
Data:
  Message = Successfully started job to remove instance(s) from DR config d3cd87de-afd4-4718-a3e6-1105b56b42d8
  Job Id = c4b0ff83-45e3-4498-98bb-e8a2f6b904e8

all instances in the DR configuration

PCA-ADMIN> drRemoveComputeInstance drConfigId=d3cd87de-afd4-4718-a3e6-1105b56b42d8 all=True
Data:
  Message = Successfully started job to remove instance(s) from DR config d3cd87de-afd4-4718-a3e6-1105b56b42d8
  Job Id = 0a2cdc62-1d13-4774-b3c6-d2a30b75db5e

Use the job ID to check the status of the operation you started.

PCA-ADMIN> drgetjob jobId=20f6735e-7702-4d27-ba19-41cf08a86d90
Data:
  Type = remove_computeinstance
  Job Id = 20f6735e-7702-4d27-ba19-41cf08a86d90
  Status = finished
  Result = success
  Message = job successfully retrieved
  Response = Successfully removed the following instances from mydrconf1 ['ocid1.instance....uniqueID']

Repeat the drRemoveComputeInstance command to remove all the required compute instances from the DR configuration.

Using the Service Web UI

Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration from which you want to remove compute instances. The DR Configuration detail page appears.
In the Resources section, click Compute Instances. All instances included in the DR configuration are displayed in the table.
To quickly remove a single instance, in the Actions column, open the quick menu (3 dots) and click Delete. To remove multiple instances, proceed to the next step.
Open the Controls menu in the top-right corner of the DR Configuration detail page and click Remove Instances.
In the Remove Compute Instances window, select compute instances for removal using one of these methods:
- By instance OCID
  
  Remove a single instance by entering its OCID. Remove multiple instances by entering their OCIDs as a comma-separated list.
- By compartment OCID
  
  Remove all instances in a compartment by entering the compartment OCID. Remove instances included from multiple compartments by entering the compartment OCIDs as a comma-separated list.
- All
  
  Remove all instances included from all mapped compartments.
Click Submit. A DR job is started. When it completes successfully, the instances are removed from the resources table.

To track progress, under Disaster Recovery Service, select Jobs. The Jobs table reports the status of each job. Click a record in the table to display the job details.
Check regularly for instances that no longer require DR protection, and repeat these steps whenever necessary.

As cloud resources across the environment are added and removed over time, verify on a regular basis that the list of compute instances in the DR configuration is up-to-date.

Refreshing a DR Configuration

To ensure that the replication information stored in a DR configuration is updated with all the latest changes in your environment, you must refresh the DR configuration after making changes to the protected resources. Refreshing the DR configuration is important for the continuation of service after a DR plan is executed, because it keeps the relevant metadata up-to-date with the current status of the resources under DR protection.

Using the Service CLI

Look up the ID of the DR configuration you want to refresh (drGetConfigs).

Refresh the data stored in the selected DR configuration with the drRefreshConfig command.

PCA-ADMIN> drRefreshConfig drConfigId=b51a51ac-043e-4b6d-ab29-20f86d905e81
Data:
  Message = Successfully started job to refresh DR config b51a51ac-043e-4b6d-ab29-20f86d905e81
  Job Id = ed3a509e-4ae9-4314-9ec5-fb12ff78615a

Use the job ID to check the status of the operation you started.

PCA-ADMIN> drGetJob jobId=ed3a509e-4ae9-4314-9ec5-fb12ff78615a
Data:
  Type = refresh_config
  Job Id = ed3a509e-4ae9-4314-9ec5-fb12ff78615a
  Status = finished
  Result = success
  Message = job successfully retrieved
  Response = Successfully refreshed DR config [mydrconf1]

Using the Service Web UI

Under Disaster Recovery Service, open the DR Configurations page.
In the table, click the DR configuration to open its detail page.
In the top-right corner, select Controls, then click Refresh Configuration.
A DR job is started. When it completes successfully, the DR configuration has been refreshed.

To track progress, under Disaster Recovery Service, select Jobs. The Jobs table reports the status of each job. Click a record in the table to display the job details.

Deleting a DR Configuration

When you no longer need a DR configuration, you can remove it with a single command. However, you must stop all compute instances that are part of the DR configuration before you can delete it. Deleting a DR configuration also removes all site mappings and cleans up the associated storage projects on the local ZFS Storage Appliance.

If required, the storage projects replicated on the standby ZFS Storage Appliance can be cleaned up manually. If the native DR service is unconfigured on both systems, all related storage project on both systems are cleaned up automatically as part of the process.

Using the Service CLI

Stop all the compute instances that are part of the DR configuration you want to delete.
Look up the ID of the DR configuration you want to delete (drGetConfigs).

Delete the selected DR configuration with the drDeleteConfig command.

PCA-ADMIN> delete DrConfig id=d3cd87de-afd4-4718-a3e6-1105b56b42d8
JobId: a687aafa-7c5c-4a68-99da-62e9384140be
Data: Successfully started job to delete config d3cd87de-afd4-4718-a3e6-1105b56b42d8

Use the job ID to check the status of the operation you started.

PCA-ADMIN> show Job id=a687aafa-7c5c-4a68-99da-62e9384140be
Data:
  Id = a687aafa-7c5c-4a68-99da-62e9384140be
  Type = Job
  Associated Work Request Id = e5f77289-b263-4a5c-9163-7ce4629cc550
  Done = true
  Name = DELETE_TYPE
  Progress Message = Successfully deleted DR config [mydrconf1]
  Run State = Succeeded
  Transcript = Created job DELETE_TYPE
  Username = admin
  WorkItemIds 1 = id:622a4f37-9222-4ea4-af2c-351cf28e739b  type:WorkItem  name:

Using the Service Web UI

Under Disaster Recovery Service, open the DR Configurations page. All configurations are displayed in the table.
In the Actions column, open the quick menu (3 dots) and click Delete.

Alternatively, click the DR configuration to display its detail page. In the top-right corner, select Controls, then click Delete.
When prompted, click Confirm. A DR job is started. When it completes successfully, the DR configuration is removed from the table.

To track progress, under Disaster Recovery Service, select Jobs. The Jobs table reports the status of each job. Click a record in the table to display the job details.