Working With Disaster Recovery Plans

A DR plan describes the operations that must be performed on the resources that are under the protection of the disaster recovery service. A DR plan is associated with a DR configuration, and is executed by an administrator either when a site-level incident is detected (failover), or when one of the sites must be taken offline (switchover). After a failover, when the affected system is back online, postfailover operations are performed to ensure that both systems are ready to run new DR operations.

About DR Operations and Default Plans

The native DR service provides plans with default steps for each type of operation. DR plan steps can be customized. The built-in plans are configured as follows:

Switchover Plan

When a switchover is performed, there is no outage, so both peered systems are online. The goal is to move all resources covered in the DR configuration from the primary system (A) to the standby system (B). When completed, system B becomes the primary and system A the standby for the resources in question.

The plan starts with prechecks to ensure that both systems meet the requirements to allow compute instances to be stopped on the primary system and started again on the standby system. The prechecks include site mappings as well as other critical elements, such as tags, security lists, or network security groups. The role reversal precheck specifically ensures that the ZFS Storage Appliance in each rack is in the correct state.

When the prechecks are completed without errors, the DR configuration on the primary system (A) is frozen and its compute instances are stopped, so the role reversal can begin. Based on resource metadata exchanged between the peered systems, and replicated data on the standby ZFS Storage Appliance, the target system (B) is prepared to assume the primary role for the instances in the DR configuration. The replication process is reversed and ready to use the source system (A) as its standby as soon as the switchover is complete.

Using the replicated volumes, the compute instances in the DR configuration are launched on the standby system (B). An identical DR configuration is created on the standby system, with all source and target resources in the site mappings inverted. The metadata of the newly launched instances is stored in the DR configuration. On the primary system (A) a cleanup is performed: the DR configuration is disabled and its compute instances are terminated.

To complete the switchover, data replication from the new primary system (B) to the standby system (A) is started, the DR plans are moved to the new standby system (A), and the storage project and metadata associated with the original DR configuration are deleted from system A.

Failover Plan

A failover is performed on the standby system, when one of the peered systems goes down. The goal is to recover all resources covered in the DR configuration on the standby system (B), allowing continuation of service. The failover steps are similar to the switchover plan, but none of the operations on the primary system (A) can be performed. The primary system cannot be cleaned up until it comes back online.

The plan starts with prechecks to ensure that the standby system and its ZFS Storage Appliance are in the correct state to bring up the resources covered in the DR configuration. When the prechecks are completed without errors, the role reversal begins.

Using the replicated metadata and resources, the compute instances in the DR configuration are launched on the standby system (B), which assumes the primary role. An identical DR configuration is created on system B, which has become the primary, with inverted site mappings and metadata collected from the newly launched instances. In preparation of the original primary system (A) coming back online, the replication process is reversed and ready to use system A as the standby.

When the original primary system (A) eventually comes online, the remaining steps to return the DR configuration to a correct working state are performed by executing the postfailover plan.

Postfailover Plan

A postfailover plan is performed after a failover, when the system that experienced an outage comes back online, and the peer connection is restored. The goal is to clean up the DR configuration on the primary system that went down (A), and set it up as the standby for the new primary system (B).

There are no prechecks in a postfailover plan. System A is back online after an outage and needs to be cleaned up: the DR configuration is disabled and its compute instances are terminated. Data replication from the new primary system (B) to the standby system (A) is started, the DR plans are moved to the new standby system (A), and the storage project and metadata associated with the original DR configuration are deleted from system A.

To move resources that were originally hosted on system A back from system B, the administrator must perform a switchover from B to A for the relevant DR configuration(s).

Creating and Maintaining DR Plans

Each DR plan defines the steps to perform when a DR operation is executed: failover, switchover, or postfailover. Each DR configuration has an associated DR plan file. This file can contain multiple DR plans, each identified by a unique name assigned by the administrator during creation. The DR plan file is stored in JSON format in the directory /mnt/dr_metadata on the standby rack.

Creating a DR Plan

The DR service provides default plans for each standard operation. You create them with a simple command to get started. If necessary, you can customize the plan steps afterward.

Using the Service CLI

  1. Look up the ID of the DR configuration for which you want to create a DR plan (drGetConfigs).

  2. Create a default DR plan with the create DrPlan command.

    Typically, a DR configuration has associated DR plans for each operation type. In this example, plans are created for failover, switchover, and postfailover.

    PCA-ADMIN> create DrPlan drConfigId=6e797d8b-7245-4d49-8e68-bf67f2d53041 operation=SWITCHOVER planName=sw1
JobId: eae66f69-7b99-420e-b324-7d8964b2202b
Data: DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1. Successfully started job for DR Plan Create for config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041

PCA-ADMIN> create DrPlan drConfigId=6e797d8b-7245-4d49-8e68-bf67f2d53041 operation=FAILOVER planName=fo1

PCA-ADMIN> create DrPlan drConfigId=6e797d8b-7245-4d49-8e68-bf67f2d53041 operation=POSTFAILOVER planName=pfo1

  3. Use the job ID to check the status of the operation you started.

    PCA-ADMIN> show Job id=eae66f69-7b99-420e-b324-7d8964b2202b
Data:
  Id = eae66f69-7b99-420e-b324-7d8964b2202b
  Type = Job
  Associated Work Request Id = ec0f39df-6256-4c4c-a839-0d00a8f326dc
  Done = true
  Name = CREATE_TYPE
  Progress Message = DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1. Successfully created [sw1] plan on STANDBY for DR operation [switchover] for config [6e797d8b-7245-4d49-8e68-bf67f2d53041]
  Run State = Succeeded
  Transcript = Created job CREATE_TYPE
  Username = admin
  WorkItemIds 1 = id:5ca6d187-e01a-40e2-bc97-3193a9a88742  type:WorkItem  name:

  4. To display a list of existing DR plans for a DR configuration, use the drListPlan command. 

    PCA-ADMIN> drListPlan drConfigId=6e797d8b-7245-4d49-8e68-bf67f2d53041
Data:
  id                                           Plan Name   Operation
  --                                           ---------   ---------
  6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1    sw1         switchover
  6e797d8b-7245-4d49-8e68-bf67f2d53041::fo1    fo1         failover
  6e797d8b-7245-4d49-8e68-bf67f2d53041::pfo1   pfo1        postfailover

  5. To display the status and details of a DR plan, use the show DrPlan command. 

    PCA-ADMIN> show DrPlan id=6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1
Data:
  Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1
  Type = DrPlan
  Plan Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1
  Plan Name = sw1
  Config Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041
  Operation = switchover
  Steps 1 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::precheck
  Steps 1 - Step Name = PRECHECK
  Steps 1 - Enabled = true
  Steps 1 - Last Status = norun
  Steps 1 - Command = None
  Steps 1 - Check Only = true
  Steps 2 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::role_reversal_precheck
  Steps 2 - Step Name = ROLE_REVERSAL_PRECHECK
  Steps 2 - Enabled = true
  Steps 2 - Last Status = norun
  Steps 2 - Command = None
  Steps 2 - Check Only = true
  Steps 3 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::stop_primary
  Steps 3 - Step Name = STOP_PRIMARY
  Steps 3 - Enabled = true
  Steps 3 - Last Status = norun
  Steps 3 - Command = None
  Steps 3 - Check Only = false
  Steps 4 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::role_reversal
  Steps 4 - Step Name = ROLE_REVERSAL
  Steps 4 - Enabled = true
  Steps 4 - Last Status = norun
  Steps 4 - Command = None
  Steps 4 - Check Only = false
  Steps 5 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::start_standby
  Steps 5 - Step Name = START_STANDBY
  Steps 5 - Enabled = true
  Steps 5 - Last Status = norun
  Steps 5 - Command = None
  Steps 5 - Check Only = false
  Steps 6 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::cleanup_primary
  Steps 6 - Step Name = CLEANUP_PRIMARY
  Steps 6 - Enabled = true
  Steps 6 - Last Status = norun
  Steps 6 - Command = None
  Steps 6 - Check Only = false
  Steps 7 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::post_config
  Steps 7 - Step Name = POST_CONFIG
  Steps 7 - Enabled = true
  Steps 7 - Last Status = norun
  Steps 7 - Command = None
  Steps 7 - Check Only = false
Using the Service Web UI

  1. Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration to which you want to add a DR plan. The DR Configuration detail page appears.

  2. In the Resources section, click Plans.

    On the right hand side of the Plans box, click Add Plan.

  3. In the Create DR Plan window, enter the following information in the respective fields:

    • Plan Name: Enter a name for this DR plan.

    • Operation: Select a standard operation: switchover, failover, postfailover.

    • Steps: Leave blank if you want the default steps for the selected operation. Otherwise, specify the steps to add to the DR plan. The options are: 'precheck', 'role_reversal_precheck', 'stop_primary', 'role_reversal', 'start_standby', 'cleanup_primary', 'post_config'.

    • Dr Plan Data File: Used for migrating first-generation DR plans. Leave blank.

    • Configurations To Migrate: Used for migrating first-generation DR plans. Leave blank.

  4. Click Create DR Plan. A DR job is started. When it completes successfully, the DR plan appears in the resources table.

    To track progress, under Disaster Recovery Service, select Jobs. The Jobs table reports the status of each job. Click a record in the table to display the job details.

  5. Repeat these steps to add all the required DR plans.

  6. Optionally, click a DR plan name to display its detail page.

Changing a DR Plan

A DR plan has a complex structure with many parameters, which makes it difficult to update from the command line in particular. Instead, you can change the individual steps that make up the DR plan. See Customizing the Steps in a DR Plan.

Deleting a DR Plan

You can remove a DR plan if it is no longer required. To delete all plans associated with a DR configuration at once, use the command drDeleteAllPlans.

Using the Service CLI

  1. Look up the ID of the DR plan you want to delete (drListPlan).

  2. Remove the selected DR plan with the delete DrPlan command. 

    PCA-ADMIN> delete DrPlan id=6e797d8b-7245-4d49-8e68-bf67f2d53041::fo2
JobId: 603d480f-1e0f-4229-b596-aaaf8588e682
Data: DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::fo2. Successfully started job for DR Plan delete for config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041

  3. Use the job ID to check the status of the operation you started.

    PCA-ADMIN> show Job id=603d480f-1e0f-4229-b596-aaaf8588e682
Data:
  Id = 603d480f-1e0f-4229-b596-aaaf8588e682
  Type = Job
  Associated Work Request Id = 391a0799-235a-4b26-aa99-4b5dd14ba19a
  Done = true
  Name = DELETE_TYPE
  Progress Message = DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::fo2. Successfully deleted [fo2] plan on STANDBY for config [6e797d8b-7245-4d49-8e68-bf67f2d53041]
  Run State = Succeeded
  Transcript = Created job DELETE_TYPE
  Username = admin
  WorkItemIds 1 = id:86d0bd13-5f9c-4513-9404-60d8980b2243  type:WorkItem  name:
Using the Service Web UI

  1. Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration for which you want to delete a DR plan. The DR Configuration detail page appears.

  2. In the Resources section, click Plans. All existing DR plans are displayed in the table.

  3. In the Actions column, open the quick menu (3 dots) and click Delete.

    Alternatively, click the DR plan name to display its detail page. In the top-right corner, click Delete.

  4. When prompted, click Confirm. A DR job is started. When it completes successfully, the DR plan is removed from the table.

    To track progress, under Disaster Recovery Service, select Jobs. The Jobs table reports the status of each job. Click a record in the table to display the job details.

  5. Repeat until all obsolete DR plans have been removed.

Deleting All DR Plans

Instead of deleting DR plans one by one, you can delete all plans associated with a DR configuration at once.

Using the Service CLI

  1. Look up the ID of the DR configuration for which you want to delete all plans (drGetConfigs).

  2. Remove the selected DR plan with the drdeleteallPlans command. 

    PCA-ADMIN> drdeleteallPlans drConfigId=6e797d8b-7245-4d49-8e68-bf67f2d53041
JobId: b595dd62-8046-4ca7-90a0-dcbbf084e663
Data: DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::all_plans. Successfully started job for DR Plan delete for config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041

  3. Use the job ID to check the status of the operation you started.

    PCA-ADMIN> show Job id=b595dd62-8046-4ca7-90a0-dcbbf084e663
Data:
  Id = b595dd62-8046-4ca7-90a0-dcbbf084e663
  Type = Job
  Associated Work Request Id = d7cfb184-4e38-469d-b189-bb809386f5d4
  Done = true
  Name = DELETE_TYPE
  Progress Message = DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::all_plans. Successfully deleted [all_plans] plan on STANDBY for config [6e797d8b-7245-4d49-8e68-bf67f2d53041]
  Run State = Succeeded
  Transcript = Created job DELETE_TYPE
  Username = admin
  WorkItemIds 1 = id:6cfec1c9-4a72-492d-a414-38bf4df6cf3a  type:WorkItem  name:
Using the Service Web UI

  1. Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration for which you want to delete a DR plan. The DR Configuration detail page appears.

  2. In the Resources section, click Plans. All existing DR plans are displayed in the table.

    Verify that all plans in the table should be deleted.

  3. In the top-right corner, select Controls, then click Delete All Plans.

  4. When prompted, click Confirm. A DR job is started. When it completes successfully, all DR plan are removed from the table.

    To track progress, under Disaster Recovery Service, select Jobs. The Jobs table reports the status of each job. Click a record in the table to display the job details.

Customizing the Steps in a DR Plan

Customizing DR plan steps is the most convenient way to change a DR plan with default configuration. The administrator can enable or disable steps, configure steps to run in check-only mode, insert steps, and remove steps.

Note:

Custom steps and custom commands are not available in appliance software version 3.0.2-b1261765.

Changing the Properties of a DR Plan Step

You can enable or disable an existing DR plan step, and decide whether it should be performed in check-only mode.

Using the Service CLI

  1. Look up the ID of the DR plan step you want to change (show DrPlan). If you don't have the DR plan ID, look it up using the commands drGetConfigs and drListPlan.

  2. Change the DR plan step with the edit DrPlanStep command. These are the properties you can change in order to affect how a DR plan is executed:

    • enabled=[True|False] – determines whether this step is performed or not

    • checkOnly=[True|False] – determines whether this step is performed in check-only mode

    • command=[string] – (NOT available) specifies a custom command to be run as part of this step

      For standard DR plan operations, the command parameter must be set to None. 

    PCA-ADMIN> edit DrPlanStep id=6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::precheck checkOnly=False
JobId: d281141c-c388-490e-b038-239598488bc6
Data: DrPlanStep id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::precheck. Successfully started job for DR Plan Step update for config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041

  3. Use the job ID to check the status of the operation you started.

    PCA-ADMIN> show Job id=d281141c-c388-490e-b038-239598488bc6
Data:
  Id = d281141c-c388-490e-b038-239598488bc6
  Type = Job
  Associated Work Request Id = 8f40d0a3-b2ac-4742-bed8-70beb26d669d
  Done = true
  Name = MODIFY_TYPE
  Progress Message = DrPlanStep id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::precheck. Successfully updated plan step [precheck] in DR Plan [sw1] for config [6e797d8b-7245-4d49-8e68-bf67f2d53041]
  Transcript = Created job MODIFY_TYPE
  Username = admin
  WorkItemIds 1 = id:92a94a5e-4773-4ba8-ac4b-990496c5c2f9  type:WorkItem  name:

  4. To display the status and updated details of a DR plan step, use the show DrPlanStep command. 

    PCA-ADMIN> show DrPlanStep id=6e797d8b-7245-4d49-8e68-bf67f2d53041::fo1::precheck
Data:
  Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::fo1::precheck
  Type = DrPlanStep
  Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::fo1::precheck
  Step Name = PRECHECK
  Config Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041
  Plan Name = fo1
  Enabled = true
  Last Status = norun
  Command = None
  Check Only = false
Using the Service Web UI

  1. Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration for which you want to modify a DR plan. The DR Configuration detail page appears.

  2. In the Resources section, click Plans. Click the name of the plan for which you want to edit the steps. The DR Plan detail page appears. The Resources section displays all steps in the plan.

  3. In the Actions column, open the quick menu (3 dots) for the step you want to change, and click Edit.

  4. In the Update DR Plan Step window, edit the properties as needed:

    • Step Name: Do not change the selected step.

    • Enabled: Select Yes or No to determine whether this step is performed or not during plan execution.

    • Insert Type: Do not change. This field is used only for adding a new step.

    • Insert Location: Do not change. This field is used only for adding a new step.

    • Command: (NOT available.) Specify a custom command to be run as part of this step.

      For standard DR plan operations, this field must remain empty. Custom commands only apply to custom DR plan steps.

    • Check Only: Select Yes or No to determine whether this step is performed or not in check-only mode.

  5. Click Update DR Plan Step to apply your changes.

Inserting a DR Plan Step

Extra steps can be added to a DR plan.

Using the Service CLI

  1. Gather the information that you need to run the command:

    • DR configuration ID (drGetConfigs)

    • DR plan name (drListPlan)

    • name of the step before or after which the new step must be inserted (show DrPlan)

  2. Add a step to the DR plan with the create DrPlanStep command.

    Note the mandatory and optional parameters. If the insertType and insertLocation parameters are not provided, the new step is added as the final step of the DR plan.

    Syntax (entered on a single line):

    create DrPlanStep
drConfigId=<DR_configuration_id> 
planName=<DR_plan_name>
stepName=<DR_plan_new_step_name>
[optional:]
insertType=[BEFORE|AFTER]
insertLocation=<DR_plan_existing_step_name>
enabled=[True|False]
checkOnly=[True|False]
command=<execution_path>

    Note:

    Custom steps and custom commands are not available in appliance software version 3.0.2-b1261765.

    • Do not use the command parameter.

    • Select a step name from this list: precheck, role_reversal_precheck, stop_primary, role_reversal, start_standby, cleanup_primary, post_config.

    Example:

    PCA-ADMIN> create DrPlanStep drConfigId=6e797d8b-7245-4d49-8e68-bf67f2d53041 planName=sw1 stepName=ROLE_REVERSAL insertType=BEFORE insertLocation=START_STANDBY
JobId: 7a162b6c-1ddc-410c-b27a-0996fb2d26df
Data: DrPlanStep id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::role_reversal. Successfully started job for DR Plan Step Create for config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041

  3. Use the job ID to check the status of the operation you started.

    PCA-ADMIN> show Job id=7a162b6c-1ddc-410c-b27a-0996fb2d26df
Data:
  Id = 7a162b6c-1ddc-410c-b27a-0996fb2d26df
  Type = Job
  Associated Work Request Id = a4c76ef9-f91d-402e-986f-9795738fb429
  Done = true
  Name = CREATE_TYPE
  Progress Message = Successfully added step [role_reversal] in plan [sw1] for config [6e797d8b-7245-4d49-8e68-bf67f2d53041]
  Run State = Succeeded
  Transcript = Created job CREATE_TYPE
  Username = admin
  WorkItemIds 1 = id:5c376fae-f281-4f7a-984f-b996b19d367d  type:WorkItem  name:

  4. Repeat the create DrPlanStep command to add all the required steps to the DR plan.

  5. To display the status and updated details of a DR plan, use the show DrPlan command. 

    PCA-ADMIN> show DrPlan id=6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1
Data:
  Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1
  Type = DrPlan
  Plan Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1
  Plan Name = sw1
  Config Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041
  Operation = switchover
  Steps 1 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::precheck
  Steps 1 - Step Name = PRECHECK
  Steps 1 - Enabled = true
  Steps 1 - Last Status = norun
  Steps 1 - Command = None
  Steps 1 - Check Only = true
  Steps 2 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::role_reversal_precheck
  Steps 2 - Step Name = ROLE_REVERSAL_PRECHECK
  Steps 2 - Enabled = true
  Steps 2 - Last Status = norun
  Steps 2 - Command = None
  Steps 2 - Check Only = true
  Steps 3 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::stop_primary
  Steps 3 - Step Name = STOP_PRIMARY
  Steps 3 - Enabled = true
  Steps 3 - Last Status = norun
  Steps 3 - Command = None
  Steps 3 - Check Only = false
  Steps 4 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::role_reversal
  Steps 4 - Step Name = ROLE_REVERSAL
  Steps 4 - Enabled = true
  Steps 4 - Last Status = norun
  Steps 4 - Command = None
  Steps 4 - Check Only = false
  Steps 5 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::start_standby
  Steps 5 - Step Name = START_STANDBY
  Steps 5 - Enabled = true
  Steps 5 - Last Status = norun
  Steps 5 - Command = None
  Steps 5 - Check Only = false
  Steps 6 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::cleanup_primary
  Steps 6 - Step Name = CLEANUP_PRIMARY
  Steps 6 - Enabled = true
  Steps 6 - Last Status = norun
  Steps 6 - Command = None
  Steps 6 - Check Only = false
  Steps 7 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::post_config
  Steps 7 - Step Name = POST_CONFIG
  Steps 7 - Enabled = true
  Steps 7 - Last Status = norun
  Steps 7 - Command = None
  Steps 7 - Check Only = false
Using the Service Web UI

  1. Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration for which you want to modify a DR plan. The DR Configuration detail page appears.

  2. In the Resources section, click Plans. Click the name of the plan to which you want to add one or more steps. The DR Plan detail page appears. The Resources section displays all steps in the plan.

  3. In the Resources section of the DR Plan detail page, on the right hand side of the Steps box, click Add Step.

  4. In the Add DR Plan Step window, enter the following information in the respective fields:

    • Step Name: Select a standard step from the list.

    • Enabled: Select Yes or No to determine whether this step is performed or not during plan execution.

    • Insert Type: Select Before or After the insert location.

    • Insert Location: Select the existing step before or after which you want the new step to be inserted.

    • Command: (NOT available.) Specify a custom command to be run as part of this step.

      For standard DR plan operations, this field must remain empty.

    • Check Only: Select Yes or No to determine whether this step is performed or not in check-only mode.

  5. Click Add DR Plan Step to insert this new step in the selected location.

  6. Repeat until all the required DR plan steps have been added.

Deleting a DR Plan Step

Steps can be removed from a DR plan.

Using the Service CLI

  1. Look up the ID of the DR plan step you want to delete (show DrPlan). If you don't have the DR plan ID, look it up using the commands drGetConfigs and drListPlan.

  2. Delete the DR plan step with the delete DrPlanStep command. 

    PCA-ADMIN> delete DrPlanStep id=6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::precheck
JobId: c75a4c71-0525-40b7-9618-c2a4e8fcb051
Data: DrPlanStep id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::precheck. Successfully started job for DR Plan Step delete for config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041

  3. Use the job ID to check the status of the operation you started.

    PCA-ADMIN> show Job id=c75a4c71-0525-40b7-9618-c2a4e8fcb051
Data:
  Id = c75a4c71-0525-40b7-9618-c2a4e8fcb051
  Type = Job
  Associated Work Request Id = d7bd3873-e5a0-4437-b6c2-8ca0cce83953
  Done = true
  Name = DELETE_TYPE
  Progress Message = DrPlanStep id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::precheck. Successfully deleted step [precheck] in plan [sw1] for config [6e797d8b-7245-4d49-8e68-bf67f2d53041]
  Run State = Succeeded
  Transcript = Created job DELETE_TYPE
  Username = admin
  WorkItemIds 1 = id:b462883b-6e6e-49b6-b455-1ceb54a5e2e3  type:WorkItem  name:

  4. Repeat the delete DrPlanStep command to remove all the required steps from the DR plan.

  5. To display the status and updated details of a DR plan, use the show DrPlan command. 

    PCA-ADMIN> show DrPlan id=6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1
Data:
  Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1
  Type = DrPlan
  Plan Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1
  Plan Name = sw1
  Config Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041
  Operation = switchover
  Steps 1 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::stop_primary
  Steps 1 - Step Name = STOP_PRIMARY
  Steps 1 - Enabled = true
  Steps 1 - Last Status = norun
  Steps 1 - Command = None
  Steps 1 - Check Only = false
  Steps 2 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::role_reversal
  Steps 2 - Step Name = ROLE_REVERSAL
  Steps 2 - Enabled = true
  Steps 2 - Last Status = norun
  Steps 2 - Command = None
  Steps 2 - Check Only = false
  Steps 3 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::start_standby
  Steps 3 - Step Name = START_STANDBY
  Steps 3 - Enabled = true
  Steps 3 - Last Status = norun
  Steps 3 - Command = None
  Steps 3 - Check Only = false
  Steps 4 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::cleanup_primary
  Steps 4 - Step Name = CLEANUP_PRIMARY
  Steps 4 - Enabled = true
  Steps 4 - Last Status = norun
  Steps 4 - Command = None
  Steps 4 - Check Only = false
  Steps 5 - Step Id = 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1::post_config
  Steps 5 - Step Name = POST_CONFIG
  Steps 5 - Enabled = true
  Steps 5 - Last Status = norun
  Steps 5 - Command = None
  Steps 5 - Check Only = false
Using the Service Web UI

  1. Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration for which you want to modify a DR plan. The DR Configuration detail page appears.

  2. In the Resources section, click Plans. Click the name of the plan for which you want to edit the steps. The DR Plan detail page appears. The Resources section displays all steps in the plan.

  3. In the Actions column, open the quick menu (3 dots) for the step you want to remove, and click Delete.

    When prompted, click Confirm.

  4. Repeat until all obsolete DR plan steps have been removed.

Executing a DR Plan

A DR plan can be executed from either the standby or the primary Private Cloud Appliance. However, in the case of a failover, the primary rack is down, so a failover plan is always executed from the standby system.

A switchover can be performed for the purpose of testing the disaster recovery setup, or when extensive maintenance is required on the primary system. To return both appliances to their normal working state after a failover, a postfailover plan is executed on each system when the primary is back online. The switchover plan has postfailover steps built in, so it does not require an additional run of the postfailover plan.

As a result of executing a DR plan, resources are moved between peered systems and the primary system changes. Those resources are not automatically moved back to their original host system. To move resources back to their original environment, you must perform another switchover for the relevant DR configuration(s).

Performing a Switchover

A switchover allows the administrator to move resources away from a system so it can be taken offline, for example in case of planned maintenance. A (second) switchover is also performed to move resources back to their original host system, after they were impacted by a failover or switchover.

Using the Service CLI

  1. Look up the ID of the switchover DR plan you want to execute. Use drGetConfigs to find the DR configuration, and display its associated DR plans using drListPlan.

  2. From the primary or standby appliance, execute the switchover DR plan with the drExecutePlan command.

    Note:

    To run the command in check-only mode, add the parameter checkOnly=True. Only the DR plan steps enabled for check-only mode will be performed. 

    PCA-ADMIN> drExecutePlan planId=6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1
JobId: 92b4acc2-2dff-492c-9ba2-0a2ac058baa5
Data: DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1. Successfully started job for DR Plan Execute for config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041, plan_name sw1

  3. Use the job ID to check the status of the operation you started.

    PCA-ADMIN> show Job id=92b4acc2-2dff-492c-9ba2-0a2ac058baa5
Data:
  Id = 92b4acc2-2dff-492c-9ba2-0a2ac058baa5
  Type = Job
  Associated Work Request Id = c6cca56c-a1cc-421c-9ded-acf0e7cd9da2
  Done = false
  Name = OPERATION-EXECUTE_DR_PLAN
  Progress Message = DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1. Successfully started job for DR Plan Execute for config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041, plan_name sw1
  Run State = Active
  Transcript = Created job OPERATION
  Username = admin
  WorkItemIds 1 = id:e06881fc-ea57-4835-bb86-e1244d3787c3  type:WorkItem  name:

  4. Ensure that the job completes successfully.

    PCA-ADMIN> show Job id=92b4acc2-2dff-492c-9ba2-0a2ac058baa5
Data:
  Id = 92b4acc2-2dff-492c-9ba2-0a2ac058baa5
  Type = Job
  Associated Work Request Id = c6cca56c-a1cc-421c-9ded-acf0e7cd9da2
  Done = true
  Name = OPERATION-EXECUTE_DR_PLAN
  Progress Message = DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1. DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::sw1. drexecuteplan succeeded for config [6e797d8b-7245-4d49-8e68-bf67f2d53041] Operation: [switchover] plan_name: [sw1]. Response: [Successfully completed checks for switchover for DR config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041. Plan Execution Status: [precheck : pass , role_reversal_precheck : pass , stop_primary : norun , role_reversal : norun , start_standby : norun , cleanup_primary : norun , post_config : norun , ]]
  Run State = Succeeded
  Transcript = Created job OPERATION
  Username = admin
  WorkItemIds 1 = id:e06881fc-ea57-4835-bb86-e1244d3787c3  type:WorkItem  name:

    After successful completion, all instances included in the DR configuration have been recovered and are running on the standby appliance.

Using the Service Web UI

  1. Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration for which you want to perform a switchover. The DR Configuration detail page appears.

  2. In the Resources section, click Plans.

  3. In the Actions column, open the quick menu (3 dots) for the switchover plan of your choice, and click Execute Plan.

    Alternatively, click the DR plan name to display its detail page. In the top-right corner, click Execute Plan.

  4. When prompted, choose whether to execute the full plan or a subset of the steps in check-only mode.

    Click Confirm. A DR job is started. When it completes successfully, all steps in the switchover DR plan have been performed as expected.

    To track progress, under Disaster Recovery Service, select Jobs. The Jobs table reports the status of each job. Click a record in the table to display the job details.

    After successful completion, all instances included in the DR configuration have been recovered and are running on the standby appliance.

Performing a Failover

The native DR service does not provide automated failover. An administrator must confirm that the primary appliance is down, and execute the failover plan from the standby appliance. A failover is meant to allow continuation of service when the primary system experiences an outage.

Note:

When one appliance is down, the peer rack reports a fault with a name containing "peerconnect" and the rack serial number. Use the Service CLI to check the fault list (list fault <parameters>) and display the details of the peer connection problem. For example: 

PCA-ADMIN> show fault id=57701191-5764-480b-826c-38c4b1970dde
Data:
  Cause =  1742XC3024 : network is not in a CONNECTED state: CONNECTING
  Action = Please contact customer support for solution
  Health Exporter = peerconnect-checker
  Diagnosing Source = peer connect health checker
  Faulted Component Type = SOFTWARE
  Description = 1749XC302P-- 1742XC3024 : network is not in a CONNECTED state: CONNECTING
  Name = 1749XC302P--PCA-8000-UY--peerconnect
Using the Service CLI

  1. Look up the ID of the failover DR plan you need to execute. Use drGetConfigs to find the DR configuration, and display its associated DR plans using drListPlan.

  2. From the standby appliance, execute the failover DR plan with the drExecutePlan command.

    Note:

    To run the command in check-only mode, add the parameter checkOnly=True. Only the DR plan steps enabled for check-only mode will be performed. 

    PCA-ADMIN> drExecutePlan planId=6e797d8b-7245-4d49-8e68-bf67f2d53041::fo1
JobId: 49521287-c148-4791-9626-13190fce3d1d
Data: DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::fo1. Successfully started job for DR Plan Execute for config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041, plan_name fo1

  3. Use the job ID to check the status of the operation you started.

    PCA-ADMIN> show Job id=49521287-c148-4791-9626-13190fce3d1d
Data:
  Id = 49521287-c148-4791-9626-13190fce3d1d
  Type = Job
  Associated Work Request Id = c8e3b554-a3ef-4e9b-a52c-c9a518f70974
  Done = false
  Name = OPERATION-EXECUTE_DR_PLAN
  Progress Message = DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::fo1. Successfully started job for DR Plan Execute for config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041, plan_name fo1
  Run State = Active
  Transcript = Created job OPERATION
  Username = admin
  WorkItemIds 1 = id:d7a09483-ef2e-4e03-81bb-fed5ee661428  type:WorkItem  name:

  4. Ensure that the job completes successfully.

    PCA-ADMIN> show Job id=49521287-c148-4791-9626-13190fce3d1d
Data:
  Id = 49521287-c148-4791-9626-13190fce3d1d
  Type = Job
  Associated Work Request Id = c8e3b554-a3ef-4e9b-a52c-c9a518f70974
  Done = true
  Name = OPERATION-EXECUTE_DR_PLAN
  Progress Message = DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::fo1. DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::fo1. drexecuteplan succeeded for config [6e797d8b-7245-4d49-8e68-bf67f2d53041] Operation: [failover] plan_name: [fo1]. Response: [Successfully completed checks for failover for DR config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041. Plan Execution Status: [precheck : pass , role_reversal_precheck : pass , role_reversal : pass , start_standby : pass , ]]
  Run State = Succeeded
  Transcript = Created job OPERATION
  Username = admin
  WorkItemIds 1 = id:d7a09483-ef2e-4e03-81bb-fed5ee661428  type:WorkItem  name:

    After successful completion, all instances included in the DR configuration have been recovered and are running on the standby appliance.

Using the Service Web UI

  1. Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration for which you want to perform a switchover. The DR Configuration detail page appears.

  2. In the Resources section, click Plans.

  3. In the Actions column, open the quick menu (3 dots) for the failover plan of your choice, and click Execute Plan.

    Alternatively, click the DR plan name to display its detail page. In the top-right corner, click Execute Plan.

  4. When prompted, choose whether to execute the full plan or a subset of the steps in check-only mode.

    Click Confirm. A DR job is started. When it completes successfully, all steps in the switchover DR plan have been performed as expected.

    To track progress, under Disaster Recovery Service, select Jobs. The Jobs table reports the status of each job. Click a record in the table to display the job details.

    After successful completion, all instances included in the DR configuration have been recovered and are running on the standby appliance.

Performing Postfailover Operations

A postfailover is performed after a failover, when the system that experienced an outage comes back online. The plan can be executed from either of the peered systems. During postfailover, the DR configuration is cleaned up on the primary system that went down. The original standby system becomes the primary for the resources covered by the DR configuration, using the original primary as the new target for DR data replication.

Using the Service CLI

  1. After a failover, confirm that the primary appliance is back online and in healthy condition.

    Ensure that the peering status is active and replication is enabled. Neither rack should report an active fault with a name containing "peerconnect". (Check with Service CLI command list fault.)

  2. Look up the ID of the postfailover DR plan you want to execute. Use drGetConfigs to find the DR configuration, and display its associated DR plans using drListPlan.

  3. From the primary or standby appliance, execute the postfailover DR plan with the drExecutePlan command.

    Note:

    For postfailover operations, the check-only mode does not apply.

    PCA-ADMIN> drExecutePlan planId=6e797d8b-7245-4d49-8e68-bf67f2d53041::pfo1
JobId: 56d040ba-30a6-4bea-b924-78ebabed2626
Data: DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::pfo1. Successfully started job for DR Plan Execute for config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041, plan_name pfo1

  4. Use the job ID to check the status of the operation you started.

    PCA-ADMIN> show Job id=56d040ba-30a6-4bea-b924-78ebabed2626
Data:
  Id = 56d040ba-30a6-4bea-b924-78ebabed2626
  Type = Job
  Associated Work Request Id = b4ad564b-e385-4688-94ff-11bf5267d72e
  Done = false
  Name = OPERATION-EXECUTE_DR_PLAN
  Progress Message = DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::pfo1. Successfully started job for DR Plan Execute for config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041, plan_name pfo1
  Run State = Active
  Transcript = Created job OPERATION
  Username = admin
  WorkItemIds 1 = id:2e4db010-239e-41a1-aa0d-cb97167c64fc  type:WorkItem  name:

  5. Ensure that the job completes successfully.

    PCA-ADMIN> show Job id=56d040ba-30a6-4bea-b924-78ebabed2626
Data:
  Id = 56d040ba-30a6-4bea-b924-78ebabed2626
  Type = Job
  Associated Work Request Id = b4ad564b-e385-4688-94ff-11bf5267d72e
  Done = true
  Name = OPERATION-EXECUTE_DR_PLAN
  Progress Message = DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::pfo1. DrPlan id: 6e797d8b-7245-4d49-8e68-bf67f2d53041::pfo1. drexecuteplan succeeded for config [6e797d8b-7245-4d49-8e68-bf67f2d53041] Operation: [postfailover] plan_name: [pfo1]. Response: [Successfully completed checks for postfailover for DR config_id 6e797d8b-7245-4d49-8e68-bf67f2d53041. Plan Execution Status: [stop_primary : pass , cleanup_primary : pass , post_config : pass , ]]
  Run State = Succeeded
  Transcript = Created job OPERATION
  Username = admin
  WorkItemIds 1 = id:2e4db010-239e-41a1-aa0d-cb97167c64fc  type:WorkItem  name:

    After successful completion, all instances impacted by the switchover or failover have been restored and are running on the appliance where they were hosted before.

Using the Service Web UI

  1. After a failover, confirm that the primary appliance is back online and in healthy condition.

    Ensure that the peering status is active and replication is enabled. Neither rack should report an active fault with a name containing "peerconnect". (Display active faults in the Service Web UI.)

  2. Under Disaster Recovery Service, open the DR Configurations page. In the table, click the configuration for which you want to perform postfailover operations. The DR Configuration detail page appears.

  3. In the Resources section, click Plans.

  4. In the Actions column, open the quick menu (3 dots) for the postfailover plan of your choice, and click Execute Plan.

    Alternatively, click the DR plan name to display its detail page. In the top-right corner, click Execute Plan.

  5. When prompted, click Confirm.

    Note:

    For postfailover operations, the check-only mode does not apply.

    A DR job is started. When it completes successfully, all steps in the postfailover DR plan have been performed as expected.

    To track progress, under Disaster Recovery Service, select Jobs. The Jobs table reports the status of each job. Click a record in the table to display the job details.

    When the job has completed successfully, all instances impacted by the switchover or failover have been restored and are running on the appliance where they were hosted before.