Note:

Automate a Switchover and Failover Plan for a Demo Application Deployed on OCI Kubernetes Engine with OCI Full Stack Disaster Recovery

Introduction

This tutorial demonstrates an Oracle Cloud Infrastructure Full Stack Disaster Recovery (OCI Full Stack DR) use case with a demo e-commerce application deployed on an Oracle Cloud Infrastructure Kubernetes Engine (OCI Kubernetes Engine or OKE) cluster.

At the time of writing this tutorial, OCI Full Stack DR has announced limited availability for OKE. In limited release, we get to try OCI Full Stack DR on OKE-based applications like MuShop, a microservices-based demo application that uses various other Oracle Cloud Infrastructure (OCI) services as one application.

We will use a warm standby approach: a disaster recovery (DR) model where some or all application components are pre-deployed in a standby region to enable a faster DR transition. Although this model involves higher operating costs, it provides a lower recovery time objective (RTO).

OCI Full Stack DR orchestrates the transition of computes, databases, and applications between OCI regions from around the globe with a single click. Customers can automate the steps needed to recover one or more business systems without redesigning or re-architecting existing infrastructure, databases, or applications without needing specialized management or conversion servers.

Deployment Architecture

OKE Deployment Architecture

Note: The primary region is Sydney and the DR region is Melbourne.

Objectives

Prerequisites

Task 1: Install and Set Up Oracle Autonomous Database

  1. Create the primary Oracle Autonomous Database.

    oci db autonomous-database create --compartment-id ${COMPARTMENT_ID} \
--db-name ${DB_NAME} --admin-password ${DB_PASSWORD} --db-version 19c \
--cpu-core-count 1 --data-storage-size-in-tbs 1 \
--display-name ${DB_DISPLAY_NAME} --region ${PRIMARY_REGION}

  2. Fetch the primary Oracle Autonomous Database OCID.

    DB_ID=$(oci db autonomous-database list -c ${COMPARTMENT_ID} \
--region ${PRIMARY_REGION} --display-name $DB_NAME \
--query "data[?\"db-name\"=='${DB_NAME}'].id | [0]" --raw-output)

  3. Create the standby DR and enable cross-region Oracle Data Guard.

    oci db autonomous-database create-adb-cross-region-data-guard-details \
--compartment-id ${COMPARTMENT_ID} --db-name ${DB_NAME} --source-id ${DB_ID} \
--cpu-core-count 1 --data-storage-size-in-tbs 1 \
--region ${FAILOVER_REGION} --db-version 19c

  4. Download and extract the autonomous database wallet from the primary Oracle Autonomous Database.

    oci db autonomous-database generate-wallet --autonomous-database-id ${DB_ID}\
--password ${WALLET_PW} --file ${WALLET_ZIP} --region $PRIMARY_REGION

  5. Unzip the wallet at the primary.

    unzip ${WALLET_ZIP} -d /tmp/wallet_primary

    Note:

    • Keep this wallet handy as we will need to add it as the OKE secret later on.
    • The wallet has to be downloaded separately for the primary and standby regions because the tnsnames.ora DNS entries are different.

  6. Fetch the standby Oracle Autonomous Database OCID.

    STANDBY_DB_ID=$(oci db autonomous-database list -c ${COMPARTMENT_ID} \
--region ${STANDBY_REGION} --display-name $STANDBY_DB_NAME \
--query "data[?\"db-name\"=='${DB_NAME}'].id | [0]" --raw-output)

  7. Download and extract the autonomous database wallet from the standby Oracle Autonomous Database.

    oci db autonomous-database generate-wallet --autonomous-database-id \
${STANDBY_DB_ID} --password ${WALLET_PW} \
--file ${STANDBY_WALLET_ZIP} --region $STANDBY_REGION

  8. Unzip the standby wallet.

    unzip ${STANDBY_WALLET_ZIP} -d /tmp/wallet_standby

Task 2: Create an OKE Cluster

Create an OKE cluster on both the primary and DR sites. For more information, see Create a Cluster with Oracle Cloud Infrastructure Container Engine for Kubernetes.

We have used the Quick Create option to create clusters with the following information:

To access your cluster, go to the OCI Console, navigate to Developer Service, Container & Artifacts, and click Kubernetes Clusters (OKE).

Or

Run the following command to access your cluster.

oci ce cluster create-kubeconfig --cluster-id <cluster-id> --file $HOME/.kube/config --region ap-sydney-1 --token-version 2.0.0  --kube-endpoint PUBLIC_ENDPOINT

Task 3: Set Up Kubernetes Secrets at the Primary Site

  1. Create a namespace.

    kubectl create ns mushop

  2. Add the Oracle Autonomous Database admin password secret.

    kubectl create secret generic oadb-admin \
      --namespace mushop \
      --from-literal=oadb_admin_pw=${DB_PASSWORD}

  3. Add the Oracle Autonomous Database connection secret.

    kubectl create secret generic oadb-connection \
      --namespace mushop \
      --from-literal=oadb_wallet_pw=${WALLET_PW} \
      --from-literal=oadb_service=${DB_SERVICE_NAME}

  4. Add the primary wallet secret.

    kubectl create secret generic oadb-wallet \
      --namespace mushop --from-file=/tmp/wallet_primary

Task 4: Set Up the MuShop Application

Note: The application is only deployed at the primary region (ap-sydney-1).

  1. Clone the repository.

    git clone git@github.com:naikvenu/fsdr-demo.git

  2. Go to the chart folder.

    cd fsdr-demo/helm-chart/

  3. Update the chart dependencies.

    helm dependency update ./setup

  4. Install and set up the chart.

    helm upgrade --install mushop-utils setup --dependency-update --namespace mushop-utilities --create-namespace

  5. Locate the ingress controller EXTERNAL-IP address.

    PRIMARY_EXTERNAL_IP=$(kubectl get svc -n mushop-utilities mushop-utils-ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

  6. Install the application on the OKE cluster.

    helm upgrade --install -f ./mushop/values-dr.yaml \
   fsdrmushop mushop -n mushop

  7. Access the primary MuShop application using the ingress IP.

    kubectl get svc mushop-utils-ingress-nginx-controller \
   --namespace mushop-utilities

  8. To verify the application, access http://<primary-site-ingress-ip-address> and ensure you see all of the MuShop catalogue products listed without errors.

Task 5: Set Up an OKE Cluster at the Standby Site

Note: Since we are using a warm standby approach, we need to create an OKE cluster and run a few basic things, like the ingress controller. The following steps will help you do that.

  1. To access your cluster at the standby site, go to the OCI Console, navigate to Developer Service, Container & Artifacts, and click Kubernetes Clusters (OKE).

    Or

    Run the following command to access your cluster at the standby site.

    oci ce cluster create-kubeconfig --cluster-id <cluster-id> --file $HOME/.kube/config --region ap-sydney-1 --token-version 2.0.0  --kube-endpoint PUBLIC_ENDPOINT

  2. Clone the repository.

    git clone git@github.com:naikvenu/fsdr-demo.git

    Or

    git clone  https://github.com/naikvenu/fsdr-demo

  3. Go to the charts folder.

    cd fsdr-demo/helm-chart/

  4. Update the chart dependencies.

    helm dependency update ./setup

  5. Install and set up the charts. This is required to deploy an ingress controller (OCI Load Balancer) to access the application.

    Note: This step will only deploy an ingress controller (OCI Load Balancer) and not the full application.

    helm upgrade --install mushop-utils setup --dependency-update --namespace mushop-utilities --create-namespace

  6. Locate the ingress controller EXTERNAL-IP address.

    STANDBY_EXTERNAL_IP=$(kubectl get svc -n mushop-utilities mushop-utils-ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

  7. Create a MuShop namespace.

    kubectl create namespace mushop

  8. Create a secret for the wallet.

    kubectl create secret generic oadb-wallet \
      --namespace mushop --from-file=/tmp/wallet_standby

    Note: We are using a standby wallet.

Task 6: Set Up DNS Zones (Optional)

At the primary region, go to the OCI Console, navigate to Networking, DNS Management, Zones, and click Create Zone.

Configuration:
The Zone type : Primary
‘A’ record: “mushop”

The zone name should match your Purchased Domain Name. Enter the nameservers to be added to your domain.

The application will be accessible at https://mushop.<your-domain>.com.

Task 7: Create Health Checks (Optional)

Health checks are required to set up OCI DNS traffic steering policies.

Run the following command to create health checks.

oci health-checks http-monitor create --compartment-id ${COMPARTMENT_ID} --display-name fsdr-test --interval-in-seconds 30 --targets '[“${PRIMARY_EXTERNAL_IP}”]' --protocol http --path "/" --port 80

Or

Go to the OCI Console, navigate to Observability & Management, Monitoring, Healthchecks, click Create Health Check, and enter the following information.

Task 8: Configure the Traffic Management Steering Policy (Optional)

Go to the OCI Console, navigate to Networking, DNS Management, Traffic Management Steering Policies, click Create Traffic Management Steering Policy, and enter the following information.

Task 9: Set Up OCI Full Stack DR

  1. Create a DRPG in both regions. Go to the OCI Console, navigate to Migration & Disaster Recovery and click DR protection groups. For example, primary-drpg-sydney and standby-drpg-melbourne.

  2. Associate the DRPGs. Go to the OCI Console, navigate to Migration & Disaster Recovery, DR protection groups, and click Associate.

  3. Add resources to the DRPG (Sydney region). Go to the OCI Console, navigate to Migration & Disaster Recovery, DR protection groups, Members, and click Add Member.

  4. Add the OKE cluster and Oracle Autonomous Database.

    DRPG Members

  5. Add resources to the DRPG (Melbourne region) - OKE cluster and Oracle Autonomous Database.

    DRPG Members

  6. Create a DR plan at the standby region (Melbourne). Go to the OCI Console, navigate to Migration & Disaster Recovery, DR protection groups, Plans, and click Create Plan.

    DRPG Members

    The following image shows a DR plan that orchestrates recovery for an entire application stack, which can include other services along with OKE.

    As you can see from the image, we have built-in steps for OKE. The OCI Full Stack DR service runs an internally developed backup tool. This tool will periodically take the cluster’s backups of Deployments, Replica Sets, Pods, CronJobs, Daemon Sets, and so on.

    The backups will be stored in the OCI Object Storage bucket that we specified in the member property.

    • OKE - Stop Backup and Cleanup (Primary): This stops the backups and terminates all the mentioned resources in the OKE cluster.

    • OKE - Restore (Standby): Using the backup, it restores the latest backup in the DR OKE cluster, so you will have all the resources created in the OKE cluster.

    • OKE - Schedule Reverse Backup (Standby): Set the reverse backup for the switchback plan.

    If you are using PersistentVolume (PV) and PersistentVolumeClaim (PVC), you need to configure cross-region volume groups (block storage) and cross-region FSS replication (file storage) and add those as members in the DRPG. This will create additional plan groups, like what we saw for OKE and Oracle Autonomous Database.

  7. Perform a switchover. This step must be performed from the standby site (Melbourne).

    Go to the OCI Console, navigate to Migration & Disaster Recovery, DR protection groups, and click Execute DR Plan.

Task 10: Test and Validate the Application

Access the application from the standby region and ensure everything works. The application should be accessible at https://mushop.domain.com or use the http://standbyloadbalancerIP.com address.

Ensure you can access the catalogue items, indicating that the standby database is fully operational.

DRPG Members

Note: In this tutorial, we have excluded steps to include SSL certificates on OCI Load Balancer and using OCI Web Application Firewall. These two components can be added to production environments.

Next Steps

You have seen how an e-commerce application based on microservices, deployed on OCI Kubernetes Engine, can be configured with the OCI Full Stack DR service to enable disaster recovery in a warm standby mode. We showed how this application can seamlessly fail over without any manual intervention. For more information, see the OCI Full Stack DR documentation in the Related Links section.

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.