1. Set up a Disaster Recovery topology for Oracle Data Integrator
  2. Set up a Disaster Recovery topology for Oracle Data Integrator

Set up a Disaster Recovery topology for Oracle Data Integrator

Oracle Data Integrator is a comprehensive data integration platform that covers all data integration requirements: from high-volume, high-performance batch loads, to event-driven, trickle-feed integration processes, to SOA-enabled data services.

Disaster Recovery is a process using which we can restore the services when a system is down. It is imperative to have a disaster recovery setup for mission critical applications as it saves the organization from data loss and also reduces the down time of an application.

Setting up a disaster recovery topology for Oracle Data Integrator will enable business users to access the latest data and improve their analytics results.

Architecture

This reference architecture describes the disaster recovery setup for an Oracle Data Integrator Marketplace image.

The architecture setup has Autonomous database, Virtual machine, boot volume, block volume, network, and file storage. Each of these components should be replicated to a secondary region from the primary region so that the instance can be restored during a disaster.

The virtual machine contains the operating system and Oracle Data Integrator software. It is stored in the boot volume and these boot volumes are replicated across regions using the cross-region replication features in OCI.

Block volume provides high-performance network storage capacity. Block volumes are attached to a Virtual machine to increase the storage capacity. The block volumes attached to Oracle Data Integrator Virtual machines can be replicated using the cross-region replication feature in OCI.

File Storage service provides a durable, scalable, secure, enterprise-grade network file system. File storage can be attached to multiple virtual machines to increase the storage and to share files across the instances. File storage attached to the virtual machine can be replicated using the cross-region replication feature in OCI.

Data Guard ensures high availability, data protection, and disaster recovery for enterprise data. Data Guard can be enabled for Autonomous database to create the standby database in secondary region. The virtual machine can be restored using the boot volumes in the secondary region. Once the virtual machine is restored, the Oracle Data Integrator can be configured to the secondary autonomous database. The boot volume and File storage can be attached to the virtual machine.

The following diagram illustrates this reference architecture.


Description of oracle-data-integrator-dr-arch.png follows
Description of the illustration oracle-data-integrator-dr-arch.png

oracle-data-integrator-dr-arch-oracle.zip

The architecture has the following components:

  • Region

    An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).

  • Availability domains

    Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain shouldn't affect the other availability domains in the region.

  • Virtual cloud network (VCN) and subnets

    A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.

  • Autonomous Database

    Oracle Autonomous Database is a fully managed, preconfigured database environments that you can use for transaction processing and data warehousing workloads. You do not need to configure or manage any hardware, or install any software. Oracle Cloud Infrastructure handles creating the database, as well as backing up, patching, upgrading, and tuning the database.

  • Virtual Machine

    Oracle Cloud Infrastructure (OCI) Compute VMs provide secure and elastic compute capacity in the cloud for workloads ranging from small development projects to large-scale, global applications such as real-time communication platforms. Flexible shapes enable users to optimize VM resources with customized processor and memory values for improved price performance.

  • Block volume

    With block storage volumes, you can create, attach, connect, and move storage volumes, and change volume performance to meet your storage, performance, and application requirements. After you attach and connect a volume to an instance, you can use the volume like a regular hard drive. You can also disconnect a volume and attach it to another instance without losing data.

  • File storage

    The Oracle Cloud Infrastructure File Storage service provides a durable, scalable, secure, enterprise-grade network file system. You can connect to a File Storage service file system from any bare metal, virtual machine, or container instance in a VCN. You can also access a file system from outside the VCN by using Oracle Cloud Infrastructure FastConnect and IPSec VPN.

Recommendations

Use the following recommendations as a starting point. Your requirements might differ from the architecture described here.
  • VCN Sizing

    When you create a VCN, determine the number of CIDR blocks required and the size of each block based on the number of resources that you plan to attach to subnets in the VCN. Use CIDR blocks that are within the standard private IP address space.Select CIDR blocks that don't overlap with any other network (in Oracle Cloud Infrastructure, your on-premises data center, or another cloud provider) to which you intend to set up private connections. After you create a VCN, you can change, add, and remove its CIDR blocks.

  • Compute shapes

    In this architecture, an Oracle Data Integrator Marketplace image and the VM.Standard 2.4 shape is used for the compute machine. Choose a shape that's appropriate for your application’s resource needs.

  • Boot volume

    When you launch a virtual machine (VM) or bare metal instance based on a platform image or custom image, a new boot volume for the instance is created in the same compartment. That boot volume is associated with that instance until you terminate the instance. Boot volume is a special type of block volumes. Enable the cross region replication for boot volume so that the data is replicated to secondary region.

  • Block volume

    As the Oracle Data Integrator marketplace image comes with a default size of 50 GB, a block volume is attached to increase the storage capacity. OCI block volume supports volume from 50 GB to 32 TB in 1 GB increments. Choose the block volume size which is sufficient for your use case. Enable the cross region replication for block volume so that the data is replicated to the disaster recovery region.

  • File storage

    A file storage can be attached to multiple compute machines and will be helpful when we want to use the File system as a source. OCI File storage scales automatically to accommodate the growth up to 8 exabytes. Enable the cross region replication for file storage.

  • Data guard

    Enable the Autonomous Data Guard on Oracle Autonomous Database so that the secondary instance can become active during disaster recovery.

Considerations

When implementing this architecture, consider your requirements for the following parameters.

  • Static IP for Compute

    Static IP address can be used for a compute instance so that instance provisioning will be easy during the switchover. If a dynamic IP address is used, the IP address should be updated in /etc/hosts.

  • DNS, Host name entries

    The DNS should be configured so that virtual machines can connect to the Autonomous Database. If DNS is not configured, the Autonomous Database instance IP address can be added in /etc/hosts.

  • Service limits

    When designing your architecture, consider the service limits for the compute instance, block storage, file storage, Autonomous Database. See the Service Limits documentation listed in the Explore More section.

  • Backups
    • Database

      OCI automatically backs up autonomous databases and retains the backups for 60 days. You can restore and recover your database to any point in time during the retention period. You can also create manual backups to supplement the automatic backups. Manual backups are stored in an OCI Object Storage bucket that you create, and are retained for 60 days.

    • Application

      The Oracle Cloud Infrastructure Block Volumes service lets you create point-in-time backups of data on a block volume. You can restore these backups to new volumes at any time. You can also use the service to make a point-in-time, crash-consistent backup of a boot volume without application interruption or downtime. Boot and block volumes have the same backup capabilities.

    • Security

      For access control, use policies to restrict who can access your resources in the cloud and the actions that they can perform.

Deploy

Oracle Data Integrator is available in the Oracle marketplace images and an instance can be provisioned using the Marketplace image.
Attach block storage and file storage
  1. A block storage can be provisioned and attached to ODI marketplace instance based on the storage requirements.
  2. A file storage can be attached if a shared drive is required for multiple compute instances.
Deploy on the Secondary region
  1. The compute instance in the secondary region should point to the Autonomous database in the secondary region.
  2. The odi-setup.properties file should be updated to the appropriate region.
    • dbTech=ADB
    • rcuCreationMode=false
    • odiSchemaPassword=<valid password>
    • odiSchemaUser=<odi schema username>
    • odiSupervisorPassword=<odi SUPERVISOR password>
    • walletZipLoc=<path_to_zipped_wallet>
    • workRepoName=<WORK REPO NAME>
  3. Create a repository.properties file in $MW_HOME/odi/common/scripts.
    masterReposDriver=oracle.jdbc.OracleDriver
masterReposUser=<odi schema username>
workReposName=<WORK REPO NAME>
  4. Navigate to the $MW_HOME/odi/common/scripts directory and execute the following Python scripts in the given order.
    1. python odiMPConfiguration.py
    2. python manageOdiApps.py start
  5. Steps 2 to 4 can be automated and can be executed when the instance is provisioned.

Explore More

Review these additional resources to learn more about the features of this reference architecture.

Acknowledgments

  • Author: Srivishnu Gullapalli
  • Contributor: Anupama Pundpal