1. Import virtual disk images to Oracle Cloud Infrastructure Block Volumes using Apache Airflow
  2. Import virtual disk images to Oracle Cloud Infrastructure Block Volumes using Apache Airflow

Import virtual disk images to Oracle Cloud Infrastructure Block Volumes using Apache Airflow

As organizations continue to migrate to the cloud, they often find themselves facing the challenge of seamlessly transferring and managing their existing block volumes. Whether it's for migrating on-premises workloads, implementing disaster recovery strategies, or optimizing their cloud infrastructure, the ability to import virtual disks with ease becomes a critical need.

This is where the synergy of Oracle Cloud Infrastructure Functions (OCI Functions), OCI Object Storage, Apache Airflow, qemu-img, and event-driven architectures enables users to automate the import of virtual disk image files into Oracle Cloud Infrastructure Block Volumes.

Note:

As this solution uses Apache Airflow for orchestration, prior experience will allow users to monitor the virtual disk import workflows, troubleshoot and retry any interrupted conversion flow.

Architecture

The architecture for a typical virtual disk image import workflow is described in the following section.

  1. The user uploads a new virtual disk image to the OCI Object Storage bucket.
  2. When the virtual disk image file upload is finalized, the event triggers the execution of an OCI Function.
  3. The function will fetch the virtual disk object metadata and call the Apache Airflow API to start DAG (Directed Acyclic Graph) execution.
  4. The Apache Airflow DAG will handle the import of the virtual disk image into OCI Block Volume in 14 steps. Some of the most important steps are:
    • Create a new worker compute instance that will handle the virtual disk image import.
    • Set up required tools on the new instance: qemu-img, oci-cli.
    • Download the virtual disk image from the bucket to the worker instance.
    • Determine the virtual disk image's real size using qemu-img and provision a new OCI Block Volume (named as the virtual disk image).
    • Attach the OCI Block Volume to the worker compute instance.
    • Write the virtual disk image content to the OCI Block Volume.
    • Run fsck on the OCI Block Volume.
    • Detach the OCI Block Volume.
    • Terminate the worker instance.
    • (Optional) Update the backup policy on the OCI Block Volume.
  5. Users can monitor the DAG execution by connecting to Apache Airflow.

The following diagram illustrates this reference architecture.


Description of virtual-disk-import-arch.png follows
Description of the illustration virtual-disk-import-arch.png

virtual-disk-import-arch-oracle.zip

The architecture has the following components:

  • Region

    An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).

  • Availability domains

    Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain is unlikely to affect the other availability domains in the region.

  • Virtual cloud network (VCN) and subnets

    A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.

  • Object storage

    Object storage provides quick access to large amounts of structured and unstructured data of any content type, including database backups, analytic data, and rich content such as images and videos. You can safely and securely store and then retrieve data directly from the internet or from within the cloud platform. You can seamlessly scale storage without experiencing any degradation in performance or service reliability. Use standard storage for "hot" storage that you need to access quickly, immediately, and frequently. Use archive storage for "cold" storage that you retain for long periods of time and seldom or rarely access.

  • Events

    Oracle Cloud Infrastructure services emit events, which are structured messages that describe the changes in resources. Events are emitted for create, read, update, or delete (CRUD) operations, resource lifecycle state changes, and system events that affect cloud resources.

  • Functions

    Oracle Cloud Infrastructure Functions is a fully managed, multitenant, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform. It is powered by the Fn Project open source engine. Functions enable you to deploy your code, and either call it directly or trigger it in response to events. Oracle Functions uses Docker containers hosted in Oracle Cloud Infrastructure Registry.

  • DevOps

    The DevOps service is an end-to-end, continuous integration and continuous delivery (CI/CD) platform for developers that facilitates the process of building artifacts and deployment to OCI compute services (Functions, Oracle Container Engine (OKE) or Compute Instances).

  • Policy

    An Oracle Cloud Infrastructure Identity and Access Management policy specifies who can access which resources, and how. Access is granted at the group and compartment level, which means you can write a policy that gives a group a specific type of access within a specific compartment, or to the tenancy.

  • Block volume

    With block storage volumes, you can create, attach, connect, and move storage volumes, and change volume performance to meet your storage, performance, and application requirements. After you attach and connect a volume to an instance, you can use the volume like a regular hard drive. You can also disconnect a volume and attach it to another instance without losing data.

  • Virtual Machines

    Oracle Cloud Infrastructure (OCI) Compute VMs provide secure and elastic compute capacity in the cloud for workloads ranging from small development projects to large-scale, global applications such as real-time communication platforms. Flexible shapes enable users to optimize VM resources with customized processor and memory values for improved price performance.

Recommendations

Use the following recommendations as a starting point. Your requirements might differ from the architecture described here.
  • VCN

    When you create a VCN, determine the number of CIDR blocks required and the size of each block based on the number of resources that you plan to attach to subnets in the VCN. Use CIDR blocks that are within the standard private IP address space.

    Select CIDR blocks that don't overlap with any other network (in Oracle Cloud Infrastructure, your on-premises data center, or another cloud provider) to which you intend to set up private connections.

    After you create a VCN, you can change, add, and remove its CIDR blocks.

    When you design the subnets, consider your traffic flow and security requirements. Attach all the resources within a specific tier or role to the same subnet, which can serve as a security boundary.

  • Security

    It is strongly recommended for the user who deploys the stack to be a member of the administrators group. In case the user exposes Apache Airflow to the internet using the allowed_source_cidr variable, it is strongly recommended to fill in specific IP, subnet CIDR block.

Considerations

Consider the following points when deploying this reference architecture.

  • Security

    The instance running Apache Airflow and temporary worker instances are authorized using a dynamic group to manage instances, block volumes and read objects in the compartment they belong to. The function is authorized to perform read operations against the object resources in the compartment.

  • Availability

    The Apache Airflow API on port 8080 of the Apache Airflow instance is reachable from the internet only when a new VCN is created, the instance has a public IP address and the allowed_source_cidr variable includes the source IP from where the connection is attempted.

  • Cost

    A temporary compute instance is created to handle each virtual disk import operation to Oracle Cloud Infrastructure Block Volumes. On successful completion of the workflow the instance is terminated.

  • Placement

    Block volume placement can be customized using the "ad_number": 1/2/3 metadata on the virtual disk image file uploaded to OCI Object Storage.

Deploy

The Terraform code to configure and deploy the resources of this reference architecture is available in GitHub.

  1. Go to GitHub.
  2. Clone or download the repository to your local computer.
  3. Follow the instructions in the README document.

Explore More

To learn more about Oracle Cloud Infrastructure, Apache Airflow, and qemu-img, see the following resources.

Acknowledgments

Author: Andrei Ilas