Optimize asynchronous workflows by integrating OCI Queue with OCI Functions

The Oracle Cloud Infrastructure (OCI) Queue service allows you to send messages between systems in an asynchronous manner, allowing you to decouple your systems and improve their performance. It is designed to handle high volumes of transactional data and ensures that messages are processed without loss or duplication.

The OCI Queue service is fully managed and automatically scales based on the throughput of producers and consumers, using open standards to support communication with any client or producer with minimal effort. Serverless applications often rely on a combination of various services (API Gateway, Functions, and so on) to function properly. However, in order for these distributed components to communicate with each other, the Queue service is crucial. It also allows your application to extend communication to other OCI components or even outside of OCI.

Architecture

This reference architecture deploys a sample Python application to Oracle Container Engine for Kubernetes (OKE), which regularly polls the configured queue to fetch new messages, invokes a configured function using fetched messages, and on successful execution, removes the messages from the queue.

The application authenticates itself as the instance_principal when it interacts with OCI services. Queue and Functions can be in any region, and queue and function OCIDs must be specified during stack creation. If it is necessary to update the queue and function OCIDs after Terraform deployment, you can edit the Kubernetes deployment artifact (part of the created OCI DevOps project).

The Terraform deployment includes the creation of the pipelines and repositories needed in an OCI DevOps service to automate the steps required to take application source code all the way into production. The application source code is hosted on a DevOps code repository. The user can then commit the code into the repository, which triggers the start of a build pipeline.

The following diagram illustrates this reference architecture.

Description of the illustration queue_fn_automation_arch.png

queue-fn-automation-arch-oracle.zip

The architecture has the following components:

• Region

  An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).

• DevOps project

  A logical grouping of DevOps resources needed to implement a CI/CD workflow. DevOps resources can be artifacts, build pipelines, deployment pipelines, external connections, triggers, and environments. DevOps projects make it easy to enable logging, monitoring, and notifications for all your DevOps resources.

• Build pipeline

  A build pipeline takes a commit ID from your source code repositories and uses that source code to run your build instructions. Build pipelines define a set of stages for the build process: building, testing, and compiling software artifacts, delivering artifacts to OCI repositories, and optionally triggering a deployment. You can define the flow and instructions of your build run in the build spec file.

• Code repository

  Private Git repositories hosted by the DevOps service. You can store, manage, and develop source code with OCI DevOps Code Repositories.

• Deployment pipeline

  A sequence of steps for delivering and deploying a set of artifacts to a target environment. The flow and logic of your software release can be controlled by defining stages that can run in serial or parallel.

• Deployment stages
  Stages are individual actions that take place during a run of a pipeline. The DevOps deployment pipeline includes the following predefined stage types for you to use in your release process:

  • Rolling deployment: An incremental release to OKE, Functions, or instance groups.
  • Wait: Wait N seconds.
  • Manual approval: Proceed if approval is given; stop if approval is rejected.
  • Invoke function: Perform custom tasks or integrations by calling a Function and passing an artifact of request parameters.
• DevOps artifact

  A DevOps artifact is a reference or pointer to any file, binary, package, manifest, or image that makes up your application. When creating an artifact, inform Oracle DevOps of the source location of the actual artifact. DevOps supports OCI Container Image Registry and OCI Artifact Registry repositories.

• OCI Logging and Notification services

  The OCI Logging service stores logs related to the deployment. The deployment runtime output and the final results of the deployment are shown as log entries. The OCI Notifications service provides visibility into the latest state of the deployment project and its resources and takes any necessary action. For example, you are notified when an important event, such as a stage in a deploy pipeline waiting for approval. When you receive the notification message, you can go to the DevOps deployment pipeline and approve the stage.

• Deployment environments

  An environment is a collection of a customer’s computing resources where artifacts are deployed. Environments can be a function, compute virtual machine (VM) or bare metal instance, or an OKE cluster.

  • OKE cluster: OCI Container Engine for Kubernetes is a fully managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud.
  • Compute instances: The OCI Compute service enables you to provision and manage Compute hosts in the cloud. You can deploy Compute instances with shapes that meet your resource requirements for CPU, memory, network bandwidth, and storage.
  • Functions: Oracle Functions is a fully managed, multitenant, highly scalable, on-demand, functions-as-a-service platform. The service is built on enterprise-grade Oracle Cloud Infrastructure and powered by the Fn Project open-source engine.

  Environments can be in different OCI regions from the region of the deployment pipeline. This separation allows developers to deploy in multiple OCI regions using the same deployment pipeline.

• IAM dynamic groups and policies

  A dynamic group is a special type of group that contains resources (such as compute instances) that match rules that you define (thus, the membership can change dynamically as matching resources are created or deleted). These instances act as "principal" actors and can make API calls to services according to policies that you write for the dynamic group.

  A policy is a document that specifies who can access which resources and how. Access is granted at the group and compartment level, which means you can write a policy that gives a group a specific type of access within a specific compartment, or to the tenancy itself.

Recommendations

Use the following recommendations as a starting point. Your requirements might differ from the architecture described here.

• Deployment

  When deploying the architecture via Resource Manager, preserve default values as much as possible if no explicit change is required.

Considerations

When deploying the solution and using the application, consider the following aspects:

• Deploying the solution

  Administrative privileges are required to setup required dynamic groups and associated policies. OCI user name and auth token are required for authorization to OCI DevOps Git repository and setup of secrets in OKE.

• Using the application

  Adjustment of the Python application parameters can be handled in the created DevOps project by modifying the variables at the beginning of the queue_automation.py program and setup of the OCIR secret in OKE.

Deploy

The Terraform code for this deployment, together with the Python application source code, are available on GitHub.

1. Click

   If you aren't already signed in, enter the tenancy and user credentials.

2. Select the region where you want to deploy the stack.
3. Follow the on-screen prompts and instructions to create the stack.
4. After creating the stack, click Terraform Actions, and select Plan.
5. Wait for the job to be completed, and review the plan.

   To make any changes, return to the Stack Details page, click Edit Stack, and make the required changes. Then, run the Plan action again.

6. If no further changes are necessary, return to the Stack Details page, click Terraform Actions, and select Apply.

Explore More

Review these additional resources to learn more about the features of this reference architecture.

• Best practices framework for Oracle Cloud Infrastructure
• OCI Queue
• OCI Functions
• Oracle Cloud Infrastructure documentation
• Oracle Cloud Cost Estimator

Acknowledgments

• Author: Andrei Ilas