Integrate OCI Functions and OCI Event services to publish messages to private OSS streams
Cloud-native, event-driven architecture leverages cloud resources and events as the main communication mechanism between services. It enables scalable, resilient, and agile systems by decoupling components and allowing real-time reactions to events. It utilizes event streaming and serverless computing for distributed event-driven communication and processing.
Oracle Cloud Infrastructure Events is a powerful tool for creating automation based on state changes in various cloud resources. It allows us to create rules that deliver events to specific OCI resources such as Oracle Cloud Infrastructure Streaming, OCI Functions, and Oracle Cloud Infrastructure Notifications.
In certain scenarios, there is a need to publish messages to private streams. Although OCI Events can deliver events directly to public streaming endpoints, currently it does not support delivering messages directly to private streaming endpoints since private endpoints can only be resolved within the same VCN and OCI Events does not have access to push messages to a private stream. This can, however, be achieved by using OCI Functions as an action type in your events rule.
This reference architecture provides a comprehensive overview of how to leverage OCI Functions with OCI Events to deliver messages to private streaming endpoints.
Architecture
This reference architecture demonstrates how to utilize OCI Functions and OCI Events to publish messages to a private OSS streaming endpoint whenever a file is created or updated in Oracle Cloud Infrastructure Object Storage.
The architecture diagram illustrates the data flow, which starts with the upload of a file to a specific bucket in OCI Object Storage. OCI Events is triggered based on defined event rule conditions. The emitted event then invokes a function that extracts data from the uploaded files and publishes both the data and event messages to a private streaming endpoint configured within OCI Functions. It's important to note that the streaming service and function should be created in the same virtual cloud network (VCN) and private subnet to ensure the OCI function can access the private streaming message endpoint.
The following diagram illustrates this reference architecture.
oci-functions-events-integration-diagram-oracle.zip
Before you begin
- Set up your OCI Functions application and OCI Streaming service (stream pool and stream).
- Please gather the following information. These values are required to
configure the environment variables for OCI Functions.
- User ID
The OCID of the user to authenticate with.
- Tenancy ID
Tenancy OCID. Can be found in user profile.
- Fingerprint
Will be used to authenticate to the OCI API.
- Region
Identifier of the region to create the requests in.
- API private key
API private key file location.
- Stream OCID
OSS Stream OCID.
- Auth Token
Auth Token from User settings.
- Stream endpoint
OSS Stream endpoint (Obtained from "Messages Endpoint" from Stream information screen).
- User ID
The architecture has the following components:
- Tenancy
A tenancy is a secure and isolated partition that Oracle sets up within Oracle Cloud when you sign up for Oracle Cloud Infrastructure. You can create, organize, and administer your resources in Oracle Cloud within your tenancy. A tenancy is synonymous with a company or organization. Usually, a company will have a single tenancy and reflect its organizational structure within that tenancy. A single tenancy is usually associated with a single subscription, and a single subscription usually only has one tenancy.
- Region
An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).
- Compartment
Compartments are cross-region logical partitions within an Oracle Cloud Infrastructure tenancy. Use compartments to organize your resources in Oracle Cloud, control access to the resources, and set usage quotas. To control access to the resources in a given compartment, you define policies that specify who can access the resources and what actions they can perform.
- Availability domains
Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain is unlikely to affect the other availability domains in the region.
- Fault domains
A fault domain is a grouping of hardware and infrastructure within an availability domain. Each availability domain has three fault domains with independent power and hardware. When you distribute resources across multiple fault domains, your applications can tolerate physical server failure, system maintenance, and power failures inside a fault domain.
- Virtual cloud network (VCN) and subnets
A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.
- Security list
For each subnet, you can create security rules that specify the source, destination, and type of traffic that must be allowed in and out of the subnet.
- Service gateway
The service gateway provides access from a VCN to other services, such as Oracle Cloud Infrastructure Object Storage. The traffic from the VCN to the Oracle service travels over the Oracle network fabric and never traverses the internet.
- Object storage
Object storage provides quick access to large amounts of structured and unstructured data of any content type, including database backups, analytic data, and rich content such as images and videos. You can safely and securely store and then retrieve data directly from the internet or from within the cloud platform. You can seamlessly scale storage without experiencing any degradation in performance or service reliability. Use standard storage for "hot" storage that you need to access quickly, immediately, and frequently. Use archive storage for "cold" storage that you retain for long periods of time and seldom or rarely access.
- Events
Oracle Cloud Infrastructure services emit events, which are structured messages that describe the changes in resources. Events are emitted for create, read, update, or delete (CRUD) operations, resource lifecycle state changes, and system events that affect cloud resources.
- Streaming
Oracle Cloud Infrastructure Streaming provides a fully managed, scalable, and durable storage solution for ingesting continuous, high-volume streams of data that you can consume and process in real time. You can use Streaming for ingesting high-volume data, such as application logs, operational telemetry, web click-stream data; or for other use cases where data is produced and processed continually and sequentially in a publish-subscribe messaging model.
- Functions
Oracle Cloud Infrastructure Functions is a fully managed, multitenant, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform. It is powered by the Fn Project open source engine. Functions enable you to deploy your code, and either call it directly or trigger it in response to events. Oracle Functions uses Docker containers hosted in Oracle Cloud Infrastructure Registry.
- LoggingLogging is a highly scalable and fully managed service that provides access to the following types of logs from your resources in the cloud:
- Audit logs: Logs related to events emitted by the Audit service.
- Service logs: Logs emitted by individual services such as API Gateway, Events, Functions, Load Balancing, Object Storage, and VCN flow logs.
- Custom logs: Logs that contain diagnostic information from custom applications, other cloud providers, or an on-premises environment.
Recommendations
- VCN
When you create a VCN, determine the number of CIDR blocks required and the size of each block based on the number of resources that you plan to attach to subnets in the VCN. Use CIDR blocks that are within the standard private IP address space.
Select CIDR blocks that don't overlap with any other network (in Oracle Cloud Infrastructure, your on-premises data center, or another cloud provider) to which you intend to set up private connections.
After you create a VCN, you can change, add, and remove its CIDR blocks.
When you design the subnets, consider your traffic flow and security requirements. Attach all the resources within a specific tier or role to the same subnet, which can serve as a security boundary.
- Cloud Guard
Clone and customize the default recipes provided by Oracle to create custom detector and responder recipes. These recipes enable you to specify what type of security violations generate a warning and what actions are allowed to be performed on them. For example, you might want to detect Object Storage buckets that have visibility set to public.
Apply Cloud Guard at the tenancy level to cover the broadest scope and to reduce the administrative burden of maintaining multiple configurations.
You can also use the Managed List feature to apply certain configurations to detectors.
- OCI Functions
The OCI function in this architecture was developed using Python. Please note that OCI Functions supports multiple programming languages and you can use the language of your choice to develop the functions and deploy them.
- OCI Streaming
The Stream pool is created using a private subnet in a VCN. And the stream is created using this stream pool. Please make sure to create the function in the same VCN as the private OSS stream and have all necessary access.
Streaming also takes advantage of the Kafka Connect ecosystem to interface directly with first-party and third-party products by using out-of-the-box Kafka source and sink connectors. See Explore More for more information about using Kafka Connect.
- OCI Object Storage
This architecture uses Standard Object Storage to upload a file. Please make sure to enable "Emit Object events" so that even service can be triggered. It is also recommended to use private buckets for sensitive data.
- OCI Events
In this architecture, the OCI Events is configured to listen to changes in OCI Object Storage for "Object Create" and "Object Update" events. The service is invoked after the object is uploaded to OCI Object Storage and calls the function for processing.
Considerations
When implementing this reference architecture, it is important to take into account the following aspects.
- Performance
OCI Events, OCI Functions, and OCI Streaming are highly scalable. Consider adjusting the number of partitions and streams, based on the size and number of files that you expect.
- Security
Use policies to restrict who can access the OCI resources.
For OCI Object Storage, encryption is enabled by default and can’t be turned off.
All access to functions deployed in OCI Functions is controlled through Oracle Cloud Infrastructure Identity and Access Management (IAM), which allows both function management and function invocation privileges to be assigned to specific users and user groups.
It is recommended to store secrets and sensitive data in OCI Vault. Please consider using OCI Vault for storing API keys and auth token used for authorization with OCI services.
- Availability
Oracle ensures high availability of the OCI Functions, OCI Events, OCI Streaming, and OCI Logging, which are cloud native and fully managed. OCI Streaming includes the following high-availability capabilities:
- Constant flow of log data
- Multi-threaded and horizontally scalable service
- Near real-time ingestion
- Resilience against short-term outages
- Optimized for efficient data usage
- Cost
You pay for only the resources used while a function is running.
Deploy
The GitHub repository contains the OCI Functions code that implements this reference architecture.
- Go to GitHub.
- Clone or download the repository to your local computer.
- Follow the instructions in the
READMEdocument.