Implement OCI Functions to Update an Oracle NoSQL Table Upon Upload to OCI Object Storage
Cloud native, event-driven, and serverless architecture is an approach to building and deploying applications that leverages cloud computing resources and services to maximize scalability, efficiency, and flexibility.
This reference architecture demonstrates the seamless integration of Oracle Cloud Infrastructure Events and OCI Functions to automatically update Oracle NoSQL Database Cloud Service (Oracle NoSQL) tables when objects are created or modified in OCI Object Storage buckets. It addresses the common business scenario of parsing and updating data in an Oracle NoSQL table whenever a file is added or modified in an OCI Object Storage bucket.
OCI Events offers a robust solution for automating tasks based on state changes in various cloud resources. It enables the creation of rules that deliver events to specific OCI services such as OCI Streaming, OCI Functions (which serve as a valuable tool for processing event data and integrating with other OCI services to execute diverse actions), and OCI Notifications.
Furthermore, OCI Functions can incorporate code to handle payload data received from the OCI Events and adapt the Oracle NoSQL table update based on specific business requirements.
This reference architecture provides a comprehensive overview of leveraging OCI Functions in conjunction with OCI Events to update Oracle NoSQL tables when objects are created or modified in OCI Object Storage buckets.
Architecture
This reference architecture demonstrates how to utilize OCI Functions and OCI Events to update an Oracle NoSQL table whenever a file is created or updated in OCI Object Storage.
The architecture diagram showcases a data flow starting with the file upload to a designated OCI Object Storage bucket. Event service triggers are configured to activate based on specified event rule conditions. The emitted event, in turn, invokes a function capable of extracting data from the uploaded file, conducting additional processing as per business requirements, and ultimately updating the Oracle NoSQL table with the processed data. This approach offers significant advantages by capitalizing on event-driven architecture and facilitating real-time data processing. It proves invaluable when there is a need to pre-process data before updating the target table and perform necessary sanity checks to ensure data integrity.
The business use case, where data needs to be parsed and updated in an Oracle NoSQL table whenever a file is created or updated in an OCI Object Storage bucket, can be achieved using the following approach:
- Set up an event rule in OCI Events to trigger OCI Functions whenever an object is created or updated in the OCI Object Storage bucket.
- Create a function that receives the event triggered by the event rule. This function will handle the parsing and updating of data in the Oracle NoSQL table.
- Within the function, implement the necessary logic to parse the data from the object in the OCI Object Storage bucket.
- Use the appropriate OCI SDK or API to interact with the Oracle NoSQL service and insert or update records in the Oracle NoSQL table based on the parsed data.
By combining OCI Events, OCI Functions, and Oracle NoSQL services, we can automate the process of parsing and updating data in the Oracle NoSQL table whenever a file is created or updated in the OCI Object Storage bucket. This approach allows for efficient and streamlined data processing and synchronization between OCI Object Storage and Oracle NoSQL.
The following diagram illustrates this reference architecture.
object-storage-function-nosql-diagram-oracle.zip
Before you begin
- Set up your function application and create an Oracle NoSQL table using the OCI console.
- Gather these values required for configuring the environment variables
for function application.
- User ID
The OCID of the user to authenticate with.
- Tenancy ID
Tenancy OCID. Can be found in user profile.
- Fingerprint
Will be used to authenticate to the OCI API.
- Region
Identifier of the region to create the requests in.
- API private key
API private key file location.
- Compartment OCID
Compartment OCID where the Oracle NoSQL table is created.
- NoSQL table
Sample table person table. Create it with the following attributes.
- User ID
| Primary key | Column name | Type | Shared key | Not null |
|---|---|---|---|---|
| Yes | id | INTEGER | Yes | Yes |
| No | name | STRING | No | No |
| No | age | INTEGER | No | No |
| No | gender | STRING | No | No |
The architecture has the following components:
- Tenancy
A tenancy is a secure and isolated partition that Oracle sets up within Oracle Cloud when you sign up for Oracle Cloud Infrastructure. You can create, organize, and administer your resources in Oracle Cloud within your tenancy. A tenancy is synonymous with a company or organization. Usually, a company will have a single tenancy and reflect its organizational structure within that tenancy. A single tenancy is usually associated with a single subscription, and a single subscription usually only has one tenancy.
- Region
An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).
- Compartment
Compartments are cross-region logical partitions within an Oracle Cloud Infrastructure tenancy. Use compartments to organize your resources in Oracle Cloud, control access to the resources, and set usage quotas. To control access to the resources in a given compartment, you define policies that specify who can access the resources and what actions they can perform.
- Availability domains
Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain is unlikely to affect the other availability domains in the region.
- Fault domains
A fault domain is a grouping of hardware and infrastructure within an availability domain. Each availability domain has three fault domains with independent power and hardware. When you distribute resources across multiple fault domains, your applications can tolerate physical server failure, system maintenance, and power failures inside a fault domain.
- Virtual cloud network (VCN) and subnets
A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.
- Security list
For each subnet, you can create security rules that specify the source, destination, and type of traffic that must be allowed in and out of the subnet.
- Service gateway
The service gateway provides access from a VCN to other services, such as Oracle Cloud Infrastructure Object Storage. The traffic from the VCN to the Oracle service travels over the Oracle network fabric and never traverses the internet.
- Object storage
Object storage provides quick access to large amounts of structured and unstructured data of any content type, including database backups, analytic data, and rich content such as images and videos. You can safely and securely store and then retrieve data directly from the internet or from within the cloud platform. You can seamlessly scale storage without experiencing any degradation in performance or service reliability. Use standard storage for "hot" storage that you need to access quickly, immediately, and frequently. Use archive storage for "cold" storage that you retain for long periods of time and seldom or rarely access.
- Events
Oracle Cloud Infrastructure services emit events, which are structured messages that describe the changes in resources. Events are emitted for create, read, update, or delete (CRUD) operations, resource lifecycle state changes, and system events that affect cloud resources.
- Functions
Oracle Cloud Infrastructure Functions is a fully managed, multitenant, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform. It is powered by the Fn Project open source engine. Functions enable you to deploy your code, and either call it directly or trigger it in response to events. Oracle Functions uses Docker containers hosted in Oracle Cloud Infrastructure Registry.
- LoggingLogging is a highly scalable and fully managed service that provides access to the following types of logs from your resources in the cloud:
- Audit logs: Logs related to events emitted by the Audit service.
- Service logs: Logs emitted by individual services such as API Gateway, Events, Functions, Load Balancing, Object Storage, and VCN flow logs.
- Custom logs: Logs that contain diagnostic information from custom applications, other cloud providers, or an on-premises environment.
- Oracle NoSQL
Oracle NoSQL Database Cloud Service is a fully managed database cloud service that is designed for database operations that require predictable, single-digit millisecond latency responses to simple queries. Once you are authenticated against your Oracle Cloud account, you can create an Oracle NoSQL table, and specify throughput and storage requirements for the table. Oracle reserves and manages the resources to meet your requirements, and provisions capacity for you. Capacity is specified using read and write units for throughput and GB for storage units.
Recommendations
- VCN
When you create a VCN, determine the number of CIDR blocks required and the size of each block based on the number of resources that you plan to attach to subnets in the VCN. Use CIDR blocks that are within the standard private IP address space.
Select CIDR blocks that don't overlap with any other network (in Oracle Cloud Infrastructure, your on-premises data center, or another cloud provider) to which you intend to set up private connections.
After you create a VCN, you can change, add, and remove its CIDR blocks.
When you design the subnets, consider your traffic flow and security requirements. Attach all the resources within a specific tier or role to the same subnet, which can serve as a security boundary.
- Cloud Guard
Clone and customize the default recipes provided by Oracle to create custom detector and responder recipes. These recipes enable you to specify what type of security violations generate a warning and what actions are allowed to be performed on them. For example, you might want to detect Object Storage buckets that have visibility set to public.
Apply Cloud Guard at the tenancy level to cover the broadest scope and to reduce the administrative burden of maintaining multiple configurations.
You can also use the Managed List feature to apply certain configurations to detectors.
- Functions
The OCI function in this architecture was developed using Python. Please note that OCI Functions supports multiple programming languages and you can use the language of your choice to develop the functions and deploy them.
- Oracle NoSQL
Ensure that you create the Oracle NoSQL table in the appropriate compartment and manage access using OCI Identity and Access Management (IAM) policies. It is important to estimate the read and write capacity accurately before creating the table. Refer to the Oracle documentation and review the recommendations for detailed guidance.
In this solution, a sample table named "person" is used as an example. Remember to customize the function and table name according to your specific requirements.
- OCI Object Storage
This architecture uses Standard Object Storage to upload a file. Please make sure to enable Emit Object events so that event services can be triggered. It is also recommended to use private buckets for sensitive data.
- OCI Events
In this architecture, the OCI Events service is configured to listen to changes in OCI Object Storage for "Object Create" and "Object Update" events. The service is invoked after the object is uploaded to OCI Functions and calls the function for processing.
Considerations
Consider the following points when deploying this reference architecture.
- Performance
OCI Events, OCI Functions, and Oracle NoSQL are highly scalable, managed services. Oracle NoSQL service provides on-demand throughput and storage provisioning.
Please make sure to estimate throughput and storage capacities for your Oracle NoSQL Database Cloud Service before provisioning.
- Security
Use policies to restrict who can access the OCI resources.
For OCI Object Storage, encryption is enabled by default and can’t be turned off.
It is recommended to store secrets and sensitive data in OCI Vault. Please consider using OCI Functions for storing API keys, DB usernames, passwords, and auth tokens used for authorization with OCI services.
- Availability
Oracle ensures high availability of the OCI Events, OCI Functions, Oracle NoSQL, and OCI Logging, which are cloud native and fully managed.
- Cost
You pay for only the resources used while a function is running.
Deploy
The GitHub repository contains the OCI Functions code that implements this reference architecture.
- Go to GitHub.
- Clone or download the repository to your local computer.
- Follow the instructions in the
READMEdocument.