About Loading Data Into Oracle Cloud ERP Using Serverless Technologies
As an Oracle SaaS customer, you may prefer the licensing model and ease of management of SaaS: pay by consumption of resources or by the number of "entities" you create, and manage nothing except your own data. When building SaaS extensions you can enjoy the same experience. By using Oracle's serverless options, you do not have to maintain an operating system, patches, a firewall, or runtime libraries. "Serverless" technology does not mean there are no servers, but rather that the developer doesn't manage them, so they are essentially hidden from the developer. In a serverless environment you deploy your code and configurations to the cloud and Oracle manages them.
Oracle Functions, Oracle Cloud data storage, and other OCI features have scaling pricing based on usage. You only pay for storage and compute resources that you consume. If your Functions aren't being used at the moment, there is no cost; if your storage is empty at the moment, there is no cost, and so on.
Oracle's premier product for moving and transforming data between systems is Oracle Integration. Oracle Integration allows integrators the ability to create integrations using declarative constructs, perform transformations in the cloud, run the actual data load in the cloud, automatically manage exceptions and retries, and do so without writing code. For heavyweight or low-code/no-code solutions, Oracle Integration can be a great option. But one use case where serverless might be useful is for customers who are loading data into SaaS infrequently, or where the integration flows are straightforward.
This solution playbook shows you how to implement a data load pattern using serverless technologies, and highlights how the fundamental pattern can be extended to include notifications and other enhancements.
Architecture
This architecture shows how you can create a serverless architecture for loading data into Oracle Cloud ERP.
To demonstrate how serverless technologies can be used to load data into Oracle Fusion Cloud we have built a sample using only OCI serverless services and a 3GL code-based approach.
load-data-serverless-architecture-oracle.zip
This architecture supports the following components:
- Functions
Oracle Functions is a fully managed, multitenant, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform. It is powered by the Fn Project open source engine. Functions enable you to deploy your code, and either call it directly or trigger it in response to events. Oracle Functions uses Docker containers hosted in Oracle Cloud Infrastructure Registry.
- Events
Oracle Cloud Infrastructure services emit events, which are structured messages that describe the changes in resources. Events are emitted for create, read, update, or delete (CRUD) operations, resource lifecycle state changes, and system events that affect cloud resources.
- Notifications
The Oracle Cloud Infrastructure Notifications service broadcasts messages to distributed components through a publish-subscribe pattern, delivering secure, highly reliable, low latency, and durable messages for applications hosted on Oracle Cloud Infrastructure.
- Vault
Oracle Cloud Infrastructure Vault enables you to centrally manage the encryption keys that protect your data and the secret credentials that you use to secure access to your resources in the cloud. You can use the Vault service to create and manage vaults, keys, and secrets.
- API Gateway
Oracle API Gateway enables you to publish APIs with private endpoints that are accessible from within your network, and which you can expose to the public internet if required. The endpoints support API validation, request and response transformation, CORS, authentication and authorization, and request limiting.
- Object storage
Object storage provides quick access to large amounts of structured and unstructured data of any content type, including database backups, analytic data, and rich content such as images and videos. You can safely and securely store and then retrieve data directly from the internet or from within the cloud platform. You can seamlessly scale storage without experiencing any degradation in performance or service reliability. Use standard storage for "hot" storage that you need to access quickly, immediately, and frequently. Use archive storage for "cold" storage that you retain for long periods of time and seldom or rarely access.
- Fusion Applications
The Oracle Fusion Cloud Applications Suite offers dozens of cloud apps across every business category, including ERP, SCM, HCM, Sales, Service, and Industry verticals.
This example architecture demonstrates how you can create a serverless architecture for loading data into Oracle Cloud ERP. It demonstrates a pattern of using OCI Buckets (a feature of Object Storage), Events, Functions, Vault, Notifications, and API Gateway to implement the integration pipeline. The pattern highlights how the flow can be extended to include notifications and further enrich the integration. This pattern is code-based and serverless, and when the system is idle the only cost to the consumer is the storage of the files in OCI Object Storage.
If you have a lot of integrations to implement and these integrations will be changed frequently then Oracle Integration is a good alternative. Oracle Integration provides a rich declarative development environment where integrations can be built, deployed quickly, and scale to meet your needs. Additionally Oracle Integration gives you a lot of extra built-in functionality, such as error handling, retrying of messages, dashboards, reporting, state tracking of previous requests, and more.
However, if you prefer a serverless approach, a code-centric style of integration, and the flexibility and extensibility this pattern provides, then use this architecture as a starting point for your own Functions-based serverless SaaS extension.
About Loading Data into Oracle Cloud ERP
To develop this application it is necessary to understand the flow for loading data into Oracle Cloud ERP.
Ther following steps are typical of the requirements when loading data into Oracle Cloud ERP:
- Enrich the incoming data with some external data, and transform the data into the Oracle Cloud ERP data loading format (FBDI)
- Load the file into Oracle Cloud ERP using REST or SOAP Web Services
- Listen for a call-back from Oracle Cloud ERP indicating that the process has completed and that the data has been processed.
Enrich and Transform
When loading data into Oracle Cloud ERP a mandatory step is to transform the input data into the required CSV format and then zip up the files into a single ZIP file. In the example in this playbook this transform step is demonstrated by accepting a simplified JSON data structure which is then transformed into the CSV format required by Oracle Cloud ERP. Then the files are combined into a single ZIP file ready to be uploaded.
Load
Once the data has been transformed it can be loaded into Oracle Cloud ERP using Fusion REST APIs. The response from Oracle Cloud ERP contains some metadata and the JobID being used to load the data. The JobID is needed later to determine which data file is being imported by which job.
Process the call-back
Oracle Cloud ERP issues a call-back to the client which indicates if the job was processed correctly or if there was an error. Note that a success code does not necessarily mean the data was all loaded: for example, there could have been duplicate records.
Before You Begin
You'll need to create various objects in an OCI instance, including storage buckets, funcitons, and Oracle Cloud Infrastructure Vault. The following references may be helpful:
- Review Example 1: ImportBulkData Operation in Create an inbound or outbound set of data, in REST API for Oracle Fusion Cloud Financials.
- To create buckets in OCI, review Putting Data into Object Storage in Oracle Cloud Infrastructure Documentation.
- To create Functions in OCI, review Creating, Deploying, and Invoking a Helloworld Function in Oracle Cloud Infrastructure Documentation.
- Review key management with Oracle Vault in Oracle Cloud Infrastructure Documentation.
About Required Services and Roles
This solution requires the following services:
- Oracle Fusion ERP
- Oracle Cloud Infrastructure, including Oracle API Gateway, Oracle Functions, and other OCI components
These are the roles needed for each service.
| Service Name: Role | Required to... |
|---|---|
| Oracle Fusion ERP: Implementor | Load Fusion ERP data via APIs |
| Oracle Cloud Infrastructure: Administrator | Create accounts in OCI IAM, deploy OCI components and Functions |
See Learn how to get Oracle Cloud services for Oracle Solutions to get the cloud services you need.