Integration Best Practices
Oracle Utilities applications integrate with each other or with other external applications to enable new functionalities or to exchange data needed to verify, process, or complete transactions.
This section presents common integration use cases and demonstrates how they can be used efficiently without impacting the overall cloud services performance. This includes:
Primary Integration Methods
As noted earlier in this chapter, the primary integration methods supported with the cloud services are:
Figure 1. Diagram illustrating integration options
Inbound and Outbound Files
File processing can be implemented in two primary ways: File Import as Inbound and Standard Extracts as Outbound, depending on the direction of data exchange. The only supported storage with all cloud services is Cloud Object Storage where separate buckets can be created to separate inbound and outbound files.
File Import or Inbound:
Files can be dropped into the Cloud Object Storage and then processed using an Upload Batch Process. There are various fit for purpose upload processes provided by the product. For example, payment upload.
In addition, the product provides a plug-in driven upload. The upload treats one file as one unit of work and streams it to an algorithm that is responsible for parsing the data and adding the data to a table with minimal validations. One run of the upload process can process multiple files, and each file is treated as a unit of work. Multi-threading may occur if there are multiple files.
For more information on the product recommendations related to the responsibility of the upload algorithm, refer to Uploading Records in the Application Framework Administrative User Guide.
Standard Extracts or Outbound:
An Extract Batch Process can be used to export data from an application object into a file, which is then placed in a designated location for consumption by other systems.
Example: Exporting financial information such as General Ledger (GL) and Accounts Payable (AP) data to be picked up by an external financial system.
Best Practice:
• The recommendation is to split larger files into manageable sizes so that the resulting multiple files can be processed multi-threaded.
• Batch processing should not be executed during peak business hours.
• Use auto archiving feature on cloud object storage to cleanup the buckets for better manageability and improve batch performance.
Inbound and Outbound Web Services
• API Data Integration: External or other utilities applications can interface to another utilities application via Inbound Web Services. Using a SOAP or a REST API to communicate and share data between applications. Inbound Web Services can be used for real-time and non-real-time integrations:
• Real-time Example: Searching for available appointments from an external scheduling tool.
• Non-real-time Example: Recording in the Customer Information System the details of customer notifications sent out, where the request is submitted via batch, Oracle Cloud Infrastructure (OCI) stream, or message queue.
• Outbound Messages: Utilities applications can initiate real time or non-real time outbound communications to other applications.
• Real-time outbound messages are used for critical or time-sensitive processes.
• Example: Reporting an outage to an external system as soon as it is logged.
• Non-real-time outbound messages are typically submitted using a batch monitor or scheduled process.
• Example: Sending updates to customer information details at scheduled intervals.
Common Use Cases
Use Case 1: Data Synchronization Between Applications
Data is generally maintained in a designated system of record, which serves as the authoritative source for that data. In many cases, this data needs to be synchronized to other systems to support or validate business processes.
Example: Copying service point information from a Customer Information System (CIS) application to an Asset Management Application or Meter Data Management application.
Use Case 2: Data Aggregations
This use case applies where a portal is required to pull data from multiple applications. The goal is to aggregate data from various sources for reporting or contextual display, helping users get a comprehensive view in one place. This improves the user experience by consolidating information (as customer, asset, or meter data) into a single, actionable interface.
Example: Building a customer 360 view or dashboard of different customer related information
Use Case 3: Data Processing and Orchestration
Additional data is needed in another application for data processing and might sometimes involve a middleware application for data orchestration and provide additional validation.
Examples: For a credit card payment to be processed, the credit card must be verified by a payment processing application or when a customer support representative is booking an appointment.
To optimize performance, real-time integration should be used only when batch processing is not feasible.
Example: In a Start Service process, creating a field activity and sending it to another system does not require immediate processing and can be done in non-real-time using a batch job.
Favoring batch over real-time where possible improves scalability, reliability, and system performance.
Use Case 4: Bulk or Batch Data Transfer
Data that does not need to be transferred in real-time or transfer of huge volumes of data can be done by a batch process. Such scenarios optimize and provide better utilization of the application's resources.
Examples: One-time migration of customer information from an old CIS application or periodic transfer of General Ledger (GL) transactions from a CIS application to an Enterprise Resource Planning (ERP) application.
Best Practices
Data Synchronization
Data is maintained and mastered in a designated system of record.. Data synchronization to another application is sometimes needed to validate or complete a business transaction.
To perform this synchronization, Business Objects (BOs) are used. A sync request BO defines the behavior and lifecycle of the synchronization. Data is typically synchronized using a batch process.This process should be asynchronous to prevent incorrect handling of a two-phase commit flow. In this situation, if an error occurs after the message to update the external system is committed but before changes are committed in the current system, then the data sent to the other system is out of synch. Retrying the update may create duplicates in the external system, and if the update is never processed again, then an incorrect record exists in the external system. The correct update flow should be to update the external system only after the changes in the current system are committed, which is by deferring the external update to the corresponding monitor batch process.
This is achieved by using the monitor batch process, which is considered best practice. As such, you should not remove the monitor process on any state of the sync request business objects.
If near real-time processing of sync requests is needed, schedule the sync request monitor jobs frequently.
Configuring Message Senders
Message Senders are used to define how outbound messages are sent out of the application (UPL, security, etc.)
Aside from configuring the HTTP User/Password, HTTP URL, HTTP Method and HTTP Header, make sure the following context values are also populated:
• Making sure customers always set a 'HTTP Timeout' Message Sender Content Value for any outbound scenarios they have.
• Customers have o/b integrations dependent on an external system but they don't include a timeout, so if the external system is down or responding slowly, our transaction pays the price. HTTP Timeout should be considered to avoid a 2 minute wait if the external system is not responding.
• Populate HTTP Transport Method to
• SendReceive - When invoking a Synchronous Flow, meaning an immediate response is expected,
• Send - When invoking an Asynchronous Flow, meaning no response is expected.
Creating Custom Inbound Web Services (IWS)
When creating a custom Inbound Web Service, ensure it adheres to performance and design standards-such as avoiding full table scans and processing only a small, manageable set of records per call.
Inbound Web Services should be built for efficient, targeted operations rather than bulk processing.
Examples of typical IWS use cases include:
• Passing a service request logged by a customer through a portal into the system.
• Creating a refund transaction in the system based on an external trigger or approval.
Design with scalability in mind to ensure responsiveness and minimize system load.
REST APIs typically operate on a single resource. They may perform basic maintenance actions on the resource, search for data related to the resource, or process a manageable list of resources. When designing a new REST API, a good way of thinking about the intent of a REST operation is to ask these questions - What is being 'operated on' or 'searched for' or 'created' or 'deleted'? Once the resource is determined, you may think through the various operations that can be done on the resource.
There are two main types of "search" operations:
• Search across the entire population of a resource using filters. This is usually a 'starting point' search for top-level objects such as Person, Billing Account or Premise, etc also referred to as 'unbounded' search where the number of results very much depends on the specificity of the search criteria. Such search must cap the number of results to some reasonable number, for example a few hundred rows, and must not offer 'total rows' or 'ordering' capability as this would involve processing the entire list of matches.
• Search across a subset of objects which are related to one particular chosen 'context' object, for example Premises for an Address, Bills for an Account, Cases for a Premise, etc. This is referred to as a 'bounded search' which can be 'paged through'. While the average number of results may be small, it is possible for it to be slightly larger. In these cases, the query should offer pagination with standard 'offset' and 'limit' query parameters. If the caller does not specify a limit, the service must impose a default limit value. Base inbound web services are provided that use this technique.
These search queries must be carefully crafted to perform well, backed by proper indexes and follow an efficient execution plan.
Usage of Inbound Web Services should be limited to the following to fetching / adding/updating one transaction record for one call:
• Not recommended: Extraction of mass data via inbound web services
• Data migration should not be performed via Inbound Web Services.
Please note:
• Performance tuning of any custom IWS will be the customer's responsibility.
• All the performance testing of any custom integration including file uploads needs to be tested before go-live.