Common Integration Style Pitfalls and Design Best Practices

Note the following best practices and integration style pitfalls to avoid when designing an integration.

Avoid Common Integration Style Pitfalls

Designing integrations correctly from the start can save you a tremendous amount of rework. This section describes common integration style pitfalls (known as antipatterns) and best practices for avoiding these pitfalls.

Chatty Integrations

Use Case: Synchronize records in a file or large data set with an external system (for example, synchronizing journal transactions or uploading employee records into Oracle HCM Cloud).

Antipattern	Why an Antipattern?	Best Practice
Use an invoke activity within a looping construct to call external APIs for every record.	Downstream applications are receiving a large number of atomic requests. This puts the entire system under duress. A usage-based pricing model translates to high costs.	Leverage application capabilities to accept multiple records in a single request: Salesforce: 200 records, Oracle Engagement Cloud/ERP Cloud: 100 records, Service Cloud: 1000 records Leverage adapter capabilities to send a large data set as attachments/files. Salesforce Adapter: 10,000 records/10 MB files, Oracle ERP Cloud Adapter supports FBDI files Use a stage file action (the append to file option) - send the file to the destination at the end.

Scheduled Job that Never Stops Trying to Process

Use Case: Process records within a set of files with a tight service level agreement (SLA) (for example, synchronize employee records into Active Directory from Oracle HCM Cloud or send benefits information).

Antipattern	Why an Antipattern?	Best Practice
The scheduled integration looks for all files to process and loops over all to sequentially process until no files remain.	If a large number of files exist, one run of a scheduled job executes for a long time and starves other jobs and may get terminated by the framework. Processing tied to a single server: this does not leverage multiple nodes in a cluster.	Limit the number of files to process in a single scheduled run. Limit results from the list file. Use schedule parameters to remember the last processed file for the next run. Invoke the run now command from the same integration to trigger processing of the next file if waiting for the next scheduled run is not feasible.

Import an Externally Updated IAR File

Use Case: Need to leverage advanced XSL constructs that may not be available in the mapper.

Antipattern	Why an Antipattern?	Best Practice
Updating the IAR file externally and then importing it into Oracle Integration.	This can lead to metadata inconsistency and validation failures. Activation failures may occur.	Use the import map feature in Oracle Integration. Keeps metadata consistent and leverages validation.

Synchronous Integration Doing Too Much

Use Case: A request triggers complex processing involving enrichment and updates across multiple systems.

Antipattern	Why an Antipattern?	Best Practice
Huge synchronous integrations modeling a large number of invokes /conditional logic. Synchronous integrations with invokes within a loop with a large number of iterations.	Susceptible to timeouts - Any marginal slowdown adds up. Blocking call – Holds resources and starves other integrations.	Explore moving completely to an asynchronous integration - fire and forget, asynchronous response. Integration platform provides an acknowledgment to clients upon receiving the message. Platform ensures guaranteed processing and also supports resubmission of failures. Split into a synchronous integration containing mandatory processing before sending out a response and triggering separate asynchronous fire and forget integrations for other processing logic. Optimize synchronous processing with a coarse-grained external API to replace multiple chatty calls.

Too Many Connections in an Integration

Use Case: As developers create integrations, they define their own connections pointing to the same application. This leads to many duplicate connections.

Antipattern	Why an Antipattern?	Best Practice
Every developer creates their own connection using a different set of configurations/credentials.	High number of connections make manageability painful, especially when you need to update the endpoint, credentials, configuration, and so on. Complicates impact analysis when there is an application upgrade or metadata/coordinate change.	Have a custodian create needed connections and ensure duplicate connections of the same types are not created. Build a best practice for naming conventions and maintaining a set of configurations.

Read Files with Many Records

Use Case: Read a file with a large number of records and process individual records.

Antipattern	Why an Antipattern?	Best Practice
Reading the entire file in memory using the read file option and processing record by record.	Consumes large amounts of memory and impacts other system processing. Sequential processing does not leverage built-in map reduction capabilities.	Download the file to the stage location using the download file option. Stage files are secured and deleted at the completion of the instance. Use the read file with segments option. You can still model record level processing inside the segment processing. Platform automatically processes segments in parallel. Platform brings in only portions of the file to memory, as needed.

Integrations Running Unchanged Despite Changing Business Needs

Use Case: Integrations/schedules created during the initial implementation continue to run even though your business requirements have changed over time.

Antipattern	Why an Antipattern?	Best Practice
Integrations and scheduled jobs created during the initial product implementation are never re-evaluated against changing business needs. A proliferation of integrations occurs without looking at existing integrations.	Unnecessary runs of jobs that handle no work. Unoptimized invocations of multiple integrations with similar functionality. Clutter with dead integrations, life cycle management (LCM) overheads, and developer confusion.	Periodically analyze existing integrations/schedules against current business needs. Look at monitoring data for historical runs. Consolidate integrations that are very similar. Integrations A and B subscribing to the Customer Created event in Oracle Engagement Cloud – integration A for invoking endpoint A and integration B for invoking endpoint B. Deactivate integrations that are no longer relevant. An integration for generating a file meant for partner consumption becomes irrelevant when the partner no longer needs it. Adjust the schedules for integrations that need lesser frequency and delete schedules no longer required. Files generated monthly have schedules run every few minutes. Changes in business needs can make certain schedules completely unnecessary.

Avoid Creating Too Many Scheduled Integrations

When there are too many scheduled integrations configured, instances can get backlogged waiting for resources to become available or previous integration runs to complete. This can cause processing delays where some instances are in a waiting state longer than they should be and schedules may not start at the scheduled time.

As a best practice, it is recommended that you do not create too many scheduled integrations that are scheduled to run at the same time. Where possible:

Use an asynchronous REST Adapter trigger instead of a scheduled trigger when an active schedule is not absolutely required.
Do not create any long-running scheduled integrations (a scheduled integration that takes longer than one hour, for example, to complete). This blocks scheduler resources impacting other scheduled runs.
Spread schedules out over time to avoid schedule clusters.

You can convert a scheduled integration to an application-driven orchestrated, REST Adapter trigger-based integration. See Convert a Scheduled Integration to a REST Adapter-Triggered Orchestration Integration.

If you absolutely require a large number of scheduled integrations, and you are encountering the issue described above, we recommend the following design changes as a solution:

For each scheduled integration, convert it to an application-driven orchestrated, REST Adapter trigger-based integration. See Convert a Scheduled Integration to a REST Adapter-Triggered Orchestration Integration.
Create a new scheduled integration that only performs an asynchronous invoke of the application-driven orchestrated integration you converted in step 1 above.
This solution enables the scheduled integration to start at the scheduled time, invoke the REST Adapter trigger-based child integration asynchronously, and complete within milliseconds. This approach reduces the backlog and contention surrounding scheduler resources.
If you have a large number of scheduled integrations to convert, a staged approach starting with the following integrations is recommended.
- Longest-running scheduled integrations.
- Scheduled integrations configured with the shortest frequency (for example, integrations that run every 10 minutes or less).
Design any new scheduled integrations with the design practices described above.

See Scheduled Integrations Are Not Executing on Time and When a Scheduled Integration Instance Gets Terminated.

Synchronous Integration Best Practices

Note the following best practices when designing a synchronous integration.

A synchronous integration calling any asynchronous request response service:
- Calling asynchronous fire and forget (one-way) is acceptable.
- Oracle Integration does not currently allow modeling an asynchronous request response service. However, all scheduled orchestration styles internally use an asynchronous request response. Therefore, a synchronous integration using a scheduled orchestration is an antipattern.
A synchronous integration calling multiple services that span more than 5 minutes is reported as a stuck thread in Oracle WebLogic Server.

Design Long-Running or Time-Consuming Integrations as Asynchronous Flows

Note the following best practice when designing long-running or time-consuming integrations.

Do not expose long-running or time-consuming integrations as synchronous flows. This action can lead to client applications (including other integrations) timing out. Synchronous integrations also have a server-side time out. Instead, model any synchronous integration taking more than two minutes as an asynchronous flow.

Time Outs in Service Calls During Synchronous Invocations

You may have scenarios in which synchronous invocations from Oracle Integration (including calls to other integrations) are blocking calls and must complete within 300 seconds.

Because the call may involve one or more proxies, each of the proxies may have a similar time out. For instance, the default proxy on Oracle Public Cloud has a time out value of 120 seconds. If the call is to an on-premises service behind a fire wall, the proxy configured may also have its own time out value.

In case time outs are defined at multiple layers, the service invocation fails at the first time out.

Parallel Processing in Outbound Integrations

While there is no specific integration design to automatically enable an outbound integration to send data to different third party systems in parallel, there are integration design approaches that allow this scenario.

Separate the integration into multiple integrations:

Create a main parent integration that only receives/processes the data.
Create separate child integrations to perform the individual outbound REST invocations.

The interface between the main and separate child integrations can follow these approaches:

Consist of dummy REST calls, but it must be asynchronous. Essentially, the asynchronous calls are not blocked by the response and the fire-and-forget design enables available threads to work on child integration processing in parallel, within the available system resources. This type of design is recommended because if all synchronous REST calls are done in the same integration, a time out error may occur if the sum of time taken for each synchronous call exceeds five minutes.
Follow a publish/subscribe design approach (for example, putting the data events in a queue, having each child flow subscribe from the queue, and so on).