Common Integration Pattern Pitfalls and Design Best Practices

Understand the following best practices and integration pattern pitfalls to avoid before you design an integration.

• Avoid Common Integration Pattern Pitfalls
• Avoid Creating Too Many Schedule Integrations
• Synchronous Integration Best Practices
• Design Long-Running or Time-Consuming Integrations as Asynchronous Flows
• Timeouts in Service Calls During Synchronous Invocations
• Parallel Processing in Outbound Integrations

Avoid Common Integration Pattern Pitfalls

Designing integrations correctly from the start can save you a tremendous amount of rework. This section describes common integration pattern pitfalls (known as antipatterns) and best practices for avoiding these pitfalls.

• Chatty Integrations
• Scheduled Job that Never Stops Trying to Process
• Import an Externally Updated IAR File
• Synchronous Integration Doing Too Much
• Too Many Connections in an Integration
• Read Files with Many Records
• Integrations Running Unchanged Despite Changing Business Needs

Chatty Integrations

Use Case: Synchronize records in a file or large data set with an external system (for example, synchronize journal transactions or upload employee records into Oracle HCM Cloud).

Antipattern	Why an Antipattern?	Best Practice
Use an invoke activity within a looping construct to call external APIs for every record.	Downstream applications are receiving a large number of atomic requests. This puts the entire system under duress. A usage-based pricing model translates to high costs.	Leverage application capabilities to accept multiple records in a single request: Salesforce: 200 records, Oracle Engagement Cloud/Oracle ERP Cloud: 100 records, Oracle Service Cloud: 1000 records Leverage adapter capabilities to send a large data set as attachments/files. Salesforce Adapter: 10,000 records/10 MB files, Oracle ERP Cloud Adapter supports FBDI files. Use a stage file action (the append to file option) to send the file to the destination at the end.

Scheduled Job that Never Stops Trying to Process

Use Case: Process records within a set of files with a tight service level agreement (SLA) (for example, synchronize employee records into Active Directory from Oracle HCM Cloud or send benefits information).

Antipattern	Why an Antipattern?	Best Practice
The schedule integration looks for all files to process and loops over all to sequentially process until no files remain.	If a large number of files exist, one run of a scheduled job processes for a long time and starves other jobs and may get terminated by the framework. Processing tied to a single server: this does not leverage multiple nodes in a cluster.	Limit the number of files to process in a single scheduled run. Limit results from the list file. Use schedule parameters to remember the last processed file for the next run. Invoke the run now command from the same integration to trigger processing of the next file if waiting for the next scheduled run is not feasible.

Import an Externally Updated IAR File

Use Case: Need to leverage advanced XSL constructs that may not be available in the mapper.

Antipattern	Why an Antipattern?	Best Practice
Update the IAR file externally and then import it into Oracle Integration.	This can lead to metadata inconsistency and validation failures. Activation failures may occur.	Use the import map feature in Oracle Integration. Keeps metadata consistent and leverages validation. The mapper can show an imported map and allow it to be edited.

Synchronous Integration Doing Too Much

Use Case: A request triggers complex processing involving enrichment and updates across multiple systems.

Antipattern	Why an Antipattern?	Best Practice
Huge synchronous integrations designed with a large number of invokes /conditional logic. Synchronous integrations with invokes within a loop with a large number of iterations.	Susceptible to timeouts. Any marginal slowdown adds up. Blocking call. Holds resources and starves other integrations.	Consider moving completely to an asynchronous integration (fire and forget, asynchronous response). The integration platform provides an acknowledgment to clients upon receiving the message. The integration platform ensures guaranteed processing and also supports resubmission of failures. Split into a synchronous integration containing mandatory processing before sending out a response and triggering separate asynchronous fire and forget integrations for other processing logic. Optimize synchronous processing with a coarse-grained external API to replace multiple chatty calls.

Too Many Connections in an Integration

Use Case: When developers create integrations, they define their own connections pointing to the same application. This leads to many duplicate connections.

Antipattern	Why an Antipattern?	Best Practice
Every developer creates their own connection using a different set of configurations/credentials.	A high number of connections make manageability painful, especially when you need to update the endpoint, credentials, configuration, and so on. Impact analysis is complicated when there is an application upgrade or metadata/coordinate change.	Have a custodian create needed connections and ensure that duplicate connections of the same types are not created. Build a best practice for naming conventions and maintaining a set of configurations. Use the Configuration Editor to edit/replace the connection in an integration.

Read Files with Many Records

Use Case: Read a file with a large number of records and process individual records.

Antipattern	Why an Antipattern?	Best Practice
Read the entire file in memory using the read file option and process it record by record.	Large amounts of memory are consumed and other system processing is impacted. Sequential processing does not leverage built-in map reduction capabilities.	Download the file to the staged location using the download file option. Stage files are secured and deleted at the completion of the instance. Use the read file with segments option. You can still model record level processing inside the segment processing. The platform automatically processes segments in parallel. The platform brings in only portions of the file to memory, when needed.

Integrations Running Unchanged Despite Changing Business Needs

Use Case: Integrations/schedules created during the initial implementation continue to run even though your business requirements have changed over time.

Antipattern	Why an Antipattern?	Best Practice
Integrations and scheduled jobs created during the initial product implementation are never re-evaluated against changing business needs. A proliferation of integrations occurs without looking at existing integrations.	Unnecessary runs of jobs that handle no work. Unoptimized invocations of multiple integrations with similar functionality. Clutter with dead integrations, lifecycle management (LCM) overhead, and developer confusion.	Periodically analyze existing integrations/schedules against current business needs. Look at monitoring data for historical runs. Consolidate integrations that are very similar. Integrations A and B subscribing to the Customer Created event in Oracle Engagement Cloud: integration A for invoking endpoint A and integration B for invoking endpoint B. Deactivate nonrelevant integrations. An integration for generating a file meant for partner consumption becomes irrelevant when the partner no longer needs it. Adjust the schedules for integrations that need lesser frequency and delete schedules no longer required. Files generated monthly have schedules run every few minutes. Changes in business needs can make certain schedules completely unnecessary.

Avoid Creating Too Many Schedule Integrations

When there are too many schedule integrations configured, instances can become backlogged waiting for resources to become available or previous integration runs to complete. This can cause processing delays where some instances are in a waiting state longer than they must be and schedules may not start at the scheduled time.

As a best practice, do not create too many schedule integrations that are scheduled to run at the same time. Where possible:

• Use an asynchronous REST Adapter trigger instead of a scheduled trigger when an active schedule is not absolutely required.
• Do not create any long-running schedule integrations (a schedule integration that takes longer than one hour, for example, to complete). This blocks scheduler resources impacting other scheduled runs.
• Spread schedules out over time to avoid schedule clusters.

You can convert a schedule integration to a REST Adapter trigger-based application integration. See Convert a Schedule Integration to a REST Adapter-Triggered, Application Integration.

If you absolutely require a large number of schedule integrations, and you encounter the previously described issue, the following design changes as a solution are recommended:

1. For each schedule integration, convert it to a REST Adapter trigger-based application integration. See Convert a Schedule Integration to a REST Adapter-Triggered, Application Integration.
2. Create a new schedule integration that only performs an asynchronous invoke of the application integration you converted in step 1 above.

   This solution enables the schedule integration to start at the scheduled time, invoke the REST Adapter trigger-based child integration asynchronously, and complete within milliseconds. This approach reduces the backlog and contention surrounding scheduler resources.

   If you have a large number of schedule integrations to convert, a staged approach starting with the following integrations is recommended.

   • Longest-running schedule integrations.
   • Schedule integrations configured with the shortest frequency (for example, integrations that run every 10 minutes or less).

   Design any new schedule integrations with the design practices described above.

See Schedule Integrations Are Not Running on Time and When a Schedule Integration Instance Gets Terminated.

Synchronous Integration Best Practices

Note the following best practices when designing a synchronous integration.

• A synchronous integration calling any asynchronous request response service:

  • Calling asynchronous fire and forget (one-way) is acceptable.

  • Oracle Integration does not currently allow modeling an asynchronous request response service. However, all schedule integrations internally use an asynchronous request response. Therefore, a synchronous integration using a schedule integration is an antipattern.

• A synchronous integration calling multiple services that span more than five minutes is reported as a stuck thread.

Design Long-Running or Time-Consuming Integrations as Asynchronous Flows

Note the following best practice when designing long-running or time-consuming integrations.

Do not expose long-running or time-consuming integrations as synchronous flows. This action can lead to client applications (including other integrations) timing out. Synchronous integrations also have a server-side timeout. Instead, model any synchronous integration taking more than two minutes as an asynchronous flow.

Timeouts in Service Calls During Synchronous Invocations

You may have scenarios in which synchronous invocations from Oracle Integration (including calls to other integrations) are blocking calls and must complete within 300 seconds.

Because the call may involve one or more proxies, each of the proxies may have a similar timeout. For instance, the default proxy on Oracle Public Cloud has a timeout value of 120 seconds. If the call is to an on-premises service is behind a fire wall, the proxy configured may also have its own timeout value.

In case timeouts are defined at multiple layers, the service invocation fails at the first timeout.

Parallel Processing in Outbound Integrations

While there is no specific integration design to automatically enable an outbound integration to send data to different third party systems in parallel, there are integration design approaches that allow this scenario.

Separate the integration into multiple integrations:

• Create a main parent integration that only receives/processes the data.

• Create separate child integrations to perform the individual outbound REST invocations.

The interface between the main and separate child integrations can follow these approaches:

• Consist of dummy REST calls, but it must be asynchronous. Essentially, the asynchronous calls are not blocked by the response and the fire-and-forget design enables available threads to work on child integration processing in parallel, within the available system resources. This type of design is recommended because if all synchronous REST calls are done in the same integration, a timeout error may occur if the sum of time taken for each synchronous call exceeds five minutes.

• Follow a publish/subscribe design approach (for example, putting the data events in a queue, having each child flow subscribe from the queue, and so on).