30 Best Practices for Designing and Building End-to-End Integration Flows

This chapter discusses best practices and recommendations for designing and building end-to-end integration flows.

Note:

Composite Business Processes (CBP) will be deprecated from next release. Oracle advises you to use BPM for modeling human/system interactions.

This chapter includes the following sections:

30.1 General Guidelines for Design, Development, and Management of AIA Processes

This section includes the following topics:

30.1.1 Interpreting Empty Element Tags in XML Instance Document

The XML Instance document should have empty element tags only when the sender intends to have the consumer nullify the values for the elements. Otherwise, these tags should not be present. Having these nonsignificant tags can have a huge impact on memory consumption and hence scalability of the application.

AIA recommends that Enterprise Business Messages (EBMs) have only significant elements.

Some applications can inspect the content and produce an XML document having only the significant tags, whereas the rest do not. Because the ABCS is intimate with the application, it can choose the appropriate style based on the application's behavior. In situations in which the applications produce Application Business Messages (ABM) containing all of the elements regardless of whether the elements underwent change or not, the requester connector services should assume that the empty elements in ABM are not significant; and should ignore them when producing EBM. The first code example shown in Example 30-1 illustrates how this could be done. In situations in which ABMs contain only elements that underwent a change in value, the connector services should treat the presence of an empty element as the sender's intent to have the consumer nullify the values for that element. Example 30-2 illustrates this use case. With service requesters producing EBM in the preceding manner, the job of the consumer becomes straightforward. It can assume that every element is a significant one-hence, the ABCS consuming the EBM should not skip the empty element. Example 30-3 illustrates how a message containing significant elements can be processed.

Use the construct in Example 30-1 when the source application's ABM contains all the elements defined in its schema regardless of whether all those elements underwent any change in value. In this situation, in the interest of conserving memory, AIA makes the assumption that an empty element is NOT a significant element and ignores it.

Example 30-1 ABM -> EBM Transformation

~--> <xsl:if test="ABMSourceElement/text()"> <EBMTargetElement> <xsl:value-of 
select="ABMSourceElement"/> </EBMTargetElement> </xsl:if>

Use the construct in Example 30-2 when the source application's ABM contains only those elements that underwent a change in value. In this situation, the presence of an empty element can be considered significant and so AIA does not ignore it.

Example 30-2 ABM -> EBM transformation

~--> <xsl:if test="ABMSourceElement/text()"> <EBMTargetElement> <xsl:value-of 
select="ABMSourceElement"/> </EBMTargetElement> </xsl:if>

If the above rules have been followed, then we can assume that any empty element in the EBM is a significant one, and therefore should not be ignored when transforming from EBM to ABM. So use the construct in Example 30-3.

Example 30-3 EBM -> ABM Transformation

~--> <xsl:if test="EBMSourceElement"> <ABMTargetElement> <xsl:value-of 
select="EBMSourceElement"/> </ABMTargetElement> </xsl:if>

30.1.2 Purging the Completed Composite Instances

AIA highly recommends having a process in place to archive and clean up the completed composite instances after a specified period.

  • Synchronous request -response-based completed composite instances should be purged from SOA dehydration data store after a month.

  • Data synchronization-related composite instances should be purged three to four months after their completion.

  • Composite Business Processes should be purged one to two years after their completion.

Even though this duration is subject to change based on your company policies, attempts should be made to do the purging at the earliest time. Purging must be preceded by archiving of the instance data.

For more information, see "Managing Database Growth" in Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

30.1.3 Syntactic / Functional Validation of XML Messages

This section discusses:

30.1.3.1 Syntactic Validation

With document style, a web service endpoint can use the capabilities of a validating parser and the runtime to perform syntactic validation on business documents against their schema definitions.

A schema specifies the data shape of the payload that could be sent to the web service operation. When a SOAP message is received by the web-services handler, the schema pertaining to the operation being called is used to validate the content of the SOAP message.

Because this validation has a performance impact, it should be used judiciously in a production environment. AIA recommends enabling the schema validation only for the service-receiving message from a requester outside its boundary. Even though the requester could be an internal application, the syntactic validation is normally applied at runtime only for messages coming from an external source. AIA highly recommends that you turn off schema validation for the intermediary services residing in AIA layer.

In a development environment, AIA highly recommends that you turn on schema validation for all services during certain testing phases to ensure that messages produced by services are syntactically valid.

For more information about how to enable schema validation, see Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

30.1.3.2 Data / Functional Validation

AIA highly recommends that the validation be done by the service provider. It could be done either by the provider ABCS or by the application itself.

30.1.4 Provide Provision for Throttling Capability

The implementation of cross-functional business processes often requires integrating endpoint systems having divergent nonfunctional characteristics such as availability and concurrent processing capability. Occasionally, the endpoints become unavailable due to planned or unplanned outages and many systems may not be able to handle large volumes of concurrent messages. Overwhelming these systems with a high-volume of requests may result in the failure of target endpoints, which in turn has a cascading impact on the overall integration. AIA recommends implementing store and forward capability using queues in most of the situations while interacting either with the requester or provider application. This gives opportunities for customers to implement throttling that helps impose a limit on the rate at which message requests are handed over to a specific endpoint.

For more information, see Section 28.1, "AIA Message Processing Patterns."

30.1.5 Artifacts Centralization

To facilitate the governance of shared artifacts such as Enterprise Business Objects, Enterprise Business Service WSDLs, Application Business Message Schemas, ABCS WSDLs, Domain Value Maps, and Cross-Reference Meta data, AIA highly recommends designing and implementing these artifacts independently from the service capabilities that use them. AIA strongly discourages the management and persistence of these artifacts as part of individual services. AIA's programming model recommends the use of MDS to persist these artifacts.

For more information about how MDS is used for centralization of these assets, see Section 2.1.3.4, "Using MDS in AIA."

30.1.6 Separation of Concerns

Separation of concerns is a core principle of SOA. It helps you introduce loose coupling in the integration flow where it is required. However, it must be applied not only at the architectural level, but at the implementation level as well.

Managing dependencies between services at development and runtime is a challenging task when developers implement requester and provider ABC services. One of the approaches is to make them de-coupled. Using this approach, the cross-dependencies in the code that handle each concern of the service can be minimized.

You may see error messages indicating invalid SOA composites after a server restart. This is caused by referring to concrete WSDLs where abstract WSDLs should have been used.

You will not see the problem at the first deployment of a new composite X when you reference the concrete WSDL of composite Y. Why? Composite Y is up and running at the time of the deployment of composite X. The problem starts when the server is restarted because there is no guarantee that the composites will be activated in the right order to resolve any dependencies. If service X is activated before composite Y, the reference cannot be resolved because Y is still down, so X remains in status Invalid.

You can begin with separating the interface from implementation.

  • Do not use the concrete WSDLs directly in your consuming artifacts. If the Service Provider (not as in ProviderABCS) changes, then you must redeploy the consumer.

  • Do not reference the deployed concrete WSDL of a Service Provider. If Provider is unavailable, then consumer cannot be compiled or be deployed.

    For example, assume you are building a Provider ABCS that consumes an adapter service, You have directly referenced the concrete WSDL of the adapter service and developed your consuming Provider ABCS. Now, when the adapter service is not available or not deployed as yet, when you are compiling or deploying your Provider ABCS, you hit a road block.

  • Introduce a common directory of abstract WSDLs of your services and store these abstract WSDL s on a centrally accessible endpoint.

SOA provides ways to fully de-couple services at design time. AIA advocates always using abstract WSDLs when composites refer to others. Ensure that abstract WSDLs fully describe a service's interface, which is all that is needed at design time. The concrete WSDLs are only needed later at execution time to provide the binding details, that is the information on how the deployed composite can be invoked.

Figure 30-1 illustrates that references between SOA composites actually consist of two parts: an abstract WSDL (design time) and a concrete WSDL (runtime).

Figure 30-1 References Between SOA Composites

The image is described in surrounding text

AIA uses Metadata Services (MDS) for storing each service's abstract WSDL. Thus MDS becomes the source of truth for all service interfaces and the composites should not have any redundant copies.

30.1.6.1 Using MDS as the Central Storage for Abstract WSDLs, and Other Shared Artifacts

SOA Suite 11g has introduced a central repository, Metadata Service (MDS), that backs your application (and hence your composites) at design time and runtime. MDS is like a version management system, where you can store and use the shared common artifacts at design and runtime.

AIA leverages the MDS repository to store common design time artifacts that are shared with all developers. AIA has introduced a common directory of abstract wsdls in the MDS, which is a centrally accessible endpoint.

The artifacts that are stored in the MDS include:

  • Schemas - EBO schemas and ABM schemas

  • WSDLs - Abstract WSDLs of EBS, ABCS, Adapter Services, CBPs, and EBFs

  • DVMs and XREFs

AIAComponents presents the various schemas and WSDLs referred to by various services. The structure is shown in Table 30-1:

Table 30-1 Structure of AIA Components

Location Artifacts

ApplicationConnectorServiceLibrary

Abstract WSDLs of various Application Business Connector Services (ABCSs)

ApplicationObjectLibrary

WSDLs of services exposed by applications and schemas of application business objects

B2BObjectLibrary

Business-to-business (B2B) schemas

B2BServiceLibrary

Abstract WSDLs of various B2B Connector Services (B2BCSs) and B2B Infrastructure Services

BusinessProcessServiceLibrary

Abstract WSDLs of Composite Business Processes (CBPs) and Enterprise Business Flows (EBFs)

EnterpriseBusinessServiceLibrary

Abstract WSDLs of Enterprise Business Services (EBSs)

EnterpriseObjectLibrary

Schemas of the Oracle AIA Canonical Model

InfrastructureServiceLibrary

Abstract WSDLs of infrastructure services

Transformations

XSLs shared among various services

UtilityArtifacts

Utility schemas and WSDLs

config

AIAConfigurationProperties.xml and AIAEHNotification.xml

dvm

Domain Value Maps

faultPolicies

Default policies applicable to all the services

xref

Metadata for Cross References

For more details on how to upload these artifacts into MDS, and how to update them in the MDS, see Section 2.1.3.4, "Using MDS in AIA."

The abstract WSDLs of all AIA services are stored in the MDS. You build your consumer AIA service composite application using the abstract WSDL of a referenced service.

So where is your MDS located, and how does an application know how to reference it?

You configure your developer environment to point to the MDS. The configuration file is available in $application_home/.adf/META-INF/adf-config.xml and by default it points to your local - JDeveloper file system based MDS. You must configure it to point to the MDS on your managed SOA server.

For details, see Section 2.1.1, "How to Set Up JDeveloper for AIA Development."

In a composite, a referenced service is defined by its abstract WSDL.

As an example, we will use the scenario which is described earlier in this section.

You build a Provider ABCS that consumes an adapter service, by referencing the abstract WSDL of the adapter service.

What does it bring to the table?

First, there is no dependency on the referenced services during the deployment of the composites. Hence there is no dependency on the order of deployment for the composites. A referenced service does not need to be deployed first, in order for it to be referenced by another service. This also resolves the cases where there are cyclic references involved among the services.

After a composite has been designed, you must perform a few tasks. Since you have used an abstract WSDL to reference an external service, the run-time bindings are not generated in the composite. For the referenced service you must open the composite in the source mode and provide this binding information - the values for element binding.ws.

For details about how to populate the binding.ws.port and binding.ws.location elements of your composite, refer to Section 17.1.7.1, "Populating the binding.ws Element in the composite.xml."

30.1.7 Adapters Inside ABCS Composite OR as Separate Composite

The most common and recommended approach to migrate an existing application is to migrate it 1-on-1. This makes every component in the old application an SCA composite. The next step is to use the benefits of SCA; combine multiple SCA composites into a single composite. But do you always gain an advantage by combining composites?

An architectural decision should be made on how you build the SCA. With SOA/SCA it is all about re-usability. Think about this - are you designing your SCA based on the business flow or are you designing our SCA from technical point of view? The answer lies in finding the correct balance.

You might want to design a service once and reuse it many times. From that point of view you would build small SCAs. But you could also choose large SCAs with multiple services that are exposed to the outside world. In that case, since the SCA is relatively large, it increases the maintainability.

What you want is to define and design services and create composites that reference these services.As a result the service is designed once and reused many times.

Going back to the question of whether the adapters should be inside the ABCS composite or should be developed as a separate composite, as a best practice, AIA recommends that you put adapters that are interfaced with ABCS in a different composite. as shown in Figure 30-2. This is also the preferred way when the same transport adapter service could be used with multiple ABCS.

Figure 30-2 Example of Adapters Interfaced with ABCS in a Different Composite

The image is described in surrounding text

However, if no such reusability is foreseen, then there is nothing wrong in using the alternative design where an ABCS and a Transport Adapter service can be in the same composite as shown in Figure 30-3.

Figure 30-3 Example of ABCS and Transport Adapter Service in the Same Composite

The image is described in surrounding text

30.1.8 AIA Governance

After developed, built and deployed, AIA artifacts must be capable of being governed.

To allow for the governance of shared artifacts such as EBOs, EBS WSDLs, ABM schemas, ABCS WSDLS, DVMs, and XREFs, AIA recommends designing and implementing these artifacts independent of the service's capabilities that use them. They are not to be treated as service artifacts and should be managed and persisted in a central location.

As mentioned in earlier sections, AIA uses MDS to persist these artifacts.

Annotations are injected during the development phase in the composite XML file, apart from annotations in the WSDL file. The annotations in composite files provide detailed information about:

  • AIA artifacts and their relationship to other AIA artifacts.

  • Composite-level descriptor properties, that are used to configure the component at deployment and runtime.

AIA Architecture categorizes SOA composites as adapter services, requester services, provider services, and so on. The meta information of these AIA services is used in maintaining OER assets of AIA asset types and linking them to OER assets with native asset types; this is accomplished with the help of the AIA Harvester which harvests the SOA Composites. In line with SOA modeling and development practices, these composites are expected to be harvested multiple times during the development cycle from conception till deployment to production environment.

AIA Harvester is built on top of the Oracle Enterprise Repository Harvester Extension Framework. It introspects SOA artifacts and publishes their ensuing metadata into the Project Lifecycle Workbench back end or Oracle Enterprise Repository (optional), or both, to aid governance and downstream automation.

The best practice for using the AIA Harvester tool is to harvest to both Lifecycle DB and OER. The AIA Harvester then parses the AIA service artifacts and captures metadata into the AIA Project Lifecycle Workbench database and the Oracle Enterprise Repository. The system uses this information to generate deployment plans.

30.1.9 Using AIA Service Constructor

The AIA Service Constructor is an Oracle JDeveloper plug-in used to generate composites conforming to AIA guidelines and naming standards. It also provides guidance for annotating the artifacts to support governance. AIA recommends that you construct the ABCS services using the Service Constructor tool.

For more information about using the AIA Service Constructor, see Chapter 4, "Working with Service Constructor."

30.2 Building Efficient BPEL Processes

This section provides some recommendations on how to keep the BPEL-based processes as lean as possible.

The section includes the following topics:

30.2.1 Using BPEL as "Glue", Not as a Programming Language

Since BPEL is an orchestration language, it should be used primarily as a glue to orchestrate a set of services exposed by the application systems. Even though BPEL does support a wide array of programming constructs, they are not meant for building a complex programming logic within the BPEL process.

30.2.1.1 Keep the Number of BPEL Activities as Minimal as Possible

Use XSL instead of Assign

Avoid the use of multiple assign activities to populate various elements in an XML message. Consider using XSL to do the transformations. Because the invocation of XSL script comes with a cost, usage of XSL for populating the message should be considered only when more than seven to ten assignments must be done.

Use XPath expressions to constrain the data set

Programming scenarios exist in which you must loop through a given array of data (A), and operate on a specific subset of the array (using condition C). So the simple way of doing this is to have a while loop for A, and then a switch condition C inside the while loop. In this approach, you invariably end up looping through all the lines, leading to inefficiency. A better way to approach this situation is to use multi-indexes. So instead of accessing A as A[i] and then checking the condition, you can access A as A[C][i], where you loop through all those elements of A(using i), where condition C is satisfied. This way, you reduce the number of BPEL activities.

30.2.1.2 Avoid Large While Loop

A typical usage pattern is the usage of the while loop to process:

  • Multiple repeating child instances in an XML message

  • Large number of discrete object instances in an XML message

In situations in which the while loop is being used to process repeating child instances (maxOccurences is unbounded) in an XML message, the BPEL activities in the while loop have to be designed keeping the possibility of large number of iterations in mind. For example, having 50 activities to process a single instance in a while loop suddenly results in creation of 5000 activities at runtime when an instance document having 100 repeating instances is processed.

30.2.2 Avoiding Global Variables Wherever Possible

Within the assign activity in BPEL, local variables should be used instead of process variables wherever possible. Local variables are limited to the scope in the BPEL process. These get deleted from memory and from the database after you close the scope. However, the life cycle of a global or process variable is tied to the instance life cycle. These variables stay in memory or disk until the instance finishes. Thus, local variables are preferred to process or global variables. However, if the same variable is being used either in every iteration in the while loop or throughout the entire process, creating one global variable and having that accessed by all iterations would be better.

The BPEL fragment in Example 30-4 illustrates the use of local variables.

Example 30-4 Using Local Variables

<scope name="Scope_1">
<variables> <variable name="Invoke_CallprocessBillingMove_InputVariable"
                  messageType=
                  
"sordmovsubabcs:ProcessFulfillmentOrderBillingBRMCommsMoveAddSubProcessRequest
Message"/>
               <variable name="Invoke_CallprocessBillingMove_OutputVariable"                
                   messageType=
               
"sordmovsubabcs:ProcessFulfillmentOrderBillingBRMCommsMoveAddSubProcessResponse
Message"/>
        </variables>
        <sequence name="Sequence_MoveAdd_SubProcess">
                <assign name="Assign_Variable_Invoke_CallMoveAdd">
                      <copy>
                         <from variable="inputSubProcess"           
                          
query="/sordsubebo:ProcessFulfillmentOrderBillingBRMCommsSubprocessMessage"/>                         
<to variable="Invoke_CallprocessBillingMove_InputVariable"
                                     part="payload"                                    
query="/sordsubebo:ProcessFulfillmentOrderBillingBRMCommsSubprocessMessage"/>                      
</copy>
                 </assign>
                 <invoke name="Invoke_CallMove-Add_Subprocess"                            
partnerLink="ProcessFulfillmentOrderBillingBRMCommsMoveAddSubProcess"                            
portType="sordmovsubabcs:ProcessFulfillmentOrderBillingBRMCommsMoveAddSubProcess"                            
operation="processBillingMove"
                            inputVariable="Invoke_CallprocessBillingMove_
InputVariable"
                            outputVariable="Invoke_CallprocessBillingMove_
OutputVariable"/>
          </sequence>
</scope>

30.2.3 Avoiding Large FlowN

Careful consideration must be given during design of the BPEL process having FlowN activity. You must have a good understanding of the upper limit of N. Depending on the type of activities performed in a flow, you must strongly consider the option of running the flows in an asynchronous mode by setting nonBlockingInvoke property in composite.xml to true.

30.2.4 Controlling the Persistence of Audit Details for a Synchronous BPEL Process

auditLevel property sets the audit trail logging level. This configuration property is applicable to both durable and transient processes. This property controls the number of audit events logged by a process. Audit events result in more database inserts into the audit_level and audit_details tables, which may impact performance. Audit information is used only for viewing the state of the process from Oracle Enterprise Manager Console.

Use the Off value if you do not want to store any audit information. Always choose the audit level according to your business requirements and use cases.

For synchronous BPEL processes, AIA recommends non persistence of instance details. For this, set the auditLevel property to Off at the service component level. This general guideline can be overridden for individual services based on use cases.

30.2.5 Using Non-Idempotent Services Only When Absolutely Necessary

Idempotent services are retryable. They reproduce the same results regardless of the number of times the service is invoked. They have absolutely no side effects.

The default value for idempotent property is true. Setting the idempotent property to false results in dehydration of the process after running the partnerlink. The decision about the configuration of idempotent property must be done at the design and development time by the developer.

For more information about idempotent property, see Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

30.2.6 Defining the Scope of the Transaction

A transaction tends to grow big when it encompasses a set of activities to process each repeating node in an XML message and when the number of repeating nodes is quite large.

The default value for the dspMaxThreadDepth property is set to 600. If the number of BPEL activities run in a transaction exceeds this value, the BPEL engine issues an automatic implicit commit to end the transaction. This might not be in alignment with the application transaction semantics. This could be eliminated by setting this property to an arbitrarily high value. This approach anticipates that the global transaction as defined by the integration developer ends much before BPEL reaches the new threshold. In most of the situations, setting the property to a very high value helps the transaction to get completed. However, it has the potential to impact the overall scalability of the application. Hence, attempts should be made to keep the scope of the transaction as small as possible.

30.2.7 Disabling the Audit for Synchronous BPEL Process Service Components

AIA recommends turning off the audit completely for synchronous BPEL-based services that have no midprocess breakpoint activities.

auditLevel

This property sets the audit trail logging level and controls the number of audit events that are logged by a process. The value set at the BPEL process service component level overrides the value specified at the SOA Infrastructure, BPEL Process Service Engine, and Composite Application levels. Override the value only for synchronous BPEL processes that have no midprocess breakpoint activities.

AIA recommends the following value to be set to this property.

Off: The BPEL service engine does not capture the payload. The payload details are not available in the flow audit trails. Payload details for other BPEL activities are collected, except for assign activities. This level is optimal for most normal operations and testing.

Example 30-5 shows how to set the bpel.config.auditLevel property to an appropriate value in the composite.xml file of your SOA project.

Example 30-5 Setting the bpel.config.auditLevel Property in the composite.xml

<component name="BPELProcess">
<implementation.bpel src="graphics/BPELProcess.bpel" />
<property name="bpel.config.auditLevel">Off</property>
</component>

30.2.8 Including No Break-Point Activity in a Request-Response Flow

AIA highly recommends that a BPEL service implementing the synchronous request-response message exchange pattern has no break-point activity such as midprocess receive, wait, onMessage, onAlarm.

Similarly, a mediator service implementing synchronous request-response message exchange pattern should have no parallel routing rules. The ESB routing service implementing the request-response message exchange pattern should have no target services invoked in asynchronous mode.