2
Methodology and Solutions

This chapter introduces you to the methodology of e-business integration and offers examples of ways in which Oracle Integration Server meets integration challenges. The topics in this chapter are:

• Selecting the Appropriate E-Business Integration Methodology

• Application Integration: The Solution Spectrum

• Business-to-Business Integration

Selecting the Appropriate E-Business Integration Methodology

We have examined the three primary integration technologies available: data synchronization technologies, component-oriented development technologies, and message-oriented middleware. Now we consider how these technologies optimally map to the three fundamental integration problems: synchronizing data between systems, isolating applications from each other, and automating multistep business processes within companies and between companies.

Although the choice of methodology usually requires a close study of the specific environment in which integration is implemented, you can use certain broad architectural principles to guide integration decisions.

This section includes:

• Synchronizing Data Among Systems

• Isolating Applications and Businesses from Each Other

• Automating Multi-step Business Processes

Synchronizing Data Among Systems

The fundamental architectural choice in determining how to synchronize data among systems is whether you need to move data between different systems or whether you need to provide data access to a variety of system from a central location. If you must provide data access, then the ideal choice of integration technology is to use gateways to access the databases and legacy systems in which the data resides.

If you must move data between systems, you can use two different mechanisms: database replication and asynchronous messaging. Three factors influence your choice of technology:

• Firstly, if data can be synchronized in a relatively simple fashion such as when extending one or two back-end applications to a Web front end, simple database replication of information is sufficient. Similarly, if distributing information from a central database to a number of smaller workgroup databases, database replication is sufficient. In these cases, message-based solutions introduce an additional level of complexity (for instance, the need to deploy specialized middleware) and are better suited for more complex applications.

• Secondly, in most cases in which replication can be used, the sender and the recipient of information must be homogeneous databases, for example, both Oracle databases. The databases must have a similar schema representation. If data needs to go through a complex transformation as part of data movement, then asynchronous messaging technology is probably more appropriate.

• Thirdly, in order to use database replication, the receiving application must provide the sending application direct access to its schema. Two issues are involved with providing such direct access to the schema: because the two applications do not communicate through a clearly defined set of interfaces, direct schema access violates encapsulation and component-based development. Further, direct schema access can circumvent application-level security policies.

Despite these concerns, in a number of cases where simple data synchronization between two data sources is the only integration requirement, using database replication simplifies how you build and deploy the applications and is probably the most appropriate choice.

Isolating Applications and Businesses from Each Other

If the integration scenario is primarily focused on isolating applications and businesses from each other, data synchronization technologies such as database replication and database gateways are not appropriate. To isolate applications from each other, communication between the applications must be limited to a standard set of well-defined public interfaces. Data synchronization technologies, by definition, violate encapsulation and as a result are not suited for application isolation.

When you need to isolate applications from each other while facilitating communication between them, the primary choice is between a synchronous or an asynchronous communication facility. Three factors determine which communication mechanism to use:

• Firstly, if the two application components are part of a larger composite application and work together in a request-reply structure, then a synchronous component-oriented middleware facility such as an Object Request Broker (ORB) and a communication protocol such as CORBA IIOP is the best choice. In this case, both applications should be wrapped as CORBA services with well-defined public interfaces defined in the CORBA Interface Definition Language (IDL) and they should communicate through a middleware hub known as an ORB. The ORB provides services to the two applications including:
  • Registering them as services

  • Making the public interfaces of each application accessible to the other applications

  • Routing each protocol request from one application to the other

  • Discovering the object by looking up its location on the network

  • Activating the object when the request is received

• Secondly, if the two application components are not related in a request-reply structure but rather form steps within a multistep business process, then an asynchronous message-oriented middleware solution is the most appropriate choice. The primary choices in such a situation depend on the complexity of the messaging environment that needs to be deployed and on the messaging architecture. These choices are similar to those that need to be made in automating multistep business processes and will be discussed in the next section.

• Thirdly, if the integration requirement is to connect the business processes of two companies using the Internet, then synchronous integration is not feasible and loosely coupled message-based integration is the only solution.

Automating Multi-step Business Processes

Automating a multi-step business process requires applications to communicate with each other in a loosely coupled structure. The only appropriate choice of integration technology in this case is asynchronous message-oriented middleware. In choosing to deploy a middleware solution, the primary architectural decisions that you make depend on the complexity of the integration problem. There are four key issues that you must consider:

• Integration Topology

• Messaging Architecture

• Data Transformation

• Business Process Management and Workflow

Integration Topology

When deploying messaging middleware, you must determine whether to use a point-to-point interface to link applications or to use a hub-and-spoke architecture to link applications. Although point-to-point connectivity is simple when connecting two applications, it quickly becomes unmanageable if more than two applications need to connect.

In such cases, use a hub-and-spoke architecture. In this architecture, applications are not connected directly with each other. Instead, each application is connected to a hub that provides connectivity between all the applications. A change or upgrade to any one application changes only its relationship with the hub and does not affect all the other applications with which it must be integrated.

Messaging Architecture

Next, you must determine the messaging architecture to use based on three questions:

Must messages be stored persistently? If messages are business critical and must be audited and tracked, for example, to resolve disputes between companies or to track information flows, the message header and contents must be stored persistently in a database.These messages can be warehoused and analyzed using standard decision support tools. If the messages are not business critical, then the message brokering facility can provide volatile queuing facilities.

Must messages be propagated with guaranteed delivery? If messages are business critical or if the messaging system connects two mission critical applications, then the messages need to propagated with guaranteed once-only, in-order delivery; otherwise messages can simply be propagated impermanently.

How must messages be routed? Messages can be routed between applications and business processes based on their subject or topic, their content or payload information, or by using the publish-subscribe method.

In most e-business integration scenarios, messages are routed based on their content: in most cases, invoking a specific workflow to process the message in a specific way before sending it to an appropriate destination.

Data Transformation

Applications that form part of a business process usually store and manage data in different data formats. For instance, the Oracle E-Business Suite of packaged applications stores and manages data in SQL format. SAP applications operate on data in iDOCs format, which is a derivative of ASCII.

Four data transformation issues must be addressed when connecting one application to another:

• Datatype transformation means converting data from the format of the originating application to that of the target application. For instance, when connecting an Oracle customer relationship application to an SAP application, Oracle data types must be converted from SQL format to an ASCII format appropriate for iDOCs.

• Semantic transformation means converting a customer name from an Oracle format into an SAP format. For instance, Oracle represents the customer's name in three fields: a first name, a last name, and middle initial. SAP represents the customer's name in a single field: a last name followed by a comma followed by the first name. A semantic transformation must be applied to concatenate the last name followed by the first name.

• Functional transformation means mapping the business events in the sending application to the business events and interfaces in the receiving application. For instance, when a purchase order is sent from an Oracle application to an SAP application, a specific set of business events must start and a specific set of public interfaces must be invoked. Such functional transformation information must be captured in order to propagate a message from one application to another.

  There are two commonly used approaches to data transformation: The first converts the data directly from the format of the originating application to the format of the target application. The second converts the data first into a canonical, intermediate format before converting it into the format of the target application. Although direct conversion is faster, converting to an intermediate format isolates one application from changes in other applications. For instance, when you connect an Oracle CRM application with a Baan, SAP, or PeopleSoft application, mapping information to an intermediate representation isolates the Oracle CRM application from an upgrade to the Baan application. The only change necessary is to modify the mapping between the intermediate representation and the Baan application.

• Locational transformation means determining the location where data transformation is conducted. In some cases, data transformation is done close to the application at the spoke. The hub simply sends the transformed message from one application to another. In most cases, however, data transformation must be conducted in the hub:
  • A message that must be audited is stored in its pretransformed form.

  • A message that must be sent to many different target applications is transformed within the originating application into an intermediate form that is sent to the hub. The hub then applies various transformations to convert the information into the format of each target application.

  • A message that must be routed to different locations based on its content is routed by the hub after it is transformed in the hub. However, this depends on the capability of the hub itself and on the kinds of services it provides to the applications that require connection.

Business Process Management and Workflow

Finally, you must decide whether messages can simply be sent from one application to another or whether they require processing before they are propagated. For instance, when a company sends a purchase order from its supply chain application to the procurement system of another company, the purchase order can require the approval of a purchasing manager at the originating company. If so, the messaging middleware must invoke a workflow application before forwarding the message to the trading partner. In this case, the messaging platform requires a facility for business process management of workflow.

Application Integration: The Solution Spectrum

Remember that all integration scenarios involve three fundamental integration problems: synchronizing data between systems, isolating applications and business from each other, and automating multistep business processes. Integration requires a range of different technologies, each of which is appropriate to a particular type of integration problem.

Oracle Corporation recognizes that integration is not a single, narrowly defined problem that can be solved by any single technology. A complete solution to this complex problem requires a variety of technologies, all of which must be seamlessly integrated. This section clarifies your choices by illustrating some typical integration scenarios. These should help you gain an appreciation of the different types of integration technology that constitute a comprehensive solution.

This section includes:

• Data Integration

• Application Integration

• Business Process Modeling and Execution

• Business Process Intelligence

Data Integration

Scenario:

Your enterprise has a number of applications, each with its own databases. The applications change the information in their databases frequently and independently of each other.

Problem:

How does an application get the most current information residing in the database of another application? For example the order entry application needs to access the same customer data as the accounts receivable application in order to use the address.

Solution:

Data Access Gateways and Replication: Gateways enable applications to easily and directly access other databases to get the information they need. Replication automatically synchronizes the information in multiple databases so that each database has the most current information. Changes in any database are immediately reflected in other databases.

Application Integration

A business process involving multiple applications requires integration of application logic and application functionality. To facilitate this integration, applications must communicate to exchange important business information. The two different communication models are synchronous and asynchronous communication.

Synchronous Communication with Functional Interfaces

Scenario:

You want to create a front-end e-commerce Web application that accepts orders. Before accepting the order, the application requires credit card authorization, customer credit rating, inventory levels, delivery schedules, and pricing. If any of these services is not available, the order application cannot complete the transaction. This might mean asking the customer to try again later.

Problem:

The services required for order entry are provided by other dedicated applications. So, how does the e-commerce application access these services? How can these applications be integrated to implement and automate the business process?

Solution:

Synchronous, request-reply protocol based on functional interfaces: Each application offers services by defining a set of specific functions as public interfaces. To request these services, applications invoke the corresponding interfaces. Examples of request-reply protocols based on functional interfaces include Remote Procedure Calls (RPC), Common Object Request Broker (CORBA), COM, and Java Remote Method Invocation (RMI). Standards for semantically richer interfaces, specifically aimed at distributed enterprise transactional applications, are emerging with component models such as Enterprise JavaBeans and COM+.

Asynchronous Communication with Message-Based Interfaces

Scenario:

You want to implement an end-to-end order-fulfillment process that involves a number of applications such as order entry, manufacturing, inventory management, distribution, and billing. The order fulfillment process originates with the order-entry application with which a customer places an order. After that, business objects and events must flow between the applications.

Problem:

These applications are distributed across a wide area network, are owned by different organizations, and have heterogeneous internal architecture. They were designed as single-function, standalone applications, with no plans for integration. The network that connects the applications might be unreliable. The organizations that own the application might change, relocate, or replace the applications at will, without informing the organizations owning the other applications. Furthermore, one application does not require a response from another in order to proceed. Hence, synchronous communication integration is not a viable option.

Solution:

Asynchronous communication with message queuing: Applications communicate with each other by exchanging information as messages through queues. Each application defines a set of messages that it accepts as input and a set of messages that it publishes as output for other applications. The messages represent business objects like customer records or business events like a new shipment requests. Asynchronous communication with message queuing enables loose coupling, so that individual applications are completely isolated from application, network, and system failures.

An example of a message queuing interface is the industry-standard Java Messaging Service (JMS).

Business Process Modeling and Execution

Scenario:

Consider the order-fulfillment scenario described in the previous section. Assume that the applications are integrated and that the entire business process is automated.

Problem:

Business managers decide that the business process must be modified. For instance, the order-fulfillment process moves from a clerk-oriented system to a Web-based self-service system that requires a different approval process. The flow of the process must be changed and new applications must be added to the process. How quickly can the business decision be translated into reality? How do you minimize the time that elapses between decision making and implementation?

Solution:

Business process modeling and execution: Firstly, a business analyst models the entire business process using a graphical modeling tool. Then, a technical analyst fills in the details and maps the model to the underlying integration infrastructure. Next, the model is validated and generated in a repository. Finally, it is executed by a business process coordinator. To change the process at any time, only the high-level model requires change. You can quickly implement the entire cycle, from identifying the change to executing the new model.

Business Process Intelligence

Scenario:

Same as in the previous section.

Problem:

The end-to-end business process is not performing to expectation, even though each of the individual applications is finely tuned. A holistic view of the business process is not available. Standard system management and monitoring tools provide performance metrics only about individual components, not about the entire process. To complicate matters further, performance varies daily and seasonally with market variations, thus making it hard to pinpoint the exact cause of the problem. So, how do you identify the inefficiencies and bottlenecks to help you to deploy your resources intelligently?

Solution:

This solution utilizes Business Process Intelligence as its strategy. The only way to get a holistic view of the entire business process is to track each transaction end-to-end. This implies tracking all the information (data, messages, and business events) in order to reconstruct the transaction later. Further, you must collect this information over an extended period of time, and it must be warehoused and analyzed to gather intelligence about the business process. The accumulated information can be mined to discover patterns and to provide insight into how resources can be optimally deployed.

Business-to-Business Integration

The previous section discussed the range of technologies required for a complete application integration solution. The focus of most integration projects in enterprises to date has been on integration of applications within the enterprise. The few projects that involved integration beyond the enterprise with partners and suppliers were limited in their scope and used simple mechanisms like FTP and e-mail. The real mission-critical, inter-enterprise transactions are conducted over private value-added networks (VANs) using proprietary protocols like EDI, HL-7, and SWIFT.

e-Business integration requires extensive, flexible, and dynamic cooperation and collaboration with customers, suppliers, and partners. Some examples of new business models that require this type of integration include:

• Corporate Web-based self-service applications that requires users to have a unified and up-to-date view of business processes

• Virtual, extended supply chains

• Procurement through online marketplaces or business-to-business commerce exchanges

• Dynamic order fulfillment

• Hosting applications by Application Service Providers

The integration must take place over the low-cost and ubiquitous Internet, so that an enterprise can choose from a worldwide selection of partners, suppliers, and vendors. This necessity places additional requirements on a solution. These requirements include:

• End-to-end security: Business partners must have complete confidence that a transaction they conduct over the Internet is completely secure throughout its entire life cycle. Information must be secure while it resides in databases, is transmitted over networks, and is processed by applications.

• Auditing and tracking: Traditionally, private network vendors have provided value-added services such as auditing and tracking of all business transactions between partners. The Internet provides a cheaper and ubiquitous alternative to private networks. However, an integration solution that exploits this inexpensive transport must provide auditing and tracking services.

• High availability: When conducting business online and globally, enterprises must be open for business all the time, with no downtime. Therefore, the integration software linking these companies and their applications is mission-critical and needs to be highly reliable, scalable, and always available. The integration software is at least as mission-critical as the applications it links.

• Complex business processes: In terms of duration, number of organizations involved, and number of applications involved, e-business integration must deal with complex business processes. The highly autonomous, heterogeneous, and distributed nature of these applications places additional requirements on all aspects of an integration solution.

• Internet standards: By their very nature, business-to-business transactions must take place within agreed-upon standards. Protocols specific to one enterprise or to one integration vendor do not necessarily scale across a broader set of partners, each of which might have its own integration solution. General purpose software must support all the standard Internet protocols such as
  • HTTP as the wire or transport protocol

  • XML for message formats with standard definitions for common business objects such as those proposed by the Open Applications Group

  • Business process protocols such as Open Buying on the Internet (OBI) and RosettaNet