BEA Logo BEA BEA Tuxedo Release [Release Number]

  BEA Home  |  Events  |  Solutions  |  Partners  |  Products  |  Services  |  Download  |  Developer Center  |  WebSUPPORT

 

   BEA Tuxedo Doc Home   |   Creating CORBA Server Applications   |   Previous Topic   |   Next Topic   |   Contents   |   Index

Scaling a BEA Tuxedo CORBA Server Application

 

This chapter shows how you can take advantage of several key scalability features of the BEA Tuxedo system to make a CORBA server application highly scalable, using the Production University sample application as an example. The Production sample application uses these scalability features to achieve the following goals:

This topic includes the following sections:

 


Overview of the Scalability Features Available in the BEA Tuxedo System

Supporting highly scalable applications is one of the strengths of the BEA Tuxedo system. Many applications may perform well in an environment characterized by 1 to 10 server processes, and 10 to 100 client applications. However, in an enterprise environment, applications need to support:

Deploying an application with such demands quickly reveals the resource shortcomings and performance bottlenecks in your application. The BEA Tuxedo system supports such large-scale deployments in several ways, three of which are described in this chapter as follows:

Other features provided in the BEA Tuxedo system to make an application highly scalable include the IIOP Listener/Handler, which is summarized in Getting Started with BEA Tuxedo CORBA Applications and fully described in Setting Up a BEA Tuxedo Application. See also Scaling, Distributing, and Tuning CORBA Applications.

 


Scaling a BEA Tuxedo Server Application

This section explains how to scale an application to meet a significantly greater processing capability, using the Production sample application as an example. The basic design goal for the Production sample application is to greatly scale up the number of client applications it can accommodate by doing the following:

To accommodate these design goals, the Production sample application does the following:

Note: To make the Production sample application easy for you to use, this application is configured on the BEA Tuxedo software kit to run on one machine, using one database. The examples shown in this chapter, however, show running this application on two machines using two databases.

The design of the Production sample application is set up so that it can be configured to run on several machines and to use multiple databases. Changing the configuration to multiple machines and databases involves modifying the UBBCONFIG file and partitioning the databases, and is described in How the Production Server Application Can Be Scaled Further.

The sections that follow describe how the Production sample application uses replicated server processes and server groups, object state management, and factory-based routing to meets its scalability goals. The first section that follows provides a description of the OMG IDL changes implemented in the Production sample application.

OMG IDL Changes for the Production Sample Application

The only OMG IDL changes for the Production sample application are limited to the find_registrar() and find_teller() operations on, respectively, the RegistrarFactory and TellerFactory objects. These two operations are modified to require, respectively, a student ID and account number, which is needed to implement factory-based routing. See the section Factory-based Routing to read about how the Production sample application implements and uses factory-based routing.

Replicating Server Processes and Server Groups

The BEA Tuxedo system offers a wide variety of choices for how you may configure your server applications, such as:

In summary:

The following sections describe replicated server processes and groups, and also explain how you can configure them in the BEA Tuxedo system.

Replicated Server Processes

When you replicate the server processes in your application:

To achieve the full benefit of replicated server processes, make sure that the objects instantiated by your server application generally have unique IDs. This way, a client invocation on an object can cause the object to be instantiated on demand, within the bounds of the number of server processes that are available, and not queued up for an already active object.

Figure 8-1 shows the following:

Both groups are shown in this figure as running on a single machine.

Figure 8-1 Replicated Server Groups in the Production Sample


 
 

When a request arrives for either of these groups, the BEA Tuxedo domain has several server processes available that can process the request, and the BEA Tuxedo domain can choose the server process that is least busy.

In Figure 8-1, note the following:

Replicated Server Groups

The notion of server groups is specific to the BEA Tuxedo system and adds value to a CORBA implementation; server groups are an important part of the scalability features of the BEA Tuxedo system. Basically, to add more machines to a deployment, you need to add more groups.

Figure 8-2 shows the Production sample application groups replicated on another machine, as specified in the application's UBBCONFIG file, as ORA_GRP2 and APP_GRP2.

Figure 8-2 Replicating Server Groups Across Machines

In Figure 8-2, the only difference between the content of the groups on Production Machines 1 and 2 is the database. The database for Production Machine 1 contains student and account information for a subset of students. The database for Production Machine 2 contains student and account information for a different subset of students. (The course information table in both databases is identical.) Note that the student information in a given database may be completely unrelated to the account information in the same database.

The way in which server groups are configured, where they run, and the ways in which they are replicated is specified in the UBBCONFIG file. When you replicate a server group, you can do the following:

The effect of having multiple server groups includes the following:

The section Factory-based Routing shows how the Production sample application uses factory-based routing to spread the application's processing load across multiple machines.

Configuring Replicated Server Processes and Groups

To configure replicated server processes and groups in your BEA Tuxedo domain:

  1. Bring your application's UBBCONFIG file into a text editor, such as WordPad.

  2. In the GROUPS section, specify the names of the groups you want to configure.

  3. In the SERVERS section, enter the following information for the server process you want to replicate:

The following example shows lines from the GROUPS and SERVERS sections of the UBBCONFIG file for the Production sample application.

*GROUPS
APP_GRP1
LMID = SITE1
GRPNO = 2
TMSNAME = TMS
APP_GRP2
LMID = SITE1
GRPNO = 3
TMSNAME = TMS
ORA_GRP1
LMID = SITE1
GRPNO = 4
OPENINFO = "ORACLE_XA:Oracle_XA+Acc=P/scott/..."
CLOSEINFO = ""
TMSNAME = "TMS_ORA"
ORA_GRP2
LMID = SITE1
GRPNO = 5
OPENINFO = "ORACLE_XA:Oracle_XA+Acc=P/scott/..."
CLOSEINFO = ""
TMSNAME = "TMS_ORA"
*SERVERS
# By default, activate 2 instances of each server
# and allow the administrator to activate up to 5
# instances of each server
DEFAULT:
MIN = 2
MAX = 5
tellp_server
SRVGRP = ORA_GRP1
SRVID = 10
RESTART = N
tellp_server
SRVGRP = ORA_GRP2
SRVID = 10
RESTART = N
billp_server
SRVGRP = APP_GRP1
SRVID = 10
RESTART = N
billp_server
SRVGRP = APP_GRP2
SRVID = 10
RESTART = N
univp_server
SRVGRP = ORA_GRP1
SRVID = 20
RESTART = N
univp_server
SRVGRP = ORA_GRP2
SRVID = 20
RESTART = N

Scaling the Application Via Object State Management

As stated in CORBA Server Application Concepts, object state management is a fundamentally important concern of large-scale client/server systems because it is critically important that such systems achieve optimized throughput and response time. This section explains how you can use object state management to increase the scalability of the objects managed by a BEA Tuxedo server application, using the Registrar and Teller objects in the Production sample applications as an example.

The following table summarizes how you can use the object state management models supported in the BEA Tuxedo system to achieve major gains in scalability in your BEA Tuxedo applications.

State Model

How You Can Use It to Achieve Scalability

Method-bound

Method-bound objects are brought into the machine's memory only for the duration of the client invocation on them. When the invocation is complete, the object is deactivated and any state data for that object is flushed from memory.

You can use method-bound objects to create a stateless server model in your application, in which thousands of objects are managed by your application. From the client application view, all the objects are available to service requests. However, because the server application is mapping objects into memory only for the duration of client invocations on them, only comparatively few of the objects managed by the server application are in memory at any given moment.

A method-bound object is said in this document to be a stateless object.

Process-bound

Process-bound objects remain in memory from the time they are first invoked until the server process in which they are running is shut down. If appropriate for your application, process-bound objects with a large amount of state data can remain in memory to service multiple client invocations, and the system's resources need not be tied up reading and writing the object's state data on each client invocation.

A process-bound object is said in this document to be a stateful object. (Note that transaction-bound objects can also be considered stateful, since they can remain in memory between invocations on them within the scope of a transaction.)


 

To achieve scalability gains, the Registrar and Teller objects are configured in the Production server application to have the method activation policy. The method activation policy assigned to these two objects results in the following behavior changes:

With the Basic through the Wrapper sample applications, the Registrar object was process-bound. All client requests on that object invariably went to the same object instance in the machine's memory. The Basic sample application design may be adequate for a small-scale deployment. However, as client application demands increase, client requests on the Registrar object eventually become queued, and response time drops.

However, when the Registrar and Teller objects are stateless, and the server processes that manage these objects are replicated, these objects can be made available to process client requests on them in parallel. The only constraint on the number of simultaneous client requests that these objects can handle is the number of server processes that are available that can instantiate these objects. These stateless objects, thereby, make for more efficient use of machine resources and reduced client response time.

Most importantly, so that the BEA Tuxedo system can instantiate copies of the Registrar and Teller objects in each of the replicated server processes, each copy of these objects must be unique. To make each instance of these objects unique, the factories for those objects must assign unique object IDs to them. This, and other design considerations on these two objects, are described in the section Additional Design Considerations for the Registrar and Teller Objects.

Factory-based Routing

Factory-based routing is a powerful feature that provides a means to send a client request to a specific server group. Using factory-based routing, you can spread that processing load for a given application across multiple machines, because you can determine the group, and thus the machine, in which a given object is instantiated.

You can use factory-based routing to expand upon the variety of load-balancing and scalability capabilities in the BEA Tuxedo system. In the case of the Production sample application, you can use factory-based routing to send requests to register one subset of students to one machine, and requests for another subset of students to another machine. As you add machines to ramp up your application's processing capability, the BEA Tuxedo system makes it easy to modify the factory-based routing in your application to add more machines.

The chief benefit of factory-based routing is that it provides a simple means to scale up an application, and invocations on a given interface in particular, across a growing deployment environment. Spreading the deployment of an application across additional machines is strictly an administrative function that does not require any recoding or rebuilding of the application.

The chief design consideration regarding implementing factory-based routing in your client/server application is in choosing the value on which routing is based. The sections that follow describe how factory-based routing works, using the Production sample application, which uses factory-based routing in the following way:

How Factory-based Routing Works

Your factories implement factory-based routing by changing the way they create object references. All object references contain a group ID, and by default the group ID is the same as the factory that creates the object reference. However, using factory-based routing, the factory creates an object reference that includes routing criteria that determines the group ID. Then when client applications send an invocation using such an object reference, the BEA Tuxedo system routes the request to the group ID specified in the object reference. This section focuses on how the group ID is generated for an object reference.

To implement factory-based routing, you need to coordinate the following:

To describe the data that needs to be coordinated, the following two sections discuss configuring for factory-based routing in the UBBCONFIG file, and implementing factory-based routing in the factory.

Configuring for Factory-based Routing in the UBBCONFIG file

For each interface for which requests are routed, you need to establish the following information in the UBBCONFIG file:

To configure for factory-based routing, the UBBCONFIG file needs to specify the following data in the INTERFACES and ROUTING sections, and also in how groups and machines are identified:

  1. The INTERFACES section lists the names of the interfaces for which you want to enable factory-based routing. For each interface, this section specifies what kinds of criteria the interface routes on. This section specifies the routing criteria via an identifier, FACTORYROUTING, as in the following example:
    INTERFACES
    "IDL:beasys.com/UniversityP/Registrar:1.0"
    FACTORYROUTING = STU_ID
    "IDL:beasys.com/BillingP/Teller:1.0"
    FACTORYROUTING = ACT_NUM

    The preceding example shows the fully qualified interface names for the two interfaces in the Production sample in which factory-based routing is used. The FACTORYROUTING identifier specifies the names of the routing values, which are STU_ID and ACT_NUM, respectively.

  2. The ROUTING section specifies the following data for each routing value:

  3. The groups specified by the RANGES identifier in the ROUTING section of the UBBCONFIG file need to be identified and configured. For example, the Production sample specifies four groups: APP_GRP1, APP_GRP2, ORA_GRP1, and ORA_GRP2. These groups need to be configured, and the machines on which they run need to be identified.

    The following example shows the GROUPS section of the Production sample UBBCONFIG file, in which the ORA_GRP1 and ORA_GRP2 groups are configured. Notice how the names in the GROUPS section match the group names specified in the ROUTING section; this is critical for factory-based routing to work correctly. Furthermore, any change in the way groups are configured in an application must be reflected in the ROUTING section. (Note that the Production sample packaged with the BEA Tuxedo software is configured to run entirely on one machine. However, you can easily configure this application to run on multiple machines.)

    *GROUPS
    APP_GRP1
    LMID = SITE1
    GRPNO = 2
    TMSNAME = TMS
    APP_GRP2
    LMID = SITE1
    GRPNO = 3
    TMSNAME = TMS
    ORA_GRP1
    LMID = SITE1
    GRPNO = 4
    OPENINFO = "ORACLE_XA:Oracle_XA+Acc=P/scott/..."
    CLOSEINFO = ""
    TMSNAME = "TMS_ORA"
    ORA_GRP2
    LMID = SITE1
    GRPNO = 5
    OPENINFO = "ORACLE_XA:Oracle_XA+Acc=P/scott/..."
    CLOSEINFO = ""
    TMSNAME = "TMS_ORA"

Implementing Factory-based Routing in a Factory

Factories implement factory-based routing by the way the invocation to the TP::create_object_reference() operation is implemented. This operation has the following C++ binding:

CORBA::Object_ptr  TP::create_object_reference (
const char* interfaceName,
const PortableServer::oid &stroid,
CORBA::NVlist_ptr criteria);

The third parameter to this operation, criteria, specifies a list of named values to be used for factory-based routing. Therefore, the work of implementing factory-based routing in a factory is in building the NVlist.

As stated previously, the RegistrarFactory object in the Production sample application specifies the value STU_ID. This value must match exactly the following in the UBBCONFIG file:

The RegistrarFactory object inserts the student ID into the NVlist using the following code:

// put the student id (which is the routing criteria)
// into a CORBA NVList:
CORBA::NVList_var v_criteria;
TP::orb()->create_list(1, v_criteria.out());
CORBA::Any any;
any <<= (CORBA::Long)student;
v_criteria->add_value("student_id", any, 0);

The RegistrarFactory object has the following invocation to the TP::create_object_reference() operation, passing the NVlist created in the preceding code example:

// create the registrar object reference using
// the routing criteria :
CORBA::Object_var v_reg_oref =
TP::create_object_reference(
UniversityP::_tc_Registrar->id(),
object_id,
v_criteria.in()
);

The Production sample application also uses factory-based routing in the TellerFactory object to determine the group in which Teller objects should be instantiated based on an account number.

Note: It is possible for an object with a given interface and OID to be simultaneously active in two different groups, if those two groups both contain the same object implementation. (However, if your factories generate unique OIDs, this situation is very unlikely.) If you need to guarantee that only one object instance of a given interface name and OID is available at any one time in your domain, either: use factory-based routing to ensure that objects with a particular OID are always routed to the same group, or configure your domain so that a given object implementation is in only one group. This assures that if multiple clients have an object reference containing a given interface name and OID, the reference is always routed to the same object instance.

To enable routing on an object's OID, specify the OID as the routing criterion in the TP::create_object_reference() operation, and set up the UBBCONFIG file appropriately.

What Happens at Run Time

When you implement factory-based routing in a factory, the BEA Tuxedo system generates an object reference. The following example shows how the client application gets an object reference to a Registrar object when factory-based routing is implemented:

  1. The client application invokes the RegistrarFactory object, requesting a reference to a Registrar object. Included in the request is a student ID.

  2. The RegistrarFactory inserts the student ID into an NVlist, which is used as the routing criteria.

  3. The RegistrarFactory invokes the TP::create_object_reference() operation, passing the Registrar interface name, a unique OID, and the NVlist.

  4. The BEA Tuxedo system compares the contents of the routing tables with the value in the NVlist to determine a group ID.

  5. The BEA Tuxedo system inserts the group ID into the object reference.

When the client application subsequently does an invocation on an object using the object reference, the BEA Tuxedo system routes the request to the group specified in the object reference.

Note: Be careful how you implement factory-based routing if you use the Process-Entity design pattern. The object can service only those entities that are contained in the group's database.

Additional Design Considerations for the Registrar and Teller Objects

The principal considerations that influence the design of the Registrar and Teller objects include:

The primary implications of these considerations are that these objects must:

The remainder of this section discusses these considerations and implications in detail.

Instantiating the Registrar and Teller Objects

In University server applications prior to the Production sample application, the run-time behavior of the Registrar and Teller objects was fairly simple:

However, since the University and Billing server processes are now replicated, the BEA Tuxedo domain must have a means to differentiate between multiple instances of the Registrar and Teller objects. That is, if there are two University server processes running in a group, the BEA Tuxedo domain must have a means to distinguish between, say, the Registrar object running in the first University server process and the Registrar object running in the second University server process.

The way to provide the BEA Tuxedo domain with the ability to distinguish among multiple instances of these objects is to make each object instance unique.

To make each Registrar and Teller object unique, the factories for those objects must change the way in which they make object references to them. For example, when the RegistrarFactory object in the Basic sample application created an object reference to the Registrar object, the TP::create_object_reference() operation specified an OID that consisted only of the string registrar. However, in the Production sample application, the same TP::create_object_reference() operation uses a generated unique OID instead.

A consequence of giving each Registrar and Teller object a unique OID is that there may be multiple instances of these objects running simultaneously in the BEA Tuxedo domain. This characteristic is typical of the stateless object model, and is an example of how the BEA Tuxedo domain can be highly scalable and at the same time offer high performance.

And last, since unique Registrar and Teller objects need to be brought into memory for each client request on them, it is critical that these objects be deactivated when the invocations on them are completed so that any object state associated with them does not remain idle in memory. The Production server application addresses this issue by assigning the method activation policy to these two objects in the ICF file.

Ensuring That Student Registration Occurs in the Correct Server Group

The chief scalability advantage of having replicated server groups is to be able to distribute processing across multiple machines. However, if your application interacts with a database, which is the case with the University sample applications, it is critical that you consider the impact of these multiple server groups on the database interactions.

In many cases, you may have one database associated with each machine in your deployment. If your server application is distributed across multiple machines, you must consider how you set up your databases.

The Production sample application, as described in this chapter, uses two databases. However, this application can easily be configured to accommodate more. The system administrator can decide how many.

In the Production sample application, the student and account information is partitioned across the two databases, but course information is identical. Having identical course information in both databases is not a problem because the course information is read-only for the purposes of course registration. However, the student and account information is read-write. If multiple databases were also to contain identical data for students and accounts (that is, the database is not partitioned), the application would need to deal with the overhead of synchronizing the updates to student and account information across all the databases each time any student or account information were to change.

The Production sample application uses factory-based routing to send one set of requests to one machine, and another set to the other machine. As mentioned earlier, factory-based routing is implemented in the RegistrarFactory object by the way in which references to Registrar objects are created.

For example, when the client application sends a request to the RegistrarFactory object to get an object reference to a Registrar object, the client application includes a student ID in that request. The client application must use the object reference that the RegistrarFactory object returns to make all subsequent invocations on a Registrar object on a particular student's behalf, because the object reference returned by the factory is group-specific. Therefore, for example, when the client application subsequently invokes the get_student_details() operation on the Registrar object, the client application can be assured that the Registrar object is active in the server group associated with the database containing data for that student. To show how this works, consider the following execution scenario, which is implemented in the Production sample application:

  1. The client application invokes the find_registrar() operation on the RegistrarFactory object. Included in this invocation is the student ID 1000003.

  2. The BEA Tuxedo domain routes the client request to any RegistrarFactory object.

  3. The RegistrarFactory object uses the student ID to create an object reference to a Registrar object in ORA_GRP1, based on the routing information in the UBBCONFIG file, and returns that object reference to the client application.

  4. The client application invokes the register_for_courses() operation on the Registrar object.

  5. The BEA Tuxedo domain receives the client request and routes it to the server group specified in the object reference. In this case, the client request goes to the University server process in ORA_GRP1, which is on Production Machine 1.

  6. The University server process instantiates a Registrar object and sends the client invocation to it.

The RegistrarFactory object from the preceding scenario returns to the client application a unique reference to a Registrar object that can be instantiated only in ORA_GRP1, which runs on Production Machine 1 and has a database containing student data for students with IDs in the range 100001 to 100005. Therefore, when the client application sends subsequent requests to this Registrar object on behalf of a given student, the Registrar object interacts with the correct database.

Ensuring That the Teller Object Is Instantiated in the Correct Server Group

When the Registrar object needs a Teller object, the Registrar object invokes the TellerFactory object, using the TellerFactory object reference cached in the University Server object, as described in Sending Requests to the Teller Object.

However, because factory-based routing is used in the TellerFactory object, the Registrar object passes the student's account number when the Registrar object requests a reference to a Teller object. This way, the TellerFactory object creates a reference to a Teller object in the group that has the correct database.

Note: For the Production sample application to work properly, it is essential that the system administrator configures the server groups and the databases properly. In particular, the system administrator must make sure that a match exists between the routing criteria specified in the routing tables and the databases to which requests using those criteria are routed. Using the Production sample as an example, the database in a given group must contain the correct student and account information for the requests that are routed to that group.

 


How the Production Server Application Can Be Scaled Further

In the future, the system administrator of the Production sample application may want to add capacity to the BEA Tuxedo domain. For example, the University may eventually have a large increase in the student population, or the Production application may be scaled up to accommodate the course registration process for an entire state university system encompassing several campuses. This can be done without modifying or rebuilding the application.

The system administrator has the following tools available to continually add capacity:

Note: If you add capacity to an application that uses a database, you must also consider the impact on how the database is set up, particularly when you are using factory-based routing. For example, if the Production sample application is spread across six machines, the database on each machine must be set up appropriately and in accordance with the routing tables in the UBBCONFIG file.

 


Choosing Between Stateless and Stateful Objects

In general, you need to balance the costs of implementing stateless objects against the costs of implementing stateful objects.

In the case where the cost to initialize an object with its durable state is expensive— because, for example, the object's data takes up a great deal of space, or the durable state is located on a disk very remote to the servant that activates it—it may make sense to keep the object stateful, even if the object is idle during a conversation. In the case where the cost to keep an object active is expensive in terms of machine resource usage, it may make sense to make such an object stateless.

By managing object state in a way that's efficient and appropriate for your application, you can maximize your application's ability to support large numbers of simultaneous client applications that use large numbers of objects. You generally do this by assigning the method activation policy to these objects, which has the effect of deactivating idle object instances so that machine resources can be allocated to other object instances. However, your specific application characteristics and needs may vary.

Note: BEA Tuxedo Release 8.0 provides support for parallel objects, as a performance enhancement. This feature allows you to designate all business objects in a particular application as stateless objects. For complete information, see the section TP Framework in the CORBA Programming Reference.

When You Want Stateless Objects

Stateless objects generally provide good performance and optimal usage of server resources, because server resources are never used when objects are idle. Stateless objects are generally a good approach to implementing server applications. Stateless objects are particularly appropriate in the following situations:

By making an object stateless, you can generally assure that server application resources are not being tied up for an arbitrarily long time waiting for input from the client application.

Note the following characteristics about an application that employs a stateless object model:

When You Want Stateful Objects

A stateful object, once activated, remains in memory until a specific event occurs, such as the process in which the object exists is shut down, or the transaction in which the object is activated is completed.

Stateful objects are typically appropriate in the following situations:

Note: Plan carefully how process objects are potentially involved in a transaction. Any object that is involved in a transaction cannot be invoked by another client application or object. Process objects meant to be used by a large number of client applications can create problems if they are involved in transactions frequently or for long durations.

Note the following behavior with stateful objects:

 

back to top previous page