---

3. Running Domains

---

What This Chapter Is About

This chapter covers the following topics:

• Domains run-time administrative commands and servers

• Transaction management

Run-time Administration

Domains integrates with an existing BEA TUXEDO application by adding entries for domain gateways groups and gateway servers in the TUXCONFIG file. The existing tmconfig(1) and tmadmin(1) commands can be used to add this new configuration dynamically to a running BEA TUXEDO application. The tmadmin command can also be used to list the information contained in the Bulletin Board for Domains gateway groups and their gateways.

In addition to tmadmin(1), a new administrative command, dmadmin(1), is provided. This command provides administrators with the run-time administration capabilities to maintain the Domains configuration and active gateway groups. Figure 3-1 shows the relationship between administrative commands and servers.

Figure 3-1 Domains Run-Time Administration

Figure 3-1 shows the components of the Domains administrative subsystem: the dmadmin(1) command, the Domains administrative server, DMADM(5), the gateway group administrative server, GWADM(5), the gateway process, GWTDOMAIN(5), and the Domains configuration file, BDMCONFIG. The dmadmin command can be used by the administrator to update the BDMCONFIG file while the BEA TUXEDO application is running. The command acts as a front-end process that translates administrative commands to service requests to the DMADMIN service, which is a generic administrative service advertised by the DMADM server. The DMADMIN service invokes the validation, retrieval, or update functions provided within the DMADM server to maintain the BDMCONFIG file. The DMADM server also provides a registration service to gateway groups. This registration service is requested by GWADM servers as part of their initialization procedure. The registration service downloads the configuration information required by the requesting gateway group. The DMADM server maintains the list of registered gateway groups, and propagates to these groups any changes made to the configuration.

Note: The gateway process, GWTDOMAIN(5), which provides connectivity to remote gateway processes, focuses on throughout of messages between BEA TUXEDO domains. This process is not get involved in Domains administration.

Figure 3-1 also shows some of the functionality provided by the GWADM server. This server registers with the DMADM server to obtain the configuration information used by the corresponding gateway group. The GWADM accepts queries from dmadmin to obtain run-time statistics or to change the run-time options of the corresponding gateway group. Periodically, the GWADM server sends an "I-am-alive" message to the DMADM server. If no reply is received from the DMADM server, the GWADM server registers again. This mechanism makes sure the GWADM server always has the latest copy of the Domains configuration for its group.

Another run-time feature is the capability to specify gateway parameters when a gateway group is booted. This capability requires the use of the CLOPT parameter when the GWADM server is defined in the *SERVERS section of the TUXCONFIG file.

The main capabilities of the run-time administration feature are as follows:

• A command, dmadmin(1), that allows administrators dynamically to configure, monitor, and tune domain gateway groups

• A domain administrative server, DMADM(5), that provides the administrative processing required for updating the Domains configuration. This server acts as a back-end to the dmadmin command.

• A gateway administrative server, GWADM(5), that provides the run-time administrative processing required for a specific gateway group. This server also acts as a back-end to the dmadmin command.

• The gateway process, GWTDOMAIN(5), that provides connectivity to remote gateway processes. Clients and servers send and receive messages across BEA TUXEDO domains via the GWTDOMAIN process.

The capabilities listed above are described in more detail later in this chapter.

Migration Restriction

The migration of the DMADM is possible. Also, the migration of a domain gateway group (GWADM/GWTDOMAIN) to another machine is possible. However, when using transactions, the domain gateway group can be migrated only across machines of the same type. You must complete the following procedures to migrate GWADM/GWTDOMAIN.

1. The DMCONFIG file should include multiple listening addresses in the following format in the DM_TDOMAIN section:

   *DM_TDOMAIN

   LDOM NWADDR="//primary:port"

   LDOM NWADDR="//backup:port"

   Note: This step is unnecessary if third party IP failover solutions are used.

2. The DMCONFIG source must be copied manually to the backup machine and built using dmloadcf.

3. If you are using transactions, the domains transaction log must also be copied manually to the backup machine.

4. The DMCONFIG files of the remote domains should include both network addresses as specified in Step 1.

Security Limitation

For security reasons, the domain administrative command, dmadmin, cannot be run from a workstation when Domains is used with an older release of BEA TUXEDO. Older releases cannot distinguish application administrators from ordinary users accessing the system from a workstation.

DMADMIN(1)

The dmadmin command is used by application administrators for the interactive administration of the information stored in the BDMCONFIG file and the different gateway groups running within a particular BEA TUXEDO application. dmadmin is used to obtain statistics or other information gathered by gateway groups, to change the gateway group parameters, and to add (or update) information to the BDMCONFIG file.

dmadmin(1) is designed for extensibility. In future releases, new commands will be integrated with only minor changes to the software.

dmadmin is implemented as a front-end to the DMADM and GWADM servers. The communication between the two servers is done via FML typed buffers.

For details, see dmadmin(1) in the BEA TUXEDO Reference Manual.

Run-time Deletions

Run-time deletions to the BDMCONFIG file can be performed only when the changes do not involve an active gateway group.

DMADM(5)

The domain administrative server, DMADM(5), is a BEA TUXEDO-supplied server that provides run-time administration of the BDMCONFIG file. Its main function is to maintain the BDMCONFIG file and to support a list of registered gateway groups. The DMADM propagates run-time configuration changes to the registered gateway groups (see Figure 3-1).

The DMADM server advertises two services:

• DMADMIN, which is used by dmadmin and the GWADM servers

• A service whose name is derived from the server identifier, DMADM_<svrid>. DMADM_<svrid> is used by registered GWADM servers for specific administrative functions (for example, to refresh the gateway group configuration information or to signal that a GWADM is still registered).

For details, see DMADM(5) in the BEA TUXEDO Reference Manual.

The DMADM server must be defined in the *SERVERS section of the TUXCONFIG file as a server running within a group (for example, DMADMGRP). There should be only one instance of the DMADM server in this group and it must be defined with no reply queue, that is, REPLYQ=N.

The administrator may define only one copy of the BDMCONFIG file per BEA TUXEDO application (that is, per domain).

GWADM(5)

The gateway administrative server, GWADM(5), is a BEA TUXEDO-supplied server that provides administrative functions for a Domains gateway group.

The main functions of the GWADM server are:

• To get Domains configuration information from the DMADM server, and to accept queries from dmadmin. The GWADM server gets the gateway group configuration information by registering with the DMADM server. The GWADM server then makes the configuration available to gateways by storing the information in shared memory.

• To provide administrative functionality for a gateway group, for example, to accept queries from dmadmin for run-time statistics or to change run-time parameters of the gateway group.

• To provide transaction logging functionality for a gateway group. The GWADM server determines what transactions need to be logged by reading information stored in shared memory. When the GWADM server is booted, it performs a scan of the log to see if any transactions need to be recovered; it then reconstructs the transaction information in shared memory. The first gateway booted, called the distinguished gateway, scans the information in shared memory and performs recovery for the corresponding transactions. The recovery procedure is performed asynchronously with new incoming or outgoing requests received by the gateway group.

The GWADM server advertises two services:

• A generic service, GWADMIN

• A service name based on the local domain name (the value of the LDOM keyword in the BDMCONFIG)

The dmadmin command uses the services to retrieve information from all active gateway groups or from a specific gateway group.

The GWADM server must be defined in the *SERVERS section of the TUXCONFIG file. It should not be part of the MSSQ used by the gateways associated with the group and it must not have a reply queue, that is, REPLYQ=N must be specified. It must be the first server booted within the gateway group; that is, either (a) it must have a SEQUENCE number, or (b) it must be defined ahead of the gateway servers.

The GWADM server requires the existence of a DMADM server. Specifically, a DMADM server must be booted before that GWADM is booted.

For details, see GWADM(5) in the BEA TUXEDO Reference Manual.

The GWADM server must create the shared memory required by the gateway group to populate the configuration tables with information received from the DMADM server. The GWADM server uses IPC_PRIVATE with shmget and stores the ipckey returned in the shmid field of its registry entry in the Bulletin Board. Gateways can learn the ipckey by retrieving the GWADM registry entry and checking the shmid field.

GWTDOMAIN(5)

The GWTDOMAIN, or gateway process, provides connectivity to remote gateway processes. A GWTDOMAIN process can communicate with one or more remote gateways simultaneously.

GWTDOMAIN gateways should not be specified as members of a MSSQ set. They should not have a reply queue (REPLYQ=N should be specified). GWTDOMAIN gateways may be restartable.

Transaction Management

Application programmers can request the execution of remote services within a transaction. Also, users at remote domains can request the execution of local services within a transaction. Therefore, Domains coordinates the mapping of remote transactions to local transactions, and the sane termination (commitment or rollback) of these transactions.

BEA TUXEDO architecture uses a separate process (the TMS) to coordinate the commitment and the recovery of transaction branches accessing a particular group. In Domains, however, this architecture would require extra messages from the gateway to the TMS server to process the commitment for incoming transactions. To simplify the Domains architecture and to reduce the number of messages, the TMS code has been integrated with the gateway code.

Hence, Domains gateways can process the transaction protocol used by the BEA TUXEDO system. BEA TUXEDO transaction protocol requires that the gateway group advertise the TMS service. This advertisement is done when the first gateway is booted. Once the TMS service is advertised, any transaction control messages directed to the gateway group are put on the gateways' queue.

Domains gateway groups should be defined in the TUXCONFIG file without the parameters TMSNAME, TMSCOUNT, OPENINFO, and CLOSEINFO. These four parameters apply to groups that use an XA-compliant resource manager; Domains gateways do not use the XA interface.

The commitment protocol across domains is strictly hierarchical. It is not possible to flatten the transaction tree because the structure of the transaction tree is not fully known at every domain, that is, a superior knows only its immediately subordinate domains. Flattening the tree would also require the root domain to be fully connected to all domains participating in the transaction.

The following sections describe the transaction management capabilities that are provided by domain gateways. These capabilities are:

• TMS functionality

• Mapping of GTRIDs (Global Transaction Identifiers)

• Logging transactions

• Recovery

TMS Functionality

In BEA TUXEDO, the TMS is a special server that is implicitly associated with server groups that use X/Open XA-compliant resource managers. The TMS server releases application servers from the delays associated with the distributed two-phase commitment protocol. TMSs coordinate the commitment of a transaction via special service requests to the TMS service, which is offered by all TMS servers.

In Domains, GWTDOMAIN gateways are not associated with an XA-compliant resource manager. The Transaction Processing Working Group (TPWG) of X/Open has proposed an advanced XA interface. This interface is not used in BEA TUXEDO because the interface does not match the highly asynchronous and non-blocking model required by the gateway. While Domains gateways do not use a separate TMS server, they do offer the TMS capability, which allows gateways to coordinate the two-phase commitment of transactions executed across domains. This is accomplished as follows:

• Domains gateways advertise the TMS service and perform all operations associated with that service. Messages sent to this service are placed on the queue used by the gateways and the gateways are able to process transaction management functions for transactions associated with the group.

• A gateway can act as a subordinate of transactions coordinated by another group within the domain. In this case, the gateway is a superior of the transaction branches executed in other remote domains. (See GWTDOMAIN in Figure 3-2.)

• When acting as a subordinate of a transaction coordinated by a remote domain, the gateway also acts as the coordinator for all groups in the local domain accessed by the transaction. (See GWTDOMAIN in Figure 3-2.)

  Figure 3-2 The Gateway as Subordinate/Coordinator

  

• A gateway can act as a coordinator of transactions within the domain. In this case, the gateway manages the transaction commitment for a particular client. (See Figure 3-3.)

  Figure 3-3 The Gateway Managing Commitment for a Client

  

• Gateways manage transaction commitment for a particular client or for a server that uses the forwarding service with the AUTOTRAN capability. When this combination is used, the last server (the Domains gateway) in the forward chain issues the commit and becomes the coordinator of the transaction; as a matter of fact, Domains gateways always act as the last server in a forward chain.

• Gateways automatically start/terminate transactions for remote services specified with the AUTOTRAN capability. This capability is required when the application administrator wants to enforce reliable network communication with remote services. Application administrators specify this capability by setting the AUTOTRAN parameter to Y in the corresponding remote service definition. (See the *DM_REMOTE_SERVICES section of the dmconfig(5) reference page in the BEA TUXEDO Reference Manual.)

• Gateways map the BEA TUXEDO transaction protocol to the networking transaction protocol used for interoperation with remote domains. This mapping is instantiation-dependent.

GTRID Mapping

A GTRID is a Global Transaction IDentifier. GTRID mapping defines how a transaction tree that crosses domain boundaries is constructed.

In the BEA TUXEDO system, a transaction tree is a two-level tree where the root is the group coordinating a global transaction and the machines are other groups involved in the transaction. Each group performs its part of the global transaction independently from the parts done by other groups. Each group, therefore, implicitly defines a transaction branch. The BEA TUXEDO system, through Transaction Manager Servers, TMSs, coordinates the completion of the global transaction, making sure each branch completes.

Tightly-coupled or Loosely-coupled

In the X/Open DTP Model, a Transaction Manager can construct transaction trees by defining either tightly-coupled or loosely-coupled relationships with a Resource Manager by the way it interprets the transaction identifiers (XIDs) used by the XA interface.

A tightly-coupled relationship is one in which the same transaction identifier, XID, is used by all processes participating in the same global transaction and accessing the same RM. This relationship maximizes data sharing between processes; XA-compliant RMs expect to share locks for resources used by processes having the same XID. The BEA TUXEDO system achieves the tightly-coupled relationship via the group concept; that is, work done by a group on behalf of a given global transaction belongs to the same transaction branch; all the processes are given the same XID.

In a loosely-coupled relationship, the TM generates a transaction branch for each part of the work in support of the global transaction. The RM handles each transaction branch separately; there is no sharing of data or of locks between the transaction branches. Deadlocks between transaction branches can occur. A deadlock results in the rollback of the global transaction. In the BEA TUXEDO system, when different groups participate in the same global transaction each group defines a transaction branch; this results in a loosely-coupled relationship.

Global Transactions across Domains

The first difference between a global transaction in a single BEA TUXEDO application and global transactions across domains is that in the Domains framework the transaction tree cannot be flattened to a two-level tree. There are two reasons for this:

• The transaction may involve more domains than can be known from the root domain (where the transaction is controlled), so the structure of the transaction tree cannot be fully known.

• If the transaction tree were flattened to a two-level tree, the root domain would have to be directly connected to all domains in the transaction.

This means that the commitment protocol across domains must be hierarchical. Even a loop-back service request defines a new branch in the transaction tree.

Note: A loop-back request is one that goes to another domain and then comes back to be processed in the original domain. For example, domain A requests a service of domain B. The service in domain B requests another service in domain A. The transaction tree has two branches at the network level: a branch b1 from A to B and a branch b2 from B to A. Domain A cannot commit the work done on branch b2 before receiving commit instructions from B.

The structure of the transaction tree for global transactions across domains also depends on the distributed transaction processing protocol used by the particular Domains instantiation. For example, in the OSI TP protocol each dialogue (the OSI TP word for a service request) is associated with a different transaction branch. In the BEA TUXEDO system, the OSI TP instantiation uses a dialogue for each service request, so each service request is mapped to a separate transaction branch. The XAP-TP interface hides this mapping and provides a mechanism by which an entire OSI TP sub-tree can be referenced by a user-defined identifier. (In the BEA TUXEDO implementation, this identifier is the GTRID.) The GTRID is used to instruct XAP-TP how a transaction tree must be constructed, that is, which dialogues must be included within a given OSI TP transaction. Therefore, from the BEA TUXEDO perspective, a whole OSI TP subtree can be managed as a single transaction branch.

This property, however, applies only to outgoing service requests (that is, service requests sent from the root domain to subordinate domains). It cannot be applied to incoming service requests. The OSI TP instantiation consequently implements a loosely-coupled relationship; each incoming service request is mapped to a new BEA TUXEDO global transaction.

The /TDOMAIN instantiation tries to optimize GTRID mapping by implementing a tightly-coupled relationship. In /TDOMAIN, service requests issued on behalf of the same global transaction are mapped to the same network transaction branch. Therefore, incoming service requests can be mapped to a single BEA TUXEDO transaction. However, the hierarchical structure of inter-domain communication and the inter-domain transaction tree must still be maintained.

The optimization that /TDOMAIN introduces applies only to a single domain. When two or more domains are involved in the transaction, the network transaction tree contains at least one branch per domain interaction. Hence, across domains, the network transaction tree remains loosely-coupled. There will be as many branches as there are domains involved in the transaction (even if all branches access the same resource manager instance).

Domains gateway groups implement a loosely-coupled relationship because they generate different transaction branches for inter-domain transactions.

Figure 3-4 shows the service request graph for a client that generates three service requests: one local request (r0) and two remote requests (r2 and r3). Request r0 goes to a local service (Svc0), which generates another remote service request (r1). Request r1 goes to remote service Rsvc1, which issues a loop-back service request r4 to local service Svc4. Svc0 and Svc4 execute in different groups (G0 and G4). The Domains gateway executes within another group (GW), and the remote services Rscv1, Rsvc2, and Rsvc3 execute in another domain (domain B).

Figure 3-4 Service Request Graph

For the service request graph shown in Figure 3-4, Figure 3-5 shows the transaction tree for BEA Connect OSI TP and Figure 3-6 shows the transaction tree for /TDomain. It is assumed, in these figures, that both domains A and B are BEA TUXEDO applications. BEA Connect OSI TP is loosely-coupled because of the OSI TP protocol. The transaction tree for this instantiation shows group G0 in Domain A coordinating the global transaction started by the client. Group G0 coordinates group GW. Requests r1, r2, and r4 are mapped each to an OSI TP dialogue and therefore to an OSI TP transaction branch. However, OSI TP uses the XAP-TP feature that allows an entire OSI TP transaction to be referred by a unique identifier (T1) and uses this identifier for requests r1, r2, and r3. It is up to XAP-TP to generate OSI TP transaction identifiers and to construct the corresponding OSI TP transaction tree. The generic part of Domains only needs to map service requests r1, r2, and r3 to the T1 identifier. In domain B, OSI TP uses the rule that new transaction branches must be mapped to a new BEA TUXEDO transaction. Therefore, OSI TP transaction branches r1, r2, and r3 get mapped to three different BEA TUXEDO transactions (the corresponding mapping is represented by identifiers T2, T3, and T4). The graph shows the gateway group GW in Domain B coordinating three BEA TUXEDO transactions on group G1. Finally, there is the loop-back service request r4 that generates another branch in the transaction tree. OSI TP maps this request to identifier T2, but XAP-TP generates a new branch in its transaction tree (r4: B to A'). This is a new transaction branch on Domain A, and therefore, the gateway generates a new mapping T5 to a new BEA TUXEDO transaction. Hence, the transaction graph shows that gateway group GW on Domain A coordinates group G4.

Notice that the hierarchical nature of the OSI TP protocol is fully enforced by these mappings. However, because these mappings introduce a loosely-coupled relationship, the probability of intra-transaction deadlock is increased (e.g., there are three BEA TUXEDO transactions accessing the RM represented by group G1).

Figure 3-5 Transaction Tree, BEA Connect OSI TP

The /TDOMAIN instantiation provides a tightly-coupled integration that solves this deadlock problem by minimizing the number of transaction branches required in the interoperation between two domains. The corresponding transaction tree is shown in Figure 3-6.

Figure 3-6 Transaction Tree, /TDOMAIN

Notice that the gateway still must perform mappings between a BEA TUXEDO transaction and a network transaction, and that the hierarchical nature of the communication between domains must be strictly enforced. The figure shows that requests r1, r2, and r3 are mapped to a single /TDOMAIN transaction branch. Therefore, on domain B only one BEA TUXEDO transaction needs to be generated; T2 represents this mapping and the graph shows gateway group GW on domain B coordinating group G1. Request r4 is mapped to identifier T2 on Domain B, but /TDOMAIN will generate a new branch in its transaction tree (r4: B to A'). This is a new transaction branch on Domain A, and therefore, the gateway generates a new mapping T3 to a new BEA TUXEDO transaction. The graph shows that gateway group GW on domain A also coordinates group G4. Hence, the hierarchical nature of inter-domain communication is fully enforced with this mapping: group G4 cannot commit before group G1.

Summary of Transaction Management

We can summarize Domains transaction management as follows:

• Gateways generate mappings from a BEA TUXEDO transaction to a network transaction. A new mapping is generated per BEA TUXEDO transaction or per incoming network transaction branch.

• Each instantiation handles its own representation of the network transaction tree. Instantiations observe the hierarchical nature of the inter-domain communication.

Logging

Logging is used to keep track of the progress of the two-phase commit protocol. The information stored in the log is used to make sure the transaction is completed in the event of a network failure or machine crash.

To ensure completion of transactions across domains, Domains gateways log the mapping between local and remote identifiers. Along with this information, the Domains transaction management facility records the decisions during the different phases of the commitment protocol, and the information about the different remote domains involved in the transaction. In the OSI TP case, the XAP-TP interface logs the information required for the recovery of the OSI TP protocol machine. The information is referred to as a blob (for binary large object); it is kept in the same log record as the commit information to make recovery easier.

Domains log records have a different structure from the log records stored in the BEA TUXEDO system's TLOG. The TLOG records are fixed size and are stored in a single page. Domains log records are variable size; more than one page may be required to store the record. The Domains logging mechanism, DMTLOG, has the ability to store variable size log records.

When a TMS is the superior of a Domains gateway group, the BEA TUXEDO TLOG is still required to coordinate the commitment.

How Logging Works

Logging is done by the GWADM administrative server. While the GWTDOMAIN process requests a log write to be performed, the GWADM performs the actual log write. Each Domains gateway group has its own log. The specification of the DMTLOG is included in the Domains configuration file, DMCONFIG, for each local domain by means of one required parameter (DMTLOGDEV=string) and two optional parameters (DMTLOGNAME=identifier and DMTLOGSIZE=numeric) in the *DM_LOCAL_DOMAINS section. (See dmconfig(5) in the BEA TUXEDO Reference Manual.) The administrator can optionally create a DMTLOG with the run-time administration facility. (See dmadmin(1) in the BEA TUXEDO Reference Manual.)

If the DMTLOG has not been created when a Domains gateway group is booted, the distinguished gateway automatically creates the log. The information for this automatic creation is extracted from the BDMCONFIG file.

Until the logging device is specified in the BDMCONFIG file the Domains gateway group cannot process requests in transaction mode and the gateway group cannot offer the TMS service.

To coordinate the commit protocol, Domains gateways require the following two log records:

• Ready record-The ready record is created by a gateway acting as a leaf or intermediate machine in a transaction tree. It records information about the superior and subordinate remote domains involved in the transaction. The log record indicates that all subordinates of the domain gateway group logging the record have been prepared.

• Commit record-The commit record notes that a transaction has been committed. A domain gateway creates the commit record when it is the coordinator of a particular transaction tree.

When the transaction has been committed at all machines, these log records of the transaction are removed.

In the OSI TP protocol, two types of heuristic records are logged:

• Log Heuristic record -This record holds the details of a heuristic decision at the domain until the outcome of the transaction is known by the superior.

• Log Damage record -This record is created for a transaction branch to indicate either a heuristic hazard (the outcome of the transaction branch for a subordinate is unknown) or a heuristic mix (the transaction subtree has a mixed outcome).

Heuristic log records persist until they are explicitly removed by the administrator. This persistence is required to provide the right information during recovery after a crash, and to provide diagnostic information for administrators.

The administrator uses the forgettran command of dmadmin(1) to remove heuristic records when they are no longer needed.

Recovery

When a Domains gateway group is booted, the distinguished gateway performs an automatic warm-start of the DMTLOG. The warm-start includes scanning the log to see if any transactions were not completed. If uncompleted transactions are found, action is taken to move them to completion.

In OSI TP any blobs stored in the DMTLOG with a transaction record are passed to the network access module, which uses the blobs to reconstruct its internal state and to recover any failed connection.

In the case of heuristic decisions, if the Domains gateway group is a subordinate of a local TMS and the heuristic decision has been indicated, the TMS generates a TMS_STATUS message to learn the final decision.

• If a gateway fails, then it cleans up after itself when it is restarted (this is called a hot-start). The gateway rolls back all undecided transactions in which it was involved.

• If a communication line failure occurs and the first phase of commit has not been completed, the gateway rolls back the transactions associated with that connection.

• In OSI TP if a transaction was in the second phase of commit when a failure is produced, recovery is managed by the XAP-TP.

---

---