The following sections provide information on administration tasks used to manage transactions:
Monitor transactions on a server using statistics and monitoring facilities. Use the Administration Console to configure these features and to display the resulting output.
In the Administration Console, monitor transactions for each server in the domain. Transaction statistics are displayed for a specific server, not the entire domain.
For instructions, see the following pages in the Oracle WebLogic Server Administration Console Help:
A heuristic completion (or heuristic decision) occurs when a resource makes a unilateral decision during the completion stage of a distributed transaction to commit or rollback updates. This can leave distributed data in an indeterminate state. Network failures or resource timeouts are possible causes for heuristic completion. In the event of an heuristic completion, one of the following heuristic outcome exceptions may be thrown:
HeuristicRollback—one resource participating in a transaction decided to autonomously rollback its work, even though it agreed to prepare itself and wait for a commit decision. If the Transaction Manager decided to commit the transaction, the resource's heuristic rollback decision was incorrect, and might lead to an inconsistent outcome since other branches of the transaction were committed.
HeuristicCommit—one resource participating in a transaction decided to autonomously commit its work, even though it agreed to prepare itself and wait for a commit decision. If the Transaction Manager decided to rollback the transaction, the resource's heuristic commit decision was incorrect, and might lead to an inconsistent outcome since other branches of the transaction were rolled back.
HeuristicMixed—the Transaction Manager is aware that a transaction resulted in a mixed outcome, where some participating resources committed and some rolled back. The underlying cause was most likely heuristic rollback or heuristic commit decisions made by one or more of the participating resources.
HeuristicHazard—the Transaction Manager is aware that a transaction might have resulted in a mixed outcome, where some participating resources committed and some rolled back. But system or resource failures make it impossible to know for sure whether a Heuristic Mixed outcome definitely occurred. The underlying cause was most likely heuristic rollback or heuristic commit decisions made by one or more of the participating resources.
When an heuristic completion occurs, a message is written to the server log. Refer to your database vendor documentation for instructions on resolving heuristic completions.
Some resource managers save context information for heuristic completions. This information can be helpful in resolving resource manager data inconsistencies. If the
ForgetHeuristics attribute is selected (set to true) on the JTA panel of the WebLogic Console, this information is removed after an heuristic completion. When using a resource manager that saves context information, you may want to set the
ForgetHeuristics attribute to false.
A server instance is identified by its URL (IP address or DNS name plus the listening port number). Changing the URL by moving the server to a new machine or changing the Listening Port of a server on the same machine effectively moves the server so the server identity may no longer match the information stored in the transaction logs.
If the new server has the same URL as the old server, the Transaction Recovery Service searches all transaction log files for incomplete transactions and completes them as described in Transaction Recovery Service Actions After a Crash.
When the coordinator server is in the same domain as the sub-coordinator and the server URL changes, the coordinator queries the Administration Server for the new URL of the sub-coordinator and the propagation of any new transactions and any transactions that are committing or rolling back use the new URL. Transaction branches for the sub-coordinator with pending commit records stored in the coordinator's transaction log files before the URL change are unrecoverable. If you wish, you can delete the transaction log files of the coordinator. This step prevents the Transaction Recovery Service from attempting to resolve these transactions until the value of the
AbandonTimeoutSeconds parameter is exceeded. See Abandoning Transactions and How to Remove Transaction Records for more information.
When transactions span multiple domains and if a server acting as a remote transaction sub-coordination fails and its URL changes, any ongoing transactions do not complete (commit or are rolled back) because the coordinator is unable to communicate with the remote domain's Admin server. The coordinator is unable to contact the sub-coordinator using the new URL and any ongoing transactions fail. The coordinator attempts the commit or rollback request until the
AbandonTimeoutSeconds value is exceeded. See Abandoning Transactions for more information. Any new transactions fail because the coordinator cannot contact the sub-coordinator. The TLOGs of the coordinator and sub-coordinators, excluding the moved server domain, must be deleted. See How to Remove Transaction Records.
Oracle recommends configuring server instances using DNS names rather than IP addresses to promote portability.
If you move a server to a new machine, follow the instructions for Recovering Transactions For a Failed Non-Clustered Server.
You can choose to abandon incomplete transactions after a specified amount of time. In the two-phase commit process for distributed transactions, the transaction manager coordinates all resource managers involved in a transaction. After all resource managers vote to commit or rollback, the transaction manager notifies the resource managers to act—to either commit or rollback changes. During this second phase of the two-phase commit process, the transaction manager continues to try to complete the transaction until all resource managers indicate that the transaction is completed. Using the
AbandonTimeoutSeconds attribute, set the maximum time, in seconds, that a transaction manager persists in attempting to complete a transaction during the second phase of the commit protocol. The default value is 86400 seconds, or 24 hours. After the abandon transaction timer expires, no further attempt is made to resolve the transaction with any resources that are unavailable or unable to acknowledge the transaction outcome. If the transaction is in a prepared state before being abandoned, the transaction manager rolls back the transaction to release any locks held on behalf of the abandoned transaction and writes an heuristic error to the server log.
You may want to review the following related information:
For instructions on how to set the
AbandonTimeoutSeconds attribute, see "Configure JTA" in the Oracle WebLogic Server Administration Console Help.
The first phase of the two-phase commit protocol is called the prepare phase. The required updates are recorded in a transaction log file, and the resource must indicate, through a resource manager, that it is ready to make the changes. Resources either vote to commit the updates or to roll back to the previous state. The second or commit phase is what happens after the resources vote. If all resources vote to commit, all the resources participating in the transaction are updated. If one or more of the resources vote to roll back, then all the resources participating in the transaction are rolled back to their previous state. WebLogic Server provides the following parameters that you can use to tune the amount of time spent processing a transaction.
The maximum amount of time that can be spent processing from the beginning of a transaction until the end of the first phase of a transaction is controlled by setting the value of the
The maximum amount of time that can be spent processing the second phase of a transaction is controlled by setting the value of the
Prior to WebLogic Server 10.3.3, the maximum amount of time spent processing the second phase was approximately twice the default
transaction-timeout value with a maximum value of 120 seconds and not tunable. For the vast majority of environments, the time allotted for completion of the second phase is adequate. However, in environments where high system stress or high network latency can occur, it is possible to exceed the maximum amount of time available to complete the commit phase and the transaction manager throws a
SystemException is non-deterministic relative to transaction outcome so an application environment must provide special exception handling for this case which often involves manually analyzing the transaction activity and state of the resources involved in the transaction. As application stacks become more complex, it becomes more difficult to resolve transacion outcomes. The
completion-timeout-seconds attribute provides the possibility for a successful or deterministic completion in many cases by allowing a longer processing time for the commit phase.
completion-timeout-seconds value exceeds the value set for
completion-timeout-seconds value. If the transaction is abandoned, a
SystemException is thrown. In general, transactions requiring a large values for the
transaction-completion-seconds attribute indicate a need for system tuning.
Note:Please note that if the
abandon-timeout-secondsvalue is set less than 60 seconds, it is voided by the default
completion-timeout-secondssetting. Also, within the first 600 seconds (ten minutes) after the transaction service's startup, the
abandon-timeout-secondssetting becomes fully void.
For configuration information, see:
"Configure advanced domain JTA options" in Oracle WebLogic Server Administration Console Online Help
"CompletionTimeoutSeconds" in Oracle WebLogic Server MBean Reference
Note:The completion-timeout-seconds attribute does not apply to imported transactions such as JCA transactions or to recovering transactions.
The WebLogic Server transaction manager is designed to recover from system crashes with minimal user intervention. The transaction manager makes every effort to resolve transaction branches that are prepared by resource managers with a commit or roll back, even after multiple crashes or crashes during recovery.
To facilitate recovery after a crash, WebLogic Server provides the Transaction Recovery Service, which automatically attempts to recover transactions on system startup. On startup, the Transaction Recovery Service parses all transaction log records for incomplete transactions and completes them as described in Transaction Recovery Service Actions After a Crash.
Because the Transaction Recovery Service is designed to gracefully handle transaction recovery after a crash, Oracle recommends that you attempt to restart a crashed server and allow the Transaction Recovery Service to handle incomplete transactions.
If a server crashes and you do not expect to be able to restart it within a reasonable period of time, you may need to take action. Procedures for recovering transactions after a server failure differ based on your WebLogic Server environment. For a non-clustered server, you can manually move the server (with the default persistent store DAT file) to another system (machine) to recover transactions. See Recovering Transactions For a Failed Non-Clustered Server for more information. For a server in a cluster, you can manually migrate the whole server or the Transaction Recovery Service to another server in the same cluster. Migrating the Transaction Recovery Service involves selecting a server with access to the transaction logs to recover transactions, and then migrating the service using the Administration Console or the WebLogic command-line interface.
Note:For non-clustered servers, you can only move the entire server to a new system. For clustered servers, you can migrate the entire server or temporarily migrate the Transaction Recovery Service.
For more information about migrating the Transaction Recovery Service, see Recovering Transactions For a Failed Clustered Server. For more information about clusters, see Using Clusters for Oracle WebLogic Server.
The following sections provide information on how to recover after a failure:
When you restart a server after a crash or when you migrate the Transaction Recovery Service to another (backup) server, the Transaction Recovery Service does the following:
Complete transactions ready for second phase of two-phase commit
For transactions for which a commit decision has been made but the second phase of the two-phase commit process has not completed (transactions recorded in the transaction log), the Transaction Recovery Service completes the commit process.
Resolve prepared transactions
For transactions that the transaction manager has prepared with a resource manager (transactions in phase one of the two-phase commit process), the Transaction Recovery Service must call
XAResource.recover() during crash recovery for each resource manager and eventually resolve (by calling the
forget() method) all transaction IDs returned by
Report heuristic completions
If a resource manager reports a heuristic exception, the Transaction Recovery Service records the heuristic exception in the server log and calls
forget() if the
Forget Heuristics configuration attribute is enabled. If the
Forget Heuristics configuration attribute is not enabled, refer to your database vendor's documentation for information about resolving heuristic completions. See Handling Heuristic Completions for more information.
The Transaction Recovery Service provides the following benefits:
Maintains consistency across resources
The Transaction Recovery Service handles transaction recovery in a consistent, predictable manner: For a transaction for which a commit decision has been made but is not yet committed before a crash, and
XAResource.recover() returns the transaction ID, the Transaction Recovery Service consistently calls
XAResource.commit(); for a transaction for which a commit decision has not been made before a crash, and
XAResource.recover() returns its transaction ID, the Transaction Recovery Service consistently calls
XAResource.rollback(). With consistent, predictable transaction recovery, a transaction manager crash by itself cannot cause a mixed heuristic completion where some branches are committed and some are rolled back.
Persists in achieving transaction resolution
If a resource manager crashes, the Transaction Recovery Service must eventually call
rollback() for each prepared transaction until it gets a successful return from
rollback(). The attempts to resolve the transaction can be limited by setting the
AbandonTimeoutSeconds configuration attribute. See Abandoning Transactions for more information.
When using Apache with the WebLogic Proxy plug-in as the front-end for a cluster, the plug-in uses several configuration parameters to determine how long to wait for connections to the WebLogic Server host and, after a connection is established, how long the plug-in waits for a response:
Verify the setting of the Apache
idempotent flag. When
idempotent is set to ON, and if the servers do not respond within the specified
WLIOTimeoutSecs value, the plug-ins fail over. The plug-ins also fail over if
idempotent is set to ON and the servers respond with an error such as
READ_ERROR_FROM_SERVER. If set to OFF, the plug-ins do not fail over. For more information, see "Parameters for Web Server Plug-Ins" in Using Web Server 1.1 Plug-Ins with Oracle WebLogic Server.
Verify the setting of WebLogic Proxy Plug-in retry mechanism; for example, whether the maximum number of retries allowed is equal to the
ConnectTimeoutSecs value divided by the
ConnectRetrySecs value. For more information, see "Failover, Cookies, and HTTP Sessions" in Using Web Server 1.1 Plug-Ins with Oracle WebLogic Server.
To recover transactions for a failed server, follow these steps:
Move (or make available) the persistent store DAT file (which contains all transaction log records) from the failed server to a new server.
Set the path for the default persistent store with the path to the data file. See Setting the Path for the Default Persistent Store.
Start the new server. The Transaction Recovery Service searches all transaction log files for incomplete transactions and completes them as described in Transaction Recovery Service Actions After a Crash.
When moving transaction log records after a server failure, make all transaction log records available on the new machine before starting the server there. Otherwise, transactions in the process of being committed at the time of a crash might not be resolved correctly, resulting in application data inconsistencies. Accomplish this by storing persistent store data files on a dual-ported disk available to both machines. As in the case of a planned migration, update the default file store
directory attribute with the new path before starting the server if the pathname is different on the new machine.
Note:The Transaction Recovery Service is designed to gracefully handle transaction recovery after a crash. Oracle recommends that you attempt to restart a crashed server and allow the Transaction Recovery Service to handle incomplete transactions, rather than move the server to a new machine.
When a clustered server fails, you have the following options for recovering transactions:
For clustered servers, WebLogic Server enables you to migrate a failing server to a new machine, including the Transaction Recovery Service. When the server migrates to another machine, it must be able to locate the transaction log records to complete or recover transactions. Transaction log records are stored in the default persistent store for the server. If you plan to migrate clustered servers in the event of a failure, you must set up the default persistent store so that it stores records in a shared storage system that is accessible to any potential machine to which a failed migratable server might be migrated. For highest reliability, use a shared storage solution that is itself highly available—for example, a storage area network (SAN).
For information about server migration, see "Whole Server Migration" in Using Clusters for Oracle WebLogic Server.
For more information about setting default persistent store options, see:
When a clustered server crashes, you can manually migrate the Transaction Recovery Service from the crashed server to another server in the same cluster using the Administration Console or the command-line interface. For instructions to manually migrate the Transaction Recovery Service using the Administration Console, see "Manually migrate the Transaction Recovery Service" in the Oracle WebLogic Server Administration Console Help.
You can also configure WebLogic Server to automatically migrate the Transaction Recovery Service to a healthy candidate server based with the help of WebLogic Server health monitoring of singleton services. See Automatic Transaction Recovery Service Migration.
When manual or automatic service migration takes place, the following events occur:
The Transaction Recovery Service on the backup server takes ownership of the transaction log from the crashed server.
The Transaction Recovery Service searches all transaction log records from the failed server for incomplete transactions and completes them as described in Transaction Recovery Service Actions After a Crash.
If the Transaction Recovery Service on the backup server successfully completes all incomplete transactions from the failed server, the server releases ownership of the Transaction Recovery Service for the failed server so the failed server can reclaim it upon restart.
A server can perform transaction recovery for multiple failed servers. While recovering transactions for other servers, the backup server continues to process and recover its own transactions. If the backup server fails during recovery, you can migrate the Transaction Recovery Service to yet another server, which continues the transaction recovery. You can also manually migrate the Transaction Recovery Service back to the original failed server using the Administration Console or the command-line interface. See Manually Migrating the Transaction Recovery Service to the Original Server for more information.
When a backup server completes transaction recovery for a server, it releases ownership of the Transaction Recovery Service for the failed server. When you restart a failed server, it attempts to reclaim ownership of its Transaction Recovery Service. If a backup server is in the process of recovering transactions when you restart the failed server, the backup server stops recovering transactions, performs some internal cleanup, and releases ownership of the Transaction Recovery service so the failed server can reclaim it and start properly. The failed server then completes its own transaction recovery.
If a backup server still owns the Transaction Recovery Service for a failed server and the backup server is inactive when you attempt to restart the failed server, the failed server does not start because the backup server cannot release ownership of the Transaction Recovery Service. This is also true if the fail back mechanism fails or if the backup server cannot communicate with the Administration Server. You can manually migrate the Transaction Recovery using the Administration Console or the command-line interface.
You can specify to have the Transaction Recovery Service automatically migrated from an unhealthy server instance to a healthy server instance, with the help of the server health monitoring services. This way the backup server can complete transaction work for the failed server. See "Roadmap for Configuring Automatic Migration of the JTA Transaction Recovery Service" in Using Clusters for Oracle WebLogic Server.
Prior to WebLogic Server 10.0, when a cluster's primary Managed Server was booted, but was unable to contact the Administration Server (mostly because that Administration Server had not started yet), then the primary Managed Server would automatically go into MSI (managed server independence) mode and continue to boot up using its local configuration information. During a manual migration of the Transaction Recovery Service, this situation posed a potential risk that a backup server was still recovering TLOG data on behalf of the primary Managed Server, which could then lead to concurrent access to TLOG and potential corruption of the TLOG.
To avoid risking potential TLOG corruption, there is a
strictOwnershipCheck property on the JTAMigratableTargetMBean. This way, when a primary Managed Server attempts to boot up and it finds that it cannot connect to the Administration Server (for the manual JTA migration policy) or the Singleton Master (for the automatic JTA migration policy), then it verifies its independence by checking the value of the
strictOwnershipCheck, as follows:
True – This is the recommended setting. The primary Managed Server throws an exception and fail to boot.
False – The primary Managed Server skips the Transaction Recovery Service failback, then it can boot successfully. This poses the same TLOG corruption risk as in WebLogic Server 9.2 or earlier.
When manually or automatically migrating the Transaction Recovery Service, the following limitations apply:
You cannot migrate the Transaction Recovery Service to a backup server from a server that is running. You must stop the server before migrating the Transactions Recovery Service.
The backup server does not accept new transaction work for the failed server. It only processes incomplete transactions.
The backup server does not process heuristic log files.
The backup server only processes log records written by WebLogic Server. It does not process log records written by gateway implementations, including WebLogic Tuxedo Connector.
In addition to the limitations described above, the following rules also apply when WebLogic Server 10.0 or later is configured to automatically migrate the Transaction Recovery Service:
If the cluster also contains servers from earlier releases of WebLogic Server, the primary server and backup servers must be WebLogic Server 10.0 or later. To enforce this when automatic migration is enabled, on the Administration Console, only WebLogic Server 10.0 or later servers appear in the Candidate Servers Available list.
Manual service migration is supported between release 9.2 or earlier servers and release 10.0 or later servers if no migration scripts are used.
To migrate the Transaction Recovery Service from a failed server in a cluster to another server (backup server) in the same cluster, the backup server must have access to the transaction log records from the failed server. Therefore, you must store default persistent store data files on persistent storage available to all potential backup servers in the cluster. Consider the following:
A storage area network (SAN) is recommended.
If you use an NFS file system, configure the NFS server so that disk writes are not cached.
Regardless of the storage solution, test failover and migration.
With some storage solutions, particularly some versions of NFS, file locking can be an issue. You may need to configure your storage server to handle file locks effectively or you may need to turn off file locking for the WebLogic file store.
Important:NFS storage may not fully protect transactional data, as it may be configured to silently buffer synchronous write requests in volatile memory. If a file store Directory is located on an NFS mount, and the file store's Synchronous Write Policy is anything other than
Disabled, check your NFS implementation and configuration to make sure that it is configured to support synchronous writes. A
Disabledsynchronous write policy does not perform synchronous writes, but, as a consequence, is generally not transactionally safe. You may detect undesirable buffering of synchronous write requests by observing high persistent message or transaction throughput that exceeds the physical capabilities of your storage device. On the NFS server, check the synchronous write setting of the exported NFS directory hosting your File Store. A SAN based file store, or a JDBC store, may provide an easier solution for safe centralized storage. See File Locking and NFS in Performance and Tuning for Oracle WebLogic Server.
The following persistent store rules apply when manually or automatically migrating the Transaction Recovery Service:
The default persistent store cannot be shared by JTA and other migratable services. Other migratable services, such as JMS services, must use another custom store if they are targeted to a migratable target.
If post-deactivation and pre-activation scripts are specified to perform any dismounting and mounting of the default store, then the Node Manager must be configure and running on all candidate machines.
The Administration Server must be available when the primary server starts up, fails over, or fails back. This is required to guarantee that the Transaction Recovery Service gets exclusive ownership to its TLOG correctly and without conflict. When the primary server starts up, the Transaction Recovery Service connects to Administration Server to get the latest information about JTA. And should failover/failback occur, the Transaction Recovery Service saves the latest information to Administration Server.
When migrating the Transaction Recovery Service from a server, you must stop the failing or failed server before actually migrating the Transaction Recovery Service. If the original server is still running, you cannot migrate the Transaction Recovery Service from it.
All servers that participate in the migration must have a listen address specified in their configuration. See "Configure listen addresses" in the Oracle WebLogic Server Administration Console Help.
You may want to limit the choices of the servers to use as a Transaction Recovery Service backup for a server in a cluster. For example, all servers in your cluster may not have access to the transaction log records for a server. You can limit the list of destination servers available on the "Servers: Configuration: Migration" page in the Administration Console. See "Configure candidate servers for Transaction Recovery Service migration" in the Oracle WebLogic Server Administration Console Help for instructions.
Note:You must include the original server in the list of chosen servers so that you can manually migrate the Transaction Recovery Service back to the original server, if need be. The Administration Console enforces this rule.
When you migrate the Transaction Recovery Service to another server in the cluster, the backup server takes ownership of the Transaction Recovery Service until it completes all incomplete transactions. After which, it releases ownership of the Transaction Recovery Service and the original server can reclaim it. You can see the current owner on the "Servers: Control: Migration" page in the Oracle WebLogic Server Administration Console Help. Follow these instructions:
In the Domain Structure tree in the Administration console, expand Environment and click Servers.
Select the original server from which the Transaction Recovery Service was migrated, then select the Control > Migration tab.
Click Advanced. Under JTA Migration Options, Hosting Server indicates the current owner of the Transaction Recovery Service.
After completing transaction recovery for a failed server, a backup server releases ownership of the Transaction Recovery Service so that the original server can reclaim it when the server is restarted. If the backup server stops (crashes) for any reason before it completes transaction recovery, the original server cannot reclaim ownership of the Transaction Recovery Service and does not start.
You can manually migrate the Transaction Recovery Service back to the original server by selecting the original server as the destination server. The backup server must not be running when you migrate the service back to the original server. Follow the instructions below.
Notes:Please note the following:
If a backup server fails before completing the transaction recovery actions, the primary server cannot reclaim ownership of the Transaction Recovery Service and recovery is not re-attempted after the rebooting server. Therefore, you must attempt to manually re-migrate the Transaction Recovery Service to another backup server.
If you restart the original server while the backup server is recovering transactions, the backup server gracefully releases ownership of the Transaction Recovery Service. You do not need to stop the backup server. See Recovering Transactions For a Failed Clustered Server.
For instructions on manually migrating the Transaction Recovery Service using the Administration Console, see "Manually migrate the Transaction Recovery Service" in the Oracle WebLogic Server Administration Console Help.
Before deleting TLOGs the WebLogic Server instance should be shutdown gracefully to allow the completion of as many transactions as possible.
Note:You should delete TLOGs only in an extreme case. Deleting the TLOGs removes transaction records, resulting in heuristic failures. For example, see Moving a Server.
The location of TLOGs is dependent on whether LLR is a participaging resource in a transaction.
When one resource involved in the transaction is a LLR, then the TLOGs are stored in two locations.
The transaction records are stored in a database table. See How to Remove the TLOG in the LLR Database.
The server and resource checkpoints are stored in the default store. See How to Remove the TLOG Files from the Default Store.
If there are no participating LLR in the transactions, the transaction records, server checkpoints, and resource checkpoints are all saved to the TLOG file in the default store. See How to Remove the TLOG Files from the Default Store.
Default name of the LLR table is
SERVERNAME is the name of the server instance. See
JDBC LLR Table Name in "Servers: Configuration: General" in Oracle WebLogic Server Administration Console Help. To delete the LLR TLOG that is kept in the database, remove all the records from the table by issuing
drop table WL_LLR_
To remove the TLOGs in a the default store, delete all files having the following pattern:
where xxxxxx are integers ranging from 0 to 9.
Note:If the default store contains a configured JMS file store, deleting the TLOG also deletes the JMS File Store. In this case, before deleting the TLOG files, first export the JMS messages to another location. You can then safely delete the TLOG files and import the JMS messages back to the original store. See “Managing JMS Messages” in Configuring and Managing JMS for Oracle WebLogic Server.