![]() |
![]() |
![]() |
![]() |
![]() |
•
• From 11.1.1.3.0, Oracle Tuxedo provides many optimizations for Oracle Exalogic platforms; Table 1 lists the Exalogic supported features:
In previous releases, servers in the same participated group use the same transaction branch in a global transaction; if these serves connect to different instances on the same RAC, the transaction branch may fail and an XA error, XAER_AFFINITY, will be reported, meaning one branch cannot go through different instances. For this reason, Tuxedo groups can only use singleton RAC services. A DTP service (if the DTP option, -x in srvctl, is specified) or a service offered by only one instance could be a singleton RAC service.Fast Application Notification (FAN) is a facility provided by Oracle Database to allow database clients to know about changes in the state of the database. These notifications let an application respond proactively to events such as a planned outage of a RAC node or an imbalance in database load. Tuxedo provides support for FAN notifications by a new system server TMFAN that can monitor Oracle RAC instance and notify Tuxedo application server to establish a new Database connection in case of Database instance up or down.Based on FAN notification, Tuxedo TMFAN server can receive load balancing advisories that include the load information of each RAC instance. If the change in advised load exceeds the threshold specified in the TMFAN command line switches, then Tuxedo request will be routed to the Tuxedo application server that has a lower Database load.This section introduces the basic Oracle Tuxedo feature configuration on Exalogic. For more information, see the Oracle Tuxedo 12c Release 2 (12.1.3) Release Notes and Setting Up an Oracle Tuxedo Application.
• There is one attribute for Direct Cross Node Communication Leveraging RDMA in the *RESOURCE section.The IP address where the Msgq_daemon is bound. It must be configured, and must be an IPoIB address (not an Ethernet based IP address). You should configure one Msgq_daemon for one logic machine.The port number where Msgq_daemon listens on. It must be configured.The EMSQ queue entry number, that is the maximum number of messages allowed in this queue. The default value is 64 if not defined in the UBBCONFIG file.The entry number for Oracle Tuxedo EMSQ cache. The function of EXALOGIC_MSGQ_CACHE_SIZE here is the same as that of the environment variable EXALOGIC_MSGQ_CACHE_SIZE; however, the environment variable has the higher priority. The value must be between 32 and 2048 inclusive. EXALOGIC_MSGQ_CACHE_SIZE can be specified in *MACHINES only when RDMA is enabled. The default value is 32 if it is not defined in UBBCONFIG.In some scenarios, Tuxedo performance can be improved when increasing this number. For more details, please see Setting EXALOGIC_MSGQ_CACHE_SIZE.After enable RDMA option in the *RESOURCES section. Attribute "TYPE" of *MACHINES section cannot be set, since by default, any machines in MP mode should be Exalogic machine (with the same type) to support RDMA feature.You can also get/change the configuration via TM_MIB. For more information, see File Formats, Data Descriptions, MIBs, and System Processes Reference.Listing 1 shows an example UBBCONFIG File with Direct Cross Node Communication Leveraging RDMA enabled.The shared memory used by Msgq_Daemon will be locked into physical memory to avoid being paged to swap area, so it is necessary to set a proper value to memlock in /etc/securitylimits.conf.Msgq_daemon shared memory size: The size of shared memory allocated by Msgq_daemon. For more information, see "Calculating Shared Memory Size for Msgq_daemon".MAXACCESSERS: An attribute in the UBBCONFIG file.Msgq_daemon shared memory size: 200*1024 kbThe default name is /u01/common/patches/tuxtmpfile, you can also set your own directory using the EXALOGIC_SHARED_PATH environment variable. It is used for Oracle Tuxedo file transfer. When the EMSQ is full, or the message size exceeds the queue size, Oracle Tuxedo puts this message into a temporary file under the /u01/common/patches/tuxtmpfile directory, and sends notification directly to the remote process queue. The remote process can then get the file as long as it receives the notification.To enable this feature, in RESOURCES section of UBBCONFIG, you must specify BYPASSDOM_ID, BYPASSDOM_SEQ_NUM, and BYPASSDOM_SHARED_DIR parameters, as well as EECS flag of OPTIONS parameter.There is an optional attribute MAXDOMAINS, which specifies the maximum number of domains within one domain group. The default is 32.You can also get or change the configuration via TM_MIB. For more information, see File Formats, Data Descriptions, MIBs, and System Processes Reference.Listing 2 shows a UBBCONFIG file example of enabling Direct Cross Domain Communication Leveraging RDMA.IPCKEY 87654MASTER site1MAXACCESSERS 40MAXSERVERS 40MAXSERVICES 40MODEL SHMOPTIONS EECSLDBAL Yslce04cn01 LMID=site1GROUP1 LMID=site1 GRPNO=1 OPENINFO=NONEsimpserv SRVGRP=GROUP1 SRVID=3For configuration information, see "Self-Tuning Lock Mechanism" in Using Oracle Tuxedo Advanced Performance Pack.To enable Oracle Tuxedo SDP Support, you must specify EECS for OPTIONS in *RESOURCES section, and set the relevant configuration in UBBCONFIG file or DMCONFIG file.You can also get or change the configuration via TM_MIB. For more information, see File Formats, Data Descriptions, MIBs, and System Processes Reference.
•
•
•
• Listing 3 shows a UBBCONFIG file SDP example.Listing 3 UBBCONFIG File SDP Configuration ExampleListing 4 shows a UBBCONFIG file IPoIB configuration example.If the node running GWTDOMAIN has multiple network interfaces (multi-homed) with multiple IP addresses, it is better to use explicit IP address when configuring GWTDOMAIN in DMCONFIG file instead of host name. Typically, every Exalogic node has at least two types of network interface, i.e., IB interface and Ethernet interface, in order to facilitate to demonstrate how to configure GWTDOMAIN, just presume IB interface is bound to IP address IB_IP, and Ethernet interface with IP address ETH_IP.Prefixed "sdp:" to the network address, and the network address must be an IPoIB address, as shown in Listing 5.Prefixed "sdp:" to the network address, and the network address must be an IPoIB address as shown in Listing 6.Prefixed "sdp:" to the network address, and the network address must be an IPoIB address as shown in Listing 9.Prefixed "sdp:" to the network address, and the network address must be an IPoIB address as shown in Listing 11.It is the same as the GWTDOMAIN NWADDR configuration in the DMCONFIG file.
2. Add additional Java Option “-Djava.net.preferIPv4Stack=true” to the java command line to start up WLS server.Only Weblogic Server 12c (12.1.1) and higher can connect to Oracle Tuxedo via SDP. For more information, seeEnable IPv4 for SDP transport, NWAddr attribute for WTC local Tuxedo Domain configuration, and NWAddr attribute for WTC remote Tuxedo Domain configuration.For configuration information, see "Shared Memory Interprocess Communication" in Using Oracle Tuxedo Advanced Performance Pack.For configuration information, see "Partial One Phase Read-Only Optimization for RAC" in Using Oracle Tuxedo Advanced Performance Pack.Please configure the database to support InfiniBand, as described in Configuring SDP Protocol Support for Infiniband Network Communication to the Database Server in the Oracle Database Net Services Administrators Guide.
Note:
• recv_poll
The receiving poll time before going to sleep waiting for data. You can improve the write performance by decreasing the recv_poll value. It is recommended to set it to zero.
• sdp_zcopy_thresh
The threshold for zero copy sending data. If the messages are longer than the sdp_zcopy_thresh value, it causes the user buffer to be pinned and the data sent directly from the original buffer. This results in decreased CPU use. The default value is 64K; it is recommended to set it to zero.For example, you can set the two parameters in “/etc/modprobe.conf” on the server node as follows:options ib_sdp sdp_zcopy_thresh=0 recv_poll=0Besides above, each node had better use distinctive TMIFRSVR repository_file, standard output/error file, AUDITLOG file, and ALOGPFX to have a clear logging system. All applications should be set distinctive names to use the Shared Applications Staging feature better.Listing 13 shows a UBBCONFIG file shared APPDIR example.Listing 13 UBBCONFIG File Shared APPDIRIf SECURITY is set in the UBBCONFIG file, only MP domains with EECS enabled can share a common APPDIR.For configuration information, see "Tightly Coupled Transactions Spanning Domains" in Using Oracle Tuxedo Advanced Performance Pack.For configuration information, see "XA Transaction Affinity" in Using Oracle Tuxedo Advanced Performance Pack.For configuration information, see "Single Group Multiple Branches (SGMB)" in Using Oracle Tuxedo Advanced Performance Pack.For configuration information, see "Failover/Failback across Database Instances" in Using Oracle Tuxedo Advanced Performance Pack.For configuration information, see "Load Balancing across RAC Instances" in Using Oracle Tuxedo Advanced Performance Pack.For configuration information, see "Concurrent Global Transaction Table Lock" in Using Oracle Tuxedo Advanced Performance Pack.The feature provides the ability for client to directly access remote server, it eliminates the bottleneck on BRIDGE. When Tuxedo is under high concurrent remote access in MP mode, the throughput will have significant improvement if this feature is enabled in UBBCONFIG.The client connects with the remote server through BRIDGE, and work on it for a relatively short duration. For example, tpinit() followed several tpcall(), then tpterm(). The overhead of creating/opening/closing for RDMA connection is much higher than it on Unix IPC queue. So it cannot have obvious performance improvement under this scenario.Each Oracle Tuxedo thread has an EMSQ runtime cache; the default entry number is 32. You can change it between 32 and 2048 using the environment EXALOGIC_MSGQ_CACHE_SIZE variable before the Oracle Tuxedo application starts, or setting it in UBBCONFIG. In some scenarios, increasing the number can improve Oracle Tuxedo performance, for example:
• If there are 40 remote Oracle Tuxedo servers providing the same service and clients call the service repeatedly, setting EXALOGIC_MSGQ_CACHE_SIZE to a value equal to or greater than 40 on the client can improve performance.
• If there are 50 WSHs, and each WSH receives response messages from the same remote server, setting EXALOGIC_MSGQ_CACHE_SIZE to a value equal to or greater than 50 on the server environment can improve performance.Listing 14 UBBCONFIG File *MACHINES SectionListing 15 tmloadcf -c ubb Output ExampleRCDMSZ increases linearly when any of following items configured in UBB increases:
• The number of logic machines in the UBBCONFIG fileAfter getting the RCDMSZ by tmloadcf, you can adjust the actual size according to the following runtime factors:
• EXALOGIC_MSGQ_CACHE_SIZE: Specify this value in UBBCONFIG file or environment variable, the default value is 32.
• MAXSERVERS: A UBBCONFIG file attribute.
• RDMAQSIZE: A UBBCONFIG file attribute.For example, if MAXSERVERS is 100, RDMAQSIZE is 100000 and processes on machine ex04 start/stop frequently, you should configure the shared memory size with at most 1300M (340M+960M) instead of 340M.The number of machines in the UBBCONFIG file is considered in calculating RCDMSZ. If new machines are added using MIB, you should increase the size linearly.In some scenarios, Msgq_daemon needs to allocate more shared memory when increasing queue cache entry number using EXALOGIC_MSGQ_CACHE_SIZE. For example, if Oracle Tuxedo process start/stop or tpinit/tpterm frequently. Then you should consider adjusting the initial Msgq_daemon shared memory size, otherwise the message transfer may fail.The default value is 32 for each Oracle Tuxedo thread, and tmloadcf -c always assumes the value of 32. If you change the default value, adjust the Msgq_daemon shared memory size using the following method:
Note: You can adjust the size using this method only when setting EXALOGIC_MSGQ_CACHE_SIZE at the machine level.
Note: For best practices, see "Self-Tuning Lock Mechanism" in Using Oracle Tuxedo Advanced Performance Pack.For best practices, see "Shared Memory Interprocess Communication" in Using Oracle Tuxedo Advanced Performance Pack.For best practices, see "Partial One Phase Read-Only Optimization for RAC" in Using Oracle Tuxedo Advanced Performance Pack.For best practices, see "Failover/Failback across Database Instances" in Using Oracle Tuxedo Advanced Performance Pack.For best practices, see "Load Balancing across RAC Instances" in Using Oracle Tuxedo Advanced Performance Pack.For best practices, see "Single Group Multiple Branches (SGMB)" in Using Oracle Tuxedo Advanced Performance Pack.The feature provides the ability for client to directly access remote service across domain; it eliminates the bottleneck on GWTDOMAIN. When Tuxedo is under high concurrent remote access in different domains, this feature significantly improves throughput's performance.The client accesses the remote service in remote domains, and work on it for a relatively short time. For example, tpinit() followed several tpcall(), then tpterm(). The overhead of creating/opening/closing for RDMA connection is much higher than the overhead on Unix IPC queue. Consequently, this feature cannot bring about obvious performance improvement in this scenario.There is a difference running Oracle Tuxedo on a non-Exalogic platform if Direct Cross Node Communication Leveraging RDMA is enabled. The tux_msgq_monitor must be started before booting an Oracle Tuxedo application. This section includes the following topics:tux_msgq_monitor functions as the Msgq_daemon default monitor. It is responsible for starting Msgq_daemon and checking it is running status. When Msgg_daemon terminates abnormally, it restarts using tux_msgq_monitor.tux_msgq_monitor should be started before booting the Oracle Tuxedo application. One monitor can only serve one Oracle Tuxedo application on one logical machine.Before starting tux_msgq_monitor, ensure the environment variables TUXCONFIG, LD_LIBRARY_PATH and TUXDIR are properly set. Run tux_msgq_monitor as follows:tux_msgq_monitor: Options lists tux_msgq_monitor options.
Table 2 tux_msgq_monitor: Options Shared memory size (in bytes) allocated by Msgq_daemon, you can get it by tmloadcf -c. For more information, see Calculating Shared Memory Size for Msgq_daemon. Shared memory key used by Msgq_daemon. -K must be great than 0; its maximum value depends on the definition of key_t by OS.When tux_msgq_monitor is running, there will be two active processes named as tux_msgq_monitor and Msgq_daemon.After starting tux_msgq_monitor successfully, you can boot the Oracle Tuxedo application.There are some shell scripts that simplify the startup/shutdown procedure. Using these tools, you can only run one command to start/stop both tux_msgq_monitor and an Oracle Tuxedo application. Before running these commands, ensure the environment variables TUXCONFIG, LD_LIBRARY_PATH and APPDIR are set properly.tmboot.sh -i daemon_ip -d daemon_port -M shm_size -K shm_key [-l nlsaddr]This script starts up tux_msgq_monitor, executes tmboot to start the Oracle Tuxedo application, and starts tlisten if option "-l" specified.This script starts the tux_msgq_monitor and tlisten.This script terminates tlisten and tux_msgq_monitor.
• tmlisten_start.sh on all the slave nodes
• tmboot.sh on the master node
• tmshut.sh on the master node
• tmlisten_stop.sh on all the slave nodesThere are no special requirements if you do not use any Exalogic optimization. For more information, see Oracle Tuxedo Interoperability Guide and Upgrading the Oracle Tuxedo System to 12c Release 1 (12.1.1).
Note: If any Exalogic optimization is specified in OPTIONS, you cannot perform a hot upgrade from any previous release.
• Oracle Home: An Oracle Home directory serves as the central support directory for all Oracle products installed on the target system.
• Compute Node: A physical machine in an Exalogic rack that is meant for running an Oracle Fusion Middleware instance at any given time.
• Shared Storage: Shared storage refers to the Sun ZFS Storage 7320 appliance that is accessible by all compute nodes in the Oracle Exalogic Machine. All compute nodes in the Exalogic machine can access this storage appliance simultaneously for both read and write operations.
• UBBCONFIG: Each Oracle Tuxedo domain is controlled by a configuration file in which installation-dependent parameters are defined. The text version of the configuration file is referred to as UBBCONFIG. The UBBCONFIG file for a Oracle Tuxedo domain contains all the information necessary to boot the application.
• TUXCONFIG: The TUXCONFIG file is a binary version of the UBBCONFIG file. It is created by running the tmloadcf(1) command, which parses UBBCONFIG and loads the binary TUXCONFIG file to the location referenced by the TUXCONFIG environment variable.
• XA: The eXtended Architecture (XA) is the standard-defined interface between the Transaction Manager and the Resource Manager. XA allows program control of Resource Managers that are involved in distributed transactions
• InfiniBand: It is a switched fabric communications link used in high-performance computing and enterprise data centers. Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable.
• EOIB: Ethernet over InfiniBand.