3 Using the WebLogic Replicated Store for WebLogic Server Messaging Services
Note:
The WebLogic Replicated Store, which is intended only for use in Oracle Exalogic Elastic Cloud environments, is deprecated as of Oracle WebLogic Server version 12.2.1.3.0 and will be removed in a future release. WebLogic Server components that enable the configuration and management of the WebLogic Replicated Store, such as the DomainMBean.ReplicatedStores
attribute and the weblogic.management.configuration.ReplicatedStoreMBean
API, are also deprecated and will be removed. As of Oracle WebLogic Server 12.2.1.3.0, Oracle recommends that you use either a JDBC store or a custom file store for JMS message storage.
Note also that the WebLogic Replicated Store is supported only on Oracle Exalogic Elastic Cloud environments hosted on Oracle Enterprise Linux systems. The WebLogic Replicated Store is not supported in virtual Exalogic environments. See Oracle Fusion Middleware Supported System Configurations for details about Oracle Enterprise Linux versions supported on Exalogic systems.
Overview of the Replicated Store
WebLogic Messaging Services (JMS) can use Replicated Stores as a high performance alternative to existing File and JDBC storage options. A Replicated Store stores data in local Exalogic node memory and replicates it to memory on a second node providing high availability with no single point of failure while yielding linearly scalable performance.
WebLogic Server Messaging services use a WebLogic Replicated Store to persist data to a Daemon Cluster using configuration information stored in a shared Global Directory. The message state is stored locally but can be recovered from any node that hosts the same Daemon Cluster by running the store instance on that node. A Replicated Store is analogous to a File or JDBC Store where a Region in a Daemon Cluster corresponds to a File Store file or JDBC Store table.
Note:
For more information about File and JDBC stores, see The WebLogic Persistent Store in Administering the WebLogic Persistent Store.
Table 3-1 defines the WebLogic services that can create connections to a Replicated Store. Each subsystem that uses the Replicated Store specifies a unique connection ID that identifies that subsystem.
Table 3-1 Replicated Store Users
Subsystem/Service | What It Stores | More Information |
---|---|---|
JMS Servers and SAF Agents |
Persistent messages and durable subscribers. |
Understanding the Messaging Models in Developing JMS Applications for Oracle WebLogic Server |
Path Service |
The mapping of a group of messages to a messaging resource. |
Using the WebLogic Path Service in Administering JMS Resources for Oracle WebLogic Server |
Store-and-Forward (SAF) Service Agents |
Messages for a sending SAF agent for retransmission to a receiving SAF agent |
Understanding the Store-and-Forward Service in Administering the Store-and-Forward Service for Oracle WebLogic Server. |
For more information about the store connection IDs, see Monitoring Store Connections.
Features of the Replicated Store
The Replicated Store leverages Oracle Exalogic Elastic Cloud hardware and software which provides a unique combination of redundant hardware, large physical memory, high bandwidth InfiniBand networking, and efficient Remote Direct Memory Access (RDMA).
Scalability and High Availability of Replicated Stores
A Replicated Store provides superior linear scalability while providing failure resilience that is much higher than non-persistent messaging.
Table 3-2 summarizes the relative performance and high availability provided by available data storage types in Oracle Exalogic Elastic Cloud environments.
Table 3-2 Comparison of Replicated Store Performance and High-Availability Level for Data Storage
Store Type | Storage | Performance | High Availability Level |
---|---|---|---|
None |
non-persistent |
Fastest |
Single point of failure. |
Replicated Store |
Data stored in replicated memory Regions of a Daemon Cluster. |
Second Fastest |
No single point of failure. Simultaneous failure of two nodes or processes required to cause data loss |
File Store |
Data stored in a file on the file system. |
Third Fastest |
No single point of failure when configured to use the Oracle ZFS Storage Appliance. |
JDBC Store |
Data stored in a JDBC table. |
Slowest |
No single point of failure when configured with an Oracle Exadata Database Machine. Optional ability to configure multi-site disaster recovery with Oracle Data Guard. |
Server and Service Migration for Replicated Stores
Replicated Store configurations support server and service level migration as described in the following sections.
Whole Server Migration for Replicated Stores
For higher availability, the WebLogic Replicated Store instance can be migrated along with its parent server as part of the Whole Server Migration (WSM) feature, which provides both automatic and manual migration at the server level, rather than on the service level. WSM automatically restarts or migrates failed WebLogic Server Replicated Store instances. When a JMS Server instance or Replicated Store instance fails, a Replicated Store instance can recover its particular Regions by restarting on any machine that hosts a running RS Daemon in its Daemon Cluster. For more information, see Whole Server Migration in Administering Clusters for Oracle WebLogic Server.
Service Level Migration for Replicated Stores
A WebLogic Replicated Store instance can also be migrated as part of Automatic Service Migration (ASM) for JMS-related services, such JMS servers, SAF agents, and the path service, which rely on stores to maintain data. Service-level migration is controlled by a migratable target, which serves as a grouping of JMS-related services, and which is hosted on only one physical server in a cluster. Such hosted services can be automatically migrated from the current unhealthy hosting server to a healthy active server with the help of the Health Monitoring subsystem. JMS services hosted by a migratable target can also be manually migrated, either in response to a server failure or as part of regularly scheduled server maintenance. When the migratable target is migrated, all pinned services hosted by that target are also migrated. ASM automatically restarts or migrates failed WebLogic Server Replicated Store instances. When a JMS Server instance or Replicated Store instance fails, a Replicated Store instance can recover its particular Regions by restarting on any machine that hosts a running RS Daemon in its Daemon Cluster. For more information on service-level migration, see Service Migration in Administering Clusters for Oracle WebLogic Server.
Note:
As a best practice, a path service should use its own Replicated Store and migratable target.
Administering a Replicated Store
The following sections provide information on how to administer the a Replicated Store:
Quick Start Guide to Use a Replicated Store
Use the following procedures start and stop a Replicated Store:
Shutting Down a Replicated Store
Use the following steps to stop a Replicated Store:
- Shutdown the associated WebLogic cluster. To temporarily stop using a store, you can disconnect from a Daemon Cluster without shutting down to WebLogic Server by untargeting the store from its server instance, dynamic cluster, or migratable target.
- Optionally, shutdown each Daemon in the Daemon Cluster by running the
stopRSDaemon.sh
script. If you choose not to stop the Daemons, the store data will remain available for later recovery by restarted stores. See Shutting Down Daemons using the stopRSDaemon.sh Script.
Administering a Global Directory
A Global Directory is shared directory requiring a custom tuned NFS mount on the host Exalogic machine's ZFS Storage Appliance for a Daemon Cluster. A WebLogic Replicated Store instance and the administration utility reference a Global Directory to access its associated Daemon Cluster.
Table 3-3 describes the structure of a Global Directory.
Table 3-3 Components of a Global Directory
File Name | Description |
---|---|
|
An administrator created configuration file. All Daemons in the same Daemon Cluster and all the clients for these Daemons share the same
|
|
Generated Daemon log files. See Logging Daemon Information. |
|
Internal runtime files. |
|
Lock files that the Daemons use to protect a region. Lock files normally exist only when a region is open. |
|
Lock file that WebLogic Replicated Store configuration uses to protect a region. This lock file continues to exist even after the region is closed. It can only be deleted after a WebLogic Replicated Store configuration is deleted. |
Configuration and Tuning Requirements for a Global Directory
The following section outlines configuration and tuning requirements for administrators creating Global Directories for associated Daemon Clusters:
-
Create a Global Directory using the
startRSDaemon.sh
script. See Starting and Stopping a Daemon Cluster. -
There is a one-to-one correspondence between Daemon Clusters and Global Directories. Different Daemon Clusters cannot share the same Global Directory.
-
A Global Directory is a tuned NFS mount that must be located on an Exalogic machine's ZFS Storage Appliance and must be centrally accessible by all Exalogic nodes that host components of an associated Daemon Cluster.
-
To ensure stability, the NFS mount requires the following tuning:
-
Set
/etc/fstab "actimeo=0"
on each Exalogic node. This significantly impacts file operation performance so Oracle recommends restricting the use a Global Directory NFS mount to a Replicated Store. -
Always use NFS v4 for all NFS clients and servers. See Configuring NFS Version 4 (NFSv4) on Exalogic in Oracle Exalogic Elastic Cloud Machine Owner's Guide.
-
A best practice is to minimize file activity in the ZFSA NFS volume hosting the Global Directory. For instance, if Daemon verbose tracing is required, use a volume other than the ZFSA NFS volume that hosts the Global Directory. See the
-logdir
parameter at Starting and Stopping a Daemon Cluster.
-
Administering a WebLogic Replicated Store
A WebLogic Server Replicated Store instance runs on a WebLogic Server, cluster, or migratable target and acts as a client to persist data to Regions in a Daemon Cluster. A particular instance can only attach to a Daemon that is running on the same node and shares the same Global Directory. Once attached, an instance creates and /or opens a set of Regions in a Daemon Cluster that are owned by (and uniquely named for) that instance. Lock files in the Global Directory ensure instances do not share Regions.
How to Create and Configure a WebLogic Replicated Store
For more information on how to create and configure a WebLogic Replicated Store, see Create replicated stores in Oracle WebLogic Server Administration Console Online Help.
Monitoring a Replicated Store
You can monitor statistics for each existing Replicated Store and for each open store connection.
Monitoring Stores
Each Replicated Store is represented at run time by an instance of the PersistentStoreRuntimeMBean, which provides the following options.
Table 3-4 Replicated Store Run-time Options
Option | What It Does |
---|---|
CreateCount |
Number of create requests issued to this store. |
ReadCount |
Number of read requests issued to this store. |
UpdateCount |
Number of update requests issued by this store. |
DeleteCount |
Number of delete requests issued by this store. |
ObjectCount |
Number of objects contained in the store. |
Connections |
Number of active connections in the store. |
PhysicalWriteCount |
Number of times the store flushes its data to durable storage. |
Monitoring Store Connections
For each open Replicated Store connection, a PersistentStoreConnectionRuntimemMBean is registered, which provides the following options.
Table 3-5 Replicated Store Connection Runtime Options
Option | What It Does |
---|---|
CreateCount |
Number of create requests issued to this connection. |
ReadCount |
Number of read requests issued to this connection. |
UpdateCount |
Number of update requests issued by this connection. |
DeleteCount |
Number of delete requests issued by this connection. |
ObjectCount |
Number of objects contained in the connection. |
Table 3-6 defines most of the run-time prefix names of the WebLogic services and subsystems that can create a connection to the Replicated Store.
Table 3-6 Replicated Store Run-Time Prefix Names
Subsystem/Service | Run-Time Prefix Name |
---|---|
JMS Service |
JMS server:
where JMS durable subscriber:
where |
Path Service |
where |
SAF Service |
SAF agent
where SAF durable subscriber:
where |
Best Practices and Considerations for WebLogic Replicated Stores
The following section provides information on best practices and other considerations when configuring WebLogic Replicated Store instances:
-
If no Daemon is available on the same node as the instance, the Replicated Store fails to start and logs an error. For configuration options that cause a failing Replicated Store to automatically restart or migrate, see Server and Service Migration for Replicated Stores.
-
If there are multiple Exalogic instances in a single node, they should connect to a single Daemon on each node. This promotes high availability by ensuring that each Daemon replicates its state to a Daemon that is running on a different node.
-
If a non-production environment configures multiple Daemons on a single node, the
Local Index
attribute is used to determine the attachment. See LocalIndex in MBean Reference for Oracle WebLogic Server. -
The size of a region is configurable using the
RegionSize
parameter, see Replicated Store: Configuration in Oracle WebLogic Server Administration Console Online Help. The size of a Region cannot be changed once it's created; changing theRegionSize
affects any new regions that an instance may create after the change is implemented. -
The region size must be larger than the largest single message, batch of messages, or SAF Window size processed when sending and receiving large messages. Additionally, you may need to consider setting the
MaxMessageSize
as described in Setting Maximum Message Size for Network Protocols in Tuning Performance of Oracle WebLogic Server. -
Choose a region size that allows several regions to be created for each store instance that might share the same Daemon memory. For example: If a store instance will host up to 512 MB of data, choose a region size smaller than 50MB.
-
Replicated Store instances recycle the space that they use within their regions. When data in a given memory location of a Region is no longer needed, the memory location is made available to store new data. A new region is created only when existing regions are too full to accommodate new data (for example, a growing backlog of unprocessed JMS messages).
-
A region is not deleted and its backing machine memory is not freed until either:
-
The entire Daemon Cluster is shut down or
-
The WebLogic Replicated Store instance that references the region is shutdown and the region is administratively deleted. See Administering a Daemon Cluster using the Administration Utility.
-
Administering a Daemon Cluster
The following sections provide information on how to administer a Daemon Cluster:
Configuring a Daemon
All Daemons in the same Daemon Cluster and all of the clients for these Daemons share the same rs_daemons.cfg
file which is located at the root of their Global Directory.
Creating a rs_daemons.cfg File
The rs_daemons.cfg
is a simple text file created by an administrator that contains a single entry for each Daemon in a Daemon Cluster. It may include optional blank lines and optional comments that are prefixed with a "#
". Each line entry specifies an address, port, shared memory key, and optional memory limit using the following format where each value is separated by one or more blank spaces:
address port starting-shared-memory-key shared-memory-limit
-
Address and Port—Each entry must specify a unique combination of address and port. No two entries can have both the same address and the same port. The address can be a name or a numeric IP but must correspond to an InfiniBand address on the Daemon's node. The port should be in the range 1024-49151.
Note:
It may be possible to configure port numbers higher than 49151 with additional OS tuning.
-
Shared Memory Key—A dynamically generated key used to address the unique location of each Region's primary or secondary shared memory. Daemons that run on the same node must have different shared memory keys. If there are conflicts with the Shared Memory Key, the Daemon will not start. See Daemon Shared Memory Keys.
-
Shared Memory Limit—Daemon's use shared memory to store Region data. A Daemon's shared memory limit is specified as an integer qualified by an "M" or "MB" for megabytes or a "G" or "GB" for gigabytes. Choose a memory limit that allows a Daemon to easily accommodate the predicted number of store instance primary and secondary Regions after a node fails. For example, in a three node cluster, each node should be able to accommodate a 50 percent surge in memory usage in case one node fails. See Managing Memory Utilization for Replicated Stores.
Example 3-1 provides an example contents for a rs_daemon.cfg
file.
Example 3-1 Example Daemon Configuration in a rs_daemon.cfg File
#ipaddress portno shmkey memlmt 123.456.78.9 4545 4545 2G 123.456.78.6 4546 4545 2G
A Daemon's number is automatically determined by its relative file location in the rs_daemons.cfg
file (comments and blank lines are excluded). Daemon numbering starts with zero so the first entry in the file corresponds with Daemon number 0 and every additional line increases the daemon number by 1. This number is used to identify a daemon for logging and runtime administration purposes.
Editing a rs_daemons.cfg File
Oracle recommends that a rs_daemons.cfg
file should not be modified if any Daemon in the Daemon Cluster is running. New clients and Daemons will fail to start if they attempt to connect to an existing Daemon Cluster after changes have been implemented. Existing running Daemons and clients continue to run using the old configuration.
Starting and Stopping a Daemon Cluster
Oracle provides a startRSDaemon.sh
and stopRSDaemon.sh
scripts to manage a Daemon Cluster. These scripts are located in the WL_HOME/server/bin
directory of your Weblogic Server installation.
Starting Daemons using the startRSDaemon.sh Script
The startRSDaemon.sh
script is used to start a Daemon using the information in the rs_daemons.cfg
file located in the specified Global Directory and set the associated logging configuration.
Table 3-7 Configuration Parameters for the startRSDaemon.sh Script
Parameter | Description |
---|---|
|
path specifies the location of the Replicated Store Global Directory that exactly matches the shared directory specified for a WebLogic Replicated Store in the WebLogic Server
This directory must be located on a specially tuned NFS mount, see Configuration and Tuning Requirements for a Global Directory. Oracle recommends using absolute paths when specifying the location of the Replicated Store Global Directory. |
|
|
|
Default value is 2. |
|
When specified as a relative path, the directory is created relative to the GlobalDir Specify an absolute directory path to save the log files to a location other than the GlobalDir Default value is "." |
|
Default value is 500. |
|
Maximum number of log files. Default value is 10. |
|
The help for this script. |
Shutting Down Daemons using the stopRSDaemon.sh Script
The stopRSDaemon.sh
script is used to stop a Daemon using the information in the rs_daemons.cfg
file located in the specified Global Directory.
Table 3-8 Configuration Parameters for the stopRSDaemon.sh Script
Parameter | Description |
---|---|
|
path specifies the location of the Replicated Store Global Directory that exactly matches the shared directory specified for a WebLogic Replicated Store in the WebLogic Server
This directory must be located on a specially tuned NFS mount, see Configuration and Tuning Requirements for a Global Directory. Oracle recommends using absolute paths when specifying the location of the Replicated Store Global Directory. |
|
Default value is 0. |
|
Where:
Default value is |
|
The help for this script. |
Logging Daemon Information
Daemons report initial bootstrapping information to the WebLogic Server Administration Console (stdout
) and generate one or more unique log/trace files in the root of the RS Global Directory. The log file contains any information that has been reported to the WebLogic Server Administration Console as well as runtime log messages. A Daemon's log file is named using the following pattern:
rs_daemon_dnum_xxx.xxx.xxx.xxx_ppppp_nnn.log
where:
-
dnum specifies the three digit, zero filled value of the Daemon number in the
rs_daemons.cfg
file. See Creating a rs_daemons.cfg File. -
xxx.xxx.xxx.xxx specifies the IP address.
-
ppppp specifies the port number.
-
nnn specifies the log file number.
The log configuration (log file location, maximum file size, and maximum number of log files) is controlled by parameters in the startRSDaemon.sh
script, see Starting Daemons using the startRSDaemon.sh Script. Log files are rotated so that old messages are moved to another file when the current log file reaches a specific size determined by the logfilesize
parameter. The newest log file has the number 000 and the oldest possible log file would have the value of logfilemax - 1
. If nnn for a log file is equal to logfilemax
, it is deleted.
Administering a Daemon Cluster using the Administration Utility
The administration utility enables administrators to troubleshoot a Daemon cluster, including commands to manage running Daemons and associated Regions. This utility can be run from a Java command line as shown in Examples Demonstrating the Administration Utility.
Note:
Replicated Store commands are not supported using WLST.
To administer a Daemon Cluster, you must first attach to a local Daemon of the Daemon Cluster using the rsattach
command. The most common use cases for the utility are to monitor memory utilization across the Daemon Cluster, delete unused Regions, and to shutdown Daemons. When finished, use the rsdetach
command to disconnect from the Daemon Cluster.
Table 3-9 defines the available Replicated Store administration commands.
Table 3-9 Replicated Store Administration Options
Java Command | Parameters | What It Does |
---|---|---|
|
If no parameter is specified, displays all available Replicated Store commands, usage, and examples. Where: parameter is the name of a command. If the name of a command is specified, additional help specific to the named command is presented. The administration utility does not need to be attached to a Daemon to use this command. |
|
|
|
Attaches administration utility to a Daemon. Prompt changes back to Where:
|
|
Where:
|
|
|
Disconnect from a Daemon Cluster. Prompt changes back to |
|
|
|
Monitors Daemons. Where:
|
[-sort name|time|size] |
Sorts list or table output. Where:
|
|
|
Format of the output. Where:
|
|
|
|
Shuts down Daemons. The administration utility automatically detaches from a shutdown Daemon. Where:
|
|
Where:
|
|
|
|
Lists the Regions of a Daemon Cluster. Where:
|
|
Sorts list or table output. Where:
|
|
|
Format of the output. Where:
|
|
|
Where |
|
|
|
Deletes all copies of specified Regions, including primary and secondary copies, unless a Region is currently opened by a Replicated Store. If required, you can use the Where:
|
|
Ends the administration session. |
Examples Demonstrating the Administration Utility
Before you start, set your shell environment by running a script such as $WL_HOME/server/bin/setWLEnv.sh
.
To open the administration utility from a Java command line, type: java weblogic.store.Admin. For example:
> java weblogic.store.Admin
> storeadmin->
Accessing rshelp for Replicated Stores
Type rshelp
for detailed descriptions on available Replicated Store administration commands, as well as examples of typical command usage. For example, the following comprehensive help is provided for the rsdetach
command, which is used to release the utility from the Daemon.
storeadmin->rshelp rsdetach Command: rsdetach Summary: detach from a RS Daemon Cluster Usage: rsdetach Description: Use "rsdetach" to detach from a RS Daemon Cluster. When an rsdetach succeeds, the command prompt will change so that it no longer includes '[RS]'. . . .
Attaching to a Daemon
This example attaches the administration utility to a Daemon.
storeadmin:-> rsattach INFO: Attached to Daemon 3 with global dir [.]. storeadmin:[RS]-> . . .
Example Table Output from the lsd Command
The following section provides example of table output from an lsd
command.
Current Time: 2014-02-18 11:50:29 AM EST Idx IP Port Up Rgns Rgns Rgns Rgns Mem Mem Address Time Closed Open Open Total Used Used HHHHH:MM:SS Prima Rplca MB % --- --------------- ------ ----------- ------ ------ ------ ------ ------- ----- 000 192.168.0.127 8000 0:27:28 0 0 2 2 257 3.14 001 192.168.0.128 8000 0:27:27 0 2 0 2 257 3.14
Example List Output from the lsd Command
The following section provides example of table output from an lsd
command.
Index : 001 Status : UP Reachable over IB : TRUE IP Address : 192.168.0.128 Port : 8000 Shared Memory Key (hex) : 0x1f40 Shared Memory Key (decimal) : 8000 Startup Time (YYYY-MM-DD HH:MM:ss) : 2014-02-18 11:23:02 AM EST Current Time (YYYY-MM-DD HH:MM:ss) : 2014-02-18 11:55:40 AM EST Up Time (HHHH:MM:ss) : 00019:32:37 Startup Time (micros since epoch) : 1392740582473595 Current Time (micros since epoch) : 1392742540074899 Up Time (micros) : 1957601304 Total Configured Memory (MB) : 8192 Current Memory Usage (MB) : 257 Number of Regions Closed : 0 Number of Regions Opened as Primary : 2 Number of Regions Opened as Replica : 0 Total Number of Regions Managed : 2 Number of Resilvers in Progress : 0 Number of Daemons in Cluster : 2
Understanding Daemon Clusters
A Daemon Cluster is replicated memory storage that spans multiple Exalogic nodes. This storage is organized as a set of uniquely named Regions and managed by one or more Daemons.
A WebLogic Replicated Store persists data to a Daemon Cluster, analogous to a File or JDBC Store where a Region in a Daemon Cluster corresponds to a File Store file or JDBC Store table. Once a WebLogic Replicated Store attaches to a Daemon, subsequent communication uses shared memory and InfiniBand RDMA.
Region Uniqueness
A WebLogic Server Replicated Store instance can open one or more Regions in a Daemon Cluster. Similar to File Store files and JDBC Store tables, a Region can only be safely accessed by a single client at a time. To ensure Region integrity, Oracle uses lock files and configuration checks where ever possible to prevent more than one client from accessing a Region at a given time.
Note:
If more than one WebLogic domain is configured to use the same Daemon Cluster, you must ensure that the WebLogic domains are unique. Otherwise, same-named WebLogic Replicates Stores between the domains will clash attempting to open the same Region resulting in lock errors in the WebLogic Server and Daemon logs.
Daemon High Availability
A Daemon Cluster uses resilvering, the process of creating a new synchronized copy of a Region from an existing Region, to ensure a Region has two copies of each Region on separate Daemons.
When a WebLogic Replicated Store opens a new Region, its local Daemon is responsible for maintaining the Region's primary copy and the next available Daemon maintains the Region's secondary copy. The next available Daemon is determined by:
-
Finding the primary's nearest subsequently defined Daemon in the
rs_daemons.cfg
file that is up and running, or -
If the primary happens to be the last Daemon in the file, then start at the top of the
rs_daemons.cfg
file and look for the next subsequently defined Daemon.
When a Replicated Store opens an existing Region that has no pre-existing copy on the store's local Daemon but already has copies elsewhere in the cluster, then the open succeeds and one of the existing copies is transparently resilvered to the local Daemon as part of the Region open. In addition, the Region's secondary is resilvered to the next available Daemon. The resilvering of the secondary helps ensure that Regions stay evenly distributed throughout a Daemon Cluster.
Note:
A Region can only be opened by a single client at a time. WebLogic Replicated Stores are expected to restart and/or migrate to a new node in order to open and recover their Regions after failing.
What Happens When a Daemon Fails?
If a Daemon fails, any attached client (for example, a Weblogic Replicated Store) also fails. Clients can recover their particular Region data by periodically trying to reattach to the failed Daemon or by trying to attach to a different Daemon in the same Daemon Cluster. For information on automating periodic retries, see Server and Service Migration for Replicated Stores.
The other Daemons in the cluster detect the failure, and each of the failed Daemon's primary and secondary Region copies are automatically resilvered to another Daemon (provided there's a Daemon with enough memory available). Resilvering occurs even if the WebLogic Replicated Store does not restart and reattach somewhere else in the cluster.
Note:
If resilvering is in progress and if a Daemon Cluster is down to a single Daemon, or if a Daemon Cluster doesn't have enough memory to host two copies of each region, then Regions are vulnerable to a single point of failure.
Additional Considerations
-
An administrator can restart a Daemon in place and it will then rejoin the Daemon Cluster.
-
A Daemon can be administratively shutdown safely using administration utility (see Administering a Daemon Cluster using the Administration Utility) or Daemon shutdown script (see Shutting Down a Replicated Store). By default, a shutdown resilvers all of a Daemon's Regions (primaries and secondaries) and blocks until the resilvering completes prior to shutting down the Daemon.
-
If a Region is resilvering a primary to a new secondary and the secondary's host Daemon is killed, then resilvering transparently and asynchronously starts over with a new secondary on another Daemon. There must be at least two Daemons still running in the cluster and there is another Daemon with sufficient free memory to host the Region copy.
Daemon Shared Memory Keys
Daemons maintain shared memory to store Region data. Each new Region (primary or secondary) has its own unique location that is internally addressed using a dynamically generated private Shared Memory Key. This memory is pinned to prevent the O/S from paging the information to disk. This shared memory is only freed when:
-
A Daemon crashes, is killed, or is administratively shutdown.
-
A Region is administratively deleted.
An administrator configures a candidate public Shared Memory Key for each Daemon in the rs_daemons.cfg
file (one public key shared memory location per Daemon). For example, if a Daemon hosts 5 Region primaries and 4 Region secondaries, then it reserves a total of 9 dynamically generated private shared memory keys and one public key.
Using a Local Index in Development Environments
A local index determines which particular local Daemon to start when more than one Daemon is configured to run on the current node. The following formula determines which Daemon is picked:
((idx) modulo (number-of-local-daemons))
Where idx is an entry in the rs_daemons.cfg
file. By default (idx=0), this resolves to the first entry in the rs_daemons.cfg
file with the same network address as the current node.
Note:
Using a local index is not recommended for production environments. To ensure high availability in a production environment, configure a Daemon Cluster on multiple nodes and assign one Daemon on each node.
Managing Memory Utilization for Replicated Stores
Each Daemon in a Daemon Cluster has a shared memory limit defined in the associated rs_daemons.cfg
file to guard against over aggressive memory usage. Oracle provides defined and tunable system properties to manage memory pressure events and prevent catastrophic failure of a Replicated Store.
For more information on setting the shared memory limit of a Daemon, see Creating a rs_daemons.cfg File.
Replicated Store Memory Usage
The following section describes actions that change Daemon's memory utilization:
-
A Daemon's memory usage only decreases when:
-
An attached Replicated Store is migrated to another Daemon.
-
A Replicated Store's regions are administratively deleted.
-
A Region's hosted secondary copy moves to another Daemon.
-
-
A Daemon memory usage only increases when an attached Replicated Store instance's current data set increases to the point where it needs to create new Regions.
-
When a Replicated Store instance deletes a message record, it's Region memory is unchanged and remains allocated for storing subsequent new records.
Note:
Similar to all other types of persistence, Replicated Store persisted messages are cached in WebLogic Server JVM memory. You can tune JMS Server paging to reduce this memory usage, however it may result in a severe performance impact.
Monitoring Memory Utilization
The following section provides information on how administrators can monitor Daemon shared memory utilization:
-
The Administration Utility provides commands that report memory usage for Regions and Daemons. See Administering a Daemon Cluster using the Administration Utility.
-
The UNIX
ipcs
command provides monitoring information on shared memory keys. -
Daemon logging when the trace level is set to 3 or higher. As the memory usage goes above or below each 10 percent increment of the
shared-memory-limit
, the Daemon logs the memory usage. Memory utilization messages are written at theINFO
logging level until memory utilization rises to the 80 percent warning threshold, where all memory utilization logging information is written at theWARNING
logging level. See Configuring a Daemon. -
WebLogic Server logs the shared memory limits of a Daemon:
-
Logs a memory usage
INFO
message as defined by theSpaceLoggingStartPercent
of theshared-memory-limit
. -
Logs a memory usage
WARNING
message as defined by theSpaceOverloadYellowPercent
of theshared-memory-limit
. -
Logs a memory usage ERROR message as defined by the
SpaceOverloadRedPercent
of theshared-memory-limit
.
See Table 3-10 for information on the tuning memory utilization behavior of WebLogic Server.
-
-
Monitoring Byte and Message current and pending counts on JMS servers and destinations.
Controlling Memory Utilization
The following section provides information on how administrators can monitor Daemon shared memory utilization:
-
The
shared-memory-limit
. Daemons check that theshared-memory-limit
is less than or equal to the operating system shmlimit and memlock limits. See Creating a rs_daemons.cfg File. -
WebLogic Server provides tunable system properties to manage the memory usage of a Replicated Store and associated Daemons and to protect against overload situations. These control the way that the system logs a Replicated Store's memory usage, the maximum size of a message that can be put into a destination that uses a replicated store, and the thresholds that trigger overload protection actions.The properties in Table 3-10 to all replicated stores on a WebLogic server instance, or a particular store named
store-name
.Table 3-10 Tunable System Properties for Memory Pressure Management
Property Type Default Value Range Description weblogic.store.replicated.MaximumMessageSizePercent
weblogic.store.replicated.
store-name
.MaximumMessageSizePercent
int
1% of Replicated Store's region size
1 to 100
The maximum message size for a JMS destination that is backed by a replicated store, specified as a percentage of the store region size. New messages that exceed this size get a
ResourceAllocationException
.The total size of all concurrently written replicated store messages must be less than the Region size or failures can result. See related settings for the “JMS Server or Destination Maximum Message Size".
weblogic.store.replicated.SpaceLoggingStartPercent
weblogic.store.replicated.
store-name
.SpaceLoggingStartPercent
int
70
1 to 100
The percentage of the Daemon Shared Memory Limit when a Replicated Store starts logging Daemon shared memory usage.
weblogic.store.replicated.SpaceLoggingDeltaPercent
weblogic.store.replicated.
store-name
.SpaceLoggingDeltaPercent
int
10
1 to 100
Replicated store daemon space usage logging delta.
For example: The default is 10 percent, which means the store will log every 10 percent of space usage change.
weblogic.store.replicated.SpaceOverloadYellowPercent
weblogic.store.replicated.
store-name
.SpaceOverloadYellowPercent
int
80
1 to 100
If the memory used by all Replicated Stores on a Daemon exceeds this percentage of the Daemon Shared Memory Limit, and also the data stored in the store exceeds this percentage of the total region memory allocated for the particular store, then a JMS server will reject new messages with a
ResourceAllocationException
.weblogic.store.replicated.SpaceOverloadRedPercent
weblogic.store.replicated.
store-name
.SpaceOverloadRedPercent
int
90
1 to 100
If the memory used by all Replicated Stores on a Daemon exceeds this percentage of the Daemon Shared Memory Limit, new or migrated stores will fail to open and log an
ERROR
message. -
Tune messaging quota attributes
Bytes Maximum
,Messages Maximum
, andMaximum Message Size
on destinations and JMS Servers. Messages that exceed these quota limits cause sending applications to receiveResourceAllocationException
s.
Replicated Store Behavior when Exceeding Available Memory
In the rare situation where there's not enough memory for an already running WebLogic Replicated Store to allocate a new Region, the Store will fail, close, and log an ERROR
message. As with any Store instance failure, stores that fail to start or close due to exceeding available memory can be automatically restarted or migrated using Automatic Service Migration or Whole Server Migration. See Server and Service Migration for Replicated Stores.
Interoperability Considerations for a Replicated Store
This section provides information on the interoperability of the Replicated Store:
-
Future releases of the replicated store may be incompatible with this release. If so, the Replicated Store fails to open, a version incompatibility message is logged, and the associated Region is left unmodified.
-
A WebLogic Replicated Store or Daemon Cluster will not start on an unsupported platform and will log an error message. For information on supported platforms, see Exalogic Elastic Cloud Software in Licensing Information.
Security Considerations for a Replicated Store
This section provides security considerations when using Replicated Stores:
Security Considerations for Administrators
The following section provides security requirements Administrators must implement to secure Replicated Stores:
-
All Daemons must be started using the same UID and be able to write to the Global Directory. A Daemon can not communicate with other Daemons in a cluster unless it has read/write permission on the Global Directory and specifies the same Global Directory as the other Daemons.
-
The administration utility must have the same UID as the Daemons. The administration utility can not attach unless it has read/write permission on the Global Directory and this directory is the same as the Daemon's Global Directory.
-
WebLogic Server must have the same GID as the Daemon it's attached to in order to access to shared memory.
-
The administration utility must have the same UID as the Daemon it's attached to.
-
Oracle recommends that Daemons should be run as regularly privileged user, but a Daemon executable binary still needs to be assigned UNIX
root
,set uid
, andset group
privileges so that a Daemon has permission to raise its own process priority and to change its runtime UID and group to the UID and group of the user that starts the Daemon. See Starting and Stopping a Daemon Cluster.Note:
If a Daemon executable doesn't have sufficient privileges, the Daemon start script will print the instructions necessary to assign the privileges to a Daemon.
General Security Considerations for Replicated Stores
The following section provides information on how secure access is enforced between Replicated Store components:
-
Global Directory Access Permissions—The Global Directory must be read/write accessible to the O/S user that launches clients and Daemons.If a user doesn't have permission to access this directory, the user can not launch Daemons or clients that use this directory.
-
Same Directory Verification—A Daemon verifies that connection requests from other Daemons, Replicated Stores, or the administration utility refer to the same Global Directory by:
-
Ensuring the directory path matches and its
rs_daemons.cfg
checksum matches. -
Verifying that the requestor can send the value of a UUID that it has written to that directory (a random shared secret). The UUID changes once per Daemon restart.
-
-
Shared Memory Access Permissions—A Daemon creates its shared memory with group only permissions, and only clients with a user that has matching group permissions can use this shared memory.
Limitations of a Replicated Store
The following limitations apply to Replicated Stores:
-
A Replicated Store should not be opened simultaneously by two server instances; otherwise, there is no guarantee that the data in a Region will not be corrupted. If possible, the Replicated Store will attempt to return an error in this case, but it will not be possible to detect this condition in every case. To help guard against this possibility, it is the responsibility of the administrator to ensure that no two same named domains, each with a same named Replicated Store, attempt to use the exact same Daemon Cluster. Two Replicated Stores will conflict if they have the same name, the same domain name, and the same Global Directory.
-
Daemon Logging is not integrated with WebLogic Server diagnostics, Java Flight Recorder (JFR), logging, or DebugMBeans.