6 Setting Up Geographic Redundancy

Best Practices for Domain-Specific GeoRedundant Services

The following best practices are recommended when configuring domain-specific georedundant services:

• When configuring a domain-specific georedundant service on two domains that share hardware, ensure the following

  • The domains are set up as separate installations with separate databases.

  • Each domain has a unique cluster configuration.

• To avoid any issues when a change is made to the configuration, do the following when you configure Coherence clusters:

  • Use unicast type of addressing.

  • Add Well Known Addresses (WKA) for the NT servers.

  • Configure Coherence servers for each NT server, if necessary.

About Recovering from a Geographic Redundant Failure

If a geographically redundant domain fails:

• An application's connection and login time-out or receive an error response from domain load balancer.

• The application then logs in to the peer geographically redundant domain (if not already logged in) and resumes sending and receiving traffic.

• SLA and policy requests, and budget limits optionally continue to be enforced in peer domain. You control this behavior with configuration settings.

• Applications should periodically attempt to login to the failed domain.

• When the failed domain recovers, the application logs back into the original domain, and logs out of the peer.

• Alternatively, an application could continue to use the peer domain until reset by the operator.

Configuring Both Sites for Geographic Redundancy

To use geographic redundancy, each site must be appropriately configured using GeoRedundantServiceMBean. Using this service, you:

• Define the ID of the local site in the GeoSiteId field.

• Define the number of failed attempts to reach a remote site before an alarm should be raised in the RemoteSiteReachabilityAlarmThreshold field.

• Define the remote site using the setSiteAddress method.

See "GeoRedundantService Reference" for details.

Defining One Site as the GeoMaster

One site of the site pair must be designated the geomaster site using the GeoStorageServiceMBean. You define geomaster site using the GeoMasterSiteId field.

See "GeoStorageService Reference" for details.

Understanding the Configuration Workflow

To set up geographic redundancy so that applications do not have to register with all sites, follow the steps in these sections:

• Understanding the Configuration Prerequisites

• Understanding the Configuration Example

• Configuring the MySQL Database

• Configuring the Services Gatekeeper Cache

• Configuring Services Gatekeeper MBeans

• Configuring Cache Types

• Protecting the Modified MBean Files

• Changing the type_id of Tables Replicated by MySQL

• Merging the Data Tables

• Restarting the MySQL Servers

• Connecting the Master and Slave Databases

Understanding the Configuration Prerequisites

Before starting the configuration process:

• Services Gatekeeper must be installed on all of the hosts.

• A MySQL database must be installed on all of the hosts

  The Services Gatekeeper domains must be installed on all of the hosts.

  You followed the instruction in "Configuring Basic Geographic Redundancy" and configured basic geographic redundancy.

Understanding the Configuration Example

The configuration instructions in this section use the example systems listed here. Both of these systems have Services Gatekeeper and MySQL 5.6 installed, and have been configured as master/slaves in a geographically redundant cluster:

• geotest1, with an IP address of 10.161.159.189

• geotest2 with an IP address of 10.161.159.166

This guide uses the geotest1 and geotest2 host names in the configuration steps for clarity; however it is best practice guideline to use IP addresses in a production system.

Table 6-1 lists values that are used in the configuration example.

Configuring the MySQL Database

Configure the MySQL servers on your master/slave nodes as both masters and slaves. You do this by using the my.cnf file. The example my.cnf files in this section shows you how.

Example 6-1 lists the my.cnf configuration setting used by geotest1:

[mysqld]
server-id              = 1
log_bin                = /var/lib/mysql/mysql-bin.log
#This is the database to log statements for
binlog-do-db           = ocsg60_geosite1
 
#The remote site uses a different name, need to map it to our local name
replicate-rewrite-db   = ocsg60_geosite2->ocsg60_geosite1
 
##Tables to replicate (local db name, ie after rewrite)
#SMS
replicate-do-table     = ocsg60_geositea.native_smpp_session_store
replicate-do-table     = ocsg60_geositea.pl_sms_delivery_notif
replicate-do-table     = ocsg60_geositea.pl_sms_offline_notif
replicate-do-table     = ocsg60_geositea.pl_sms_online_notif
replicate-do-table     = ocsg60_geositea.pl_sms_smpp_mt_sms
replicate-do-table     = ocsg60_geositea.pl_sms_smpp_mt_dr
replicate-do-table     = ocsg60_geositea.pl_sms_smpp_mo_sms
 
 
#MMS
replicate-do-table     = ocsg60_geositea.pl_mms_mt_dr_mms
replicate-do-table     = ocsg60_geositea.pl_mms_mo_mms
replicate-do-table     = ocsg60_geositea.pl_mms_mo_content_mms
replicate-do-table     = ocsg60_geositea.pl_legacy_mms_mt
replicate-do-table     = ocsg60_geositea.pl_mms_dr_subscribe
replicate-do-table     = ocsg60_geositea.pl_mms_offline_notif
replicate-do-table     = ocsg60_geositea.pl_mms_online_notif
replicate-do-table     = ocsg60_geositea.pl_legacy_mm7_notif
replicate-do-table     = ocsg60_geositea.pl_legacy_mms_status
replicate-do-table     = ocsg60_geositea.pl_legacy_mms_vas_id
replicate-do-table     = ocsg60_geositea.pl_legacy_mms_vasp_id
 
#TL
replicate-do-table     = ocsg60_geositea.pl_tl_mlp_trigger_info

Example 6-2 lists the my.cnf configuration file used by geotest2:

[mysqld]
server-id              = 2
log_bin                = /var/lib/mysql/mysql-bin.log
 
#The database to log statements for
binlog-do-db           = ocsg60_geosite2
 
#The remote site uses a different name, need to map it to our local name
replicate-rewrite-db   = ocsg60_geosite1->ocsg60_geosite2
 
##Tables to replicate (local db name, ie after rewrite)
#SMS
replicate-do-table     = ocsg60_geositeb.native_smpp_session_store
replicate-do-table     = ocsg60_geositeb.pl_sms_delivery_notif
replicate-do-table     = ocsg60_geositeb.pl_sms_offline_notif
replicate-do-table     = ocsg60_geositeb.pl_sms_online_notif
replicate-do-table     = ocsg60_geositeb.pl_sms_smpp_mt_sms
replicate-do-table     = ocsg60_geositeb.pl_sms_smpp_mt_dr
replicate-do-table     = ocsg60_geositeb.pl_sms_smpp_mo_sms
 
 
#MMS
replicate-do-table     = ocsg60_geositeb.pl_mms_mt_dr_mms
replicate-do-table     = ocsg60_geositeb.pl_mms_mo_mms
replicate-do-table     = ocsg60_geositeb.pl_mms_mo_content_mms
replicate-do-table     = ocsg60_geositeb.pl_legacy_mms_mt
replicate-do-table     = ocsg60_geositeb.pl_mms_dr_subscribe
replicate-do-table     = ocsg60_geositeb.pl_mms_offline_notif
replicate-do-table     = ocsg60_geositeb.pl_mms_online_notif
replicate-do-table     = ocsg60_geositeb.pl_legacy_mm7_notif
replicate-do-table     = ocsg60_geositeb.pl_legacy_mms_status
replicate-do-table     = ocsg60_geositeb.pl_legacy_mms_vas_id
replicate-do-table     = ocsg60_geositeb.pl_legacy_mms_vasp_id
 
#TL
replicate-do-table     = ocsg60_geositeb.pl_tl_mlp_trigger_info

Configuring the Services Gatekeeper Cache

Change the relevant Services Gatekeeper cache configuration so that data is read and written directly to the database instead of being cached.

This server-cache configuration guarantees that for example, when you send an SMS message, the correlation data is available to both geographically-redundant sites when the delivery receipt is received. There is a trade off in performance.

Replicate the following database tables for geographic redundancy, both in MySQL and in the Services Gatekeeper cache:

• native_smpp_session_store

• pl_legacy_mm7_notif

• pl_legacy_mms_mt

• pl_legacy_mms_status

• pl_legacy_mms_vas_id

• pl_legacy_mms_vasp_id

• pl_mms_dr_subscribe

• pl_mms_mo_content_mms

• pl_mms_mo_mms

• pl_mms_mt_dr_mms

• pl_mms_offline_notif

• pl_mms_online_notif

• pl_sms_delivery_notif

• pl_sms_offline_notif

• pl_sms_online_notif

• pl_sms_smpp_mo_sms

• pl_sms_smpp_mt_dr

• pl_sms_smpp_mt_sms

• pl_tl_mlp_trigger_info

Configuring Services Gatekeeper MBeans

See Table 6-1 for the list of MBeans and the settings required.

Configuring Cache Types

Manually modify these storage schema jar files for geographic redundancy:

• oracle.ocsg.plugin.sms.native.smpp.store_6.0.0.0.jar

• com.bea.wlcp.wlng.plugin.multimediamessaging.mm7.store_6.0.0.0.jar

• com.bea.wlcp.wlng.plugin.multimediamessaging.px21.mm7.store_6.0.0.0.jar

• com.bea.wlcp.wlng.plugin.sms.common.store_6.0.0.0.jar

• com.bea.wlcp.wlng.plugin.terminallocation.mlp.store_6.0.0.0.jar

• oracle.ocsg.plugin.sms.px21.smpp.store_6.0.0.0.jar

Protecting the Modified MBean Files

In order to keep subsequent patches from overwriting the jar files you modified in "Configuring Cache Types", protect them with these steps:

Run these commands on each file in sequence:

% jar xf store_schema_filename.jar wlng-cachestore-config-extensions.xml
% mv wlng-cachestore-config-extensions.xml store_schema_filename.xml
% chmod 640 chmod 640 store_schema_filename.xml.xml

For example:

% jar xf com.bea.wlcp.wlng.plugin.multimediamessaging.mm7.store_6.0.0.0.jar wlng-cachestore-config-extensions.xml
% mv wlng-cachestore-config-extensions.xml com.bea.wlcp.wlng.plugin.multimediamessaging.mm7.store_6.0.0.0.xml
% chmod 640 chmod 640 com.bea.wlcp.wlng.plugin.multimediamessaging.mm7.store_6.0.0.0.xml

Changing the type_id of Tables Replicated by MySQL

For each table that is replicated by MySQL, you must change the type_id (cache_name for native_smpp_session_store table) to match the value for wlng.db.direct in the wlng-cachestore-config-extensions.xml file.

This example changes the type_id for the pl_legacy_mm7_notif table:

1. Open com.bea.wlcp.wlng.plugin.multimediamessaging.mm7.store_6.0.0.0.xml for editing.

2. Search for the table name (pl_legacy_mm7_notif).

3. Change the value for type_id from wlng.db.wt.plugin.mms.legacy.mm7.mo_notif_store to wlng.db.direct.plugin.mms.legacy.mm7.mo_notif_store.

Table 6-2 lists the table names and type_ids.

Configuring the my.cnf File for MySQL Replication

Table 6-3 lists the my.cnf file configuration settings to configure two systems as a geographically redundant pair. This table continues using the geotest1 and geotest2 example systems.

Merging the Data Tables

Once you have merged the tables, avoid writing any data to them until you have completed configuration.

Restarting the MySQL Servers

Restart both MySQL servers to make you changes take effect with this command:

sudo service mysql restart

Connecting the Master and Slave Databases

You must connect the databases in your geographically redundant systems with the instructions in this section:

1. Create a user called replicator on each system to do the replication, and grant them rights with these commands:

   mysql> create user 'replicator'@'%' identified by 'password';
   mysql> grant replication slave on *.* to 'replicator'@'%';
   

   See the MySQL create user and grant commands for details.

2. Obtain each master's File and Position to connect to. Run these commands on each system:

   mysql -u root -p -h geotest1
   mysql> show master status
   

   Record the File and Position for each system. The output will look something like this:

   geotest1:

   +------------------+----------+-----------------+------------------+
   | File             | Position | Binlog_Do_DB    | Binlog_Ignore_DB |
   +------------------+----------+-----------------+------------------+
   | mysql-bin.000002 | 85527658 | ocsg60_geositea |                  |
   +------------------+----------+-----------------+------------------+
   1 row in set (0.03 sec)
   

   geotest2:

   +------------------+----------+-----------------+------------------+
   | File             | Position | Binlog_Do_DB    | Binlog_Ignore_DB |
   +------------------+----------+-----------------+------------------+
   | mysql-bin.000006 | 12927236 | ocsg60_geositeb |                  |
   +------------------+----------+-----------------+------------------+
   1 row in set (0.07 sec)
   

3. Run these commands on all systems to connect slaves to the masters:

   On geotest1:

   mysql -u root -p -h ubuntu
   slave stop; 
   CHANGE MASTER TO MASTER_HOST = '10.161.159.166', MASTER_USER = 'replicator', 
   MASTER_PASSWORD = 'password', MASTER_LOG_FILE = 'mysql-bin.000006', MASTER_LOG_POS = 12927236; 
   slave start; 
   

   On geotest2:

   mysql -u root -p -h centos
   slave stop; 
   CHANGE MASTER TO MASTER_HOST = '10.161.159.189', MASTER_USER = 'replicator', 
   MASTER_PASSWORD = 'password', MASTER_LOG_FILE = 'mysql-bin.000002', MASTER_LOG_POS = 85527658; 
   slave start; 
   

The MySQL databases are now connected and replicated. You can confirm the replication by sending an MT message to geotest1, then confirm that geotest2 received the delivery report. Have geotest1 px21 SMS plug-in binding transmitter only, and geotest2 px21 SMS plug-in binding receiver only.

Removing the GeoRedundant Service from the Administration Console

To remove geographic redundancy from domains, and return them to a standalone configuration, repeat these steps on every geographically redundant domain.

To Remove Geographic Redundancy from a Domain: 

1. Start the Administration Console.

2. In the Change Center, click Lock & Edit.

3. In the Domain Structure, navigate toe domain_name, OCSG, then server_name, the Container Services, then GeoRedundantService.

4. In the Attributes tab, empty the text field opposite the GeoSiteId attribute.

5. Navigate to Operations.

6. Select RemoveSite from the Select an Operation menu.

7. Click Invoke.

8. In the Change Center, click Release Configuration.

Removing the GeoRedundant Service Programmatically

This section explains how to remove georedundant services from domains programmatically using the WebLogic Scripting Tool (WLST), or a different MBean editor.

Access the GeoRedundantServiceMBean and change its settings to reconfigure the domains to act as standalone domains.

Retrieve a list of the Remote Sites in the Domain. 

1. Retrieve the list of all the remote sites in the domain.

   To do so, call the listRemoteSites() method of the GeoRedundantServiceMBean Mbean. This method returns the names of all the sites registered in the domain.

   For information about GeoRedundantServiceMBean, see ”All Classes” section of the OAM Java API Reference.

Remove each remote site in the list, one at a time. 

1. Remove the configuration for the remote site.

   To do so, call the removeSite() method of the GeoRedundantServiceMBean Mbean. This method takes the remote site name as a parameter.

2. Set the local site ID to a blank string.

   To do so, call the setGeoSiteId(String siteName) method of the GeoRedundantServiceMBean Mbean. When you call this method, place a blank string as the value of the input parameter, siteName.

3. Set the master site ID to a blank string.

   To do so, call the setGeoMasterSiteId(String geoMasterSiteId) method of the GeoStorageServiceMBean Mbean. When you call this method, place a blank string as the value of the input parameter, geoMasterSiteId.

   For information about GeoStorageServiceMBean, see ”All Classes” section of the OAM Java API Reference.

Troubleshooting

At times, when you attempt to revert from a georedundant configuration to a standalone domain configuration, some of the georedundancy configuration settings seem to persist in your domain database.

If this scenario occurs, you can remove these georedundancy configuration settings from the database.

To remove these settings from your testing or non-production database:

1. Stop the servers.

2. Do the following for each georedundant domain:

   1. To delete the settings for each georedundant domain, enter the following SQL commands:

      DELETE FROM wlng_configuration WHERE instance = 'GeoStorageService';
      DELETE FROM wlng_configuration WHERE instance = 'GeoRedundantService';
      DELETE FROM wlng_blob_configuration WHERE instance = 'GeoRedundantService';
      

3. Repeat step 2 for each domain.

4. Verify that you have removed the settings from the domains in your non-production environment.

5. Restart the servers.