1. Provision and deploy a maximum availability solution for PeopleSoft on Oracle Cloud
  2. Implement PIA Web Resiliency with Coherence*Web

Implement PIA Web Resiliency with Coherence*Web

Add web resiliency with Coherence*Web for your PeopleSoft Internet Architecture (PIA) web servers. Coherence*Web is configured on each PIA web server and runs parallel to, but separate from the PIA WebLogic servers. This is called Out-of-Process Topology for Coherence*Web. A cluster is formed when two or more Coherence*Web cache servers are started.After you configure Coherence*Web, you'll configure the PIA web servers to work with Coherence*Web and then they'll join this cluster.

Configure Coherence*Web

Set up the infrastructure and scripting needed to start and manage Coherence*Web. While you will create the script that starts the product, there are a few more tasks for you to complete before you can run the script.

With PeopleTools 8.57, Coherence*Web is bundled with the Fusion Middleware installation. In this project, it is in the directory: /u01/app/psft/pt/bea/coherence. This is a shared directory that all PIA web servers can access.

  1. Make directories for Coherence*Web configuration and log files.
    • Node: All PIA web server nodes
    • User: psadm2
    $ . ./psft.env
$ mkdir -p $PS_CFG_HOME/coherence/config
$ mkdir -p $PS_CFG_HOME/coherence/log
  2. Edit the psft.env file to add the following environment variables to the psft.env file.
    • Node: All PIA web server nodes
    • User: psadm2
    export COHERENCE_HOME=$BASE_DIR/pt/bea/coherence
export COHERENCE_CONFIG=$PS_CFG_HOME/coherence/config
export COHERENCE_LOG=$PS_CFG_HOME/coherence/log
    Re-source the environment file.
  3. Obtain the private IP address of each PIA web server compute instance.

    In our project, each PIA web server has a private IP address, referred to as a well-known-address (WKA) in the Coherence*Web documentation. You will need these private IP addresses in later steps.

    • Node: All PIA web server nodes
    • User: psadm2, OCI Console user
    1. Log in to the OCI Console.
    2. Click Compute, then click Instances.
    3. Select the compartment holding the web servers.
    4. Click on each of the PIA web servers listed to see their private IP address. Record the addresses.
  4. Create a Coherence override file in the COHERENCE_CONFIG directory.
    You can choose any name for the file, but it is generally called: tangosol-coherence-override.xml, and is referenced with that name in several Coherence*Web documents.
    • Node: All PIA web server nodes
    • User: psadm2
    The XML file will have content similar to the following:
    <?xml version="1.0"?>
<!DOCTYPE coherence SYSTEM "coherence-override.dtd">
<!--
This operational configuration override file is set up for use with Coherence
-->
<coherence>
<cluster-config>
<unicast-listener>
  <well-known-addresses>
   <socket-address id="1">
      <address>10.0.103.85</address>
   </socket-address>
   <socket-address id="2">
      <address>10.0.103.203</address>
  </socket-address>
</well-known-addresses>
<port system-property="tangosol.coherence.localport">8088</port>
<address system-property="coherence.localhost">10.0.103.0/24</address>
</unicast-listener>
</cluster-config>
</coherence>
    1. Specify the private IP address for every PIA web server.
      This example configuration uses two web servers.
    2. Place the file on ALL PIA web servers. The content must be identical.
    The parameter <address system-property="coherence.localhost">10.0.103.0/24</address> specifies the entire private subnet serving the PIA web servers. When the Coherence*Web cache server starts, it will obtain the local host private IP address that it is running on.
  5. Create a script that will start the Coherence*Web cache server in the background on each PIA web server.
    See startCacheServer.sh for an example script. It's located in the Wrapper directory in GitHub.
    • Node: All PIA web server nodes
    • User: psadm2

Note:

You'll run the script after you configure the Coherence*Web network.

Configure the Coherence*Web Network

Each PIA web server runs in isolation and does not communicate with the other web servers. To provide cross-server resiliency at this layer, the Coherence*Web servers form a cluster and require a network configuration that allows their cache servers to communicate with each other.

To allow Coherence*Web to form a cache cluster, use the OCI Console to add ingress and egress rules to the security list associated with the app_private_seclist. In addition to ports 8088 and 8089, you'll use the default ports specified in the Coherence*Web documentation.

Ingress rules:

Stateless Source CIDR IP Protocol Source Port Range Destination Port Range
NO 10.0.103.0/24 TCP All 7
NO 10.0.103.0/24 UDP All 7
NO 10.0.103.0/24 TCP All 7574
NO 10.0.103.0/24 UDP All 7574
NO 10.0.103.0/24 TCP All 8088-8089

Egress rules:

Stateless Destination CIDR IP Protocol Source Port Range Destination Port Range
NO 0.0.0.0/0 TCP All All
NO 0.0.0.0/0 UDP All All
  1. Log into the OCI Console.
  2. Add the ingress and egress rules to the security list associated with the app_private_seclist.
  3. Run the following firewall-cmd commands as root on each compute instance to enable the ports listed in the table on each of the PIA web compute instances.
    Modify for your environment.
    $ firewall-cmd --zone=public --add-rich-rule 'rule family=ipv4 source address=10.0.103.0/24 port port=7 protocol=tcp accept' –permanent
$ firewall-cmd --zone=public --add-rich-rule 'rule family=ipv4 source address=10.0.103.0/24 port port=7 protocol=udp accept' –permanent
$ firewall-cmd --zone=public --add-rich-rule 'rule family=ipv4 source address=10.0.103.0/24 port port=8088-8089 protocol=tcp accept' –permanent
$ firewall-cmd --zone=public --add-rich-rule 'rule family=ipv4 source address=10.0.103.0/24 port port=7574 protocol=tcp accept' –permanent
$ firewall-cmd --zone=public --add-rich-rule 'rule family=ipv4 source address=10.0.103.0/24 port port=7574 protocol=udp accept' --permanent
$ firewall-cmd –reload
  4. Start the first Coherence*Web cache server (iad-psft-hcm-web01).

    Use the startCacheServer.sh script located in the Basic Tasks directory in GitHub to start a cache server, by issuing the following commands on one compute instance as psadm2. The following assumes that the script directory is in the administrator’s PATH. 

    $ startCacheServer.sh
    When the first cache server is started, the output will look similar to the following:
    …
2022-08-31 21:15:11.539/0.639 Oracle Coherence 12.2.1.3.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/u01/app/psft/pt/bea/coherence/lib/coherence.jar!/tangosol-coherence.xml" 
…
2022-08-31 21:15:16.137/5.237 Oracle Coherence GE 12.2.1.3.0 <Info> (thread=main, member=n/a): Started cluster Name=psadm2's cluster, ClusterPort=7574

WellKnownAddressList(
  10.0.103.203
  10.0.103.85
  )

MasterMemberSet(
  ThisMember=Member(Id=1, Timestamp=2022-08-31 21:15:12.825, Address=10.0.103.85:8088, MachineId=10879, Location=site:appprivatesu.ebscloudmaavcn.oraclevcn.com,machine:iad-psft-hcm-web01,process:21537, Role=CoherenceServer)
  OldestMember=Member(Id=1, Timestamp=2022-08-31 21:15:12.825, Address=10.0.103.85:8088, MachineId=10879, Location=site:appprivatesu.ebscloudmaavcn.oraclevcn.com,machine:iad-psft-hcm-web01,process:21537, Role=CoherenceServer)
  ActualMemberSet=MemberSet(Size=1
    Member(Id=1, Timestamp=2022-08-31 21:15:12.825, Address=10.0.103.85:8088, MachineId=10879, Location=site:appprivatesu.ebscloudmaavcn.oraclevcn.com,machine:iad-psft-hcm-web01,process:21537, Role=CoherenceServer)
    )
  MemberId|ServiceJoined|MemberState|Version
    1|2022-08-31 21:15:12.825|JOINED|12.2.1.3.0
  RecycleMillis=1200000
  RecycleSet=MemberSet(Size=0

    You can verify that the configuration is being picked up correctly by seeing the IP addresses listed in the WellKnownAddresslist section.

    Since you only started one cache server, there is only one member (the compute instance on which the cache server was started): iad-psft-hcm-web01.

  5. Use the same script to start the cache server on the second middle tier compute instance (iad-psft-hcm-web02).
    $ startCacheServer.sh
    It should join the cluster and the log file should look similar to the following:
    …

2022-08-31 21:15:36.513/0.628 Oracle Coherence 12.2.1.3.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/u01/app/psft/pt/bea/coherence/lib/coherence.jar!/tangosol-coherence.xml" 
…
2022-08-31 21:15:38.307/2.421 Oracle Coherence GE 12.2.1.3.0 <Info> (thread=main, member=n/a): Started cluster Name=psadm2's cluster, ClusterPort=7574

WellKnownAddressList(
  10.0.103.203
  10.0.103.85
  )

MasterMemberSet(
  ThisMember=Member(Id=2, Timestamp=2022-08-31 21:15:37.981, Address=10.0.103.203:8088, MachineId=10880, Location=site:appprivatesu.ebscloudmaavcn.oraclevcn.com,machine:iad-psft-hcm-web02,process:12859, Role=CoherenceServer)
  OldestMember=Member(Id=1, Timestamp=2022-08-31 21:15:12.825, Address=10.0.103.85:8088, MachineId=10879, Location=site:appprivatesu.ebscloudmaavcn.oraclevcn.com,machine:iad-psft-hcm-web01,process:21537, Role=CoherenceServer)
  ActualMemberSet=MemberSet(Size=2
    Member(Id=1, Timestamp=2022-08-31 21:15:12.825, Address=10.0.103.85:8088, MachineId=10879, Location=site:appprivatesu.ebscloudmaavcn.oraclevcn.com,machine:iad-psft-hcm-web01,process:21537, Role=CoherenceServer)
    Member(Id=2, Timestamp=2022-08-31 21:15:37.981, Address=10.0.103.203:8088, MachineId=10880, Location=site:appprivatesu.ebscloudmaavcn.oraclevcn.com,machine:iad-psft-hcm-web02,process:12859, Role=CoherenceServer)
    )
  MemberId|ServiceJoined|MemberState|Version
    1|2022-08-31 21:15:12.825|JOINED|12.2.1.3.0,
    2|2022-08-31 21:15:37.981|JOINED|12.2.1.3.0
  RecycleMillis=1200000
  RecycleSet=MemberSet(Size=0
    )

…

    The log file snippet above shows two members in the cluster, both with a role of CoherenceServer.

  6. Look at the log file of the cache server that you started first on compute instance iad-psft-hcm-web01. You should see entries similar to the following showing that a connection to iad-psft-hcm-web02 has been established. 
    2022-08-31 21:15:38.179/27.279 Oracle Coherence GE 12.2.1.3.0 <D6> (thread=Cluster, member=1): TcpRing connected to Member(Id=2, Timestamp=2022-08-31 21:15:37.981, Address=10.0.103.203:8088, MachineId=10880, Location=site:appprivatesu.ebscloudmaavcn.oraclevcn.com,machine:iad-psft-hcm-web02,process:12859, Role=CoherenceServer)

    The first time you start the cache servers one at a time and check each one to verify that it is configured correctly. Going forward, when you need to start your middle tiers, you can start all of the Coherence*Web cache servers at one time.

  7. If on each compute instance you only see one member, then the Coherence*Web cache servers cannot “see” each other.
    This is caused by one or more network ports being blocked. One way to test for ports being blocked is to use the Netcat (nc) utility, which must be installed on each PIA web compute instance.
    1. Use yum to install the nc) utility on each PIA web compute instance as the user root 
      # yum install -y nc
    2. Log in as root to test ports for both TCP and UDP packets on your PIA web servers.
      For example, to test to see if port 7574 is open on iad-psft-web01, start a Netcat listener that listens for UDP packets on port 7574: 
      # nc -u -l iad-psft-web01 7574

      The nc command will wait, let it sit there.

    3. In a separate session, log in as root on iad-psft-web02. Start a Netcat UDP client that connects to iad-psft-web01 on port 7574:
       # nc -u iad-psft-web01 7574

      The nc command will wait.

    4. On that terminal, type something like “test” and press Enter.
      The word “test” should display on the other terminal.
    5. Type “test” on the server terminal where nc is listening and press Enter.
      It should show up in the client terminal.
    6. If the test is successful, then switch roles and run the nc listener on iad-psft-web02 and the client on iad-psft-web01 and repeat the test.
    7. If the test is not successful, then one or more of the following are likely the cause:
      • There is no ingress rule allowing UDP protocol for port 7574 in the relevant security list for the PIA web server subnet.
      • There is no egress rule allowing UDP protocol for port 7574 in the relevant security list for the PIA web server subnet.
      • The firewall-cmd command has not been run on each PIA web server to allow UDP traffic for port 7574.
      Coherence*Web provides a datagram test utility to test connections for both TCP and UDP. Refer to Using the Datagram Test Utility for further details.

Configure PIA Web Server for Coherence*Web

With the Coherence*Web cache servers configured and running on all PIA web server compute instances, you can now configure the PIA web servers.

Recent versions of PeopleTools configuration files already have most of the required configuration, commented out. To enable access to Coherence*Web cache servers, follow the steps, adjusting as appropriate for your environment.

Perform the following on all PIA web server nodes, as the psadm2 user.

  1. Ensure that the session cookie and network domain names are the same for all PIA web servers in the weblogic.xml file. The weblogic.xml file is located in $PS_CFG_HOME/ webserv/HR92U033/applications/peoplesoft/PORTAL.war/WEB-INF. For this example, the domain is HR92U033.
  2. In the same weblogic.xml file, move the end comment string from below the <persistent-store-type>coherence-web</persistent-store-type> tag to above it, so it is no longer part of the comment:
    The following is an example of the weblogic.xml file:
    <?xml version="1.0" encoding="ISO-8859-1" standalone="no"?>
<weblogic-web-app xmlns="http://www.bea.com/ns/weblogic/weblogic-web-app">
<description>PeopleSoft Internet Architecture</description>
  <session-descriptor>
  <id-length>32</id-length>
  <cookie-name>iad-hcm-8080-PORTAL-PSJSESSIONID</cookie-name>
  <cookie-domain>.appprivatesu.ebscloudmaavcn.oraclevcn.com</cookie-domain>
  <monitoring-attribute-name>USERID</monitoring-attribute-name>
  <persistent-store-table>wl_servlet_sessions</persistent-store-table>
  <http-proxy-caching-of-cookies>true</http-proxy-caching-of-cookies>
<!-- Coherence*Web
-->
  <persistent-store-type>coherence-web</persistent-store-type>
  </session-descriptor>
    <container-descriptor>
      <servlet-reload-check-secs>-1</servlet-reload-check-secs>
      <session-monitoring-enabled>true</session-monitoring-enabled>
  </container-descriptor>
  <context-root>/</context-root>
</weblogic-web-app>
  3. In the web.xml file, uncomment the section starting with “Coherence*Web parameters”. This section contains all the required parameters for Coherence*Web.
    The web.xml file is in the same directory as the weblogic.xml file.
    The following is an example of this specific section:
    <!-- Coherence*Web parameters
-->
  <context-param>
    <description>With this set to "true", attributes that are deemed to be
      mutable (detected with a simple check) and which are accessed through a get,
      are deemed to be suspect in that they may have been changed in application code.
      Suspect attributes are treated as changed. Defaults to "false".</description>
    <param-name>coherence-enable-suspect-attributes</param-name>
    <param-value>true</param-value>
  </context-param>
  <context-param>
    <description>This value specifies a class name of the optional
      com.tangosol.coherence.servlet.HttpSessionCollection.AttributeScopeController
      interface implementation to use.</description>
    <param-name>coherence-scopecontroller-class</param-name>
    <param-value>com.tangosol.coherence.servlet.AbstractHttpSessionCollection$GlobalScopeController</param-value>
  </context-param>
  <context-param>
    <description>Specifies a cache delegator class that is responsible for manipulating
     (getting, putting, or deleting) data in the distributed cache
    </description>
    <param-name>coherence-cache-delegator-class</param-name>
    <param-value>com.tangosol.coherence.servlet.LocalSessionCacheDelegator</param-value>
  </context-param>
  <context-param>
    <description>This value, if set to true, specifies that non-serializable attributes
      should be preserved as local ones. This parameter requires a load balancer to
      be present to retrieve non-serializable attributes for a session
    </description>
    <param-name>coherence-preserve-attributes</param-name>
    <param-value>true</param-value>
  </context-param>
  <context-param>
    <description>
      This value specifies a class name of the com.tangosol.coherence.servlet.HttpSessionCollection
      $SessionDistributionController interface implementation
    </description>
    <param-name>coherence-distributioncontroller-class</param-name>
    <param-value>com.tangosol.coherence.servlet.AbstractHttpSessionCollection$HybridController</param-value>
  </context-param>
  4. Add the additional parameters to the PIA setEnv.sh located in the $PS_CFG_HOME/webserver/HR92U033/bin directory.
    There is a comment in this file for Coherence. Here is a snippet for the setEnv.sh file where we have added the PIA web startup parameters:
    The following is an example of this specific section:
    # Coherence*Web related parameters
# COHERENCE_PARAMETERS=-Dtangosol.coherence.session.localstorage=false
# Optionally add -Dtangosol.coherence.override=file:OVERRIDE_FILE_PATH
COHERENCE_PARAMETERS="-Dtangosol.coherence.override=file:/peoplesoft/local/ps_config/coherence/config/tangosol-coherence-override.xml -Dtangosol.coherence.session.localstorage=false -Djava.net.preferIPv4Stack=true"
# Example: JAVA_OPTIONS_LINUX="$JAVA_OPTIONS_LINUX $COHERENCE_PARAMETERS"
# export JAVA_OPTIONS_LINUX
JAVA_OPTIONS_LINUX="$JAVA_OPTIONS_LINUX $COHERENCE_PARAMETERS"
export JAVA_OPTIONS_LINUX
# Refer to Coherence*Web documentation
    Notes:
    • There are hyphens immediately before each parameter (word wrap may split the display)
    • You must define the COHERENCE_PARAMETERS before exporting JAVA_OPTION_LINUX
    • You must specify the Dtangosol.coherence.override=file:/peoplesoft/local/ps_config/coherence/config/tangosol-coherence-override.xml so that the PIA WebLogic server can properly register as a Coherence*Web client.
  5. Restart the PIA web server, using the custom PeopleSoft Startup and Shutdown Scripts.
    $ stopWS.sh 
$ startWS.sh
  6. Repeat these steps for each PIA web server node.
As each PIA web server is restarted, it will join the Coherence*Web cache server cluster as clients. Each cache server log should show the WebLogic servers establishing a connection and joining the cluster with a role of WeblogicServer. Here is a snippet from the log:
2021-10-28 23:27:01.237/160.748 Oracle Coherence GE 12.2.1.3.0 <D6> (thread=Cluster, member=1): TcpRing connected to Member(Id=3, Timestamp=2021-10-28 23:27:01.044, Address=10.0.103.85:8089, MachineId=10879, Location=site:appprivatesu.ebscloudmaavcn.oraclevcn.com,machine:iad-psft-hcm-web01,process:16067, Role=WeblogicServer)
…
2021-10-28 23:28:14.958/234.469 Oracle Coherence GE 12.2.1.3.0 <D6> (thread=Cluster, member=1): TcpRing connecting to Member(Id=4, Timestamp=2021-10-28 23:28:14.768, Address=10.0.103.203:8089, MachineId=10880, Location=site:appprivatesu.ebscloudmaavcn.oraclevcn.com,machine:iad-psft-hcm-web02,process:14708, Role=WeblogicServer)

The cache server logs on each PIA web server should show something like the above. The PeopleSoft deployment now has web server resiliency.

Test the PIA Web Resiliency

Perform the following to test and validate that Coherence*Web is providing web resiliency.

  1. Shut down all but one PIA web server.
  2. Using a web browser, log in to the PeopleSoft application.
  3. Navigate around, do some work. Start a transaction, but do not complete it.
  4. Start up a second PIA web server and allow it to completely come up. Check the load balancer from the OCI Console to ensure the status of the second PIA web server shows OK.
  5. Shut down the PIA web server that was left up in Step 1.
  6. In your web browser that is connected to the PeopleSoft application, continue your transaction.
    You may detect a slight pause, but you should not receive an error, need to log in again, or lose any work.
    If an error such as “Unauthorized token” occurs, review the following:
    • The session cookie and domain names must be the same for all PIA web server weblogic.xml files.
    • The OCI Load Balancer backend set must be configured with Enable application cookie persistence and the cookie name must be specified.
    • All Coherence*Web cache servers must be able to see each other and have formed a cluster.
    • All PIA web servers must have joined the cache server cluster as WeblogicServer clients.
    • Double-check networking configurations, including the security list ingress and egress rules, firewall-cmd commands, and so on.