Site Switchover

An Oracle Maximum Availability Architecture (Oracle MAA) best practice is to perform a full stack site switchover semi-annually, reversing the roles of the primary and secondary sites, to test switchover procedures and to catch and correct any unmanaged changes or other issues that might have occurred. You can also switch to the secondary site to continue providing services while the primary site is undergoing major maintenance.

Perform a Site Switchover Within OCI

Perform a full stack PeopleSoft switchover from Site 1 (originally the primary) to Site 2 (originally the secondary).

You can use Oracle Cloud Infrastructure (OCI) to perform site switchover either step-by-step, manually, or by scripting the steps into a single flow. In either case, you'll use a combination of REST APIs for the database tier and scripts for the application and web tiers.

This section describes the manual steps. The example assumes that the on-premises database has already been dropped from the Data Guard Broker configuration.

In this example, Site 1 is originally the primary and Site 2 is originally the secondary. They switch roles during this exercise. The following are the high-level tasks for performing a switchover in OCI:

Site 1:

Drain or place on hold batch jobs in the PeopleSoft Process Scheduler ahead of the planned switchover event.
Shut all PeopleSoft application servers, the process scheduler, and all PeopleSoft Internet Architecture (PIA) web servers down.
Validate that the PeopleSoft database is ready for switchover.
Perform Oracle Data Guard switchover.
Perform OCI File Storage role reversal.

Site 2:

Validate that role-based database services have been started.
Start PeopleSoft application servers, the process scheduler, and all PIA web servers.
Validate the status of backend servers on the new primary region load balance (green OK).
Validate you can log in to the PeopleSoft PIA.

The following example provides the detailed steps for performing a full-stack PeopleSoft switchover. These examples use names from our test environment for the Primary database in Ashburn (CDBHCM_iad1dx) and the Standby database in Phoenix (CDBHCM_phx5s).

Shut down the process scheduler on Site 1 for each compute instance.
- Site: Site 1
- Node: Each process scheduler server compute instance
- User: psadm2
In preparation for site switchover, it may be necessary to shut down the process scheduler at some point ahead of the scheduled switchover. This will place any recurring and new jobs in “queued” status.

When shutting down the process scheduler ahead of the scheduled switchover time, use the individual script stopPS.sh located in the Basic Tasks directory in GitHub. Do NOT use the wrapper script at this time. Step 4 below runs the wrapper script as part of the actual switchover process.
```
$ stopPS.sh
```

Validate that the standby is ready for switchover.

Site: Site 1
Node: One Oracle Exadata Database Service on Dedicated Infrastructure domU
User: oracle

Log in to one of the primary Oracle Exadata Database Service on Dedicated Infrastructure domUs hosting a PeopleSoft Oracle RAC instance, and become the oracle user.
Source the environment.
```
$ . ./CDBHCM.env
```

Start the Oracle Data Guard command-line interface.

$ dgmgrl sys/sys password
DGMGRL> show configuration lag
Configuration - fsc
  Protection Mode: MaxPerformance
  Members:
  CDBHCM_iad1dx - Primary database
    CDBHCM_phx5s  - Physical standby database 
                    Transport Lag:      0 seconds (computed 1 second ago)
                    Apply Lag:          0 seconds (computed 1 second ago)

Fast-Start Failover:  Disabled
Configuration Status:
SUCCESS   (status updated 35 seconds ago)

Validate the standby database.

DGMGRL> validate database 'CDBHCM_phx5s'

  Database Role:     Physical standby database
  Primary Database:  CDBHCM_iad1dx

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Managed by Clusterware:
    CDBHCM_iad1dx:  YES            
    CDBHCM_phx5s :  YES

The standby database is ready for switchover.

Shut the PIA web servers down.
- Site: Site 1
- Node: PIA web server compute instances
- User: psadm2
1. Log in to the PIA middle tier servers and become psadm2.
2. Use the wrapper scripts to shut down the PIA web servers and Coherence*Web cache servers.
  The wrapper scripts are located in the Wrapper directory in GitHub.
```
$ stopPSFTWEB.sh
```
Shut down the application server and process scheduler.
- Site: Site 1
- Node: application/process scheduler server compute instances
- User: psadm2
1. Log in to the compute instances hosting the PeopleSoft application servers and process scheduler and become psadm2.
2. Run the wrapper script from stopPSFTAPP.sh.
```
$ stopPSFTAPP.sh
```
  Note:
  The first instance to run the stopPSFTAPP.sh script will perform one final rsync of the file systems after the rest of the application server and process scheduler domains are down, then will disable rsync.
3. Use the SQL script in PeopleSoft Application and Process Scheduler Domains to monitor database sessions.
4. Once all of the stopPS scripts have completed, check the rsync log to verify that a final rsync was performed.
  Go to the Replication directory in GitHub for the rsync_psft.sh script.

Use the Data Guard Broker command-line interface to perform the switchover.

Site: Site 1
Node: One Oracle Exadata Database Service on Dedicated Infrastructure domU
User: oracle

$ dgmgrl sys/sys password
DGMGRL> switchover to CDBHCM_phx5s;
Performing switchover NOW, please wait...
New primary database " CDBHCM_phx5s" is opening...
Oracle Clusterware is restarting database "CDBHCM_iad1dx" ...
Connected to " CDBHCM_iad1dx"
Connected to " CDBHCM_iad1dx"
Switchover succeeded, new primary is " CDBHCM_phx5s"

Use the Data Guard Broker command-line interface to monitor and verify that the switchover succeeded.

Site: Site 1
Node: One Oracle Exadata Database Service on Dedicated Infrastructure domU
User: oracle

$ dgmgrl sys/sys password
DGMGRL> show configuration lag
Configuration - fsc
  Protection Mode: MaxPerformance
  Members:
  CDBHCM_phx5s  - Primary database
    CDBHCM_iad1dx - Physical standby database 
                    Transport Lag:      0 seconds (computed 2 seconds ago)
                    Apply Lag:          0 seconds (computed 2 seconds ago)

Fast-Start Failover:  Disabled
Configuration Status:
SUCCESS   (status updated 22 seconds ago)

If Active Data Guard support is configured, then ensure that the Active Data Guard service for PeopleSoft (PSQUERY) has been started on the new standby database post switchover.
- Site: Site 1
- Node: One Oracle Exadata Database Service on Dedicated Infrastructure domU
- User: oracle
```
$ srvctl status service -db CDBHCM_iad1dx -s PSQUERY
Service PSQUERY is running on instance(s) CDBHCM1,CDBHCM2
```
This service should be running on all Oracle RAC instances.

Note:
This service must be started before starting the process scheduler. Otherwise, the process scheduler will fail on startup.
Verify that the role-based database services are up on the new primary.
- Site: Site 2
- Node: All Oracle Exadata Database Service on Dedicated Infrastructure domUs
- User: oracle
For example, issue the following command on each domU hosting a PeopleSoft Oracle RAC database instance:
```
$ srvctl status service -db CDBHCM_phx5s -s HR92U033_BATCH
Service HR92U033_BATCH is running on instance(s) CDBHCM1,CDBHCM2
$ srvctl status service -db CDBHCM_phx5s -s HR92U033_ONLINE
Service HR92U033_ONLINE is running on instance(s) CDBHCM1,CDBHCM2
```
This service should be running on all Oracle RAC instances.

Start the application server and process scheduler domains.

Site: Site 2
Node: Application and process scheduler server compute instances
User: psadm2

Log in to the compute instances hosting the PeopleSoft application servers and process scheduler and become psadm2.
Use the startPSFTAPP.sh script located in the Wrapper directory in GitHub:
```
$ startPSFTAPP.sh
```

Monitor the startup.

You can use the query from PeopleSoft Application and Process Scheduler Domains.

col service_name format a20
select a.inst_id,a.instance_name,b.service_name, count(*)
from gv$instance a, gv$session b
where a.inst_id = b.inst_id
and service_name not like 'SYS%'
group by a.inst_id,a.instance_name,b.service_name
order by 1

SQL> /

   INST_ID INSTANCE_NAME    SERVICE_NAME          COUNT(*)
---------- ---------------- ------------------- ----------
         1 CDBHCM1          CDBHCM_phx5s                 2
SQL> /

   INST_ID INSTANCE_NAME    SERVICE_NAME          COUNT(*)
---------- ---------------- ------------------- ----------
         1 CDBHCM1          CDBHCM_phx5s                 2
         1 CDBHCM1          HR92U033_BATCH               8
         1 CDBHCM1          HR92U033_ONLINE             52
         2 CDBHCM2          HR92U033_BATCH               7
         2 CDBHCM2          HR92U033_ONLINE             50

Start web services.
- Site: Site 2
- Node: All PIA web server compute instances
- User: psadm2
If Coherence*Web is configured, then you will first start the cache cluster on all compute instances that host the PIA web servers, then start the PIA web servers. In this example, one script is used to start both in the proper order.
1. Log in to the PIA web servers and become psadm2.
2. Using the script from startPSFTAPP.sh, start the web servers.
```
$ startPSFTWEB.sh
```
Check the load balancer.
- Site: Site 2 Region
- Node: OCI Console
- User: Tenancy administrator
1. Log in to the OCI Console and change the region to your new primary (Phoenix in our example).
2. Select Networking, then Load Balancer from the main menu.
3. Select the appropriate compartment.
4. Click Backend Set, then click Backends.
  Each backend should show OK. It may take a few minutes after each PIA web server has been started.
Attempt to log in to the PIA web server from a web browser.
- User: PeopleSoft PIA web user
For this example, the URL is:
```
https://psfthcm.appprivad1.maacloud2vcn.oraclevcn.com/psp/ps/EMPLOYEE/HRMS/?cmd=login
```

When the above steps have successfully completed, production is running at Site 2.