5 Maintenance

This chapter describes routine maintenance and procedures necessary during normal operations. If you have an issue not covered here, refer to the appropriate Related Documents at the beginning of the book or contact Oracle Support.

Manually Placing a Service Offline

When a service is experiencing issues, Microsoft Cluster detects that it is offline and restarts the service on the active node. You can take the service offline for maintenance to avoid the service restart using the following procedures:

  1. Open the Failover Cluster Manager.

  2. Expand the Cluster Object (DIVA-CL-ORC) in the navigation tree on the left side of the screen.

  3. Select Roles in the expanded tree on the left side of the screen.

  4. Select the failing service in the Roles area in the middle of the screen.

  5. Right-click the selected service, and then click Take Offline from the resulting menu.

  6. The status of the selected service should now show Offline in the Roles area in the middle of the screen.

Adding a Network for Client Access

You can configure additional client access using the Failover Cluster Manager. This is useful when another subnet is configured for automation. Each node must have one static IP address on the same subnet as listed in the Network Requirements. Use the following procedure to configure additional clients:

  1. Configure the new interfaces and subnetwork on each node.

  2. Click Start, Administrative Tools, and then Failover Cluster Management Console.

  3. Expand the Cluster Object (DIVA-CL-ORC) in the navigation tree on the left side of the screen.

  4. Select Networks in the expanded tree on the left side of the screen.

  5. Select the new network to use for automation from the Networks list in the middle of the screen.

  6. Click Properties under the listed network on the right side of the screen.

  7. Enter a new name for the network used for automation in the Name field.

    Using the name Automation for the network makes it easily identifiable.

  8. Select the Allow clients to connect through this network check box.

  9. Click Apply, and then click OK.

  10. Right-click Roles in the navigation tree on the left side of the screen.

  11. Click Add Resource from the resulting menu, and then click Client Access Point to open the Client Access Point Wizard.

  12. On the Client Access Point screen enter an access point name (for example, DIVA-CL-AUTO) in the Name field.

  13. Select the proper network and associated IP address in the Networks list.

    You must add the FQDN to the DNS. Refer to the procedures in Registering the Required Host Names to the DNS Manager and Creating the Windows 2012 R2 Server Cluster if necessary.

  14. Click Next.

  15. Verify the selected configuration on the Confirmation screen, and then click Next.

  16. When the configuration is complete, verify that all configurations were successful by clicking View Report.

  17. Click Finish after you have confirmed that the configuration was successful.

Rebuilding the Cluster after a Node Hardware Failure

Use this procedure when one node fails. The procedure requires downtime during Fail Safe configuration. To rebuild the cluster, complete the steps in the following sections:

  1. Evicting a Failed Node

  2. Preparing New Hardware

  3. Joining a New Node Server to a Cluster

  4. Installing DIVArchive

  5. Installing and Configuring Oracle Fail Safe

Evicting a Failed Node

Do not perform this procedure as the primary troubleshooting method. Eviction should only be used when:

  • Replacing a node with different hardware.

  • Reinstalling the operating system.

  • Permanently removing a node from a cluster.

  • Renaming a node in a cluster.

Use the following procedure to evict a node:

  1. Log in to the Active Node server as a dedicated cluster domain account user (DIVAClusterAdmin).

  2. Click Start, Administrative Tools, and then Failover Cluster Management Console.

  3. Expand the Cluster Object (DIVA-CL-ORC) in the navigation tree on the left side of the screen.

  4. Right-click the failed node in the Nodes list in the middle of the screen.

  5. Click More Actions from the resulting menu, and then click Evict.

  6. A confirmation dialog box asks if you are sure you want to evict the node from the cluster - click Yes to evict the node (or No to leave the node in the cluster).

Preparing New Hardware

When the new hardware is ready, install Windows Server 2012 R2 Standard and all patches to match the Active Node.

Note:

Both nodes must be at the same patch level.

Refer to the following procedures:

  1. Configuring the Operating System

  2. Installing the Windows 2012 R2 Standard Server Clustering Feature

  3. Enabling the Remote Registry Service

Joining a New Node Server to a Cluster

Use the following procedure to add a new server to the cluster:

  1. Follow the procedure in Validating the Nodes Configuration for MSCS Clustering.

  2. Before connecting the external disk, ensure there are no local partitions using the D:, E:, F:, or H: drives.

    Use the Windows Server Manager to view the disks and assigned drive letters.

  3. Follow the procedure in Replacing a Host Bus Adapter (HBA).

  4. Add the node to the cluster as follows:

    1. Log in to the Active Node server as a dedicated cluster domain account user (DIVAClusterAdmin).

    2. Click Start, Administrative Tools, and then Failover Cluster Management Console.

    3. Expand the Cluster Object (DIVA-CL-ORC) in the navigation tree on the left side of the screen.

    4. Right-click Nodes in the expanded tree on the left side of the screen.

    5. Click Add Node in the resulting menu to open the Add Node Wizard.

    6. Click Next on the first wizard screen.

    7. Proceed through the wizard to add the new node to the cluster.

Installing DIVArchive

Refer to Configuring DIVArchive to complete DIVArchive installation and configuration. Since the DIVArchive Database schema is already in place, do not reinstall the schema on the Active node.

Installing and Configuring Oracle Fail Safe

Use the following procedure to install and configure Oracle Fail Safe:

  1. To install Oracle Fail Safe, refer to Installing Oracle Fail Safe

  2. Complete the Oracle Fail Safe configuration as follows:

    1. Confirm the Fail Safe service was created during the installation.

    2. Confirm the LIB5 service instance was created during the installation.

      Note:

      The initLIB5.ora file must be replicated on both nodes.
    3. Confirm the Oracle TNS Listener service was created during installation.

    4. Restart the new node and run the tests described in Testing the Configuration.

Replacing a Host Bus Adapter (HBA)

The SAS HBA interfaces external disks dedicated for the database and quorum partitions. Use the following procedure if a SAS HBA fails, or if a node fails and you must rebuild the node using new hardware:

  1. Replace the failed SAS HBA in the server following the manufacture's installation and configuration instructions and recommendations.

  2. Launch the Storage Manager software on the Active Node.

  3. Locate the Host Mapping area of your Storage Manager.

  4. Expand the DIVA Host Group and select the host that contains the new HBA.

  5. Right-click the host and click Manage Host Port Identifiers (your menu item listing may be different) from the resulting menu.

  6. Select the failed port in the list, and then click Replace.

  7. On the following screen, click the Replace by creating a new host port identifier option under Choose a method for replacing the host port identifier.

  8. Enter the new host port identifier in the New host port identifier (16 characters required) field, and then click Replace.

  9. When the replacement process completes, you should see the Cluster Volumes from the Active Node.

Configuring Windows Firewall with Advanced Security

Microsoft Best Practices recommend enabling the Windows Firewall, however it is not mandatory for DIVArchive. To use the Windows Firewall, use the DIVACloud_Firewall_Exceptions_2012.ps1 PowerShell script to enable DIVArchive exceptions through the firewall. Use the following procedure to create and run the Firewall Exceptions script in PowerShell:

  1. Open Notepad to create a text file.

  2. Copy the following script content and paste it into the file you just created.

    Note:

    You may (or may not) need to make adjustments to the line breaks, and so on due to formatting.
    ### Oracle DIVACloud Firewall Exception list. This will enable the Windows Firewall for all profiles and exclude common DIVA ports. ###
    ### WINDOWS 2012 Only BELOW ###
    Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled True
    New-NetFirewallRule -DisplayName "DIVACloud SSH" -Description "Oracle DIVACloud
    (SSH Remote Access)" -Direction Inbound -LocalPort 22 -Protocol TCP -Action
    Allow
    New-NetFirewallRule -DisplayName "DIVACloud DIVADirector HTTP" -Description
    "Oracle DIVACloud (DIVADirector HTTP)" -Direction Inbound -LocalPort 80
    -Protocol TCP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud Remote Administration" -Description
    "Oracle DIVACloud (Remote Administration)" -Direction Inbound -LocalPort 135
    -Protocol TCP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud DIVADirector HTTPS" -Description
    "Oracle DIVACloud (DIVADirector HTTPS)" -Direction Inbound -LocalPort 443
    -Protocol TCP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud CIFS" -Description "Oracle
    DIVACloud (Req. Collection Script)" -Direction Inbound -LocalPort 445
    -Protocol TCP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud RSYNC" -Description "Oracle
    DIVACloud (RSYNC)" -Direction Inbound -LocalPort 873 -Protocol TCP -Action
    Allow
    New-NetFirewallRule -DisplayName "DIVACloud Oracle TNS Listener" -Description
    "Oracle DIVACloud (Oracle Database - Transparent Network Substrate)"
    -Direction Inbound -LocalPort 1521 -Protocol TCP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud VACP" -Description "Oracle
    DIVACloud (Automation (Harris) Control)" -Direction Inbound -LocalPort 5010
    -Protocol TCP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud DataExpedition" -Description
    "Oracle DIVACloud (ExpeDat - Accelerated File Transfer)" -Direction Inbound
    -LocalPort 8080 -Protocol UDP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud DIVArchive Robot Manager"
    -Description "Oracle DIVACloud (DIVArchive Robot Manager)" -Direction Inbound
    -LocalPort 8500 -Protocol TCP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud DIVArchive Manager" -Description
    "Oracle DIVACloud (DIVA API Listener / Systems Monitoring)" -Direction Inbound
    -LocalPort 9000 -Protocol TCP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud DIVArchive Webservices"
    -Description "Oracle DIVACloud (DIVA Systems Monitoring)" -Direction Inbound
    -LocalPort 9443,9763 -Protocol TCP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud DIVArchive AccessGateway"
    -Description "Oracle DIVACloud (DIVA Communications)" -Direction Inbound
    -LocalPort 9500 -Protocol TCP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud DIVArchive Actor" -Description
    "Oracle DIVACloud (DIVActor)" -Direction Inbound -LocalPort 9900 -Protocol TCP
    -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud SNMP" -Description "Oracle
    DIVACloud (Systems Monitoring)" -Direction Inbound -LocalPort 161 -Protocol
    UDP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud RDP" -Description "Oracle DIVACloud
    (Remote Desktop Protocol)" -Direction Inbound -LocalPort 3389 -Protocol TCP
    -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud NRPE" -Description "Oracle
    DIVACloud (Icinga Systems Monitoring - Nagios NRPE)" -Direction Inbound
    -LocalPort 5666 -Protocol TCP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud NSClient++" -Description "Oracle
    DIVACloud (NSClient++ Monitoring w/Icinga)" -Direction Inbound -LocalPort
    12489 -Protocol TCP -Action Allow
    New-NetFirewallRule -DisplayName "DIVACloud ICMP" -Description "Oracle
    DIVACloud (Packet Internet Groper ICMPv4)" -Protocol ICMPv4 -IcmpType 8
    -Enabled True -Profile Any -Action Allow
    ### OPTIONAL LOGRHYTHM ONLY### New-NetFirewallRule -DisplayName "DIVACloud
    LogRhythm TCP" -Description "Oracle DIVACloud (LogRhythm Log Collection TCP)"
    -Direction Inbound -LocalPort 135, 137, 138, 139, 445, 49153 -Protocol TCP
    -Action Allow
    ### OPTIONAL LOGRHYTHM ONLY### New-NetFirewallRule -DisplayName "DIVACloud
    LogRhythm UDP" -Description "Oracle DIVACloud (LogRhythm Log Collection UDP)"
    -Direction Inbound -LocalPort 514 -Protocol UDP -Action Allow
    ### OPTIONAL NEVERFAIL ONLY### New-NetFirewallRule -Program "C:\Program
    Files\Neverfail\R2\bin\nfgui.exe" -Action Allow -Profile Domain, Private,
    Public -DisplayName "DIVACloud Neverfail" -Description "Oracle DIVACloud
    (Neverfail)" -Direction Inbound
    New-NetFirewallRule -Program "%SystemDrive%\Oracle\Ofs41_
    1\FailSafe\Server\FsSurrogate.exe" -Action Allow -Profile Domain, Private,
    Public -DisplayName "DIVACloud Oracle Fail Safe" -Description "Oracle DIVACloud
    (Fail Safe)" -Direction Inbound
    ### WINDOWS 2012 Only ABOVE ###
    
  3. Save the file with the file name DIVACloud_Firewall_Exceptions_2012.ps1.

  4. Open a Windows PowerShell command prompt. You may have to open the PowerShell as a Windows Administrator to successfully execute the script.

  5. Navigate to the folder where the script is located.

  6. Execute the script by entering DIVACloud_Firewall_Exceptions_2012.ps1 at the command prompt.

  7. All necessary exceptions required for DIVArchive operations should now be included in the Windows Firewall configuration.

If you require additional information or assistance refer to the Microsoft TechNet document named Windows Firewall with Advanced Security located at http://technet.microsoft.com/en-us/library/hh831365.aspx.

Cluster-Aware Updating

Cluster-Aware updating automates the Microsoft software updating process on clustered servers while maintaining availability. It is a Microsoft best practice to perform regular Windows updates, however it is not mandatory for DIVArchive. Refer to the following Microsoft TechNet documentation for details on Cluster-Aware updating: