1.4 Compliance Framework (Oracle ORAchk and Oracle EXAchk) Prerequisites

Review the list of prerequisites to run Oracle ORAchk and Oracle EXAchk.

• SSH Connectivity and Access
  In a clustered database environment, Oracle ORAchk and Oracle EXAchk run compliance checks on a single node and remotely run on all other cluster nodes.
• Handling of Root Passwords
  Handling of root passwords depends on whether you have installed the Expect utility.
• Deciding Which User Should Run Oracle ORAchk and Oracle EXAchk
  Run compliance checks as root. Also, run compliance checks as the Oracle Database home owner or the Oracle Grid Infrastructure home owner.
• Data Entry Terminal Considerations
  Use any supported UNIX and Linux terminal type (character mode terminal, ILOM, VNC server) to run Oracle Autonomous Health Framework.
• Running Oracle EXAchk on Oracle Exadata and Zero Data Loss Recovery Appliance
  Review the list of additional prerequisites for running Oracle EXAchk on Oracle Exadata and Zero Data Loss Recovery Appliance.
• Running Oracle Autonomous Health Framework in Non-English Environments
  Set globalization environment variables to run Oracle Autonomous Health Framework in non-English environments.

1.4.1 SSH Connectivity and Access

In a clustered database environment, Oracle ORAchk and Oracle EXAchk run compliance checks on a single node and remotely run on all other cluster nodes.

Note:

This is only required if installed as non-root or using storage servers. As the Oracle Trace File Analyzer secure sockets are used instead.

This change means that it is no longer required to configure passwordless SSH user equivalency for root to run checks on remote database nodes. User equivalency is still required for a cluster install, however, this can be skipped by doing multiple standalone installs and running tfactl syncnodes. Oracle ORAchk and Oracle EXAchk still use SSH to connect to storage servers.

Remotely running compliance checks on cluster nodes involves remotely copying files to and from the targets and running commands without providing the passwords.

If security restrictions block, then some commands fail to run. To run those commands successfully, develop alternate plans.

To run compliance checks remotely on all other cluster nodes from the database server:

• Configure passwordless SSH equivalency for the same user on each cluster node that runs Oracle ORAchk and Oracle EXAchk on the database server

Note:

If passwordless SSH is not configured in the environment, then Oracle ORAchk or Oracle EXAchk prompts you if it should configure permanent or temporary passwordless SSH in the environment.

Or

• Provide the private key file for the remote nodes, or allow Oracle ORAchk or Oracle EXAchk to auto-generate the private key file for the remote nodes.

The process used by Oracle ORAchk and Oracle EXAchk to generate the private key is as follows:

1. SSH as the desired user to remote node and enter the password password to add the system to the SSH known_hosts file.

   You should see something like:

   The authenticity of host '<hostname> (<ip>)' can't be established.
   RSA key fingerprint is fb:78:d1:6a:5c:62:ea:c4:85:20:76:f6:a9:01:1e:b4.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added 'hostname>,<ip>' (RSA) to the list of known hosts.

2. Log in to the remote node, and generate private and public key pair on the remote node exactly as follows.

   Replace hostname and username with your actual hostname and desired username.

   # ssh-keygen -f $HOME/.ssh/id_dsa.host.user -N ''

   For example, if your remote node is cehaovmsp1080 and your desired run user is root, then log in to that host and run:

   # ssh-keygen -f $HOME/.ssh/id_dsa.cehaovmsp1080.root -N ''

   Running the command creates two files in the $HOME/.ssh directory.

   [root@cehaovmsp1080 .ssh]# ls -ltr
   total 8
   -rw-------. 1 root root 1675 May 9 12:27 id_dsa.cehaovmsp1080.root
   -rw-r--r--. 1 root root 400 May 9 12:27 id_dsa.cehaovmsp1080.root.pub

3. Copy the contents of public key into the .ssh/authorized_keys file of remote node and then delete the public key from the remote node.

   # cat $HOME/.ssh/id_dsa.hostname.username.pub >> $HOME/.ssh/authorized_keys

   # rm -rf $HOME/.ssh/id_hostname.username.pub

   For example:

   # cat $HOME/.ssh/id_dsa.cehaovmsp1080.root.pub >> $HOME/.ssh/authorized_keys
   # rm -rf $HOME/.ssh/id_dsa.cehaovmsp1080.root.pub

4. Copy the private key $HOME/.ssh/id_dsa.hostname.username of remote node into the local node, where you will run Oracle ORAchk from, into the $HOME/.ssh directory.

5. Test through SSH.

   ssh -i $HOME/.ssh/id_dsa.hostname.username hostname date

6. If the test is successful, then run the Oracle ORAchk daemon.

   # export RAT_SSH_IDENTITY=$HOME/.ssh
   # orachk -d start

If you wish to generate your own private key files, then on remote machine use the following commands:

ssh-keygen -f $HOME/.ssh/id_dsa.host.user -N ''

For example:

ssh-keygen -f $HOME/.ssh/id_dsa.myhost67.root -N ''

Running this command generates the following key pair in the $HOME/.ssh/ directory:

id_dsa.myhost67.root (private key / Identity file)
id_dsa.myhost67.root.pub (Public key)

If you are unable to configure passwordless SSH or use the private key file, then

1. Run compliance checks on each database server in the cluster using the -localonly command-line option.

2. Merge the results using the -merge command-line option.

• Storage Servers that are Configured to Deny SSH Access
  The following discussion applies to any Oracle engineered system that uses Oracle Exadata storage servers.

1.4.1.1 Storage Servers that are Configured to Deny SSH Access

The following discussion applies to any Oracle engineered system that uses Oracle Exadata storage servers.

Optionally, you can prevent SSH access, also known as "locking" or "locked". All Oracle EXAchk functions involving locked storage servers are run with standard exacli commands from the database server upon which Oracle EXAchk is launched. To temporarily unlock the storage servers that Oracle EXAchk finds locked, provide the user name and credentials that you specified when configuring exacli to lock/unlock storage servers.

See Configuring Security for Oracle Exadata System Software in the Exadata System Software User's Guide.

Oracle EXAchk does not operate upon the storage server attribute accessLevelPerm. If you have set that attribute to remoteLoginDisabled before an Oracle EXAchk run, then it will remain unchanged during and after the Oracle EXAchk run.

Oracle EXAchk operates only upon the storage server attribute accessLevelTemp. For example, starting with the storage servers locked with remoteLoginDisabled:

-bash-4.1# ssh randomceladm01
ssh: connect to host randomceladm01 port 22: Connection refused
-bash-4.1# ssh randomceladm02
ssh: connect to host randomceladm02 port 22: Connection refused
-bash-4.1# ssh randomceladm03
ssh: connect to host randomceladm03 port 22: Connection refused

-bash-4.1# exachk -unlockcells all
Enter exacli user name: celluser
Is EXACLI password same on all Storage Servers?[y/n][y] y
Enter password for EXACLI user celluser to unlock Storage Server 192.168.178.225:
.  .  .  .  .  .  .  .  .  .  . 
Storage cell 192.168.178.225 successfully unlocked
Storage cell 192.168.178.226 successfully unlocked
Storage cell 192.168.178.227 successfully unlocked
-bash-4.1# ssh randomceladm03
Last login: Tue Mar  6 12:32:36 2018 from randomadm01.us.oracle.com
-bash-4.1# ssh randomceladm02
Last login: Tue Mar  6 12:32:09 2018 from randomadm01.us.oracle.com
-bash-4.1# ssh randomceladm01
Last login: Tue Mar  6 12:18:57 2018 from randomadm01.us.oracle.com

-bash-4.1# exacli -c celluser@randomceladm01
Password: ************
exacli celluser@randomceladm01> list cell attributes accessLevelPerm,accessLevelTemp
remoteLoginDisabled ((accesslevel=remoteLoginEnabled,starttime=2018-03-06T13:49:15-08:00,
endtime=2018-03-06T14:39:15-08:00,duration=50m,reason=Running Exachk))

As can be seen from the example, Oracle EXAchk implements a temporary window 
with a default expiration time of 50 minutes, to cover the period that Oracle EXAchk 
may be executing on the storage server.  
In normal operation, this temporary window is closed with "-lockcells" during the exachk cleanup routine.  
If exachk is blocked from the cleanup routine, say because of "kill -9", 
the temporary window will expire in it's own good time.

The following example shows the typical Oracle EXAchk execution sequence starting with the storage servers locked.  
You can see by the commands at the end that "remoteLoginDisabled" is still set and there is no temporary window:

exachk -c X4-2 -profile storage
...
...
Copying plug-ins
. .
Enter exacli user name: celluser
Is EXACLI password same on all Storage Servers?[y/n][y]
Enter password for EXACLI user celluser to unlock Storage Server 192.168.178.225:
.  .  .  .  .  .  .  .  .  .  .  .  . 
Node randomcel01 is configured for ssh user equivalency for root user
Node randomcel02 is configured for ssh user equivalency for root user
Node randomcel03 is configured for ssh user equivalency for root user
.  .  .  .  .  .  .  .  .  .  .  .  .
...
...
Starting to run root privileged commands in background on STORAGE SERVER randomcel01 (192.168.178.225)
Starting to run root privileged commands in background on STORAGE SERVER randomcel02 (192.168.178.226)
Starting to run root privileged commands in background on STORAGE SERVER randomcel03 (192.168.178.227)
Collections from STORAGE SERVER:
------------------------------------------------------------
Collecting - Exadata Critical Issue EX10
...
...
Detailed report (html) -  /root/vern_wagman/exachk_122014/production/lock_doc/exachk_randomclient01_030618_140319/exachk_randomclient01_030618_140319.html
UPLOAD [if required] - /root/vern_wagman/exachk_122014/production/lock_doc/exachk_randomclient01_030618_140319.zip

-bash-4.1# ssh randomceladm01
ssh: connect to host randomceladm01 port 22: Connection refused
-bash-4.1# exacli -c celluser@randomceladm01
Password: ************
exacli celluser@randomceladm01> list cell attributes accessLevelPerm,accessLevelTemp
         remoteLoginDisabled

1.4.2 Handling of Root Passwords

Handling of root passwords depends on whether you have installed the Expect utility.

Expect automates interactive applications such as Telnet, FTP, passwd, fsck, rlogin, tip, and so on.

To handle root passwords:

1. If you have installed the Expect utility, then specify the root password when you run the compliance checks for the first time.

   The Expect utility stores the password and uses the stored password for subsequent sessions.

   The Expect utility prompts you to check if the root password is same for all the remote components such as databases, switches, and so on.

2. Specify the password only once if you have configured the same root password for all the components.

   If root password is not same for all the components, then the Expect utility prompts you to validate root password every time you run the compliance checks.

   If you enter the password incorrectly or the password is changed between the time it is entered and used, then Oracle Autonomous Health Framework:

   • Notifies you

   • Skips relevant checks

3. Run the compliance checks after resolving the issues.

   If Oracle Autonomous Health Framework skips any of the compliance checks, then the tools log details about the skipped checks in the report output.

1.4.3 Deciding Which User Should Run Oracle ORAchk and Oracle EXAchk

Run compliance checks as root. Also, run compliance checks as the Oracle Database home owner or the Oracle Grid Infrastructure home owner.

Most compliance checks do not require root access. However, you need root privileges to run a subset of compliance checks.

To run root privilege checks, Oracle ORAchk uses the script root_orachk.sh and Oracle EXAchk uses the script root_exachk.sh.

By default, the root_orachk.sh and root_exachk.sh scripts are created in the $HOME directory used by Oracle ORAchk and Oracle EXAchk. Change the directory by setting the environment variable RAT_ROOT_SH_DIR.

Specify a location for sudo remote access as follows:

export RAT_ROOT_SH_DIR=/mylocation

Add an entry in the /etc/sudoers as follows:

oracle ALL=(root) NOPASSWD:/mylocation/root_orachk.sh

For security reasons, create the root scripts outside of the standard temporary directory in a custom directory.

To decide which user to run Oracle ORAchk and Oracle EXAchk:

1. Specify the custom directory using the RAT_ROOT_SH_DIR  environment variable.

   export RAT_ROOT_SH_DIR=/orahome/oradb/

2. Specify a location for sudo  remote access.

   export RAT_ROOT_SH_DIR=/mylocation

3. Add an entry in the /etc/sudoers  file.

   oracle ALL=(root) NOPASSWD:/mylocation/root_orachk.sh

   Note:

   Specify full paths for the entries in the /etc/sudoers  file. Do not use environment variables.

4. (recommended) Run Oracle ORAchk and Oracle EXAchk as root.

   Use root user credentials to run Oracle ORAchk and Oracle EXAchk.

   The Oracle ORAchk and Oracle EXAchk processes that run as root, perform user lookups for the users who own the Oracle Database home and Oracle Grid Infrastructure home. If root access is not required, then the Oracle ORAchk and Oracle EXAchk processes use the su command to run compliance checks as the applicable Oracle Database home user or Oracle Grid Infrastructure home user. Accounts with lower privileges cannot have elevated access to run compliance checks that require root access.

   Running compliance checks as root has advantages in role-separated environments or environments with more restrictive security.

5. Run Oracle ORAchk and Oracle EXAchk as Oracle Database home owner or Oracle Grid Infrastructure home owner:

   Use Oracle Database home owner or Oracle Grid Infrastructure home owner credentials to run Oracle ORAchk and Oracle EXAchk.

   The user who runs Oracle ORAchk and Oracle EXAchk must have elevated access as root to run compliance checks that need root access.

   Running compliance checks as Oracle Database home owner or Oracle Grid Infrastructure home owner requires multiple runs in role-separated environments. More restrictive security requirements do not permit elevated access.

   There are several other options:

   • Skip the checks that require root access.

   • Specify the root  user ID and password when prompted.

   • Configure sudo.

     If you are using sudo, then add an entry for the root script, located in $HOME in the /etc/sudoers file that corresponds to the user who is running the compliance checks.

     To determine what $HOME is set to, run the echo $HOME  command.

     For example:

     user ALL=(root) NOPASSWD:/root/root_orachk.sh

     user ALL=(root) NOPASSWD:/root/root_exachk.sh

   • Pre-configure passwordless SSH connectivity.

1.4.4 Data Entry Terminal Considerations

Use any supported UNIX and Linux terminal type (character mode terminal, ILOM, VNC server) to run Oracle Autonomous Health Framework.

Respond to the prompts during the interactive run, or while configuring the daemon.

Each terminal type has advantages and disadvantages. The effect of a dropped network connection varies based on the terminal type used.

For example, in an interactive run using a character mode terminal, if all the prompts are answered prior to the network drop, then the running process completes successfully even if the network connection drops. If the network connection drops before all the input prompts are answered, then all the running processes hang. Clean up the hung processes manually when the network connection is restored.

Using a remote connection to a VNC server running on the database where Oracle Autonomous Health Framework is running minimizes the network drop interruptions.

If you use accessibility software or devices that prevent the use of a VNC server, and cause network failures, then you must work with your network team and system administrator to determine the root cause and adjust the environment as required.

For example, an accessibility aid might insert a suspension and restart the interactive process that runs Oracle Autonomous Health Framework. If this causes an operating system timeout due to terminal inactivity, then increase the inactivity timeouts of the environment before running the commands.

The timeout caused by an assistive tool at the operating system level due to terminal inactivity is not specific to Oracle Autonomous Health Framework. The timeout could happen to any process that assistive technology manages.

1.4.5 Running Oracle EXAchk on Oracle Exadata and Zero Data Loss Recovery Appliance

Review the list of additional prerequisites for running Oracle EXAchk on Oracle Exadata and Zero Data Loss Recovery Appliance.

• Storage Servers
  
• InfiniBand Switches
  

1.4.5.1 Storage Servers

On the database, if you configure passwordless SSH equivalency for the user that launched Oracle Autonomous Health Framework to the root  user on each storage server, then Oracle Autonomous Health Framework uses SSH equivalency credentials to complete the storage server checks.

You can run Oracle Autonomous Health Framework from the Oracle Exadata storage server, if there is no SSH connectivity from the database to the storage server.

To lock and unlock cells, use the –unlockcells and –lockcells options for Oracle Exadata and Zero Data Loss Recovery Appliance.

exachk -unlockcells all | -cells [comma-delimited list of cell names or cell IPs]

exachk -lockcells all | -cells [comma-delimited list of cell names or cell IPs]

Once the cells have been unlocked, if they are not locked again within the default timeout of 50 minutes, then they will be automatically locked again. You can adjust the timeout period using the RAT_CELLUNLOCK_TIMEOUT environment variable.

For example to change the timeout to 2 hours:

export RAT_CELLUNLOCK_TIMEOUT=120m

exachk -unlockcells all

1.4.5.2 InfiniBand Switches

On the database, if you configure passwordless SSH equivalency for the user that launched Oracle Autonomous Health Framework to the nm2user user on each InfiniBand switch, then Oracle Autonomous Health Framework uses SSH equivalency credentials to complete the InfiniBand switch checks.

If you have not configured passwordless SSH equivalency, then Oracle Autonomous Health Framework prompts you for the nm2user user password on each of the InfiniBand switches.

1.4.6 Running Oracle Autonomous Health Framework in Non-English Environments

Set globalization environment variables to run Oracle Autonomous Health Framework in non-English environments.

Oracle Autonomous Health Framework supports only English language. However, you can run Oracle Autonomous Health Framework by setting the globalization environment variables.

1. To run Oracle Autonomous Health Framework in a non-English environment, set the environment variable NLS_LANG to AMERICAN_AMERICA.[NLS_CHARACTERSET].

   For example:

   export NLS_LANG=AMERICAN_AMERICA.JA16SJISTILDE

   For more information on setting globalization environment variables, see Setting Up a Globalization Support Environment in the Oracle Database Globalization Support Guide.