9 Cloud Native Environment (CNE) Security Recommendations and Procedures

After installation, audit the OC-CNE security system stance before placing the system into service. This primarily consists of changing credentials and sequestering SSH keys to trusted servers. The following table lists all the credentials that need to be checked, changed and retained:

Credential Name	Deployment	Type	Associated Resource	Initial Setting	Credential Rotation
TOR Switch	Bare Metal Only	username and password	Cisco Top or Rack Switch	username and password from PreFlight Checklist	Reset post-install
Enclosure Switch	Bare Metal Only	username and password	HP Enclosure Switch	username and password from PreFlight Checklist	Reset post-install
OA Admin	Bare Metal Only	username and password	On-board Administrator Console	username and password from PreFlight Checklist	Reset post-install
ILO Admin	Bare Metal Only	username and password	HP Integrated Lights Out Manger	username and password from PreFlight Checklist	Reset post-install
Server Super User (root)	All	username and password	Server Super User	Set to well-known Oracle default during server installation	Reset post-install
Server Admin User SSH	All	SSH Key Pair	Server Admin User	Key Pair generated at install time	Can rotate keys at any time; key distribution manual procedure

If factory or Oracle defaults were used for any of these credentials, it must be changed prior to placing the system into operation. The customer must store these credentials in a safe and secure way offsite. It is recommended that the customer must plan a regular schedule for updating (rotating) these credentials. Specific procedures and recommendations for OC-CNE credential management are provided below.

The procedures are:

• Network Security Recommendations and Procedures
• Credential Management Procedures
  • Procedure 1: Setting Top Of Rack Switch Credentials
  • Procedure 2: Setting Enclosure Switch Credentials
• Hosting Environment Security Recommendations and Procedures
• Credential Management Procedures
  • Procedure 1: Setting HP Onboard Administrator (OA) Credentials.
  • Procedure 2: Setting HP Integrated Lights Out Manger (ILO) Credentials
  • Procedure 3: Setting Root Passwords for All Cluster Nodes
  • Procedure 4: Updating admusr SSH Keys for All Cluster Nodes
• Update the bastion host keys
• General Security Administration Recommendations and Procedures
  • Password Policy Administration Procedures
  • SSHD Policy Administration Procedures
  • Auditd Policy Administration Procedures

1.1.Network Security Recommendations and Procedures

Recommendation: Review and Follow TOR installation procedures

The OC-CNE on-premise installation guide provides detailed procedures on how to configure the TOR switches and configure them for remote monitoring. Deviations from the standard installation time configurations are not recommended.

Credential Management Procedures

Procedure 1: Setting Top Of Rack Switch Credentials

This procedure is used to set the credentials on the Cisco TOR switch as deployed with the bare metal deployment option. Steps for creating and deleting accounts and for setting account passwords is given below.

1. Login to the TOR switch (from the bastion host):
   $ ssh <username>@<switchIP address> User Access Verification
   Password: <password>
   Cisco Nexus Operating System (NX-OS) Software
   TAC support: www.cisco.com/tac
   <switchname>#
2. Change the password for <username>:

   # configure

   Enter configuration commands, one per line. End with CNTL/Z.

   (config)# username <username> password<newpassword>

   (config)#exit

3. Create a new user (if desired):
   # configureEnter configuration commands, one per line. End with CNTL/Z.
   (config)# username <newusername> password <newpassword> role [network-operator|network-admin|vdc-admin|vdc-operator] (config)#exit
4. Verify the account changes by exiting the ssh session (type exit) and repeat step 1.
   # exit
   Connection to <switchIP address> closed.

   $ ssh <newusername>@<switchIP address>  

   User Access Verification Password: <newpassword>
   Cisco Nexus Operating System (NX-OS) SoftwareTAC support: www.cisco.com/tac
   <switchname>#
5. Delete an unrequired user account:
   # configureEnter configuration commands, one per line. End with CNTL/Z.

   (config)# no username <username>

   (config)#exit
6. Change the enable secret:

   (config)# enable secret <newenablepassword>

   (config)# exit
7. Save the configuration changes:
   # copy running-config startup-config
   100%
   Copy complete, now saving to disk (please wait)...
   Copy complete.

Note:

Recommendation: Change TOR passwords before placing site into service. The TOR switch credentials show the changed prior to placing the site into service.

Recommendation: Use Strong Passwords.The Network Administrator must choose complex TOR Switch passwords as per their organization's security guidelines.

Procedure 2: Setting Enclosure Switch Credentials

This procedure is used to set the credentials on the HP enclosure switch as deployed with the bare metal deployment option. Steps for creating and deleting accounts and for setting account passwords is given below. For additional information, refer to: HP commands to configure enclosure switch username and password.

Setting Enclosure Switch Credentials

1. Login to the HP enclosure switch (from the bastion host): $ ssh <username>@< switchIP address>

   Copyright (c)2010-2017Hewlett Packard Enterprise Development LP ** Without the owner's prior written consent, 
   ** no decompiling or reverse-engineering shall be allowed. 
   <switchname> 
   <switchname> 
   sysSystem View:returnto User View with Ctrl+Z. 

2. Change the password for the current username:

   [switchname]local-user <username>class <currentclass> 
   [switchname-luser-manage-<username>]password simple <newpassword> 
   [switchname-luser-manage-<username>]quit

3. Create a new user account:

   [switchname]local-user <newusername>class[manage|network] 

   New local user added

   [switchname-luser-manage-<newusername>]password simple <newpassword>
   [switchname-luser-manage-<newusername>]quit

4. Delete the user account that is not required:

   [switchname]undo local-user <username>class <currentclass>

5. Delete the user account that is not required:

   [switchname]undo local-user <username>class <currentclass>

6. Save the configuration changes:

   [switchname]save 
   The current configuration will be written to the device. Are you sure? [Y/N]: y 
   Please input the file name(*.cfg)[flash:/<filename>] 
   (To leave the existing filename unchanged, press the enter key): 
   flash:/<filename> exists, overwrite? [Y/N]: yValidating file. Please wait... 
   Saved the current configuration to mainboard device successfully. 
   Slot1: 
   Save next configuration file successfully. 
   [switchname] 
   [switchname]save 
   The current configuration will be written to the device. Are you sure? [Y/N]: y 
   Please input the file name(*.cfg)[flash:/<filename>] 
   (To leave the existing filename unchanged, press the enter key): 
   flash:/<filename> exists, overwrite? [Y/N]: yValidating file. Please wait... 
   Saved the current configuration to mainboard device successfully. 
   Slot1: 
   Save next configuration file successfully. 
   [switchname] 
    
    

Note:

Recommendation: Set Enclosure Switch Credentials before Placing Into Service

The HP Enclosure switch credentials show be changed prior to placing the site into service.

Recommendation: Use Strong Passwords

The Network Administrator must choose complex Enclosure Switch passwords as per their organization's security guidelines.

1.2 Hosting Environment Security Recommendations and Procedures

The best way to keep your CNE environment secure is to keep it up-to-date. New OC-CNE releases are typically released every 2 months. The OC-CNE upgrade is not service affecting and will typically install newer versions of:

• Host OSs
• Kubernetes and associated containers
• DB-Tier binaries
• Common service containers

The upgrade process ensures that the uplifts do not affect active service. Refer Cloud Native Environment (OC-CNE) Upgrade Guide for more details.

Repository Management Recommendations

System Update (YUM) Recommendations

Recommendation: Keep central yum repositories up to date.

Keep central repositories up-to date with latest yum packages; yum updates are performed on-site whenever a fresh install or upgrade is performed. An up-to date yum repository will help ensure that fixes for all publish vulnerabilities are applied.

Docker Repository Recommendations

Recommendation: Scan docker image repositories regularly.

Scan your docker image repositories regularly using a tool such as clair or anchore-engine. All images are scanned and vulnerabilities assessed at product development time, but new exploits /vulnerabilities may be reported/fixed later. Scan tools typically use a database of known vulnerabilities - refer to tool vendor for instructions on creating off-line (internet isolated) vulnerability databases.

1.3 Credential Management Procedures

Procedure 1: Setting HP Onboard Administrator (OA) Credentials.

This procedure is used to set the credentials on the HP Onboard Administrator as deployed with the bare metal deployment option. Steps for creating and deleting accounts and for setting account passwords is shown. For additional information, please refer to: HP commands to configure OA username and password.

1. Login to the OA:

   $ ssh <username>@<OA address>

   WARNING: This is a private system. Do not attempt to login unless you are anauthorized user. Any authorized or unauthorized access and use may be monitored and can result in criminal or civil prosecution under applicable law .

   
   Firmware Version: 4.85 
   Built:04/06/2018@06:14OA 
    Bay Number:1 
   OA Role: Active 
   <username>@<OA address>'s password: <password>  
   HPE BladeSystem Onboard Administrator 
   (C) Copyright 2006-2018 Hewlett Packard Enterprise Development LP 
   Type 'HELP' to display a list of valid commands. 
   Type 'HELP <command>' to display detailed information about a specific command. 
   Type 'HELP HELP' to display more detailed information about the help system. 
   OA-A45D36FD5FB1> 

2. Change the current password:

   
   OA-A45D36FD5FB1> set password <newpassword> 
   Changed password for the"<username>"user account. 
   OA-A45D36FD5FB1> 

3. Add new user:

   OA-A45D36FD5FB1> add user <newusername> 
   New Password: <newpassword> 
   Confirm : <newpassword> 
   User"<newusername>"created. 
   You may set user privileges with the 'SET USER ACCESS' and 'ASSIGN' commands. 
   OA-A45D36FD5FB1> set user access <newusername> [ADMINISTRATOR|OPERATOR|USER] "<newusername>" 
   has been given [administrator|operator|user] level privileges. 

4. Assign full access to the enclosure for the user:

   
   OA-A45D36FD5FB1> assign server all <newusername> 
   <newusername> has been granted access to the valid requested bay (sOA-A45D36FD5FB1> assign interconnect all <newusername> 
   <newusername> has been granted access to the valid requested bay(s)OA-A45D36FD5FB1> assign oa <newusername> 
   <newusername> has been granted access to the OA. 
   

5. Verify the new account

   OA-A45D36FD5FB1> exit 
   Connection to <OA address> closed.
   [bastion host]# ssh <newusername>@<OA address> 
   WARNING: This is a private system. Do not attempt to login unless you are unauthorized user. 
   Any authorized or unauthorized access and use may be monitored and can result in criminal or 
   civil prosecution under applicable law. 
   Firmware Version : 4.85 
   Built : 04/06/2018 @ 06:14 
   OA Bay Number : 1 
   OA Role : Active 
   <newusername>@<OA address>'s password: <newpassword>  
   HPE BladeSystem Onboard Administrator 
   (C) Copyright 2006-2018 Hewlett Packard Enterprise Development LP 
   Type 'HELP' to display a list of valid commands. 
   Type 'HELP <command>' to display detailed information about a specific command. 
   Type 'HELP HELP' to display more detailed information about the help system. OA-A45D36FD5FB1> 

6. Delete the user account:

   OA-A45D36FD5FB1> remove user <username> 
   Entering anything other than 'YES' will result in the command not executing. 
   Are you sure you want to remove testuser1? yes 
   User"<username>"removed.
   

Procedure 2: Setting HP Integrated Lights Out Manger (ILO) Credentials

This procedure is used to set the credentials on the HP Integrated Lights Out Managers as deployed with the bare metal deployment option. Steps for creating and deleting accounts and for setting account passwords is shown.

1. Login to the iLO:

   $ ssh <username>@<iLO address> 

   <username>@<iLO address>'s password: <password>User:<username> 
   logged-in to ...(<iLO address> / <ipv6 address>)
   iLO Advanced2.61at Jul272018 
   Server Name: <server name> 
   Server Power: On 
   </>hpiLO->  

2. Change the current password:

   </>hpiLO-> set /map1/accounts1/ <username> password= <newpassword>  
   status=0 
   status_tag=COMMAND COMPLETED 
   Tue Aug2013:27:082019 
   </>hpiLO-> 

3. Create a new user account:

   </>hpiLO-> create /map1/accounts1 username= <newusername> password= <newpassword> 
   group=admin,config,oemHP_rc,oemHP_power,oemHP_vm 
   status=0 
   status_tag=COMMAND COMPLETED 
   Tue Aug2013:47:562019 
   User added successfully.

4. Verify the new user account:

   
   </>hpiLO-> exit 
   status=0 
   status_tag=COMMAND COMPLETED 
   Tue Aug2013:30:522019CLI session stoppedReceived disconnect from <iLO address> port22:11: Client Disconnect 
   Disconnected from <iLO address> port22 
   [bastion host]# ssh <newusername>@<iLO address> 
   <newusername>@<iLO address>'s password: <newpassword> 
   User:<newusername> logged-in to ...(<iLO address> / <ipv6 address>) 
   iLO Advanced2.61at Jul272018 
   Server Name: <server name>Server 
    Power: On</>hpiLO->

5. Delete an unneeded account:

   
   </>hpiLO-> delete /map1/accounts1/ <username> 
   status=0 
   status_tag=COMMAND COMPLETED 
   Tue Aug2013:59:042019 
   User deleted successfully. 

Procedure 3: Setting Root Passwords for All Cluster Nodes

The procedure to reset the root account requires that the administrator login to each and every server.

To reset the root account, for each and every server in the cluster perform the following steps:

1. Login to the next server:

   $ ssh admusr@ <cluster server IP>

2. Perform the root password change:

   $ sudo passwd root

   New password: <new password>
   Retype new password: <new password>
   Retype new password:<new password> 

3. Repeat steps 1 - 2 for each and every server in the cluster.

Note:

The administrator (admusr) account is provided without a usable password hash. Thus requiring the use of SSH keys to access the account. The SUDO users access is configured without the requirement of a password. If you would like to enable the SUDO passwords for the administrator, you also need to assign a password to the administrator account using a procedure very similar to the one outlined above.

Procedure 4: Updating admusr SSH Keys for All Cluster Nodes

There are two sets of SSH keys used in a deployed cluster - the key used to access the bastion host, and the key used to access the cluster servers. These key-pairs are generated at install time and are only usable on the cluster they were generated for. The public key portion of the bastion host key pair is typically provided to administrators who will manage the cluster. The key pair used to access the cluster servers should kept local to the cluster:

Table 9-1 Updating admusr SSH Keys

Key Pair Name	Public Key Distribution	Private Key Distribution
Bastion Host	Place copy in the authorized_keys file on the bastion host	Cluster Admin - place in the cluster admin key agent (e.g., ssh-agent or pageant) external to the cluster. Do not copy to any host on the cluster.
Cluster Hosts	Place a copy in the authorized_keys files on each and every cluster host; do not configure on the bastion host.	Bastion Host - ~admusr/.ssh directory. This will be used when performing orchestration activities (install / upgrade).

To replace either of these key pairs starts with an openssh request to generate a new keypair:

ssh-keygen -b 4096 -t rsa -C "New SSH Key" -f
      .ssh/new_occne_id_rsa -q -N ""

This command generates the following key pair:

Key Name	Purpose
new_occne_id_rsa	The private key
new_occne_id_rsa.pub	The public key

Update the bastion host keys

1. Login to the bastion host and generate a new key pair using the ssh-keygen command show above.

   $ ssh-keygen -b 4096 -t rsa -C "New
               Bastion Key" -f ~/.ssh/new_occne_id_rsa -q -N
       ""

2. Copy the private key portion of the key off cluster and make it available to your ssh agent of choice or store it in the .ssh directory of your client machine. See instructions for your specific SSH client (e.g., putty or openssh)

3. Add the new public key to the authorized key file on the bastion host:

   $ cat ~/.ssh/new_occne_id_rsa.pub >>
       ~/.ssh/authorized_keys

4. Confirm the permissions of the .ssh directory and files:

   $ ls -la ~/.ssh
   total 32
   drwx------. 2 admusr admusr 4096 Feb 25 15:48 .
   drwx------. 42 admusr admusr4096 Feb 24 15:14 ..
   -rw-------. 1 admusr admusr 796 Jan 28 14:43 authorized_keys
   -rw-------. 2 admusr admusr 545 Feb 12 13:58 config
   -rw-------. 1 admusr admusr 3239 Feb 25 15:48 new_occne_id_rsa
   -rw-r--r–. 1 admusr admusr 737 Feb 25 15:48 new_occne_id_rsa.pub

   In general, the .ssh directory should be mode 700 and the files under that directory should be mode 600

5. Confirm that the new key works; remove the old key from your ssh client's agent (see instructions for your client), and confirm that you can still login.

6. Assuming that you were able to login using the new key pair, remove the old key pair from the authorized_keys file using your favorite editor.

   In general, the authorized_keys file should at this point have two keys in it - the old one and the new one. The new one should be at the bottom.

General Security Administration Recommendations and Procedures

Note:

Recommendation: Record configuration changes

Note that in a disaster recovery scenario, Oracle provided procedures will only restore base system behavior (they will not include restoration of an special configurations or tweaks). We recommend that all post-delivery customization be logged or automated using tools such as Ansible.

Password Policy Administration Procedures

In general, the host environments use a user account named admusr which is not configured with a password; the only way to access this account is using SSH keys. We recommend using SSH keys rather than passwords for all non-root accounts. The root account cannot be accessed via ssh; the only access is via the console. For this account, we recommend setting a password and storing it off-site to be used only for break-glass console access to the host.

User Administration Recommendations

Customers may want to create additional accounts to manage separate concerns (Example: a dbadmin account, a k8sadmin account, etc). This can be done using normal linux user administration procedures.

SSHD Policy Administration Procedures

Customers may want to create augment the standard sshd configuration to perform additional hardening; this can be done using normal linux ssh administration procedures. Note that in a disaster recovery scenario, Oracle provided procedures will only restore base system behavior (they will not include restoration of an special configurations or tweaks).

Note:

Recommendation: Review changes with Oracle Support

We recommend reviewing any planned changes to sshd configuration with your Oracle Support contact. Improper sshd configuration can either open the system up to attacks or prevent proper system operation.

Auditd Policy Administration Procedures

Customers may want to augment the standard auditd configuration to perform additional monitoring; this can be done using normal linux auditd administration procedures. Place all customizations in a seperate file in the /etc/audit/rules.d directory; do not modify any of the other existing audit configuration files