9 Cloud Native Environment (CNE) Security Recommendations and Procedures

After installation, audit the OC-CNE security system stance before placing the system into service. This primarily consists of changing credentials and sequestering SSH keys to trusted servers. The following table lists all the credentials that need to be checked, changed and retained:

Credential Name Deployment Type Associated Resource Initial Setting Credential Rotation
TOR Switch Bare Metal Only username and password Cisco Top or Rack Switch username and password from PreFlight Checklist Reset post-install
Enclosure Switch Bare Metal Only username and password HP Enclosure Switch username and password from PreFlight Checklist Reset post-install
OA Admin Bare Metal Only username and password On-board Administrator Console username and password from PreFlight Checklist Reset post-install
ILO Admin Bare Metal Only username and password HP Integrated Lights Out Manger username and password from PreFlight Checklist Reset post-install
Server Super User (root) All username and password Server Super User Set to well-known Oracle default during server installation Reset post-install
Server Admin User SSH All SSH Key Pair Server Admin User Key Pair generated at install time Can rotate keys at any time; key distribution manual procedure

If factory or Oracle defaults were used for any of these credentials, it must be changed prior to placing the system into operation. The customer must store these credentials in a safe and secure way offsite. It is recommended that the customer must plan a regular schedule for updating (rotating) these credentials. Specific procedures and recommendations for OC-CNE credential management are provided below.

1.1.Network Security Recommendations and Procedures

Recommendation: Review and Follow TOR installation procedures

The OC-CNE on-premise installation guide provides detailed procedures on how to configure the TOR switches and configure them for remote monitoring. Deviations from the standard installation time configurations are not recommended.

Credential Management Procedures

Procedure 1: Setting Top Of Rack Switch Credentials

This procedure is used to set the credentials on the Cisco TOR switch as deployed with the bare metal deployment option. Steps for creating and deleting accounts and for setting account passwords is given below.

Table 9-1 Setting Top Of Rack Switch Credentials

Step No. Description Est time
1. Login to the TOR switch (from the bastion host):

$ ssh <username>@<switchIP address> User Access Verification

Password: <password>

Cisco Nexus Operating System (NX-OS) Software

TAC support: www.cisco.com/tac

<switchname>#
1m
2. Change the password for <username>:

# configure

Enter configuration commands, one per line. End with CNTL/Z.

(config)# username <username> password<newpassword>

(config)#exit

1m
3. Create a new user (if desired):

# configureEnter configuration commands, one per line. End with CNTL/Z. (config)# username <newusername> password <newpassword> role [network-operator|network-admin|vdc-admin|vdc-operator] (config)#exit
1m
4. Verify the account changes by exiting the ssh session (type exit) and repeat step 1.

# exit

Connection to <switchIP address> closed.

$ $ ssh <newusername>@<switchIP address>

User Access Verification Password: <newpassword>

Cisco Nexus Operating System (NX-OS) SoftwareTAC support: www.cisco.com/tac

......

<switchname>#
1m
5. Delete an unrequired user account:

# configureEnter configuration commands, one per line. End with CNTL/Z.

(config)# no username <username>

(config)#exit
1m
6. Change the enable secret:

(config)# enable secret <newenablepassword>

(config)# exit
1m
7. Save the configuration changes: # copy running-config startup-config

100%

Copy complete, now saving to disk (please wait)...

Copy complete.
1m

Note:

Recommendation: Change TOR passwords before placing site into service. The TOR switch credentials show the changed prior to placing the site into service.

Recommendation: Use Strong Passwords.The Network Administrator must choose complex TOR Switch passwords as per their organization's security guidelines.

Procedure 2: Setting Enclosure Switch Credentials

This procedure is used to set the credentials on the HP enclosure switch as deployed with the bare metal deployment option. Steps for creating and deleting accounts and for setting account passwords is given below. For additional information, refer to: HP commands to configure enclosure switch username and password.

Table 9-2 Setting Enclosure Switch Credentials

Step Description Est. Time
1. Login to the HP enclosure switch (from the bastion host): $ ssh <username>@< switchIP address>

Copyright (c)2010-2017Hewlett Packard Enterprise Development LP ** Without the owner's prior written consent, ** no decompiling or reverse-engineering shall be allowed.

<switchname>

<switchname>

sysSystem View:returnto User View with Ctrl+Z.

1m
2. Change the password for the current username: [switchname]local-user <username>class <currentclass>

[switchname-luser-manage-<username>]password simple <newpassword>

[switchname-luser-manage-<username>]quit

1m
3. Create a new user account: [switchname]local-user <newusername>class[manage|network]

New local user added

[switchname-luser-manage-<newusername>]password simple <newpassword>[switchname-luser-manage-<newusername>]quit
1m
4. Verify the account changes by exiting the ssh session (type exit) and repeat step 1. <switchname> quitConnection to <switchIP address>closed.

$

$ ssh <newusername>@< switchIP address> <newusername>@<switchIP address>'s password: <newpassword>

Copyright (c)2010-2017Hewlett Packard Enterprise Development LP *

* Without the owner's prior written consent, *

* no decompiling or reverse-engineering shall be allowed.

<switchname>

<switchname> sys

System View:returnto User View with Ctrl+Z.

1m
5. Delete the user account that is not required: [switchname]undo local-user <username>class <currentclass>

1m
6. Save the configuration changes:

[switchname]save

The current configuration will be written to the device. Are you sure? [Y/N]: y

Please input the file name(*.cfg)[flash:/<filename>]

(To leave the existing filename unchanged, press the enter key):

flash:/<filename> exists, overwrite? [Y/N]: yValidating file. Please wait...

Saved the current configuration to mainboard device successfully.

Slot1:

Save next configuration file successfully.

[switchname]

1

Note:

Recommendation: Set Enclosure Switch Credentials before Placing Into Service

The HP Enclosure switch credentials show be changed prior to placing the site into service.

Recommendation: Use Strong Passwords

The Network Administrator must choose complex Enclosure Switch passwords as per their organization's security guidelines.

1.2 Hosting Environment Security Recommendations and Procedures

The best way to keep your CNE environment secure is to keep it up-to-date. New OC-CNE releases are typically released every 2 months. The OC-CNE upgrade is not service affecting and will typically install newer versions of:

  • Host OSs
  • Kubernetes and associated containers
  • DB-Tier binaries
  • Common service containers

The upgrade process ensures that the uplifts do not affect active service. Refer Cloud Native Environment (OC-CNE) Upgrade Guide for more details.

Repository Management Recommendations

System Update (YUM) Recommendations

Recommendation: Keep central yum repositories up to date.

Keep central repositories up-to date with latest yum packages; yum updates are performed on-site whenever a fresh install or upgrade is performed. An up-to date yum repository will help ensure that fixes for all publish vulnerabilities are applied.

Docker Repository Recommendations

Recommendation: Scan docker image repositories regularly.

Scan your docker image repositories regularly using a tool such as clair or anchore-engine. All images are scanned and vulnerabilities assessed at product development time, but new exploits /vulnerabilities may be reported/fixed later. Scan tools typically use a database of known vulnerabilities - refer to tool vendor for instructions on creating off-line (internet isolated) vulnerability databases.

1.3 Credential Management Procedures

Procedure 1: Setting HP Onboard Administrator (OA) Credentials.

This procedure is used to set the credentials on the HP Onboard Administrator as deployed with the bare metal deployment option. Steps for creating and deleting accounts and for setting account passwords is shown. For additional information, please refer to: HP commands to configure OA username and password.

Table 9-3 Setting HP Onboard Administrator (OA) Credentials

Step Description Est Time
1 Login to the OA:

$ ssh <username>@<OA address>

WARNING: This is a private system. Do not attempt to login unless you are anauthorized user. Any authorized or unauthorized access and use may be moni-tored and can result in criminal or civil prosecution under applicable law

Firmware Version: 4.85

Built:04/06/2018@06:14OA

Bay Number:1

OA Role: Active

<username>@<OA address>'s password: <password>

HPE BladeSystem Onboard Administrator

(C) Copyright 2006-2018 Hewlett Packard Enterprise Development LP

Type 'HELP' to display a list of valid commands.

Type 'HELP <command>' to display detailed information about a specific command.

Type 'HELP HELP' to display more detailed information about the help system.

OA-A45D36FD5FB1>

1m
2 Change the current password:

OA-A45D36FD5FB1> set password <newpassword>

Changed password for the"<username>"user account.

OA-A45D36FD5FB1>

1m
3 Add new user:

OA-A45D36FD5FB1> add user <newusername>

New Password: <newpassword>

Confirm : <newpassword>

User"<newusername>"created.

You may set user privileges with the 'SET USER ACCESS' and 'ASSIGN' commands.

OA-A45D36FD5FB1> set user access <newusername> [ADMINISTRATOR|OPERATOR|USER] "<newusername>" has been given [administrator|operator|user] level privileges.

1m
4 Assign full access to the enclosure for the user:

OA-A45D36FD5FB1> assign server all <newusername>

<newusername> has been granted access to the valid requested bay (sOA-A45D36FD5FB1> assign interconnect all <newusername>

<newusername> has been granted access to the valid requested bay(s)OA-A45D36FD5FB1> assign oa <newusername>

<newusername> has been granted access to the OA.

1m
5 Verify the new account:

OA-A45D36FD5FB1> exit

Connection to <OA address> closed.

[bastion host]# ssh <newusername>@<OA address>

WARNING: This is a private system. Do not attempt to login unless you are unauthorized user. Any authorized or unauthorized access and use may be monitored and can result in criminal or civil prosecution under applicable law.

Firmware Version : 4.85

Built : 04/06/2018 @ 06:14

OA Bay Number : 1

OA Role : Active

<newusername>@<OA address>'s password: <newpassword>

HPE BladeSystem Onboard Administrator

(C) Copyright 2006-2018 Hewlett Packard Enterprise Development LP

Type 'HELP' to display a list of valid commands.

Type 'HELP <command>' to display detailed information about a specific command.

Type 'HELP HELP' to display more detailed information about the help system. OA-A45D36FD5FB1>

1m
6 Delete an unneeded user account:

OA-A45D36FD5FB1> remove user <username>

Entering anything other than 'YES' will result in the command not executing.

Are you sure you want to remove testuser1? yes

User"<username>"removed.

1m

Procedure 2: Setting HP Integrated Lights Out Manger (ILO) Credentials

This procedure is used to set the credentials on the HP Integrated Lights Out Managers as deployed with the bare metal deployment option. Steps for creating and deleting accounts and for setting account passwords is shown.

Table 9-4 Setting HP Integrated Lights Out Manger (ILO) Credentials

Step Description Est Time
1 Login to the iLO:

$ ssh <username>@<iLO address>

<username>@<iLO address>'s password: <password>User:<username> logged-in to ...(<iLO address> / <ipv6 address>)

iLO Advanced2.61at Jul272018

Server Name: <server name>

Server Power: On

</>hpiLO->

1m
2 Change the current password:

</>hpiLO-> set /map1/accounts1/ <username> password= <newpassword>

status=0

status_tag=COMMAND COMPLETED

Tue Aug2013:27:082019

</>hpiLO->

1m
3 Create a new user account:

</>hpiLO-> create /map1/accounts1 username= <newusername> password= <newpassword> group=admin,config,oemHP_rc,oemHP_power,oemHP_vm

status=0

status_tag=COMMAND COMPLETED

Tue Aug2013:47:562019

User added successfully.

1m
4 Verify the new user account:

</>hpiLO-> exit

status=0

status_tag=COMMAND COMPLETED

Tue Aug2013:30:522019CLI session stoppedReceived disconnect from <iLO address> port22:11: Client Disconnect

Disconnected from <iLO address> port22

[bastion host]# ssh <newusername>@<iLO address>

<newusername>@<iLO address>'s password: <newpassword>

User:<newusername> logged-in to ...(<iLO address> / <ipv6 address>)

iLO Advanced2.61at Jul272018

Server Name: <server name>Server

Power: On</>hpiLO->

1m
5 Delete an unneeded account: </>hpiLO-> delete /map1/accounts1/ <username>

status=0

status_tag=COMMAND COMPLETED

Tue Aug2013:59:042019

User deleted successfully.

Procedure 3: Setting Root Passwords for All Cluster Nodes

The procedure to reset the root account requires that the administrator login to each and every server.

To reset the root account, for each and every server in the cluster perform the following steps:

Table 9-5 Setting Root Passwords for All Cluster Nodes

Step Description Est. time
1 Login to the next server:

$ ssh admusr@ <cluster server IP>
1m
2 Perform the root password change:

$ sudo passwd root

New password: <new password>

Retype new password: <new password>

Retype new password:<new password>
1m
3 Repeat steps 1 - 2 for each and every server in the cluster.

Note:

The administrator (admusr) account is provided without a usable password hash. Thus requiring the use of SSH keys to access the account. The SUDO users access is configured without the requirement of a password. If you would like to enable the SUDO passwords for the administrator, you also need to assign a password to the administrator account using a procedure very similar to the one outlined above.
Procedure 4: Updating admusr SSH Keys for All Cluster Nodes

There are two sets of SSH keys used in a deployed cluster - the key used to access the bastion host, and the key used to access the cluster servers. These key-pairs are generated at install time and are only usable on the cluster they were generated for. The public key portion of the bastion host key pair is typically provided to administrators who will manage the cluster. The key pair used to access the cluster servers should kept local to the cluster:

Table 9-6 Updating admusr SSH Keys

Key Pair Name Public Key Distribution Private Key Distributionm
Bastion Host Place copy in the authorized_keys file on the bastion host Cluster Admin - place in the cluster admin key agent (e.g., ssh-agent or pageant) external to the cluster. Do not copy to any host on the cluster.
Cluster Hosts Place a copy in the authorized_keys files on each and every cluster host; do not configure on the bastion host. Bastion Host -
~admusr/.ssh
directory. This will be used when performing orchestration activities (install / upgrade).

To replace either of these key pairs starts with an openssh request to generate a new keypair:

ssh-keygen -b 4096 -t rsa -C "New SSH Key" -f
      .ssh/new_occne_id_rsa -q -N ""

This command generates the following key pair:

Key Name Purpose
new_occne_id_rsa The private key
new_occne_id_rsa.pub The public key
Update the bastion host keys

Table 9-7 Update the bastion host keys

Step No. Description Est time
1.

Login to the bastion host and generate a new key pair using the ssh-keygen command show above.

$ ssh-keygen -b 4096 -t rsa -C "New
            Bastion Key" -f ~/.ssh/new_occne_id_rsa -q -N
    ""
1m
2. Copy the private key portion of the key off cluster and make it available to your ssh agent of choice or store it in the .ssh directory of your client machine. See instructions for your specific SSH client (e.g., putty or openssh) 1m
3.

Add the new public key to the authorized key file on the bastion host:

$ cat ~/.ssh/new_occne_id_rsa.pub >>
    ~/.ssh/authorized_keys
1m
4.

Confirm the permissions of the .ssh directory and files:

$ ls -la ~/.ssh
total 32
drwx------. 2 admusr admusr 4096 Feb 25 15:48 .
drwx------. 42 admusr admusr4096 Feb 24 15:14 ..
-rw-------. 1 admusr admusr 796 Jan 28 14:43 authorized_keys
-rw-------. 2 admusr admusr 545 Feb 12 13:58 config
-rw-------. 1 admusr admusr 3239 Feb 25 15:48 new_occne_id_rsa
-rw-r--r–. 1 admusr admusr 737 Feb 25 15:48 new_occne_id_rsa.pub

In general, the .ssh directory should be mode 700 and the files under that directory should be mode 600

1m
5. Confirm that the new key works; remove the old key from your ssh client's agent (see instructions for your client), and confirm that you can still login. 1m
6.

Assuming that you were able to login using the new key pair, remove the old key pair from the authorized_keys file using your favorite editor.

In general, the authorized_keys file should at this point have two keys in it - the old one and the new one. The new one should be at the bottom.

1m

General Security Administration Recommendations and Procedures

Note:

Recommendation: Record configuration changes

Note that in a disaster recovery scenario, Oracle provided procedures will only restore base system behavior (they will not include restoration of an special configurations or tweaks). We recommend that all post-delivery customization be logged or automated using tools such as Ansible.

Password Policy Administration Procedures

In general, the host environments use a user account named admusr which is not configured with a password; the only way to access this account is using SSH keys. We recommend using SSH keys rather than passwords for all non-root accounts. The root account cannot be accessed via ssh; the only access is via the console. For this account, we recommend setting a password and storing it off-site to be used only for break-glass console access to the host.

User Administration Recommendations

Customers may want to create additional accounts to manage separate concerns (Example: a dbadmin account, a k8sadmin account, etc). This can be done using normal linux user administration procedures.

SSHD Policy Administration Procedures

Customers may want to create augment the standard sshd configuration to perform additional hardening; this can be done using normal linux ssh administration procedures. Note that in a disaster recovery scenario, Oracle provided procedures will only restore base system behavior (they will not include restoration of an special configurations or tweaks).

Note:

Recommendation: Review changes with Oracle Support

We recommend reviewing any planned changes to sshd configuration with your Oracle Support contact. Improper sshd configuration can either open the system up to attacks or prevent proper system operation.

Auditd Policy Administration Procedures

Customers may want to augment the standard auditd configuration to perform additional monitoring; this can be done using normal linux auditd administration procedures. Place all customizations in a seperate file in the /etc/audit/rules.d directory; do not modify any of the other existing audit configuration files.