9 Cloud Native Environment (CNE) Security Recommendations and Procedures
After installation, audit the OC-CNE security system stance before placing the system into service. This primarily consists of changing credentials and sequestering SSH keys to trusted servers. The following table lists all the credentials that need to be checked, changed and retained:
Credential Name | Deployment | Type | Associated Resource | Initial Setting | Credential Rotation |
---|---|---|---|---|---|
TOR Switch | Bare Metal Only | username and password | Cisco Top or Rack Switch | username and password from PreFlight Checklist | Reset post-install |
Enclosure Switch | Bare Metal Only | username and password | HP Enclosure Switch | username and password from PreFlight Checklist | Reset post-install |
OA Admin | Bare Metal Only | username and password | On-board Administrator Console | username and password from PreFlight Checklist | Reset post-install |
ILO Admin | Bare Metal Only | username and password | HP Integrated Lights Out Manger | username and password from PreFlight Checklist | Reset post-install |
Server Super User (root) | All | username and password | Server Super User | Set to well-known Oracle default during server installation | Reset post-install |
Server Admin User SSH | All | SSH Key Pair | Server Admin User | Key Pair generated at install time | Can rotate keys at any time; key distribution manual procedure |
If factory or Oracle defaults were used for any of these credentials, it must be changed prior to placing the system into operation. The customer must store these credentials in a safe and secure way offsite. It is recommended that the customer must plan a regular schedule for updating (rotating) these credentials. Specific procedures and recommendations for OC-CNE credential management are provided below.
1.1.Network Security Recommendations and Procedures
Recommendation: Review and Follow TOR installation procedures
The OC-CNE on-premise installation guide provides detailed procedures on how to configure the TOR switches and configure them for remote monitoring. Deviations from the standard installation time configurations are not recommended.
Credential Management Procedures
Procedure 1: Setting Top Of Rack Switch Credentials
This procedure is used to set the credentials on the Cisco TOR switch as deployed with the bare metal deployment option. Steps for creating and deleting accounts and for setting account passwords is given below.
Table 9-1 Setting Top Of Rack Switch Credentials
Step No. | Description | Est time |
---|---|---|
1. | Login to the TOR switch (from the bastion host):
$ ssh <username>@<switchIP address>
User Access Verification
Password: <password>
Cisco Nexus Operating System (NX-OS) Software
TAC support: www.cisco.com/tac
<switchname>#
|
1m |
2. | Change the password for <username>:
# configure Enter configuration commands, one per line. End with CNTL/Z. (config)# username <username>
password<newpassword>
(config)#exit |
1m |
3. | Create a new user (if desired):
# configureEnter configuration commands, one per line. End with CNTL/Z. (config)# username <newusername>
password <newpassword> role
[network-operator|network-admin|vdc-admin|vdc-operator]
(config)#exit
|
1m |
4. | Verify the account changes by exiting the ssh session (type exit) and
repeat step 1.
# exit Connection to <switchIP address> closed. $ $ ssh <newusername>@<switchIP address>
User Access Verification Password:
<newpassword>
Cisco Nexus Operating System (NX-OS) SoftwareTAC support: www.cisco.com/tac ...... <switchname># |
1m |
5. | Delete an unrequired user account:
# configureEnter configuration commands, one per line. End with CNTL/Z.
|
1m |
6. | Change the enable secret:
|
1m |
7. | Save the configuration changes: # copy
running-config startup-config
100% Copy complete, now saving to disk (please wait)... Copy complete. |
1m |
Note:
Recommendation: Change TOR passwords before placing site into service. The TOR switch credentials show the changed prior to placing the site into service.Recommendation: Use Strong Passwords.The Network Administrator must choose complex TOR Switch passwords as per their organization's security guidelines.
This procedure is used to set the credentials on the HP enclosure switch as deployed with the bare metal deployment option. Steps for creating and deleting accounts and for setting account passwords is given below. For additional information, refer to: HP commands to configure enclosure switch username and password.
Table 9-2 Setting Enclosure Switch Credentials
Step | Description | Est. Time |
---|---|---|
1. | Login to the HP enclosure switch (from the
bastion host): $ ssh <username>@< switchIP
address>
Copyright (c)2010-2017Hewlett Packard Enterprise Development LP ** Without the owner's prior written consent, ** no decompiling or reverse-engineering shall be allowed. <switchname> <switchname> sysSystem View:returnto User View with Ctrl+Z. |
1m |
2. | Change the password for the current username:
[switchname]local-user <username>class
<currentclass>
[switchname-luser-manage-<username>]password
simple <newpassword>
[switchname-luser-manage-<username>]quit
|
1m |
3. | Create a new user account:
[switchname]local-user
<newusername>class[manage|network]
New local user added [switchname-luser-manage-<newusername>]password
simple
<newpassword>[switchname-luser-manage-<newusername>]quit
|
1m |
4. | Verify the account changes by exiting the ssh
session (type exit) and repeat step 1.
<switchname> quitConnection to <switchIP
address>closed.
$ $ ssh <newusername>@< switchIP address>
<newusername>@<switchIP address>'s password:
<newpassword>
Copyright (c)2010-2017Hewlett Packard Enterprise Development LP * * Without the owner's prior written consent, * * no decompiling or reverse-engineering shall be allowed. <switchname> <switchname> sys System View:returnto User View with Ctrl+Z. |
1m |
5. | Delete the user account that is not required:
[switchname]undo local-user <username>class
<currentclass>
|
1m |
6. | Save the configuration changes:
[switchname]save The current configuration will be written to the device. Are you sure? [Y/N]: y Please input the file name(*.cfg)[flash:/<filename>] (To leave the existing filename unchanged, press the enter key): flash:/<filename> exists, overwrite? [Y/N]: yValidating file. Please wait... Saved the current configuration to mainboard device successfully. Slot1: Save next configuration file successfully. [switchname] |
1 |
Note:
Recommendation: Set Enclosure Switch Credentials before Placing Into Service
The HP Enclosure switch credentials show be changed prior to placing the site into service.
Recommendation: Use Strong Passwords
The Network Administrator must choose complex Enclosure Switch passwords as per their organization's security guidelines.
1.2 Hosting Environment Security Recommendations and Procedures
The best way to keep your CNE environment secure is to keep it up-to-date. New OC-CNE releases are typically released every 2 months. The OC-CNE upgrade is not service affecting and will typically install newer versions of:
- Host OSs
- Kubernetes and associated containers
- DB-Tier binaries
- Common service containers
The upgrade process ensures that the uplifts do not affect active service. Refer Cloud Native Environment (OC-CNE) Upgrade Guide for more details.
Repository Management Recommendations System Update (YUM) RecommendationsRecommendation: Keep central yum repositories up to date.
Keep central repositories up-to date with latest yum packages; yum updates are performed on-site whenever a fresh install or upgrade is performed. An up-to date yum repository will help ensure that fixes for all publish vulnerabilities are applied.
Docker Repository RecommendationsRecommendation: Scan docker image repositories regularly.
Scan your docker image repositories regularly using a tool such as clair or anchore-engine. All images are scanned and vulnerabilities assessed at product development time, but new exploits /vulnerabilities may be reported/fixed later. Scan tools typically use a database of known vulnerabilities - refer to tool vendor for instructions on creating off-line (internet isolated) vulnerability databases.
1.3 Credential Management Procedures
Procedure 1: Setting HP Onboard Administrator (OA) Credentials.
This procedure is used to set the credentials on the HP Onboard Administrator as deployed with the bare metal deployment option. Steps for creating and deleting accounts and for setting account passwords is shown. For additional information, please refer to: HP commands to configure OA username and password.
Table 9-3 Setting HP Onboard Administrator (OA) Credentials
Step | Description | Est Time |
---|---|---|
1 | Login to the OA:
WARNING: This is a private system. Do not attempt to login unless you are anauthorized user. Any authorized or unauthorized access and use may be moni-tored and can result in criminal or civil prosecution under applicable law Firmware Version: 4.85 Built:04/06/2018@06:14OA Bay Number:1 OA Role: Active
HPE BladeSystem Onboard Administrator (C) Copyright 2006-2018 Hewlett Packard Enterprise Development LP Type 'HELP' to display a list of valid commands. Type 'HELP <command>' to display detailed information about a specific command. Type 'HELP HELP' to display more detailed information about the help system. OA-A45D36FD5FB1> |
1m |
2 | Change the current password:
OA-A45D36FD5FB1> set password <newpassword> Changed password for the"<username>"user account. OA-A45D36FD5FB1> |
1m |
3 | Add new user:
OA-A45D36FD5FB1> add user <newusername> New Password: <newpassword> Confirm : <newpassword> User"<newusername>"created. You may set user privileges with the 'SET USER ACCESS' and 'ASSIGN' commands. OA-A45D36FD5FB1> set user access <newusername> [ADMINISTRATOR|OPERATOR|USER] "<newusername>" has been given [administrator|operator|user] level privileges. |
1m |
4 | Assign full access to the enclosure for the user:
OA-A45D36FD5FB1> assign server all <newusername> <newusername> has been granted access to the valid requested bay (sOA-A45D36FD5FB1> assign interconnect all <newusername> <newusername> has been granted access to the valid requested bay(s)OA-A45D36FD5FB1> assign oa <newusername> <newusername> has been granted access to the OA. |
1m |
5 | Verify the new account:
OA-A45D36FD5FB1> exit Connection to <OA address> closed. [bastion host]# ssh <newusername>@<OA address> WARNING: This is a private system. Do not attempt to login unless you are unauthorized user. Any authorized or unauthorized access and use may be monitored and can result in criminal or civil prosecution under applicable law. Firmware Version : 4.85 Built : 04/06/2018 @ 06:14 OA Bay Number : 1 OA Role : Active
HPE BladeSystem Onboard Administrator (C) Copyright 2006-2018 Hewlett Packard Enterprise Development LP Type 'HELP' to display a list of valid commands. Type 'HELP <command>' to display detailed information about a specific command. Type 'HELP HELP' to display more detailed information about the help system. OA-A45D36FD5FB1> |
1m |
6 | Delete an unneeded user account:
OA-A45D36FD5FB1> remove user <username> Entering anything other than 'YES' will result in the command not executing. Are you sure you want to remove testuser1? yes User"<username>"removed. |
1m |
Procedure 2: Setting HP Integrated Lights Out Manger (ILO) Credentials
This procedure is used to set the credentials on the HP Integrated Lights Out Managers as deployed with the bare metal deployment option. Steps for creating and deleting accounts and for setting account passwords is shown.
Table 9-4 Setting HP Integrated Lights Out Manger (ILO) Credentials
Step | Description | Est Time |
---|---|---|
1 | Login to the iLO:
$ ssh <username>@<iLO address> <username>@<iLO address>'s password: <password>User:<username> logged-in to ...(<iLO address> / <ipv6 address>) iLO Advanced2.61at Jul272018 Server Name: <server name> Server Power: On </>hpiLO-> |
1m |
2 | Change the current password:
status=0 status_tag=COMMAND COMPLETED Tue Aug2013:27:082019 </>hpiLO-> |
1m |
3 | Create a new user account:
status=0 status_tag=COMMAND COMPLETED Tue Aug2013:47:562019 User added successfully. |
1m |
4 | Verify the new user account:
</>hpiLO-> exit status=0 status_tag=COMMAND COMPLETED Tue Aug2013:30:522019CLI session stoppedReceived disconnect from <iLO address> port22:11: Client Disconnect Disconnected from <iLO address> port22 [bastion host]# ssh <newusername>@<iLO address> <newusername>@<iLO address>'s password: <newpassword> User:<newusername> logged-in to ...(<iLO address> / <ipv6 address>) iLO Advanced2.61at Jul272018 Server Name: <server name>Server Power: On</>hpiLO-> |
1m |
5 | Delete an unneeded account: </>hpiLO->
delete /map1/accounts1/ <username>
status=0 status_tag=COMMAND COMPLETED Tue Aug2013:59:042019 User deleted successfully. |
Procedure 3: Setting Root Passwords for All Cluster Nodes
The procedure to reset the root account requires that the administrator login to each and every server.
Table 9-5 Setting Root Passwords for All Cluster Nodes
Step | Description | Est. time |
---|---|---|
1 | Login to the next server:
$ ssh admusr@ <cluster server IP>
|
1m |
2 | Perform the root password change:
$ sudo passwd root New password: <new password> Retype new password: <new password> Retype new password:<new password> |
1m |
3 | Repeat steps 1 - 2 for each and every server in the cluster. |
Note:
The administrator (admusr) account is provided without a usable password hash. Thus requiring the use of SSH keys to access the account. The SUDO users access is configured without the requirement of a password. If you would like to enable the SUDO passwords for the administrator, you also need to assign a password to the administrator account using a procedure very similar to the one outlined above.There are two sets of SSH keys used in a deployed cluster - the key used to access the bastion host, and the key used to access the cluster servers. These key-pairs are generated at install time and are only usable on the cluster they were generated for. The public key portion of the bastion host key pair is typically provided to administrators who will manage the cluster. The key pair used to access the cluster servers should kept local to the cluster:
Table 9-6 Updating admusr SSH Keys
Key Pair Name | Public Key Distribution | Private Key Distributionm |
---|---|---|
Bastion Host | Place copy in the authorized_keys file on the bastion host | Cluster Admin - place in the cluster admin key agent (e.g., ssh-agent or pageant) external to the cluster. Do not copy to any host on the cluster. |
Cluster Hosts | Place a copy in the authorized_keys files on each and every cluster host; do not configure on the bastion host. | Bastion Host -
directory. This will be used when performing orchestration
activities (install / upgrade). |
To replace either of these key pairs starts with an openssh request to generate a new keypair:
ssh-keygen -b 4096 -t rsa -C "New SSH Key" -f
.ssh/new_occne_id_rsa -q -N ""
This command generates the following key pair:
Key Name | Purpose |
---|---|
new_occne_id_rsa | The private key |
new_occne_id_rsa.pub | The public key |
Table 9-7 Update the bastion host keys
Step No. | Description | Est time |
---|---|---|
1. |
Login to the bastion host and generate a new key pair using the ssh-keygen command show above.
|
1m |
2. | Copy the private key portion of the key off cluster and make it available to your ssh agent of choice or store it in the .ssh directory of your client machine. See instructions for your specific SSH client (e.g., putty or openssh) | 1m |
3. |
Add the new public key to the authorized key file on the bastion host:
|
1m |
4. |
Confirm the permissions of the .ssh directory and files:
In general, the .ssh directory should be mode 700 and the files under that directory should be mode 600 |
1m |
5. | Confirm that the new key works; remove the old key from your ssh client's agent (see instructions for your client), and confirm that you can still login. | 1m |
6. |
Assuming that you were able to login using the new key pair, remove the old key pair from the authorized_keys file using your favorite editor. In general, the authorized_keys file should at this point have two keys in it - the old one and the new one. The new one should be at the bottom. |
1m |
General Security Administration Recommendations and Procedures
Note:
Recommendation: Record configuration changes
Note that in a disaster recovery scenario, Oracle provided procedures will only restore base system behavior (they will not include restoration of an special configurations or tweaks). We recommend that all post-delivery customization be logged or automated using tools such as Ansible.
Password Policy Administration Procedures
In general, the host environments use a user account named admusr which is not configured with a password; the only way to access this account is using SSH keys. We recommend using SSH keys rather than passwords for all non-root accounts. The root account cannot be accessed via ssh; the only access is via the console. For this account, we recommend setting a password and storing it off-site to be used only for break-glass console access to the host.
User Administration Recommendations
Customers may want to create additional accounts to manage separate concerns (Example: a dbadmin account, a k8sadmin account, etc). This can be done using normal linux user administration procedures.
SSHD Policy Administration Procedures
Note:
Recommendation: Review changes with Oracle Support
We recommend reviewing any planned changes to sshd configuration with your Oracle Support contact. Improper sshd configuration can either open the system up to attacks or prevent proper system operation.
Auditd Policy Administration Procedures
Customers may want to augment the standard auditd configuration to perform
additional monitoring; this can be done using normal linux auditd administration
procedures. Place all customizations in a seperate file in the
/etc/audit/rules.d
directory; do not modify any of the other
existing audit configuration files.