Post-Installation Security Hardening

Introduction

After installation, the OC-CNE system security stance should be audited prior to placing the system into service. This primarily consists of changing credentials and sequestering SSH keys to trusted servers. The following table lists all the credentials that need to be checked / changed / retained:

Table 4-2 Credentials

Credential Name Type Associated Resource Initial Setting Credential Rotation
TOR Switch username / password Cisco Top or Rack Switch username/password from PreFlight Checklist Reset post-install
Enclosure Switch username / password HP Enclosure Switch username/password from PreFlight Checklist Reset post-install
OA Admin username / password HP On-board Administrator Console username/password from PreFlight Checklist Reset post-install
ILO Admin username / password HP Integrated Lights Out Manger username/password from PreFlight Checklist Reset post-install
Server Super User (root) username / password Server Super User Set to well-known Oracle default during server installation Reset post-install
Server Admin User (admusr) username / password Server Admin User Set to well-known Oracle default during server installation Reset post-install
Server Admin User SSH SSH Key Pair Server Admin User Key Pair generated at install time Can rotate keys at any time; key distribution manual procedure
MySQL Admin username / password MySQL Database Set by customer during initial install Reset post-install

If factory or Oracle defaults were used for any of these credentials, they should be changed prior to placing the system into operation. The customer should then store these credentials in a safe a secure way off site. It is recommended that the customer may plan a regular schedule for updating (rotating) these credentials.

Prerequisites

This procedure is performed after the site has been deployed and prior to placing the site into service.

Limitations and Expectations

The focus of this procedure is to secure the various credentials used or created during the install procedure. There are additional security audits that the CNE operator should perform such as scanning repositories for vulnerabilities, monitoring the system for anomalies, regularly checking security logs. These are outside the scope of this post-installation procedure.

References

  1. Nexus commands to configure Top of Rack switch username and password:https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/security/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide_chapter_01001.html
  2. HP commands to configure Enclosure switch username and password:https://support.hpe.com/hpsc/doc/public/display?docId=c04763521
  3. HP OA commands to configure OA username and password:https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00040582en_us&docLocale=en_US#N101C8
  4. HP iLO commands to configure iLO username and password:https://www.golinuxhub.com/2018/02/hp-ilo4--cli-guide-cheatsheet-example.html
  5. See ToR switch procedure for initial username/password configuration: Configure Top of Rack 93180YC-EX Switches
  6. See procedure to configure initial iLO/OA username/password: Configure Addresses for RMS iLOs, OA, EBIPA
  7. See Enclosure switch procedure for initial username/password: Configure Enclosure Switches

Procedure

Table 4-3

Step No. Procedure Description
1.

Reset Credentials on the TOR Switch
  1. From bastion host, login to the switch with username and password from the procedure
    [bastion host]# ssh <username>@<switch IP address>
    User Access Verification
    Password: <password>
     
    Cisco Nexus Operating System (NX-OS) Software
    TAC support: http://www.cisco.com/tac
    ...
    ...
    <switch name>#
  2. Change the password for current username:
    #
    # configure
    Enter configuration commands, one per line. End with CNTL/Z.
    (config)# username <username> password <newpassword>
    (config)#exit
    #
  3. Create new username:
    #
    # configure
    Enter configuration commands, one per line. End with CNTL/Z.
    (config)# username <newusername> password <newpassword> role [network-operator|network-admin|vdc-admin|vdc-operator]
    (config)#exit
    #
  4. Exit from the switch and login with the new username and password to verify the new change works:
    # exit
    Connection to <switch IP address> closed.
    [bastion host]#
     
    [some server]# ssh <newusername>@<switch IP address>
    User Access Verification
    Password: <newpassword>
     
    Cisco Nexus Operating System (NX-OS) Software
    TAC support: http://www.cisco.com/tac
    ...
    ...
    <switch name>#
  5. Delete the previous old username if it is not needed:
    #
    # configure
    Enter configuration commands, one per line. End with CNTL/Z.
    (config)# no username <username>
    (config)#exit
    #
  6. Change the enable secret when needed:
    #
    (config)# enable secret <newenablepassword>
    (config)# exit
    #
  7. Save the above configuration:
    # copy running-config startup-config
    [########################################] 100%
    Copy complete, now saving to disk (please wait)...
    Copy complete.
    #
    
2.

Reset Credentials on the Enclosure Switch
  1. From bastion host, login to the switch with username and password from the procedure:
    [bastion host]# ssh <username>@<switch IP address>
    <username>@<switch IP address>'s password: <password>
     
    ******************************************************************************
    * Copyright (c) 2010-2017 Hewlett Packard Enterprise Development LP          *
    * Without the owner's prior written consent,                                 *
    * no decompiling or reverse-engineering shall be allowed.                    *
    ******************************************************************************
     
    <switchname>
    <switchname>sys
    System View: return to User View with Ctrl+Z.
    [switchname]
  2. Change the password for current username:
    [switchname]local-user <username> class <current class>
    [switchname-luser-manage-<username>]password simple <newpassword>
    [switchname-luser-manage-<username>]quit
    [switchname]
  3. Create new username:
    [switchname]local-user <newusername> class [manage|network]
    New local user added.
    [switchname-luser-manage-<newusername>]password simple <newpassword>
    [switchname-luser-manage-<newusername>]quit
    [switchname]
  4. Exit from the switch and login with the new username and password to verify the new change works:
    <switchname>quit
    Connection to <switch IP address> closed.
    [bastion host]#
      
    [bastion host]# ssh <newusername>@<switch IP address>
    <newusername>@<switch IP address>'s password: <newpassword>
     
    ******************************************************************************
    * Copyright (c) 2010-2017 Hewlett Packard Enterprise Development LP          *
    * Without the owner's prior written consent,                                 *
    * no decompiling or reverse-engineering shall be allowed.                    *
    ******************************************************************************
     
    <switchname>
    <switchname>sys
    System View: return to User View with Ctrl+Z.
    [switchname]
  5. Delete the previous old username if it is not needed:
    [switchname]undo local-user <username> class <current class>
  6. Save the above configuration:
    [switchname]save
    The current configuration will be written to the device. Are you sure? [Y/N]:y
    Please input the file name(*.cfg)[flash:/<filename>]
    (To leave the existing filename unchanged, press the enter key):
    flash:/<filename> exists, overwrite? [Y/N]:y
    Validating file. Please wait...
    Saved the current configuration to mainboard device successfully.
    Slot 1:
    Save next configuration file successfully.
    [switchname]
3.

Reset Credentials for the OA Admin Console
  1. From bastion host, login to the OA with username and password from the procedure: (Note: If Standby OA, exit and login with the other OA address)
    [bastion host]# ssh <username>@<OA address>
     
    -----------------------------------------------------------------------------
    WARNING: This is a private system.  Do not attempt to login unless you are an
    authorized user.  Any authorized or unauthorized access and use may be moni-
    tored and can result in criminal or civil prosecution under applicable law.
    -----------------------------------------------------------------------------
     
    Firmware Version: 4.85
    Built: 04/06/2018 @ 06:14
    OA Bay Number:  1
    OA Role:        Active
    <username>@<OA address>'s password:<password>
     
     
     
     
     
     
    HPE BladeSystem Onboard Administrator
    (C) Copyright 2006-2018 Hewlett Packard Enterprise Development LP
     
     
    Type 'HELP' to display a list of valid commands.
    Type 'HELP <command>' to display detailed information about a specific command.
    Type 'HELP HELP' to display more detailed information about the help system.
     
     
    OA-A45D36FD5FB1>
  2. Change the password for current username:
    OA-A45D36FD5FB1> set password <newpassword>
     
    Changed password for the "<username>" user account.
     
    OA-A45D36FD5FB1>
  3. Add new user:
    OA-A45D36FD5FB1> add user <newusername>
     
    New Password: <newpassword>
    Confirm     : <newpassword>
    User "<newusername>" created.
    You may set user privileges with the 'SET USER ACCESS' and 'ASSIGN' commands.
     
    OA-A45D36FD5FB1> set user access <newusername> [ADMINISTRATOR|OPERATOR|USER]
     
    "<newusername>" has been given [administrator|operator|user] level privileges.
  4. Assign full access to the enclosure for the user:
    OA-A45D36FD5FB1> assign server all <newusername>
     
    <newusername> has been granted access to the valid requested bay(s
     
    OA-A45D36FD5FB1> assign interconnect all <newusername>
     
    <newusername> has been granted access to the valid requested bay(s)
     
    OA-A45D36FD5FB1> assign oa <newusername>
     
    <newusername> has been granted access to the OA.
  5. Exit from the OA and login with the new username and password to verify the new change works:
    OA-A45D36FD5FB1> exit
     
    Connection to <OA address> closed.
    [bastion host]# ssh <newusername>@<OA address>
     
    -----------------------------------------------------------------------------
    WARNING: This is a private system.  Do not attempt to login unless you are an
    authorized user.  Any authorized or unauthorized access and use may be moni-
    tored and can result in criminal or civil prosecution under applicable law.
    -----------------------------------------------------------------------------
     
    Firmware Version: 4.85
    Built: 04/06/2018 @ 06:14
    OA Bay Number:  1
    OA Role:        Active
    <newusername>@<OA address>'s password:<newpassword>
     
     
     
     
     
     
    HPE BladeSystem Onboard Administrator
    (C) Copyright 2006-2018 Hewlett Packard Enterprise Development LP
     
     
    Type 'HELP' to display a list of valid commands.
    Type 'HELP <command>' to display detailed information about a specific command.
    Type 'HELP HELP' to display more detailed information about the help system.
     
     
    OA-A45D36FD5FB1>
  6. Delete previous user if not needed:
    OA-A45D36FD5FB1> remove user <username>
     
    Entering anything other than 'YES' will result in the command not executing.
     
    Are you sure you want to remove testuser1? yes
     
    User "<username>" removed.
4.

Reset Credentials for the ILO Admin Console
  1. From bastion host, login to the iLO with username and password from the procedure:
    [root@winterfell ~]# ssh <username>@<iLO address>
    <username>@<iLO address>'s password: <password>
    User:<username> logged-in to ...(<iLO address> / <ipv6 address>)
     
    iLO Advanced 2.61 at  Jul 27 2018
    Server Name: <server name>
    Server Power: On
     
    </>hpiLO->
  2. Change current password:
    </>hpiLO-> set /map1/accounts1/<username> password=<newpassword>
     
    status=0
    status_tag=COMMAND COMPLETED
    Tue Aug 20 13:27:08 2019
     
    </>hpiLO->
  3. Create new user:
    </>hpiLO-> create /map1/accounts1 username=<newusername> password=<newpassword> group=admin,config,oemHP_rc,oemHP_power,oemHP_vm
    status=0
    status_tag=COMMAND COMPLETED
    Tue Aug 20 13:47:56 2019
     
    User added successfully.
  4. Exit from the iLO and login with the new username and password to verify the new change works:
    </>hpiLO-> exit
     
    status=0
    status_tag=COMMAND COMPLETED
    Tue Aug 20 13:30:52 2019
     
     
     
     
    CLI session stopped
    Received disconnect from <iLO address> port 22:11:  Client Disconnect
    Disconnected from <iLO address> port 22
     
    [bastion host]# ssh <newusername>@<iLO address>
    <newusername>@<iLO address>'s password: <newpassword>
    User:<newusername> logged-in to ...(<iLO address> / <ipv6 address>)
     
    iLO Advanced 2.61 at  Jul 27 2018
    Server Name: <server name>
    Server Power: On
     
    </>hpiLO->
  5. Delete the previous username if not needed:
    </>hpiLO-> delete /map1/accounts1/<username>
     
    status=0
    status_tag=COMMAND COMPLETED
    Tue Aug 20 13:59:04 2019
     
    User deleted successfully.
    
5.

Reset Credentials for the root account on Each and Every Server Login to each and every server in the cluster (ssh admusr@cluster_host ) and perform the following command:
	sudo passwd root
6.

Reset (or Delete) Credentials for the admusr account on Each and Every Server Login to each and every server in the cluster (ssh admusr@cluster_host ) and perform the following command:
sudo passwd -l admusr
7.

Reset Credentials for the MySQL Accounts See Database Tier Installer for details on how to reset the DB Account Passwords.
8.

Regenerate / Redistribute SSH Keys Credentials for the admusr Account Log into the Bastion Host VM and generate a new cluster-wide keypair by perform the following:
ssh-keygen -b 4096 -t rsa -C "New SSH Key" -f .ssh/new_occne_id_rsa -q -N ""

Now, for each and every server in the cluster, perform these actions:
# for each cluster_host in the cluster; do
    # copy the public key to the node
    scp .ssh/new_occne_id_rsa.pub admusr@cluster_host:.ssh/
 
 
    # install the key
    ssh admusr@cluster_host "cat .ssh/new_occne_id_rsa.pub >> .ssh/authorized_keys"
# done

At this point, the new key should be usable. Switch from using the old key to the new key, and confirm that each and every cluster host is still reachable. On the Bastion Host VM, perform these actions:
# remove the old keys from the agent (assuming you are using an agent)
ssh-add -D
# add the new key to the agent
ssh-add .ssh/new_occne_is_rsa
 
# for each cluster_host in the cluster; do
    # confirm access to the cluster host(s) and remove the old key
    ssh admusr@cluster_host "sed -i '/ occne installer key$/d' .ssh/authorized_keys"
# done

The new private key (new_occne_id_rsa) should also be copied to any secondary Bastion Host VM, and possibly copied off site and securely saved.