1. Cloud Native Environment (OCCNE) Installation Guide
  2. Installation Procedures

2 Installation Procedures

The installation procedures in this document provision and configure an Oracle Communications Signaling, Network Function Cloud Native Environment (OCCNE). OCCNE offers the choice of deployment platform; the CNE can be deployed directly onto dedicated hardware, (referred to as a bare metal CNE), or deployed onto OpenStack-hosted VMs. (referred to as a virtualized CNE).

Regardless of which deployment platform is selected, OCCNE installation is highly automated. A collection of container-based utilities are used to automate the provisioning, installation, and configuration of OCCNE. These utilities are based on the following automation tools:

  • PXE helps reliably automate provisioning the hosts with a minimal operating system.
  • Terraform is used to create the virtual resources that the virtualized CNE is hosted on.
  • Kubespray helps reliably install a base Kubernetes cluster, including all dependencies (like etcd), using the Ansible provisioning tool.
  • Ansible is used to orchestrate the overall deployment.
  • Helm is used to deploy and configure common services such as Prometheus, Grafana, ElasticSearch and Kibana.

Note:

Make sure that the shell is configured with Keepalive to avoid unexpected timeout.

Bare Metal Installation

This section describes the procedure to install OCCNE onto dedicated bare metal hardware.

OCCNE Installation Overview

Frame and Component Overview

The initial release of the OCCNE system provides support for on-prem deployment to a very specific target environment consisting of a frame holding switches and servers. This section describes the layout of the frame and describes the roles performed by the racked equipment.

Note:

In the installation process, some of the roles of servers change as the installation procedure proceeds.
Frame Overview

The physical frame is comprised of HP c-Class enclosure (BL460c blade servers), 5 DL380 rack mount servers, and 2 Top of Rack (ToR) Cisco switches. The frame components are added from the bottom up, thus designations found in the next section number from the bottom of the frame to the top of the frame.

Figure 2-1 Frame Overview


Frame Overview

Host Designations

Each physical server has a specific role designation within the CNE solution.

Figure 2-2 Host Designations


Host Designations

Node Roles

Along with the primary role of each host, a secondary role may be assigned. The secondary role may be software related, or, in the case of the Bootstrap Host, hardware related, as there are unique OOB connections to the ToR switches.

Figure 2-3 Node Roles


Node Roles

Transient Roles

Transient role is unique in that it has OOB connections to the ToR switches, which includes the designation of Bootstrap Host. This role is only relevant during initial switch configuration and disaster recovery of the switch. RMS1 also has a transient role as the Installer Bootstrap Host, which is only relevant during initial install of the frame, and subsequent to getting an official install on RMS2, this host is re-paved to its Storage Host role.

Figure 2-4 Transient Roles


Transient Roles

Create OCCNE Instance

This section describes the steps and procedures required to create an OCCNE instance at a customer site. The following diagrams shows the installation context:

Figure 2-5 OCCNE Installation Overview


OCCNE Installation Overview

The following is an overview or basic install flow for reference to understand the overall effort contained within these procedures:

  1. Check that the hardware is on-site and properly cabled and powered up.
  2. Pre-assemble the basic ingredients needed to perform a successful install:
    1. Identify
      1. Download and stage software and other configuration files using provided manifests. Refer to Artifact Acquisition and Hosting for manifests information.
      2. Identify the layer 2 (MAC) and layer 3 (IP) addresses for the equipment in the target frame
      3. Identify the addresses of key external network services (for example, NTP, DNS, etc.)
      4. Verify / Set all of the credentials for the target frame hardware to known settings
    2. Prepare
      1. Software Repositories: Load the various SW repositories (YUM, Helm, Docker, etc.) using the downloaded software and configuration
      2. Configuration Files: Populate the hosts inventory file with credentials and layer 2 and layer 3 network information, switch configuration files with assigned IP addresses, and yaml files with appropriate information.
  3. Bootstrap the System:
    1. Manually configure a Minimal Bootstrapping Environment (MBE); perform the minimal set of manual operations to enable networking and initial loading of a single Rack Mount Server - RMS1 - the transient Installer Bootstrap Host. In this procedure, a minimal set of packages needed to configure switches, iLOs, PXE boot environment, and provision RMS2 as an OCCNE Storage Host are installed.
    2. Using the newly constructed MBE, automatically create the first (complete) Management VM on RMS2. This freshly installed Storage Host will include a virtual machine for hosting the Bastion Host.
    3. Using the newly constructed Bastion Host on RMS2, automatically deploy and configure the OCCNE on the other servers in the frame
  4. Final Steps
    1. Perform post installation checks
    2. Perform recommended security hardening steps

Cluster Bootstrapping Overview

This install procedure is targeted at installing OCCNE onto a new hardware absent of any networking configurations to switches, or operating systems provisioned. Therefore, the initial step in the installation process is to provision RMS1 (see Installation Procedures) as a temporary Installer Bootstrap Host. The Bootstrap Host is configured with a minimal set of packages needed to configure switches, iLOs, PXE boot environment, and provision RMS2 as an OCCNE Storage Host. A virtual Bastion Host is also provisioned on RMS2. The Bastion Host is then used to provision (and in the case of the Bootstrap Host, re-provision) the remaining OCCNE hosts and install Kubernetes, Database services, and Common Services running within the Kubernetes cluster.

Installation Prerequisites

Obtain Mate Site DB Replication Service Load Balancer IP

Complete the procedures outlined in this section before moving on to the Install Procedures section. OCCNE installation procedures require certain artifacts and information to be made available prior to executing installation procedures. Refer to Configure Artifact Acquisition and Hosting for the prerequisites.

While installing MYSQL NDB on the second site the Mate Site DB Replication Service Load Balancer IP must be provided as the configuration parameter for the geo-replication process to start.

  1. Login to Bastion Host of the first site and execute the following command to retrieve DB Replication Service Load Balancer IP
  2. Fetch DB Replication Service Load Balancer IP of Mate Site MYSQL NDB. 
    $ kubectl get svc --namespace=occne-infra | grep replication
 
Example:
$ kubectl get svc --namespace=occne-infra | grep replication
occne-db-replication-svc     LoadBalancer   10.233.3.117    10.75.182.88     80:32496/TCP   2m8s
    In the above example IPv4: 10.75.182.88 is the Mate Site DB Replication Service Load Balancer IP.
Configure Artifact Acquisition and Hosting

OCCNE requires artifacts from Oracle eDelivery and certain open-source projects. OCCNE deployment environments are not expected to have direct internet access. Thus, customer-provided intermediate repositories are necessary for the OCCNE installation process. These repositories will need OCCNE dependencies to be loaded into them. This section will address the artifacts list needed to be in these repositories.

Oracle eDelivery Artifact Acquisition

The OCCNE artifacts are posted on Oracle Software delivery Cloud (OSDC) and/or OHC.

Third Party Artifacts

OCCNE dependencies that come from open-source software must be available in repositories reachable by the OCCNE installation tools. For an accounting of third party artifacts needed for this installation, refer to the Artifact Acquisition and Hosting.

Populate the MetalLB Configuration

Introduction

The metalLB configMap file (mb_configmap.yaml) contains the manifest for the metalLB configMap, this defines the BGP peers and address pools for metalLB. This file (mb_configmap.yaml) must be placed in the same directory (/var/occne/<cluster_name>) as the hosts.ini file.

Following is the procedure to configure MetalLB pools and peers:
  1. Add BGP peers and address groups: Referring to the data collected in the Preflight Checklist, add BGP peers (ToRswitchA_Platform_IP, ToRswitchB_Platform_IP) and address groups for each address pool. Address-pools list the IP addresses that metalLB is allowed to allocate.
  2. Edit the mb_configmap.yaml file with the site-specific values found in the Preflight Checklist

    Note:

    The name "signaling" is prone to different spellings (UK vs US), therefore pay special attention to how this signaling pool is referenced. 
    configInline:
     peers:
     - peer-address: <ToRswitchA_Platform_IP>
       peer-asn: 64501
       my-asn: 64512
     - peer-address: <ToRswitchB_Platform_IP>
       peer-asn: 64501
       my-asn: 64512
     address-pools:
     - name: signaling
       protocol: bgp
       auto-assign: false
       addresses:
       - '<MetalLB_Signal_Subnet_IP_Range>'
     - name: oam
       protocol: bgp
       auto-assign: false
       addresses:
       - '<MetalLB_OAM_Subnet_IP_Range>'
Install Backup Bastion Host

Introduction

This procedure details the steps necessary to install the Backup Bastion Host on the Storage Host db-1/RMS1 and backing up the data from the active Bastion Host on db-2/RMS2 to the Backup Bastion Host.

Prerequisites

  1. Bastion Host is already created on Storage Host db-2/RMS2.
  2. Storage Host db-2/RMS2 and the Backup Bastion Host are defined in the Customer hosts.ini file as defined in procedure: Inventory File Preparation.

    Note:

    If the initial bootstrap host is RMS1 then the bastion host is created on the RMS2 and backup bastion host is created on the RMS1.
  3. Host names and IP Address, network information assigned to Backup Management VM are captured in the Installation PreFlight Checklist.
  4. All the Network information should be configured in Inventory File Preparation.

Expectations

  1. Bastion Host VM on Storage Host db-1/RMS1 is created as a backup for Bastion Host VM on Storage Host db-2/RMS2.
  2. All the required config files and data configured in the Backup Bastion Host on Storage Host db-1/RMS1 are copied from the active Bastion Host on Storage Host db-2/RMS2.

Procedure

Create the Backup Bastion Host on Storage Host db-1/RMS1

All commands are executed from the active Bastion Host on db-2/RMS2.

  1. Login to the active Bastion Host (VM on RMS2) using the admusr/****** credentials.
  2. Execute the deploy.sh script from the /var/occne/ directory with the required parameters set. 
    $ export CENTRAL_REPO=<customer specific repo name>
$ export CENTRAL_REPO_IP=<customer_specific_repo_ipv4>
$ export OCCNE_CLUSTER=<cluster_name>
$ export OCCNE_BASTION=<bastion_full_name>
$ ./deploy.sh
 
Customer Example:
$ export CENTRAL_REPO=central-repo
$ export CENTRAL_REPO_IP=10.10.10.10
$ export OCCNE_CLUSTER=rainbow
$ export OCCNE_BASTION=bastion-1.rainbow.lab.us.oracle.com
$ ./deploy.sh
 
Note: The above example can be executed like the following:
CENTRAL_REPO=central-repo CENTRAL_REPO_IP=10.10.10.10 OCCNE_CLUSTER=rainbow OCCNE_BASTION=bastion-1.rainbow.lab.us.oracle.com ./deploy.sh
  3. Verify installation of the Backup Bastion Host.

    Note:

    The IP of the backup Bastion Host can be derived from the hosts.ini file under the group host_kernel_virtual for db-1 ansible_host IP.
    $ ssh -i ~/.ssh/id_rsa admusr@<backup_bastion_ip_address>

Initial Configuration - Prepare a Minimal Boot Strapping Environment

In the first step of the installation, a minimal bootstrapping environment is established that is to support the automated installation of the CNE environment. The steps in this section provide the details necessary to establish this minimal bootstrap environment on the Installer Bootstrap Host using a Keyboard, Video, Mouse (KVM) connection.

Installation of Oracle Linux 7.5 on Bootstrap Host

This procedure outlines the installation steps for installing the OL7 onto the OCCNE Installer Bootstrap Host. This host is used to configure the networking throughout the system and install OL7 onto RMS1. It is re-paved as a Database Host in a later procedure.

Prerequisites

  1. USB drive of sufficient size to hold the ISO (approximately 5Gb)
  2. Oracle Linux 7.5 iso
  3. YUM repository file
  4. Keyboard, Video, Mouse (KVM)

Limitations and Expectations

  1. The configuration of the Installer Bootstrap Host is meant to be quick and easy, without a lot of care on appropriate OS configuration. The Installer Bootstrap Host is re-paved with the appropriate OS configuration for cluster and DB operation at a later stage of installation. The Installer Bootstrap Host needs a Linux OS and some basic network to get the installation process started.
  2. All steps in this procedure are performed using Keyboard, Video, Mouse (KVM).

References

  1. Oracle Linux 7 Installation guide: https://docs.oracle.com/cd/E52668_01/E54695/html/index.html
  2. HPE Proliant DL380 Gen10 Server User Guide

Bootstrap Install Procedure

  1. Create Bootable USB Media:
    1. Download the Oracle Linux 7.5.

      On the installer's notebook, download the OL ISO from the customer's repository. Since Installer notebook may be Windows or Linux OS, and the Customer repository location may vary so the user executing this procedure determines the appropriate detail to execute this task.

    2. Push the OL ISO image onto the USB Flash Drive.

      Since the installer's notebook may be Windows or Linux OS-based, the user executing this procedure determines the appropriate detail to execute this task. For a Linux based notebook, insert a USB Flash Drive of the appropriate size into a Laptop (or some other linux host where the iso can be copied to), and run the dd command to create a bootable USB drive with the Oracle Linux 7 iso. 

      $ dd -if=<path to ISO> -of=<USB device path> -bs=1m
  2. Install OL7 on the Installer Bootstrap Host:
    1. Connect a Keyboard, Video, and Mouse (KVM) into the Installer Bootstrap Host's monitor and USB ports.
    2. Plug the USB flash drive containing the bootable iso into an available USB port on the Bootstrap host (usually in the front panel).
    3. Reboot the host by momentarily pressing the power button on the host's front panel. The button will go yellow. If it holds at yellow, press the button again. The host should auto-boot to the USB flash drive.

      Note:

      If the host was previously configured and the USB is not a bootable path in the boot order, it may not boot successfully.
    4. If the host does not boot to the USB, repeat step 3, and interrupt the boot process by pressing F11 which brings up the Boot Menu. If the host has been recently booted with an OL, the Boot Menu will display Oracle Linux at the top of the list. Select Generic USB Boot as the first boot device and proceed.
    5. The host attempts to boot from the USB. The following menu is displayed on the screen. Select Test this media & install Oracle Linux 7.x and click ENTER. This begins the verification of the media and the boot process. After the verification reaches 100%, the Welcome screen is displayed. When prompted for the language to use, select the default setting: English (United States) and click Continue in the lower left corner.
    6. The INSTALLATION SUMMARY page, is displayed. The following settings are expected. If any of these are not set correctly then please select that menu item and make the appropriate changes.
      1. LANGUAGE SUPPORT: English (United States)
      2. KEYBOARD: English (US)
      3. INSTALLATION SOURCE: Local Media
      4. SOFTWARE SELECTION: Minimal Install

      INSTALLATION DESTINATION should display No disks selected. Select INSTALLATION DESTINATION to indicate the drive to install the OS on.

      Select the first HDD drive ( in this case that would be the first one listed or the 1.6 TB disk) and select DONE in the upper right corner.

      If the server has already been installed a red banner at the bottom of the page may indicate there is an error condition. Selecting that banner causes a dialog to appear indicating there is not enough free space (which might mean an OS has already been installed). In the dialog it may show both 1.6 TB HDDs as claimed or just the one.

      If only one HDD is displayed (or it could be both 1.6 TB drives selected, select the Reclaim space button. Another dialog appears. Select the Delete all button and the Reclaim space button again. Select DONE to return to the INSTALLATION SUMMARY screen.

      If the disk selection dialog appears (after selecting the red banner at the bottom of the page), this implies a full installation of the RMS has already been performed (usually this is because the procedure had to be restarted after it was successfully completed). In this case select the Modify Disk Selection. This will return to the disk selection page. Select both HDDs and hit done. The red banner should now indicate the space must be reclaimed. The same steps to reclaim the space can be performed.

    7. Select DONE. This returns to the INSTALLATION SUMMARY page.
    8. At the INSTALLATION SUMMARY screen, select Begin Installation. The CONFIGURATION screen is displayed.
    9. At the CONFIGURATION screen, select ROOT PASSWORD.

      Enter a root password appropriate for this installation. It is good practice to use a customer provided secure password to minimize the host being compromised during installation.

    10. At the conclusion of the install, remove the USB and select Reboot to complete the install and boot to the OS on the host. At the end of the boot, the login prompt appears.

Install Additional Packages

Additional packages are needed to complete the installation and move on to the next step in the overall procedure. These additional packages are available within the OL install media on the USB. To install these packages, a YUM repo file is configured to use the install media. The additional packages to install are:
  • dnsmasq
  • dhcp
  • xinetd
  • tftp-server
  • dos2unix
  • nfs-utils
  1. Login with the root user and password configured above.
  2. Create the mount directory:

    $ mkdir /media/usb

  3. Insert the USB into an available USB port (usually the front USB port) of the Installer Bootstrap Host.
  4. Find and mount the USB partition.

    Typically the USB device is enumerated as /dev/sda but that is not always the case. Use the lsblk command to find the USB device. An example lsblk output is below. The capacity of the USB drive is expected to be approximately 30GiB, therefore the USB drive is enumerated as device /dev/sda in the example below: 

    $ lsblk
sdd           8:48   0 894.3G  0 disk
sde           8:64   0   1.7T  0 disk
sdc           8:32   0 894.3G  0 disk
├─sdc2        8:34   0     1G  0 part /boot
├─sdc3        8:35   0 893.1G  0 part
│ ├─ol-swap 252:1    0     4G  0 lvm  [SWAP]
│ ├─ol-home 252:2    0 839.1G  0 lvm  /home
│ └─ol-root 252:0    0    50G  0 lvm  /
└─sdc1        8:33   0   200M  0 part /boot/efi
sda           8:0    1  29.3G  0 disk
├─sda2        8:2    1   8.5M  0 part
└─sda1        8:1    1   4.3G  0 part

    The dmesg command also provides information about how the operating system enumerates devices. In the example below, the dmesg output indicates the USB drive is enumerated as device /dev/sda.

    Note: The output is shortened here for display purposes. 

    $ dmesg
...
[8850.211757] usb-storage 2-6:1.0: USB Mass Storage device detected
[8850.212078] scsi host1: usb-storage 2-6:1.0
[8851.231690] scsi 1:0:0:0: Direct-Access     SanDisk  Cruzer Glide     1.00 PQ: 0 ANSI: 6
[8851.232524] sd 1:0:0:0: Attached scsi generic sg0 type 0
[8851.232978] sd 1:0:0:0: [sda] 61341696 512-byte logical blocks: (31.4 GB/29.3 GiB)
[8851.234598] sd 1:0:0:0: [sda] Write Protect is off
[8851.234600] sd 1:0:0:0: [sda] Mode Sense: 43 00 00 00
[8851.234862] sd 1:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[8851.255300] sda: sda1 sda2
...

    The USB device should contain at least two partitions. One is the boot partition and the other is the install media. The install media is the larger of the two partitions. To find information about the partitions use the fdisk command to list the filesystems on the USB device. Use the device name discovered via the steps outlined above. In the examples above, the USB device is /dev/sda. 

    $ fdisk -l /dev/sda
Disk /dev/sda: 31.4 GB, 31406948352 bytes, 61341696 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x137202cf
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           0     8929279     4464640    0  Empty
/dev/sda2            3076       20503        8714   ef  EFI (FAT-12/16/32)

    In the example output above, the /dev/sda2 partition is the EFI boot partition. Therefore the install media files are on /dev/sda1. Use the mount command to mount the install media file system. The same command without any options is used to verify the device is mounted to /media/usb. 

    $ mount /dev/sda1 /media/usb
 
$ mount
...
/dev/sda1 on /media/usb type iso9660 (ro,relatime,nojoliet,check=s,map=n,blocksize=2048)
  5. Create a yum config file to install packages from local install media.

    Create a repo file /etc/yum.repos.d/Media.repo with the following information: 

    [ol7_base_media]
name=Oracle Linux 7 Base Media
baseurl=file:///media/usb
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1

  6. Disable the default public yum repo. This is done by renaming the current .repo file to end with something other than .repo. Adding .disabled to the end of the file name is standard.

    Note: This can be left in this state as the Installer Bootstrap Host is re-paved in a later procedure. 

    $ mv /etc/yum.repos.d/public-yum-ol7.repo /etc/yum.repos.d/public-yum-ol7.repo.disabled
  7. Use the yum repolist command to check the repository configuration.

    The output of yum repolist should look like the example below. Verify there no errors regarding un-reachable yum repos. 

    $ yum repolist
Loaded plugins: langpacks, ulninfo
repo id                           repo name                         status                       
ol7_base_media                    Oracle Linux 7 Base Media         5,134                                                                          
repolist: 5,134
  8. Use yum to install the additional packages from the USB repo. 
    $ yum install dnsmasq
$ yum install dhcp
$ yum install xinetd
$ yum install tftp-server
$ yum install dos2unix
$ yum install nfs-utils

  9. Verify installation of dhcp, xinetd, and tftp-server.

    Note: Currently dnsmasq is not being used. The verification of tftp makes sure the tftp file is included in the /etc/xinetd.d directory. Installation/Verification does not include actually starting any of the services. Service configuration/starting is performed in a later procedure. 

    Verify dhcp is installed:
-------------------------
$ cd /etc/dhcp
$ ls
dhclient.d  dhclient-exit-hooks.d  dhcpd6.conf  dhcpd.conf  scripts
 
Verify xinetd is installed:
---------------------------
$ cd /etc/xinetd.d
$ ls
chargen-dgram  chargen-stream  daytime-dgram  daytime-stream  discard-dgram  discard-stream 
echo-dgram  echo-stream  tcpmux-server  time-dgram  time-stream
 
Verify tftp is installed:
-------------------------
$ cd /etc/xinetd.d
$ ls
chargen-dgram  chargen-stream  daytime-dgram  daytime-stream  discard-dgram  discard-stream 
echo-dgram  echo-stream  tcpmux-server  tftp  time-dgram  time-stream
  10. Unmount the USB and remove the USB from the host. The mount command can be used to verify the USB is no longer mounted to /media/usb. 
    $ umount /media/usb
 
$ mount
Verify that /dev/sda1 is no longer shown as mounted to /media/usb.
Configure the Installer Bootstrap Host BIOS

Introduction

These procedures define the steps necessary to set up the Legacy BIOS changes on the Bootstrap host using the KVM. Some of the procedures in this document require a reboot of the system and are indicated in the procedure.

Prerequisites

Procedure OCCNE Installation of Oracle Linux 7.5 on Bootstrap Host is complete.

Limitations and Expectations

  1. Applies to HP Gen10 iLO 5 only.
  2. The procedures listed here applies to the Bootstrap host only.

Steps to OCCNE Configure the Installer Bootstrap Host BIOS

  1. Expose the System Configuration Utility: This step details how to expose the HP iLO 5 System Configuration Utility main page from the KVM. It does not provide instructions on how to connect the console as these may be different on each installation.
    1. After making the proper connections for the KVM on the back of the Bootstrap host to have access to the console, the user should reboot the host by momentarily pressing the power button on the front of the Bootstrap host.
    2. Expose the HP Proliant DL380 Gen10 System Utilities.

      Once the remote console has been exposed, the system must be reset to force it through the restart process. When the initial window is displayed, hit the F9 key repeatedly. Once the F9 is highlighted at the lower left corner of the remote console, it should eventually bring up the main System Utility.

    3. The System Utilities screen is exposed in the remote console.
  2. Change over from UEFI Booting Mode to Legacy BIOS Booting Mode: The System Utility must default the booting mode to UEFI or has been changed to UEFI, it will be necessary to switch the booting mode to Legacy.
    1. Expose the System Configuration Utility by following Step 1.
    2. Select System Configuration.
    3. Select BIOS/Platform Configuration (RBSU).
    4. Select Boot Options.

      If the Boot Mode is set to UEFI Mode then this procedure should be used to change it to Legacy BIOS Mode.

      Note: The server reset must go through an attempt to boot before the changes will actually apply.

    5. The user is prompted to select the Reboot Required popup dialog. This will drop back into the boot process. The boot must go into the process of actually attempting to boot from the boot order. This should fail since the disks have not been installed at this point. The System Utility can be accessed again.
    6. After the reboot and the user re-enters the System Utility, the Boot Options page should appear.
    7. Select F10: Save if it's desired to save and stay in the utility or select the F12: Save and Exit if its desired to save and exit to complete the current boot process.
  3. Adding a New User Account: This step provides the steps required to add a new user account to the server iLO 5 interface.

    Note:

    This user must match the pxe_install_lights_out_usrfields as provided in the hosts inventory files created using the template: OCCNE Inventory File Preparation.
    1. Expose the System Utility by following Step 1.
    2. Select System Configuration.
    3. Select iLO 5 Configuration Utility.
    4. Select User Management → Add User .
    5. Select the appropriate permissions. For the root user set all permissions to YES. Enter root as New User Name and Login Name fields, and enter <password> in the Password field.
    6. Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit, to complete the current boot process.
  4. Force PXE to boot from the first Embedded FlexibleLOM HPE Ethernet 10Gb 2-port Adapter. During host PXE, the DHCP DISCOVER requests from the hosts must be broadcast over the 10Gb port. This step provides the steps necessary to configure the broadcast to use the 10Gb ports before it attempts to use the 1Gb ports. Moving the 10Gb port up on the search order helps to speed up the response from the host servicing the DHCP DISCOVER. Enclosure blades have 2 10GE NICs which default to being configured for PXE booting. The RMS are re-configured to use the PCI NICs using this step.
    1. Expose the System Utility by following Step 1.
    2. Select System Configuration.
    3. Select BIOS/Platform Configuration (RBSU).
    4. Select Boot Options.

      This menu defines the boot mode which should be set to Legacy BIOS Mode, the UEFI Optimized Boot which should be disabled, and the Boot Order Policy which should be set to Retry Boot Order Indefinitely (this means it will keep trying to boot without ever going to disk). In this screen select Legacy BIOS Boot Order. If not in Legacy BIOS Mode, please follow procedure 2.2 Change over from UEFI Booting Mode to Legacy BIOS Booting Mode to set the Configuration Utility to Legacy BIOS Mode.

    5. Select Legacy BIOS Boot Order

      This page defines the legacy BIOS boot order. This includes the list of devices from which the server will listen for the DHCP OFFER (includes the reserved IPv4) after the PXE DHCP DISCOVER message is broadcast out from the server.

      In the default view, the 10Gb Embedded FlexibleLOM 1 Port 1 is at the bottom of the list. When the server begins the scan for the response, it scans down this list until it receives the response. Each NIC will take a finite amount of time before the server gives up on that NIC and attempts another in the list. Moving the 10Gb port up on this list should decrease the time that is required to finally process the DHCP OFFER.

      To move an entry, select that entry, hold down the first mouse button and move the entry up in the list below the entry it must reside under.

    6. Move the 10 Gb Embedded FlexibleLOM 1 Port 1 entry up above the 1Gb Embedded LOM 1 Port 1 entry.
    7. Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit, to complete the current boot process.
  5. Enabling Virtualization: This step provides the steps required to enable virtualization on a given Bare Metal Server. Virtualization can be configured using the default settings or via the Workload Profiles.
    1. Verifying Default Settings
      1. Expose the System Configuration Utility by following Step 1.
      2. Select System Configuration.
      3. Select BIOS/Platform Configuration (RBSU)
      4. Select Virtualization Options

        This screen displays the settings for the Intel(R) Virtualization Technology (IntelVT), Intel(R) VT-d, and SR-IOV options (Enabled or Disabled). The default values for each option is Enabled.

      5. Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit, to complete the current boot process.
  6. Disable RAID Configurations:
    1. Expose the System Configuration Utility by following Step 1.
    2. Select System Configuration.
    3. Select Embedded RAID 1 : HPE Smart Array P408i-a SR Gen 10.
    4. Select Array Configuration.
    5. Select Manage Arrays.
    6. Select Array A (or any designated Array Configuration if there are more than one).
    7. Select Delete Array.
    8. Select Submit Changes.
    9. Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit, to complete the current boot process.
  7. Enable the Primary Boot Device: This step provides the steps necessary to configure the primary bootable device for a given Gen10 Server. In this case the RMS would include two devices as Hard Drives (HDDs). Some configurations may also include two Solid State Drives (SSDs). The SSDs are not to be selected for this configuration. Only the primary bootable device is set in this procedure since RAID is being disabled. The secondary bootable device remains as Not Set.
    1. Expose the System Configuration Utility by following Step 1.
    2. Select System Configuration.
    3. Select Embedded RAID 1 : HPE Smart Array P408i-a SR Gen 10.
    4. Select Set Bootable Device(s) for Legacy Boot Mode. If the boot devices are not set then it will display Not Set for the primary and secondary devices.
    5. Select Select Bootable Physical Drive.
    6. Select Port 1| Box:3 Bay:1 Size:1.8 TB SAS HP EG00100JWJNR.

      Note: This example includes two HDDs and two SSDs. The actual configuration may be different.

    7. Select Set as Primary Bootable Device.
    8. Select Back to Main Menu.

      This will return to the HPE Smart Array P408i-a SR Gen10 menu. The secondary bootable device is left as Not Set.

    9. Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit, to complete the current boot process.
  8. Configure the iLO 5 Static IP Address: When configuring the Bootstrap host, the static IP address for the iLO 5 must be configured.

    Note:

    This step requires a reboot after completion.
    1. Expose the System Configuration Utility by following Step 1.
    2. Select System Configuration.
    3. Select iLO 5 Configuration Utility.
    4. Select Network Options.
    5. Enter the IP Address, Subnet Mask, and Gateway IP Address fields provided in Installation PreFlight Checklist.
    6. Select F12: Save and Exit to complete the current boot process. A reboot is required when setting the static IP for the iLO 5. A warning appears indicating that the user must wait 30 seconds for the iLO to reset and then a reboot is required. A prompt appears requesting a reboot. Select Reboot.
    7. Once the reboot is complete, the user can re-enter the System Utility and verify the settings if necessary.
Configure Top of Rack 93180YC-EX Switches

Introduction

This procedure provides the steps required to initialize and configure Cisco 93180YC-EX switches as per the topology defined in Physical Network Topology Design.

Note:

All instructions in this procedure are executed from the Bootstrap Host.

Prerequisites

  1. Procedure OCCNE Installation of Oracle Linux 7.5 on Bootstrap Host has been completed.
  2. The switches are in factory default state.
  3. The switches are connected as per Installation PreFlight Checklist. Customer uplinks are not active before outside traffic is necessary.
  4. DHCP, XINETD, and TFTP are already installed on the Bootstrap host but are not configured.
  5. The Utility USB is available containing the necessary files as per: Installation PreFlight checklist: Create Utility USB.

Limitations/Expectations

All steps are executed from a Keyboard, Video, Mouse (KVM) connection.

References

Configuration Procedure

Following is the procedure to configure Top of Rack 93180YC-EX Switches:

  1. Login to the Bootstrap host as root.

    Note:

    All instructions in this step are executed from the Bootstrap Host.
  2. Insert and mount the Utility USB that contains the configuration and script files. Verify the files are listed in the USB using the ls /media/usb command.

    Note:

    Instructions for mounting the USB can be found in: Installation of Oracle Linux 7.5 on Bootstrap Server : Install Additional Packages. Only steps 2 and 3 need to be followed in that procedure.
  3. Create bridge interface: Create bridge interface to connect both management ports and setup the management bridge to support switch initialization.

    Note:

    <CNE_Management_IP_With_Prefix> is from Installation PreFlight Checklist : Complete Site Survey Host IP Table. Row 1 CNE Management IP Addresess (VLAN 4) column.

    <ToRSwitch_CNEManagementNet_VIP> is from Installation PreFlight Checklist : Complete OA and Switch IP Table. 

    $ nmcli con add con-name mgmtBridge type bridge ifname mgmtBridge
$ nmcli con add type bridge-slave ifname eno2 master mgmtBridge
$ nmcli con add type bridge-slave ifname eno3 master mgmtBridge
$ nmcli con mod mgmtBridge ipv4.method manual ipv4.addresses 192.168.2.11/24
$ nmcli con up mgmtBridge
 
$ nmcli con add type team con-name team0 ifname team0 team.runner lacp
$ nmcli con add type team-slave con-name team0-slave-1 ifname  eno5 master team0
$ nmcli con add type team-slave con-name team0-slave-2 ifname  eno6 master team0
    The following commands are related to the vlan and ip address for this bootstrap server, the <mgmt_vlan_id> is same as in hosts.ini, <bootstrap team0 address> is same as ansible_host ip for this bootstrap server:
    
nmcli con mod team0 ipv4.method manual ipv4.addresses <bootstrap team0 address>
nmcli con add con-name team0.<mgmt_vlan_id> type vlan id <mgmt_vlan_id> dev team0
nmcli con mod team0.<mgmt_vlan_id> ipv4.method manual ipv4.addresses <CNE_Management_IP_Address_With_Prefix> ipv4.gateway <ToRswitch_CNEManagementNet_VIP>nmcli con up team0.<mgmt_vlan_id>
    Example:
     
nmcli con mod team0 ipv4.method manual ipv4.addresses 172.16.3.4/24
nmcli con add con-name team0.4 type vlan id 4 dev team0
nmcli con mod team0.4 ipv4.method manual ipv4.addresses <CNE_Management_IP_Address_With_Prefix> ipv4.gateway <ToRswitch_CNEManagementNet_VIP>
nmcli con up team0.4
  4. Edit the /etc/xinetd.d/tftp file to enable TFTP service. Change the disable option to no, if it is set to yes. 
    $ vi /etc/xinetd.d/tftp
# default: off
# description: The tftp server serves files using the trivial file transfer \
#       protocol.  The tftp protocol is often used to boot diskless \
#       workstations, download configuration files to network-aware printers, \
#       and to start the installation process for some operating systems.
service tftp
{
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -s /var/lib/tftpboot
        disable                 = no
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}
  5. Enable tftp on the Bootstrap host:
    $ systemctl start tftp
$ systemctl enable tftp
 
Verify tftp is active and enabled:
$ systemctl status tftp
$ ps -elf | grep tftp
  6. Copy the dhcpd.conf file from the Utility USB in Installation PreFlight checklist : Create the dhcpd.conf File to the /etc/dhcp/ directory. 
    $ cp /media/usb/dhcpd.conf /etc/dhcp/
  7. Restart and enable dhcpd service.
    # /bin/systemctl restart dhcpd
# /bin/systemctl enable dhcpd
 
Use the systemctl status dhcpd command to verify active and enabled.
# systemctl status dhcpd
  8. Copy the switch configuration and script files from the Utility USB to directory /var/lib/tftpboot/. 
    $ cp /media/usb/93180_switchA.cfg /var/lib/tftpboot/.
$ cp /media/usb/93180_switchB.cfg /var/lib/tftpboot/.
$ cp /media/usb/poap_nexus_script.py /var/lib/tftpboot/.
  9. Modify POAP script File. Change Username and password credentials used to login to the Bootstrap host. 
    
# vi /var/lib/tftpboot/poap_nexus_script.py
# Host name and user credentials
options = {
   "username": "<username>",
   "password": "<password>",
   "hostname": "192.168.2.11",
   "transfer_protocol": "scp",
   "mode": "serial_number",
   "target_system_image": "nxos.9.2.3.bin",
}
 
Note: The version nxos.9.2.3.bin is used by default. If different version is to be used, modify the "target_system_image" with new version.
  10. Modify POAP script file md5sum by executing the md5Poap.sh script from the Utility USB created from Installation PreFlight checklist: Create the md5Poap Bash Script. 
    # cd /var/lib/tftpboot/
# /bin/bash md5Poap.sh
  11. Create the files necessary to configure the ToR switches using the serial number from the switch. The serial number is located on a pullout card on the back of the switch in the left most power supply of the switch.


    ToR switch

    Note:

    The serial number is located on a pullout card on the back of the switch in the left most power supply of the switch. Be careful in interpreting the exact letters. If the switches are preconfigured then you can even verify the serial numbers using 'show license host-id' command.
  12. Copy the /var/lib/tftpboot/93180_switchA.cfg into a file called /var/lib/tftpboot/conf.<switchA serial number> Modify the switch specific values in the /var/lib/tftpboot/conf.<switchA serial number> file, including all the values in the curly braces as following code block.

    These values are contained at Installation PreFlight checklist : ToR and Enclosure Switches Variables Table (Switch Specific) and Installation PreFlight Checklist : Complete OA and Switch IP Table. Modify these values with the following sed commands, or use an editor such as vi etc. 

    # sed -i 's/{switchname}/<switch_name>/' conf.<switchA serial number>
# sed -i 's/{admin_password}/<admin_password>/' conf.<switchA serial number>
# sed -i 's/{user_name}/<user_name>/' conf.<switchA serial number>
# sed -i 's/{user_password}/<user_password>/' conf.<switchA serial number>
# sed -i 's/{ospf_md5_key}/<ospf_md5_key>/' conf.<switchA serial number>
# sed -i 's/{OSPF_AREA_ID}/<ospf_area_id>/' conf.<switchA serial number>
 
# sed -i 's/{NTPSERVER1}/<NTP_server_1>/' conf.<switchA serial number>
# sed -i 's/{NTPSERVER2}/<NTP_server_2>/' conf.<switchA serial number>
# sed -i 's/{NTPSERVER3}/<NTP_server_3>/' conf.<switchA serial number>
# sed -i 's/{NTPSERVER4}/<NTP_server_4>/' conf.<switchA serial number>
# sed -i 's/{NTPSERVER5}/<NTP_server_5>/' conf.<switchA serial number>
 
# Note: If less than 5 ntp servers available, delete the extra ntp server lines such as command:
# sed -i 's/{NTPSERVER5}/d' conf.<switchA serial number>
 
 Note: different delimiter is used in next two commands due to '/' sign in the variables
# sed -i 's#{ALLOW_5G_XSI_LIST_WITH_PREFIX_LEN}#<MetalLB_Signal_Subnet_With_Prefix>#g' conf.<switchA serial number>
# sed -i 's#{CNE_Management_SwA_Address}#<ToRswitchA_CNEManagementNet_IP>#g' conf.<switchA serial number>
# sed -i 's#{CNE_Management_SwB_Address}#<ToRswitchB_CNEManagementNet_IP>#g' conf.<switchA serial number>
# sed -i 's#{CNE_Management_Prefix}#<CNEManagementNet_Prefix>#g' conf.<switchA serial number>
# sed -i 's#{SQL_replication_SwA_Address}#<ToRswitchA_SQLreplicationNet_IP>#g' conf.<switchA serial number>
# sed -i 's#{SQL_replication_SwB_Address}#<ToRswitchB_SQLreplicationNet_IP>#g' conf.<switchA serial number>
# sed -i 's#{SQL_replication_Prefix}#<SQLreplicationNet_Prefix>#g' conf.<switchA serial number>
# ipcalc -n  <ToRswitchA_SQLreplicationNet_IP/<SQLreplicationNet_Prefix> | awk -F'=' '{print $2}'
# sed -i 's/{SQL_replication_Subnet}/<output from ipcalc command as SQL_replication_Subnet>/' conf.<switchA serial number>
 
# sed -i 's/{CNE_Management_VIP}/<ToRswitch_CNEManagementNet_VIP>/g' conf.<switchA serial number>
# sed -i 's/{SQL_replication_VIP}/<ToRswitch_SQLreplicationNet_VIP>/g' conf.<switchA serial number>
# sed -i 's/{OAM_UPLINK_CUSTOMER_ADDRESS}/<ToRswitchA_oam_uplink_customer_IP>/' conf.<switchA serial number>
 
# sed -i 's/{OAM_UPLINK_SwA_ADDRESS}/<ToRswitchA_oam_uplink_IP>/g' conf.<switchA serial number>
# sed -i 's/{SIGNAL_UPLINK_SwA_ADDRESS}/<ToRswitchA_signaling_uplink_IP>/g' conf.<switchA serial number>
# sed -i 's/{OAM_UPLINK_SwB_ADDRESS}/<ToRswitchB_oam_uplink_IP>/g' conf.<switchA serial number>
# sed -i 's/{SIGNAL_UPLINK_SwB_ADDRESS}/<ToRswitchB_signaling_uplink_IP>/g' conf.<switchA serial number>
# ipcalc -n  <ToRswitchA_signaling_uplink_IP>/30 | awk -F'=' '{print $2}' 
# sed -i 's/{SIGNAL_UPLINK_SUBNET}/<output from ipcalc command as signal_uplink_subnet>/' conf.<switchA serial number>
 
# ipcalc -n  <ToRswitchA_SQLreplicationNet_IP> | awk -F'=' '{print $2}'
# sed -i 's/{MySQL_Replication_SUBNET}/<output from the above ipcalc command appended with prefix >/' conf.<switchA serial number>
 
Note: The version nxos.9.2.3.bin is used by default and hard-coded in the conf files. If different version is to be used, run the following command: 
# sed -i 's/nxos.9.2.3.bin/<nxos_version>/' conf.<switchA serial number>
 
Note: access-list Restrict_Access_ToR
# The following line allow one access server to access the switch management and SQL vlan addresses while other accesses are denied. If no need, delete this line. If need more servers, add similar line. 
# sed -i 's/{Allow_Access_Server}/<Allow_Access_Server>/' conf.<switchA serial number>
  13. Copy the /var/lib/tftpboot/93180_switchB.cfg into a file called /var/lib/tftpboot/conf.<switchB serial number>

    Modify the switch specific values in the /var/lib/tftpboot/conf.<switchA serial number> file, including: hostname, username/password, oam_uplink IP address, signaling_uplink IP address, access-list ALLOW_5G_XSI_LIST permit address, prefix-list ALLOW_5G_XSI.

    These values are contained at Installation PreFlight checklist : ToR and Enclosure Switches Variables Table and Installation PreFlight Checklist : Complete OA and Switch IP Table. 

    # sed -i 's/{switchname}/<switch_name>/' conf.<switchB serial number>
# sed -i 's/{admin_password}/<admin_password>/' conf.<switchB serial number>
# sed -i 's/{user_name}/<user_name>/' conf.<switchB serial number>
# sed -i 's/{user_password}/<user_password>/' conf.<switchB serial number>
# sed -i 's/{ospf_md5_key}/<ospf_md5_key>/' conf.<switchB serial number>
# sed -i 's/{OSPF_AREA_ID}/<ospf_area_id>/' conf.<switchB serial number>
 
# sed -i 's/{NTPSERVER1}/<NTP_server_1>/' conf.<switchB serial number>
# sed -i 's/{NTPSERVER2}/<NTP_server_2>/' conf.<switchB serial number>
# sed -i 's/{NTPSERVER3}/<NTP_server_3>/' conf.<switchB serial number>
# sed -i 's/{NTPSERVER4}/<NTP_server_4>/' conf.<switchB serial number>
# sed -i 's/{NTPSERVER5}/<NTP_server_5>/' conf.<switchB serial number>
 
# Note: If less than 5 ntp servers available, delete the extra ntp server lines such as command:
# sed -i 's/{NTPSERVER5}/d' conf.<switchB serial number>
 
Note: different delimiter is used in next two commands due to '/' sign in in the variables
# sed -i 's#{ALLOW_5G_XSI_LIST_WITH_PREFIX_LEN}#<MetalLB_Signal_Subnet_With_Prefix>#g' conf.<switchB serial number>
# sed -i 's#{CNE_Management_SwA_Address}#<ToRswitchA_CNEManagementNet_IP>#g' conf.<switchB serial number>
# sed -i 's#{CNE_Management_SwB_Address}#<ToRswitchB_CNEManagementNet_IP>#g' conf.<switchB serial number>
# sed -i 's#{CNE_Management_Prefix}#<CNEManagementNet_Prefix>#g' conf.<switchB serial number>
# sed -i 's#{SQL_replication_SwA_Address}#<ToRswitchA_SQLreplicationNet_IP>#g' conf.<switchB serial number>
# sed -i 's#{SQL_replication_SwB_Address}#<ToRswitchB_SQLreplicationNet_IP>#g' conf.<switchB serial number>
# sed -i 's#{SQL_replication_Prefix}#<SQLreplicationNet_Prefix>#g' conf.<switchB serial number>
# ipcalc -n  <ToRswitchB_SQLreplicationNet_IP/<SQLreplicationNet_Prefix> | awk -F'=' '{print $2}'
# sed -i 's/{SQL_replication_Subnet}/<output from ipcalc command as SQL_replication_Subnet>/' conf.<switchB serial number>
 
# sed -i 's/{CNE_Management_VIP}/<ToRswitch_CNEManagementNet_VIP>/' conf.<switchB serial number>
# sed -i 's/{SQL_replication_VIP}/<ToRswitch_SQLreplicationNet_VIP>/' conf.<switchB serial number>
# sed -i 's/{OAM_UPLINK_CUSTOMER_ADDRESS}/<ToRswitchB_oam_uplink_customer_IP>/' conf.<switchB serial number>
 
# sed -i 's/{OAM_UPLINK_SwA_ADDRESS}/<ToRswitchA_oam_uplink_IP>/g' conf.<switchB serial number>
# sed -i 's/{SIGNAL_UPLINK_SwA_ADDRESS}/<ToRswitchA_signaling_uplink_IP>/g' conf.<switchB serial number>
# sed -i 's/{OAM_UPLINK_SwB_ADDRESS}/<ToRswitchB_oam_uplink_IP>/g' conf.<switchB serial number>
# sed -i 's/{SIGNAL_UPLINK_SwB_ADDRESS}/<ToRswitchB_signaling_uplink_IP>/g' conf.<switchB serial number>
# ipcalc -n  <ToRswitchB_signaling_uplink_IP>/30 | awk -F'=' '{print $2}'
# sed -i 's/{SIGNAL_UPLINK_SUBNET}/<output from ipcalc command as signal_uplink_subnet>/' conf.<switchB serial number>
 
Note: The version nxos.9.2.3.bin is used by default and hard-coded in the conf files. If different version is to be used, run the following command: 
# sed -i 's/nxos.9.2.3.bin/<nxos_version>/' conf.<switchB serial number>
 
Note: access-list Restrict_Access_ToR
# The following line allow one access server to access the switch management and SQL vlan addresses while other accesses are denied. If no need, delete this line. If need more servers, add similar line. 
# sed -i 's/{Allow_Access_Server}/<Allow_Access_Server>/' conf.<switchB serial number>
  14. Generate the md5 checksum for each conf file in /var/lib/tftpboot and copy that into a new file called conf.<switchA/B serial number>.md5. 
    
$ md5sum conf.<switchA serial number> > conf.<switchA serial number>.md5
$ md5sum conf.<switchB serial number> > conf.<switchB serial number>.md5
  15. Verify the /var/lib/tftpboot directory has the correct files.

    Make sure the file permissions are set as given below.

    Note: The ToR switches are constantly attempting to find and execute the poap_nexus_script.py script which uses tftp to load and install the configuration files. 

    
# ls -l /var/lib/tftpboot/
total 1305096
-rw-r--r--. 1 root root       7161 Mar 25 15:31 conf.<switchA serial number>
-rw-r--r--. 1 root root         51 Mar 25 15:31 conf.<switchA serial number>.md5
-rw-r--r--. 1 root root       7161 Mar 25 15:31 conf.<switchB serial number>
-rw-r--r--. 1 root root         51 Mar 25 15:31 conf.<switchB serial number>.md5
-rwxr-xr-x. 1 root root      75856 Mar 25 15:32 poap_nexus_script.py
  16. Disable firewalld. 
    $ systemctl stop firewalld
$ systemctl disable firewalld
 
To verify:
$ systemctl status firewalld

    Once this is complete, the ToR Switches will attempt to boot from the tftpboot files automatically. Eventually the verification steps can be executed below. It may take about 5 minutes for this to complete.

  17. Un-mount the Utility USB and remove it: umount /media/usb

Verification

Following is the procedure to verify Top of Rack 93180YC-EX Switches:
  1. After the ToR switches configured, ping the switches from bootstrap server. The switches mgmt0 interfaces are configured with the IP addresses which are in the conf files. Note: Wait till the device responds. 
    # ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=255 time=0.419 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=255 time=0.496 ms
64 bytes from 192.168.2.1: icmp_seq=3 ttl=255 time=0.573 ms
64 bytes from 192.168.2.1: icmp_seq=4 ttl=255 time=0.535 ms
^C
--- 192.168.2.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.419/0.505/0.573/0.063 ms
# ping 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=255 time=0.572 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=255 time=0.582 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=255 time=0.466 ms
64 bytes from 192.168.2.2: icmp_seq=4 ttl=255 time=0.554 ms
^C
--- 192.168.2.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.466/0.543/0.582/0.051 ms
  2. Attempt to ssh to the switches with the username/password provided in the conf files.
    # ssh plat@192.168.2.1
The authenticity of host '192.168.2.1 (192.168.2.1)' can't be established.
RSA key fingerprint is SHA256:jEPSMHRNg9vejiLcEvw5qprjgt+4ua9jucUBhktH520.
RSA key fingerprint is MD5:02:66:3a:c6:81:65:20:2c:6e:cb:08:35:06:c6:72:ac.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.2.1' (RSA) to the list of known hosts.
User Access Verification
Password:
 
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2019, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source.  This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or
GNU General Public License (GPL) version 3.0  or the GNU
Lesser General Public License (LGPL) Version 2.1 or
Lesser General Public License (LGPL) Version 2.0.
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.
#
  3. Verify the running-config has all expected configurations in the conf file using the show running-config command. 
    # show running-config
!Command: show running-config
!Running configuration last done at: Mon Apr  8 17:39:38 2019
!Time: Mon Apr  8 18:30:17 2019
version 9.2(3) Bios:version 07.64
hostname 12006-93108A
vdc 12006-93108A id 1
  limit-resource vlan minimum 16 maximum 4094
  limit-resource vrf minimum 2 maximum 4096
  limit-resource port-channel minimum 0 maximum 511
  limit-resource u4route-mem minimum 248 maximum 248
  limit-resource u6route-mem minimum 96 maximum 96
  limit-resource m4route-mem minimum 58 maximum 58
  limit-resource m6route-mem minimum 8 maximum 8
feature scp-server
feature sftp-server
cfs eth distribute
feature ospf
feature bgp
feature interface-vlan
feature lacp
feature vpc
feature bfd
feature vrrpv3
....
....
  4. Verify license on the switches. In case some of the above features are missing, verify license on the switches and at least NXOS_ADVANTAGE level license is "In use". If license not installed or too low level, contact vendor for correct license key file, following Licensing document mentioned in reference section to install license key. Then run "write erase" and "reload" to set back to factory default. The switches will go to POAP configuration again. 
    # show license

Example output:
# show license
MDS20190215085542979.lic:
SERVER this_host ANY
VENDOR cisco
INCREMENT NXOS_ADVANTAGE_XF cisco 1.0 permanent uncounted \
        VENDOR_STRING=<LIC_SOURCE>MDS_SWIFT</LIC_SOURCE><SKU>NXOS-AD-XF</SKU> \
        HOSTID=VDH=FDO22412J2F \
        NOTICE="<LicFileID>20190215085542979</LicFileID><LicLineID>1</LicLineID> \
        <PAK></PAK>" SIGN=8CC8807E6918

# show license usage

Example output:
# show license usage
Feature                      Ins  Lic   Status Expiry Date Comments
                                 Count
--------------------------------------------------------------------------------
...
NXOS_ADVANTAGE_M4             No    -   Unused             -
NXOS_ADVANTAGE_XF             Yes   -   In use never       -
NXOS_ESSENTIALS_GF            No    -   Unused             -
...
#
  5. Verify the RMS1 can ping the CNE_Management VIP. 
    # ping <ToRSwitch_CNEManagementNet_VIP>
PING <ToRSwitch_CNEManagementNet_VIP> (<ToRSwitch_CNEManagementNet_VIP>) 56(84) bytes of data.
64 bytes from <ToRSwitch_CNEManagementNet_VIP>: icmp_seq=2 ttl=255 time=1.15 ms
64 bytes from <ToRSwitch_CNEManagementNet_VIP>: icmp_seq=3 ttl=255 time=1.11 ms
64 bytes from <ToRSwitch_CNEManagementNet_VIP>: icmp_seq=4 ttl=255 time=1.23 ms
^C
--- 10.75.207.129 ping statistics ---
4 packets transmitted, 3 received, 25% packet loss, time 3019ms
rtt min/avg/max/mdev = 1.115/1.168/1.237/0.051 ms
  6. Enable customer uplink.
  7. Verify the RMS1 can be accessed from laptop. Use application such as putty etc to ssh to RMS1. 
    
$ ssh root@<CNE_Management_IP_Address>
Using username "root".
root@<CNE_Management_IP_Address>'s password:<root password>
Last login: Mon May  6 10:02:01 2019 from 10.75.9.171
[root@RMS1 ~]#

SNMP Trap Configuration

The following procedure explains the steps to configure SNMP Trap:
  1. SNMPv2c Configuration.

    When SNMPv2c configuration is needed, ssh to the two switches, run the following commands:

    These values <SNMP_Trap_Receiver_Address>and <SNMP_Community_String> are from Installation Preflight Checklist. 

    [root@RMS1 ~]# ssh <user_name>@<ToRswitchA_CNEManagementNet_IP>
# configure terminal
(config)# snmp-server host <SNMP_Trap_Receiver_Address> traps version 2c <SNMP_Community_String>
(config)# snmp-server host <SNMP_Trap_Receiver_Address> use-vrf default
(config)# snmp-server host <SNMP_Trap_Receiver_Address> source-interface Ethernet1/51
(config)# snmp-server enable traps
(config)# snmp-server community <SNMP_Community_String> group network-admin
  2. Restrict direct access to ToR switches. In order to restrict direct access to ToR switches, IP access list is created and applied on the uplink interfaces, the following commands are needed on ToR switches: 
    [root@RMS1 ~]# ssh <user_name>@<ToRswitchA_CNEManagementNet_IP>
# configure terminal
(config)#
ip access-list Restrict_Access_ToR
  permit ip {Allow_Access_Server}/32 any
  permit ip {NTPSERVER1}/32 {OAM_UPLINK_SwA_ADDRESS}/32
  permit ip {NTPSERVER2}/32 {OAM_UPLINK_SwA_ADDRESS}/32
  permit ip {NTPSERVER3}/32 {OAM_UPLINK_SwA_ADDRESS}/32
  permit ip {NTPSERVER4}/32 {OAM_UPLINK_SwA_ADDRESS}/32
  permit ip {NTPSERVER5}/32 {OAM_UPLINK_SwA_ADDRESS}/32
  deny ip any {CNE_Management_VIP}/32
  deny ip any {CNE_Management_SwA_Address}/32
  deny ip any {CNE_Management_SwB_Address}/32
  deny ip any {SQL_replication_VIP}/32
  deny ip any {SQL_replication_SwA_Address}/32
  deny ip any {SQL_replication_SwB_Address}/32
  deny ip any {OAM_UPLINK_SwA_ADDRESS}/32
  deny ip any {OAM_UPLINK_SwB_ADDRESS}/32
  deny ip any {SIGNAL_UPLINK_SwA_ADDRESS}/32
  deny ip any {SIGNAL_UPLINK_SwB_ADDRESS}/32
  permit ip any any
 
interface Ethernet1/51
  ip access-group Restrict_Access_ToR in
 
interface Ethernet1/52
  ip access-group Restrict_Access_ToR in
  3. Traffic egress out of cluster, including snmptrap traffic to SNMP trap receiver, and traffic goes to signal server: 
    [root@RMS1 ~]# ssh <user_name>@<ToRswitchA_CNEManagementNet_IP>
# configure terminal
(config)#
feature nat
ip access-list host-snmptrap
 10 permit udp 172.16.3.0/24 <snmp trap receiver>/32 eq snmptrap log
 
ip access-list host-sigserver
 10 permit ip 172.16.3.0/24 <signal server>/32
 
ip nat pool sig-pool 10.75.207.211 10.75.207.222 prefix-length 27
ip nat inside source list host-sigserver pool sig-pool overload add-route
ip nat inside source list host-snmptrap interface Ethernet1/51 overload
 
interface Vlan3
 ip nat inside
 
interface Ethernet1/51
 ip nat outside
 
interface Ethernet1/52
 ip nat outside
 
 
Run the same commands on ToR switchB
Configure Addresses for RMS iLOs, OA, EBIPA

Introduction

This procedure is used to configure RMS iLO addresses and add a new user account for each RMS other than the Bootstrap Host. When the RMSs are shipped and out of box after hardware installation and powerup, the RMSs are in a factory default state with the iLO in DHCP mode waiting for DHCP service. DHCP is used to configure the ToR switches, OAs, Enclosure switches, and blade server iLOs, so DHCP can be used to configure RMS iLOs as well.

Prerequisites

Procedure Configure Top of Rack 93180YC-EX Switches has been completed.

Limitations/Expectations

All steps are executed from the ssh session of the Bootstrap server.

References

HPE BladeSystem Onboard Administrator User Guide

Procedure

Following is the procedure to configure Addresses for RMS iLOs, OA, EBIPA:

  1. Setup the vlan interface to access ilo subnet. The ilo_vlan_id and ilo_subnet_cidr are the same value as in hosts.ini:
    $ nmcli con add con-name team0.<ilo_vlan_id> type vlan id <ilo_vlan_id> dev team0
$ nmcli con mod team0.<ilo_vlan_id> ipv4.method manual ipv4.addresses <unique ip in ilo subnet>/<ilo_subnet_cidr>
$ nmcli con up team0.<ilo_vlan_id>

    Example:

    $ nmcli con add con-name team0.2 type vlan id 2 dev team0
$ nmcli con mod team0.2 ipv4.method manual ipv4.addresses 192.168.20.11/24
$ nmcli con up team0.2
  2. Subnet and conf file address.

    The /etc/dhcp/dhcpd.conf file should already have been configured in OCCNE Configure Top of Rack 93180YC-EX Switches procedure.

    Configure Top of Rack 93180YC-EX Switches and dhcp started/enabled on the bootstrap server. The second subnet 192.168.20.0 is used to assign addresses for OA and RMS iLOs. The "next-server 192.168.20.11" option is same as the server team0.2 IP address.
  3. Display the dhcpd leases file at /var/lib/dhcpd/dhcpd.leases. The DHCPD lease file will display the DHCP addresses for all RMS iLOs, Enclosure OAs.
    # cat /var/lib/dhcpd/dhcpd.leases
# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.2.5
lease 192.168.20.101 {
  starts 4 2019/03/28 22:05:26;
  ends 4 2019/03/28 22:07:26;
  tstp 4 2019/03/28 22:07:26;
  cltt 4 2019/03/28 22:05:26;
  binding state free;
  hardware ethernet 48:df:37:7a:41:60;
}
lease 192.168.20.103 {
  starts 4 2019/03/28 22:05:28;
  ends 4 2019/03/28 22:07:28;
  tstp 4 2019/03/28 22:07:28;
  cltt 4 2019/03/28 22:05:28;
  binding state free;
  hardware ethernet 48:df:37:7a:2f:70;
}
lease 192.168.20.102 {
  starts 4 2019/03/28 22:05:16;
  ends 4 2019/03/28 23:03:29;
  tstp 4 2019/03/28 23:03:29;
  cltt 4 2019/03/28 22:05:16;
  binding state free;
  hardware ethernet 48:df:37:7a:40:40;
}
lease 192.168.20.106 {
  starts 5 2019/03/29 11:14:04;
  ends 5 2019/03/29 14:14:04;
  tstp 5 2019/03/29 14:14:04;
  cltt 5 2019/03/29 11:14:04;
  binding state free;
  hardware ethernet b8:83:03:47:5f:14;
  uid "\000\270\203\003G_\024\000\000\000";
}
lease 192.168.20.105 {
  starts 5 2019/03/29 12:56:23;
  ends 5 2019/03/29 15:56:23;
  tstp 5 2019/03/29 15:56:23;
  cltt 5 2019/03/29 12:56:23;
  binding state free;
  hardware ethernet b8:83:03:47:5e:54;
  uid "\000\270\203\003G^T\000\000\000";
}
lease 192.168.20.104 {
  starts 5 2019/03/29 13:08:21;
  ends 5 2019/03/29 16:08:21;
  tstp 5 2019/03/29 16:08:21;
  cltt 5 2019/03/29 13:08:21;
  binding state free;
  hardware ethernet b8:83:03:47:64:9c;
  uid "\000\270\203\003Gd\234\000\000\000";
}
lease 192.168.20.108 {
  starts 5 2019/03/29 09:57:02;
  ends 5 2019/03/29 21:57:02;
  tstp 5 2019/03/29 21:57:02;
  cltt 5 2019/03/29 09:57:02;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet fc:15:b4:1a:ea:05;
  uid "\001\374\025\264\032\352\005";
  client-hostname "OA-FC15B41AEA05";
}
lease 192.168.20.107 {
  starts 5 2019/03/29 12:02:50;
  ends 6 2019/03/30 00:02:50;
  tstp 6 2019/03/30 00:02:50;
  cltt 5 2019/03/29 12:02:50;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 9c:b6:54:80:d7:d7;
  uid "\001\234\266T\200\327\327";
  client-hostname "SA-9CB65480D7D7";
}
server-duid "\000\001\000\001$#\364\344\270\203\003Gim";
lease 192.168.20.107 {
  starts 5 2019/03/29 18:09:47;
  ends 6 2019/03/30 06:09:47;
  cltt 5 2019/03/29 18:09:47;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 9c:b6:54:80:d7:d7;
  uid "\001\234\266T\200\327\327";
  client-hostname "SA-9CB65480D7D7";
}
lease 192.168.20.108 {
  starts 5 2019/03/29 18:09:54;
  ends 6 2019/03/30 06:09:54;
  cltt 5 2019/03/29 18:09:54;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet fc:15:b4:1a:ea:05;
  uid "\001\374\025\264\032\352\005";
  client-hostname "OA-FC15B41AEA05";
}
lease 192.168.20.106 {
  starts 5 2019/03/29 18:10:04;
  ends 5 2019/03/29 21:10:04;
  cltt 5 2019/03/29 18:10:04;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet b8:83:03:47:5f:14;
  uid "\000\270\203\003G_\024\000\000\000";
  client-hostname "ILO2M2909004B";
}
lease 192.168.20.104 {
  starts 5 2019/03/29 18:10:35;
  ends 5 2019/03/29 21:10:35;
  cltt 5 2019/03/29 18:10:35;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet b8:83:03:47:64:9c;
  uid "\000\270\203\003Gd\234\000\000\000";
  client-hostname "ILO2M2909004F";
}
lease 192.168.20.105 {
  starts 5 2019/03/29 18:10:40;
  ends 5 2019/03/29 21:10:40;
  cltt 5 2019/03/29 18:10:40;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet b8:83:03:47:5e:54;
  uid "\000\270\203\003G^T\000\000\000";
  client-hostname "ILO2M29090048";
  4. Access RMS iLO from the DHCP address with default Administrator password. From the above dhcpd.leases file, find the IP address for the iLO name, the default username is Administrator, the password is on the label which can be pulled out from front of server.

    Note:

    The DNS Name on the pull-out label. The DNS Name on the pull-out label should be used to match the physical machine with the iLO IP since the same default DNS Name from the pull-out label is displayed upon logging in to the iLO command line interface, as shown in the example below. 
    # ssh Administrator@192.168.20.104
Administrator@192.168.20.104's password:
User:Administrator logged-in to ILO2M2909004F.labs.nc.tekelec.com(192.168.20.104 / FE80::BA83:3FF:FE47:649C)
iLO Standard 1.37 at  Oct 25 2018
Server Name:
Server Power: On
  5. Create RMS iLO new user. Create new user with customized username and password. 
    </>hpiLO-> create /map1/accounts1 username=root password=TklcRoot group=admin,config,oemHP_rc,oemHP_power,oemHP_vm

status=0
status_tag=COMMAND COMPLETED
Tue Apr  2 20:08:30 2019
User added successfully.
  6. Disable the DHCP before able to setup static IP. Setup static failed before DHCP is disabled.
    </>hpiLO-> set /map1/dhcpendpt1 EnabledState=NO
status=0
status_tag=COMMAND COMPLETED
Tue Apr  2 20:04:53 2019
Network settings change applied.
Settings change applied, iLO 5 will now be reset.
Logged Out: It may take several minutes before you can log back in.
CLI session stopped
packet_write_wait: Connection to 192.168.20.104 port 22: Broken pipe
  7. Setup RMS iLO static IP address. After a while after previous step, can login back with the same address (which is static IP now) and new username/password. If don't want to use the same address, go to next step to change the IP address. 
    # ssh <new username>@192.168.20.104
<new username>@192.168.20.104's password: <new password>
User: logged-in to ILO2M2909004F.labs.nc.tekelec.com(192.168.20.104 / FE80::BA83:3FF:FE47:649C)
iLO Standard 1.37 at  Oct 25 2018
Server Name:
Server Power: On
 
</>hpiLO-> set /map1/enetport1/lanendpt1/ipendpt1 IPv4Address=192.168.20.122 SubnetMask=255.255.255.0
 
status=0
status_tag=COMMAND COMPLETED
Tue Apr  2 20:22:23 2019
 
Network settings change applied.
Settings change applied, iLO 5 will now be reset.
Logged Out: It may take several minutes before you can log
back in.
  
CLI session stopped
 
packet_write_wait: Connection to 192.168.20.104 port 22:
Broken pipe
#
  8. Set EBIPA addresses for InterConnect Bays (Enclosure Switches). From bootstrap server, login to OA, set EBIPA addressed for the two enclosure switches. The addresses have to be in the subnet with server team0.2 address in order for TFTP to work. 
    Set address for each enclosure switch, note the last number 1 or 2 is the interconnect bay number.
  
OA-FC15B41AEA05> set ebipa interconnect 192.168.20.133 255.255.255.0 1
Entering anything other than 'YES' will result in the command not executing.
It may take each interconnect several minutes to acquire the new settings.
Are you sure you want to change the IP address for the specified
interconnect bays? yes
Successfully set 255.255.255.0 as the netmask for interconnect bays.
Successfully set interconnect bay # 1 to IP address 192.168.20.133
For the IP addresses to be assigned EBIPA must be enabled.
  
OA-FC15B41AEA05> set ebipa interconnect 192.168.20.134 255.255.255.0 2
Entering anything other than 'YES' will result in the command not executing.
It may take each interconnect several minutes to acquire the new settings.
Are you sure you want to change the IP address for the specified
interconnect bays? yes
Successfully set 255.255.255.0 as the netmask for interconnect bays.
Successfully set interconnect bay # 2 to IP address 192.168.20.134
For the IP addresses to be assigned EBIPA must be enabled.
  9. Set EBIPA addresses for Blade Servers. Set EBIPA addressed for all the blade servers. The addresses are in the same subnet with first server team0.2 address and enclosure switches. 
    OA-FC15B41AEA05> set ebipa server 192.168.20.141 255.255.255.0 1-16
Entering anything other than 'YES' will result in the command not executing.
Changing the IP address for device (iLO) bays that are enabled
causes the iLOs in those bays to be reset.
Are you sure you want to change the IP address for the specified
device (iLO) bays? YES
Successfully set 255.255.255.0 as the netmask for device (iLO) bays.
Successfully set device (iLO) bay # 1 to IP address 192.168.20.141
Successfully set device (iLO) bay # 2 to IP address 192.168.20.142
Successfully set device (iLO) bay # 3 to IP address 192.168.20.143
Successfully set device (iLO) bay # 4 to IP address 192.168.20.144
Successfully set device (iLO) bay # 5 to IP address 192.168.20.145
Successfully set device (iLO) bay # 6 to IP address 192.168.20.146
Successfully set device (iLO) bay # 7 to IP address 192.168.20.147
Successfully set device (iLO) bay # 8 to IP address 192.168.20.148
Successfully set device (iLO) bay # 9 to IP address 192.168.20.149
Successfully set device (iLO) bay #10 to IP address 192.168.20.150
Successfully set device (iLO) bay #11 to IP address 192.168.20.151
Successfully set device (iLO) bay #12 to IP address 192.168.20.152
Successfully set device (iLO) bay #13 to IP address 192.168.20.153
Successfully set device (iLO) bay #14 to IP address 192.168.20.154
Successfully set device (iLO) bay #15 to IP address 192.168.20.155
Successfully set device (iLO) bay #16 to IP address 192.168.20.156
For the IP addresses to be assigned EBIPA must be enabled.
OA-FC15B41AEA05>
  10. Add New User for OA.

    Create new user, set access level as ADMINISTRATOR, and assign access to all blades, all enclosure switches and OAs. After that, the username and password can be used to access OAs. 

    OA-FC15B41AEA05> ADD USER <username>
New Password: ********
Confirm     : ********
User "<username>" created.
You may set user privileges with the 'SET USER ACCESS' and 'ASSIGN' commands.
 
OA-FC15B41AEA05> set user access <username> ADMINISTRATOR
 
"<username>" has been given administrator level privileges.
 
OA-FC15B41AEA05> ASSIGN SERVER ALL <username>
 
<username> has been granted access to the valid requested bay(s)
 
OA-FC15B41AEA05> ASSIGN INTERCONNECT ALL <username>
 
<username> has been granted access to the valid requested bay(s)
 
OA-FC15B41AEA05> ASSIGN OA <username>
 
<username> has been granted access to the OA.
  11. From OA, go to each blade with "connect server <bay number>", add New User for each blade. 
    OA-FC15B41AEA05> connect server 4
 
Connecting to bay 4 ...
User:OAtmp-root-5CBF2E61 logged-in to ILO2M290605KP.(192.168.20.144 / FE80::AF1:EAFF:FE89:460)
iLO Standard Blade Edition 1.37 at  Oct 25 2018
Server Name:
Server Power: On
 
</>hpiLO->
 
</>hpiLO->  create /map1/accounts1 username=root password=TklcRoot group=admin,config,oemHPE_rc,oemHPE_power,oemHPE_vm
 
status=2
status_tag=COMMAND PROCESSING FAILED
error_tag=COMMAND SYNTAX ERROR
Tue Apr 23 16:18:58 2019
User added successfully.
  12. Change to static IP on OA. In order not reply on DHCP and make the OA address stable, change to static IP.

    Note:

    After the following change, on the active OA (could be the bay1 OA or bay2 OA), the OA session will be stuck due to the address change, make another server session ready to ssh with the new IP address and new root user. The change on the standby OA will not stuck the OA session. 
    OA-FC15B41AEA05> SET IPCONFIG STATIC 1 192.168.20.131 255.255.255.0
Static IP settings successfully updated.
These setting changes will take effect immediately.
  
OA-FC15B41AEA05> SET IPCONFIG STATIC 2 192.168.20.132 255.255.255.0
Static IP settings successfully updated.
These setting changes will take effect immediately.
OA-FC15B41AEA05>
Configure Legacy BIOS on Remaining Hosts
These procedures define the steps necessary to configure additional Legacy BIOS for all hosts in OCCNE. This includes steps that cannot be performed from the HP iLO 5 CLI prompt such as RAID configuration, changing the boot mode, and setting the primary and secondary boot devices.

Note:

The procedures in this document apply to the HP iLO console accessed via KVM. Each procedure is executed in the order listed.

Prerequisites

Procedure OCCNE Configure Addresses for RMS iLOs, OA, EBIPA is complete.

Limitations and Expectations

  1. Applies to HP iLO 5 only.
  2. Should the System Utility indicate (or defaults to) UEFI booting, then the user must go through the steps to reset booting back to the Legacy BIOS mode by following step: Change over from UEFI Booting Mode to Legacy BIOS Booting Mode.
  3. The procedures listed here apply to both Gen10 DL380 RMSs and Gen10 BL460c Blades in a C7000 enclosure.
  4. Access to the enclosure blades in these procedures is via the Bootstrap host using SSH on the KVM. This is possible because the prerequisites are complete. If the prerequisites are not completed before executing this procedure, the enclosure blades are only accessible via the KVM connected directly to the active OA. In this case the mouse is not usable and screen manipulations are performed using the keyboard ESC and directional keys.
  5. This procedure does NOT apply to the Bootstrap Host.

References

  1. HPE iLO 5 User Guide
  2. UEFI System Utilities User Guide for HPE ProLiant Gen10 Servers and HPE Synergy
  3. UEFI Workload-based Performance and Tuning Guide for HPE ProLiant Gen10 Servers and HPE Synergy
  4. HPE BladeSystem Onboard Administrator User Guide
  5. OCCNE Inventory File Preparation

Procedure

Following is the procedure to configure the Legacy BIOS on Remaining Hosts:
  1. Expose the System Configuration Utility on a RMS Host on the KVM. This procedure does not provide instructions on how to connect the KVM as this may be different on each installation.
    1. Once the remote console has been exposed, the system must be reset by manually pressing the power button on the front of the RMS host to force it through the restart process. When the initial window is displayed, hit the F9 key repeatedly. Once the F9 is highlighted at the lower left corner of the remote console, it should eventually bring up the main System Utility.
    2. The System Utilities screen is exposed in the remote console.
  2. Expose the System Utility for an Enclosure Blade.
    1. The blades are maintained via the OAs in the enclosure. Because each blade iLO has already been assigned an IP address from the prerequisites, the blades can each be reached using SSH from the Bootstrap host login shell on the KVM.
      1. SSH to the blade using the iLO IP address and the root user and password. This brings up the HP iLO prompt. 
        $ ssh root@<blade_ilo_ip_address>
Using username "root".
Last login: Fri Apr 19 12:24:56 2019 from 10.39.204.17
[root@localhost ~]# ssh root@192.168.20.141
root@192.168.20.141's password:
User:root logged-in to ILO2M290605KM.(192.168.20.141 / FE80::AF1:EAFF:FE89:35E)
iLO Standard Blade Edition 1.37 at  Oct 25 2018
Server Name:
Server Power: On
 
</>hpiLO->
      2. Use VSP to connect to the blade remote console. 
        </>hpiLO->vsp

      3. Power cycle the blade to bring up the System Utility for that blade.

        Note:

        The System Utility is a text based version of that exposed on the RMS via the KVM. The user must use the directional (arrow) keys to manipulate between selections, ENTER key to select, and ESC to go back from the current selection.
      4. Access the System Utility by hitting ESC 9.
    2. Enabling Virtualization

      This procedure provides the steps required to enable virtualization on a given Bare Metal Server. Virtualization can be configured using the default settings or via the default Workload Profiles.

      Verifying Default Settings
      1. Expose the System Utility by following step 1 or 2 depending on the hardware being configured.
      2. Select System Configuration
      3. Select BIOS/Platform Configuration (RBSU)
      4. Select Virtualization Options

        This view displays the settings for the Intel(R) Virtualization Technology (IntelVT), Intel(R) VT-d, and SR-IOV options (Enabled or Disabled). The default values for each option is Enabled.

      5. Select F10 if it is desired to save and stay in the utility or select the F12 if it is desired to save and exit to continue the current boot process.
  3. Change over from UEFI Booting Mode to Legacy BIOS Booting Mode:
    1. Expose the System Utility by following step 1 or 2 depending on the hardware being configured.
    2. Select System Configuration
    3. Select BIOS/Platform Configuration (RBSU)
    4. Select Boot Options.

      This menu defines the boot mode.

      If the Boot Mode is set to UEFI Mode then continue this procedure. Otherwise there is no need to make any of the changes below.
    5. Select Boot Mode
      This generates a warning indicating the following: 
      Boot Mode changes require a system reboot in order to take effect. Changing the Boot Mode can impact the ability of the server to boot the installed operating system. An operating system is installed in the same mode as the platform during the installation. If the Boot Mode does not match the operating system installation, the system cannot boot. The following features require that the server be configured for UEFI Mode: Secure Boot, IPv6 PXE Boot, Boot > 2.2 TB Disks in AHCI SATA Mode, and Smart Array SW RAID.
      Hit the ENTER key and two selections appear: UEFI Mode(highlighted) and Legacy BIOS Mode
    6. Use the down arrow key to select Legacy BIOS Mode and hit the ENTER. The screen indicates: A reboot is required for the Boot Mode changes.
    7. Hit F12. This displays the following: Changes are pending. Do you want to save changes? Press 'Y" to save and exit, 'N' to discard and stay, or 'ESC' to cancel.
    8. Hit the y key and an additional warning appears indicating: System configuration changed. A system reboot is required. Press ENTER to reboot the system.
      1. Hit ENTER to force a reboot.

        Note:

        The boot must go into the process of actually trying to boot from the boot devices using the boot order (not just go back through initialization and access the System Utility again). The boot should fail and the System Utility can be accessed again to continue any further changes needed.
      2. After the reboot, hit the ESC 9key sequence to re-enter the System Utility. Selecting System Configuration->BIOS/Platform Configuration (RBSU)->Boot Options. Verify the Boot Mode is set to Legacy Boot Mode UEFI Optimized Boot is set to Disabled
    10. Select F10 if it is desired to save and stay in the utility or select the F12 if it is desired to save and exit to complete the current boot process.
  4. Force PXE to boot from the first Embedded FlexibleLOM HPE Ethernet 10Gb 2-port Adapter.
    1. Expose the System Utility by following step 1 or 2 depending on the hardware being configured.
    2. Select System Configuration.
    3. Select BIOS/Platform Configuration (RBSU) .
    4. Select Boot Options. This menu defines the boot mode.
    5. Confirm the following settings: Boot Mode Legacy BIOS Mode UEFI Optimized Boot, and Boot Order Policy Retry Boot Order Indefinitely(this means it keeps trying to boot without ever going to disk). If not in Legacy BIOS Mode, follow procedure Change over from UEFI Booting Mode to Legacy BIOS Booting Mode.
    6. Select Legacy BIOS Boot Order In the default view, the 10Gb Embedded FlexibleLOM 1 Port 1 is at the bottom of the list.
    7. Move the 10 Gb Embedded FlexibleLOM 1 Port 1 entry up above the 1Gb Embedded LOM 1 Port 1 entry. To move an entry press the '+' key to move an entry higher in the boot list and the '-' key to move an entry lower in the boot list. Use the arrow keys to navigate through the Boot Order list.
    8. Select F10 if it is desired to save and stay in the utility or select the F12 it is desired to save and exit to continue the current boot process.
  5. Enabling Virtualization:

    This step provides the steps required to enable virtualization on a given Bare Metal Server. Virtualization can be configured using the default settings or via the Workload Profiles.

    Verifying Default Settings
    1. Expose the System Utility by following step 1 or 2 depending on the hardware being configured.
    2. Select System Configuration
    3. Select BIOS/Platform Configuration (RBSU)
    4. Select Virtualization Options

      This view displays the settings for the Intel(R) Virtualization Technology (IntelVT), Intel(R) VT-d, and SR-IOV options (Enabled or Disabled). The default values for each option is Enabled.

    5. Select F10 if it is desired to save and stay in the utility or select the F12 if it is desired to save and exit to continue the current boot process.
  6. Disable RAID Configurations:

    OCCNE does not currently support any RAID configuration. Follow this step to disable RAID settings if the default settings of the System Utility include any RAID configuration(s).

    Note: There may be more than one RAID Array set up. This procedure should be repeated for any RAID configuration.

    1. Expose the System Utility by following step 1 or 2 depending on the hardware being configured.
    2. Select System Configuration.
    3. Select Embedded RAID 1: HPE Smart Array P408i-a SR Gen 10.
    4. Select Array Configuration.
    5. Select Manage Arrays.
    6. Select Array A (or any designated Array Configuration if there are more than one).
    7. Select Delete Array. A warning is displayed indicating the following: 
      Deletes an Array. All the data on the logical drives that are part of deleted array will be lost. Also if the deleted array is the only one on the controller, the controller settings will be erased and its default configuration is restored.
    8. Hit ENTER, the changes are submitted and Delete Array Successful is displayed.
    9. Hit ENTER to go back to the main menu for the HPE Smart Array.
    10. Select F10 if it is desired to save and stay in the utility or select the F12 it is desired to save and exit to continue the current boot process.
  7. Enable the Primary and Secondary Boot Devices:

    This steps provide necessary to configure the primary and secondary bootable devices for a Gen10 Server.

    Note:

    There can be multiple configurations of hardware drives on the server that include both Hard Drives (HDD) and Solid State Hard Drives (SSD). SSDs are indicated by SATA-SSD ATA in the drive description. The commands below include two HDDs and two SSDs. The SSDs are not to be selected for this configuration. The actual selections may be different based on the hardware being updated.
    1. Expose the System Utility by following step 1 or 2 depending on the hardware being configured.
    2. Select System Configuration.
    3. Select Embedded RAID 1 : HPE Smart Array P408i-a SR Gen 10.
    4. Select Set Bootable Device(s) for Legacy Boot Mode.

      If the boot devices are not set then Not Set is displayed for the primary and secondary devices.

    5. Examine the list of available hardware drives. If one or more HDDs are available, continue with this step.

      Note:

      A single drive can be set as both the primary and secondary boot device but that is not part of this configuration.
    6. Select Bootable Physical Drive
    7. Select Port 1| Box:3 Bay:1 Size:1.8 TB SAS HP EG00100JWJNR. Note: This example includes two HDDs and two SSDs. The actual configuration may be different.
    8. Select Set as Primary Bootable Device.
    9. Hit ENTER.

      Note:

      There is no need to set the secondary boot device. Leave it as Not Set.
    10. Hit the ESC key to back out to the System Utilitiesmenu.
    11. Select F10 if it Is desired to save and stay in the utility or select the F12 if it Is desired to save and exit to continue the current boot process.
Configure Enclosure Switches

Introduction

This procedure is used to configure the 6127XLG enclosure switches.

Prerequisites

Limitations/Expectations

All steps are executed from a Keyboard, Video, Mouse (KVM) connection.

References

  1. https://support.hpe.com/hpsc/doc/public/display?docId=c04763537

Procedure

Following is the procedure to configure enclosure switches:
  1. Copy the 6127XLG configuration file from the Utility USB (See Installation PreFlight checklist: Create the OA 6127XLG Switch Configuration File) to the /var/lib/tftpboot directory on the Installer Bootstrap Host and verify it exists and the permissions. 
    $ cp /media/usb/6127xlg_irf.cfg /var/lib/tftpboot/6127xlg_irf.cfg

$ ls -l /var/lib/tftpboot/

total 1305096

-rw-r--r--. 1 root root        311 Mar 25 08:41 6127xlg_irf.cfg
  2. Modify the switch specific values in the /var/lib/tftpboot/6127xlg_irf.cfg file.

    These values are contained at Installation PreFlight checklist : Create the OA 6127XLG Switch Configuration File from column Enclosure_Switch. 

    
$ cd /var/lib/tftpboot
$ sed -i 's/{switchname}/<switch_name>/' 6127xlg_irf.cfg
$ sed -i 's/{admin_password}/<admin_password>/' 6127xlg_irf.cfg
$ sed -i 's/{user_name}/<user_name>/' 6127xlg_irf.cfg
$ sed -i 's/{user_password}/<user_password>/' 6127xlg_irf.cfg
  3. Access the InterConnect Bay1 6127XLG switch to configure the IRF (Intelligent Resilient Framework).

    Note:

    On a new switch the user is presented with the following when connecting to the console and must type CTRL_C or CTRL_D to break out of the loop.

    When trying to save the config, the following prompt is received: [HPE] [HPE] save The current configuration will be written to the device. Are you sure? [Y/N]: Before pressing ENTER you must choose 'YES' or 'NO'[Y/N]:y Please input the file name(*.cfg)[flash:/startup.cfg] (To leave the existing filename unchanged, press the enter key): User can leave this default startup.cfg unchanged, or change to another name. The cfg file will be used for next reboot.

    
$ ssh <oa username>@<oa address>

If it shows standby, ssh to the other OA address. 
 

OA-FC15B41AEA05> connect interconnect 1

....

<HPE>system-view

System View: return to User View with Ctrl+Z.

 

(Note: Run the following commands:) 

 irf member 1 priority 32

 interface range Ten-GigabitEthernet 1/0/17 to Ten-GigabitEthernet 1/0/20 

   shutdown 

   quit 

 irf-port 1/1

   port group interface Ten-GigabitEthernet1/0/17

   port group interface Ten-GigabitEthernet1/0/18

   port group interface Ten-GigabitEthernet1/0/19

   port group interface Ten-GigabitEthernet1/0/20

   quit 

 interface range Ten-GigabitEthernet 1/0/17 to Ten-GigabitEthernet 1/0/20 

   undo shutdown 

   quit 

 save 

 irf-port-configuration active
  4. Access the InterConnect Bay2 6127XLG switch to re-number to IRF 2. 
    OA-FC15B41AEA05> connect interconnect 2

....

<HPE>system-view

System View: return to User View with Ctrl+Z.

[HPE] irf member 1 renumber 2

Renumbering the member ID may result in configuration change or loss. Continue?[Y/N]Y

[HPE]save

The current configuration will be written to the device. Are you sure? [Y/N]:Y

Please input the file name(*.cfg)[flash:/startup.cfg]

(To leave the existing filename unchanged, press the enter key):

Validating file. Please wait...

Saved the current configuration to mainboard device successfully.

[HPE]quit

<HPE>reboot

Start to check configuration with next startup configuration file, please wait.........DONE!

This command will reboot the device. Continue? [Y/N]:Y

Now rebooting, please wait...

 

System is starting...
  5. Configure the IRF on Bay2 6127XLG switch

    After rebooting, the interfaces will begin with number 2 such as Ten-GigabitEthernet2/0/17, Ten-GigabitEthernet2/1/5. Run the following commands: 

    system-view 

 interface range Ten-GigabitEthernet 2/0/17 to Ten-GigabitEthernet 2/0/20

   shutdown 

   quit 

 irf-port 2/2

   port group interface Ten-GigabitEthernet2/0/17

   port group interface Ten-GigabitEthernet2/0/18

   port group interface Ten-GigabitEthernet2/0/19

   port group interface Ten-GigabitEthernet2/0/20

   quit 

 interface range Ten-GigabitEthernet 2/0/17 to Ten-GigabitEthernet 2/0/20 

   undo shutdown 

   quit 

 save 

 irf-port-configuration active
  6. Run "reboot" command on both switches.
    <HPE>reboot
Start to check configuration with next startup configuration file, please wait.........DONE!
This command will reboot the device. Continue? [Y/N]:Y
Now rebooting, please wait...
 
System is starting...
  7. Verify the IRF for the 6127XLG switches. When reboot is finished, verify IRF is working with both member and ports from previous two switches, which form IRF to act as one switch now. 
    
<HPE>system-view

System View: return to User View with Ctrl+Z.

[HPE]display irf configuration

 MemberID NewID    IRF-Port1                     IRF-Port2

 1        1        Ten-GigabitEthernet1/0/17     disable

                   Ten-GigabitEthernet1/0/18

                   Ten-GigabitEthernet1/0/19

                   Ten-GigabitEthernet1/0/20

 2        2        disable                       Ten-GigabitEthernet2/0/17

                                                 Ten-GigabitEthernet2/0/18

                                                 Ten-GigabitEthernet2/0/19

                                                 Ten-GigabitEthernet2/0/20

[HPE]
  8. Configure the IRF switch with predefined configuration file.
    
<HPE>tftp 192.168.20.11 get 6127xlg_irf.cfg startup.cfg

startup.cfg already exists. Overwrite it? [Y/N]:Y

Press CTRL+C to abort.

 % Total % Received % Xferd Average Speed Time Time Time Current

 Dload Upload Total Spent Left Speed

100 9116 100 9116 0 0 167k 0 --:--:-- --:--:-- --:--:-- 178k

 

<HPE>system-view

System View: return to User View with Ctrl+Z.

[HPE]configuration replace file flash:/startup.cfg

Current configuration will be lost, save current configuration? [Y/N]:N

Now replacing the current configuration. Please wait ...

Succeeded in replacing current configuration with the file flash:/startup.cfg.

[<switch_name>]save flash:/startup.cfg

The current configuration will be saved to flash:/startup.cfg. Continue? [Y/N]:Y

flash:/startup.cfg exists, overwrite? [Y/N]:Y

Now saving current configuration to the device.

Saving configuration flash:/startup.cfg.Please wait...

Configuration is saved to device successfully.

[<switch_name>]

Bastion Host Installation

This section outlines the use of the Installer Bootstrap Host to provision db-2/RMS2 with an operating system and configure it to fulfill the role of Database Host. After the Bastion Host is created, it is used to complete the installation of OCCNE.

Provision Second Database Host (RMS2) from Installer Bootstrap Host (RMS1)

Table 2-1 Terminology used in Procedure

Name Description
bastion_full_name This is the full name of the Bastion Host as defined in the hosts.ini file.

Example: bastion-2.rainbow.us.labs.oracle.com

bastion_kvm_host_full_name This is the full name of the KVM server (usually RMS2/db-2) that hosts the Bastion Host VM.

Example: db-2.rainbow.us.labs.oracle.com

bastion_kvm_host_ip_address

This is the IPv4 ansible_host IP address of the server (usually RMS2/db-2) that hosts the Bastion Host VM.

Example: 172.16.3.5

bastion_short_name This is the name of the Bastion Host derived from the bastion_full_name up to the first ".".

Example: bastion-2

bastion_external_ip_address This is the external address for the Bastion Host

Example : 10.75.148.5 for bastion-2

bastion_ip_address

This is the internal IPv4 "ansible_host" address of the Bastion Host as defined within the hosts.ini file.

Example: 172.16.3.100 for bastion-2

cluster_full_name This is the name of the cluster as defined in the hosts.ini file field: occne_cluster_name.

Example: rainbow.us.labs.oracle.com

cluster_short_name This is the short name of the cluster derived from the cluster_full_name up to the the first ".".

Example: rainbow

Note:

The Bootstrap Host must be setup to use root/<customer_specific_root_password> as the credentials to access it. Setting that user/password is part of the instructions at: Installation of Oracle Linux 7.x on Bootstrap Host.

Procedure

Following is the procedure to install Bastion:
  1. Copy the necessary files from the Utility USB to support the OS Install:

    This procedure is used to provide the steps for copying all supporting files from the Utility USB to the appropriate directories so that the Provision Container successfully installs OL7 onto RMS2.

    1. Login to the Bootstrap Host using the root credentials configured during OS installation of the Bootstrap Host.
    2. Create the directories needed on the Installer Bootstrap Host. 
      $ mkdir -p /var/occne/cluster/<cluster_short_name>/yum.repos.d
    3. Mount the Utility USB.

      Note:

      Follow the instructions for mounting a USB in Linux are at: Installation of Oracle Linux 7.x on Bootstrap Host.
    4. Copy the hosts.ini file (created using procedure: OCCNE Inventory File Preparation) into the /var/occne/cluster/<cluster_short_name>/ directory.

      This hosts.ini file defines the Bastion KVM Host to the Provision Container running the provision image downloaded from the repo. 

      $ cp /<path_to_usb>/hosts.ini /var/occne/cluster/<cluster_short_name>/hosts.ini
 
Example:
$ cp /media/usb/hosts.ini /var/occne/cluster/rainbow/hosts.ini
    5. Edit the /var/occne/cluster/<cluster_short_name>/hosts.ini file to include the ToR host_net (vlan3) VIP for NTP clock synchronization. Use the ToR VIP address (ToRswitch_Platform_VIP ) as defined in procedure: Installation PreFlight Checklist : Complete OA and Switch IP SwitchTable as the NTP source. Update the ntp_server field with the VIP address. Update the occne_repo_host_address to point to this Bootstrap Host internal address. This is being used for PXE booting and accessing the NFS share on the Installer Bootstrap Host (db-1/RMS1). 
      Example (from hosts.sample.ini):
 
ntp_server='172.16.3.1'
occne_repo_host_address='172.16.3.4'
    6. Follow procedure Populate the MetalLB Configuration File to copy the MetalLB configuration file into the /var/occne/cluster/<cluster_short_name>/ directory and configure it. 
      $ cp /<path_to_usb>/mb_configmap.yaml /var/occne/cluster/<cluster_short_name>/mb_configmap.yaml
 
Example:
$ cp /media/usb/mb_configmap.yaml /var/occne/cluster/rainbow/mb_configmap.yaml
    7. Copy the customer specific .repo file from the Utility USB to the Installer Bootstrap Host.

      This is the .repo file created by the customer that provides access to the onsite (within their network) yum repositories needed to complete the full deployment of OCCNE onto the Installer Bootstrap Host. It needs to be in two places, one for the local system, and one for the target systems. 

      $ cp /<path_to_usb>/<customer_specific_repo>.repo /var/occne/cluster/<cluster_short_name>/yum.repos.d/.
$ cp -r /var/occne/cluster/<cluster_short_name>/yum.repos.d /var/occne/.
$ echo "reposdir=/var/occne/yum.repos.d" >> /etc/yum.conf
 
Example:
$ cp /media/usb/ol7-mirror.repo /var/occne/cluster/rainbow/yum.repos.d/
$ cp -r /var/occne/cluster/rainbow/yum.repos.d /var/occne/
$ echo "reposdir=/var/occne/yum.repos.d" >> /etc/yum.conf
  2. Set up the /etc/hosts file for the Central Repo and Verify Access:
    1. Add an entry to the /etc/hosts file on the Installer Bootstrap Host to provide a name mapping for the Customer Central Repository. 
      $ vi /etc/hosts
 
Example:
10.75.200.217 rainbow-reg

To Verify:
$ ping <central_repo_name>
 
Example:
# ping rainbow-reg
PING reg-1 (10.75.200.217) 56(84) bytes of data.
64 bytes from reg-1 (10.75.200.217): icmp_seq=1 ttl=61 time=0.248 ms
64 bytes from reg-1 (10.75.200.217): icmp_seq=2 ttl=61 time=0.221 ms
64 bytes from reg-1 (10.75.200.217): icmp_seq=3 ttl=61 time=0.239 ms
    2. Verify the repo access execute the following command: 
      $ yum repolist

Example:
$ yum repolist
Loaded plugins: ulninfo
repo id                repo name                                                                   status
!UEKR5/x86_64          Unbreakable Enterprise Kernel Release 5 for Oracle Linux 7 (x86_64)             80
!addons/x86_64         Oracle Linux 7 Addons (x86_64)                                                  91
!developer/x86_64      Packages for creating test and development environments for Oracle Linux 7     226
!developer_EPEL/x86_64 Packages for creating test and development environments for Oracle Linux 7  13,246
!ksplice/x86_64        Ksplice for Oracle Linux 7 (x86_64)                                            393
!latest/x86_64         Oracle Linux 7 Latest (x86_64)                                               5,401
repolist: 19,437
  3. Copy the OL7 ISO to the Installer Bootstrap Host:

    The iso file must be accessible from a Customer Site Specific repository. This file should be accessible because the ToR switch configurations were completed in procedure: Configure Top of Rack 93180YC-EX Switches

    Copy the OL7 ISO file to the /var/occne directory. The example below uses OracleLinux-7.5-x86_64-disc1.iso. If this file was copied to the Utility USB, it can be copied from there into the same directory on the Bootstrap Host.

    Note:

    If the user copies this ISO from their laptop then they must use an application like WinSCP pointing to the Management Interface IP. 
    $ scp <usr>@<site_specific_address>:/<path_to_iso>/OracleLinux-7.5-x86_64-disc1.iso /var/occne/OracleLinux-7.5-x86_64-disc1.iso
  4. Install Packages onto the Installer Bootstrap Host. Use YUM to install necessary packages onto the installer Bootstrap Host. 
    $ yum install docker-engine nfs-utils ansible
  5. Set up access to the Docker Registry on the Installer Bootstrap Host:
    1. Copy the docker registry certificate to two places on the Bootstrap Host.

      Note:

      How to obtain the docker registry certificate <source> is not necessarily covered in the procedure. The user can use the instructions at reference 1 to understand this more thoroughly. 
      $ mkdir -p /var/occne/certificates
$ cp <source>.crt /var/occne/certificates/<occne_private_registry>:<occne_private_registry_port>.crt
$ mkdir -p /etc/docker/certs.d/<occne_private_registry>:<occne_private_registry_port>
$ cp <source>.crt /etc/docker/certs.d/<occne_private_registry>:<occne_private_registry_port>/ca.crt
 
Example:
$ mkdir -p /var/occne/certificates
$ cp <source>.crt /var/occne/certificates/rainbow_reg:5000.crt
$ mkdir -p /etc/docker/certs.d/rainbow_reg:5000
$ cp <source>.crt /etc/docker/certs.d/rainbow_reg:5000/ca.crt
    2. Start the docker daemon. 
      $ systemctl daemon-reload
$ systemctl restart docker
$ systemctl enable docker
  
Verify docker is running:
$ ps -elf | grep docker
$ systemctl status docker
  6. Setup NFS on the Installer Bootstrap Host:

    Run the following commands using sudo (nfs-utils has already been installed in previous steps).

    Note:

    The IP address used in the echo command is the Platform VLAN IP Address (VLAN 3)of the Bootstrap Host (RMS 1) as given in: Installation PreFlight Checklist : Site Survey Host Table. 
    $ echo '/var/occne 172.16.3.4/24(ro,no_root_squash)' >> /etc/exports
$ systemctl start nfs-server
$ systemctl enable nfs-server
  
Verify nfs is running:
$ ps -elf | grep nfs
$ systemctl status nfs-server
  7. Set up the Boot Loader on the Installer Bootstrap Host:
    $ mkdir -p /var/occne/pxelinux
$ mount -t iso9660 -o loop /var/occne/OracleLinux-7.5-x86_64-disc1.iso /mnt
$ cp /mnt/isolinux/initrd.img /var/occne/pxelinux
$ cp /mnt/isolinux/vmlinuz /var/occne/pxelinux
  8. Verify and Set the PXE Configuration File Permissions on the Installer Bootstrap Host:

    Note:

    Each file configured in the step above must be open for read and write permissions. 
    $ chmod -R 777 /var/occne/pxelinux
  9. Disable DHCP and TFTP on the Installer Bootstrap Host. The TFTP and DHCP services running on the Installer Bootstrap Host may still be running. These services must be disabled. 
    $ systemctl stop dhcpd
$ systemctl disable dhcpd
$ systemctl stop tftp
$ systemctl disable tftp
  10. Disable SELINUX: Set SELINUX to permissive mode. In order to successfully set the SELINUX mode, a reboot of the system is required. The getenforce command is used to determine the status of SELINUX. 
    $ getenforce
active
  
If the output of this command displays "active", change it to "permissive" by editing the /etc/selinux/config file.
  
$ vi /etc/selinux/config
  
Change the SELINUX variable to passive: SELINUX=permissive
save the file
  
Reboot the system: reboot
  11. Generate the SSH private and public keys on Bootstrap Host.

    This command generates a private and public key for the cluster. These keys are passed to the Bastion Host and used to communicate to other nodes from that Bastion Host. The public key is passed to each node on OS install. Do not supply a passphrase when it asks for one. Click Enter.

    Note:

    The private key (occne_id_rsa) must be copied to a server that going to access the Bastion Host because the Bootstrap Host is repaved. This key is used later in the procedure to access the Bastion Host after it has been created.
    Execute the following commands on the Bootstrap Host: 
    $ mkdir -m 0700 /var/occne/cluster/<cluster_short_name>/.ssh
$ ssh-keygen -b 4096 -t rsa -C "occne installer key" -f "/var/occne/cluster/<cluster_short_name>/.ssh/occne_id_rsa" -q -N ""
  12. Execute the OS Install and Bastion VM Creation on Bastion KVM Host (RMS2) from the Installer Bootstrap Host:
    1. Run the docker commands below to perform the OS install and Bastion Host VM creation on the Bastion KVM Host (RMS2): 
      $ docker run --rm --network host --cap-add=NET_ADMIN -v /var/occne/cluster/<cluster_short_name>/:/host -v /var/occne/:/var/occne:rw -e "OCCNEARGS=--limit=<bastion_full_name>,<bastion_kvm_host_full_name>,localhost" <image_name>:<image_tag>
  
  
Example:
  
$ docker run -it --rm --network host --cap-add=NET_ADMIN -v /var/occne/cluster/rainbow/:/host -v /var/occne/:/var/occne:rw -e "OCCNEARGS=--limit=bastion-2.rainbow.lab.us.oracle.com,db-2.rainbow.lab.us.oracle.com,localhost" winterfell:5000/occne/provision:1.3.0
    2. Verify that Bastion Host VM is installed by logging into RMS2/db-2 and issuing the following command. The <ansible_host> field (which is an IPv4 address) is derived from the hosts.ini file db-2 entry for host_hp_gen_x groups.

      Note:

      This command is optional. Had a failure actually occurred, the docker run command would have experienced failures. 
      ssh -i /var/occne/cluster/<cluster_short_name>/.ssh/occne_id_rsa  admusr@<oam_host>
 
$ sudo virsh list
  
Example:
ssh -i /var/occne/cluster/rainbow/.ssh/occne_id_rsa  admusr@10.75.148.6
$ sudo virsh list
 
 Id    Name                           State
----------------------------------------------------
 11    bastion-2.rainbow.lab.us.oracle.com running
    3. Login to Bastion Host from the Boostrap Host as admusr using the generated key from the /var/occne/cluster/<cluster_short_name> directory to confirm the VM is set up correctly. The <oam_host> field (which is an IPv4 address) is derived from the hosts.ini file bastion-2 entry for the host_kernel_virtual group.

      Note:

      This command is optional. Had a failure actually occurred, the docker run command would have experienced failures. 
      ssh -i /var/occne/cluster/<cluster_short_name>/.ssh/occne_id_rsa  admusr@<oam_host>
 
Example:
ssh -i /var/occne/cluster/rainbow/.ssh/occne_id_rsa admusr@10.75.148.5

Automated Installation

This section details the steps required to execute the automated configuration of the Bastion Host VM. This consists of two main section:

  1. Setting up and executing the deploy.sh script on the Bootstrap Host.
  2. Accessing the Bastion Host and executing the final commands to execute the pipeline.sh script to complete the Bastion Host configuration and deploy the OCCNE cluster.
Following is the procedure to perform automated installation of the Bastion Host VM:
  1. Setting up for and executing the deploy.sh script on the Bootstrap Host:
    The deploy.sh script performs the initial pre-configuration of the Bastion host. This includes installing ansible, executing the ansible playbook configBastionHost.yaml to setup the initial files and staging directories on the Bastion Host and executing the pipeline to setup the artifacts directory. The script is executed on the Bootstrap Host using a set of environment variables that can be initialized on the command line along with the deploy.sh script. These variables include the following:

    Table 2-2 Environmental Variables

    Name Comment Example usage
    CENTRAL_REPO Defines the customer specific repository host name. Note: This would be used in conjunction with CENTRAL_REPO_IP and CENTRAL_REPO_DOCKER_PORT. CENTRAL_REPO=customer_repo \ CENTRAL_REPO_IP=<IP_Address> \ CENTRAL_REPO_DOCKER_PORT=5000
    CENTRAL_REPO_IP Defines the customer specific repository IPv4 address. See above.
    CENTRAL_REPO_DOCKER_PORT Defines the customer specific repository docker port number. Defaults to 5000. See above.
    OCCNE_CLUSTER Defines the cluster short name. OCCNE_CLUSTER=rainbow
    OCCNE_BASTION Bastion Host full name OCCNE_BASTION=bastion-2.rainbow.us.labs.oracle.com
    OCCNE_VERSION The version tag of the image releases OCCNE_VERSION=1.6.0
  2. Copy necessary files from Utility USB to the Bootstrap Host staging directory:
    1. The MySQL .zip file (ex: V980756-01.zip) must be copied to the staging directory /var/occne directory. This file should be provided from the Utility USB. This file is used for installing the ndb MySQL nodes. 
      $ cp /<usb_dev>/db/*.zip /var/occne/*.zip
    2. Copy the configBastionHost.yaml file from the Customer Utility USB to the staging directory /var/occne/. 
      $ cp /<usb_dev>/configBastionHost.yaml /var/occne/.
    3. Copy the deploy.sh script from the Customer Utility USB to the staging directory at /var/occne/ and set the file to executable mode.
      $ cp /<usb_dev>/deploy.sh /var/occne/. 
$ chmod +x /var/occne/deploy.sh
  3. Execute the deploy.sh script from the /var/occne/ directory with the required parameters set. 
    $ export CENTRAL_REPO=<customer_specific_repo_name>
$ export CENTRAL_REPO_IP=<customer_specific_repo_ipv4>
$ export OCCNE_CLUSTER=<cluster_short_name>
$ export OCCNE_BASTION=<bastion_full_name>
$ ./deploy.sh
 
Customer Example:
$ export CENTRAL_REPO=central-repo
$ export CENTRAL_REPO_IP=10.10.10.10
$ export OCCNE_CLUSTER=rainbow
$ export OCCNE_BASTION=bastion-2.rainbow.us.labs.oracle.com 		
$ ./deploy.sh
 
The command above can be executed like the following:
$ CENTRAL_REPO=central-repo CENTRAL_REPO_IP=10.10.10.10 OCCNE_CLUSTER=rainbow OCCNE_BASTION=bastion-2.rainbow.us.labs.oracle.com ./deploy.sh
  4. Execute the final deploy on Bastion Host:

    The following commands are executed from the Bastion Host to complete the Bastion Host configuration and deploy OCCNE on the Bare Metal system.

    Note:

    The Bootstrap Host cannot be used to access the Bastion Host as it will be re-paved from execution of this command.
    1. Login to the Bastion Host as admusr. The private key that was saved earlier should be used to access the Bastion Host from a server other than the Bootstrap Host using the ssh command. This private key should be copied to the /home/<user>/.ssh directory on that server as id_rsa using scp (or winSCP from a desktop PC). The permissions of the key must be set to 0600 using the command: chmod 0600 ~/.ssh/id_rsa 
      $ ssh -i ~/.ssh/id_rsa admusr@<bastion_external_ip_address>
    2. Execute the following command to complete the deployment of OCCNE from the Bastion Host (excluding re-install on the Bastion Host and its KVM host, which are already setup). This action will re-pave the Bootstrap Host RMS. 
      $ export OCCNE_PROV_DEPLOY_ARGS='--limit=!<bastion_full_name>,!<bastion_kvm_host_full_name>'
$ export OCCNE_DB_ARGS='--extra-vars=mate_site_db_replication_ip=<remote_site_db_replication_service_ip>,occne_mysqlndb_cluster_id=<mysqlndb_cluster_identifier>'
$ /var/occne/cluster/<cluster_short_name>/artifacts/pipeline.sh
 
Customer Example:
$ export OCCNE_PROV_DEPLOY_ARGS='--limit=!bastion-2.rainbow.lab.us.oracle.com,!db-2.rainbow.lab.us.oracle.com'
$ export OCCNE_DB_ARGS='--extra-vars=mate_site_db_replication_ip=10.75.182.88,occne_mysqlndb_cluster_id=2'
$ /var/occne/cluster/rainbow/artifacts/pipeline.sh
 
To save the output from the pipeline.sh script the command can be written as:
$ /var/occne/cluster/rainbow/artifacts/pipeline.sh | tee pipeline$(date +"%F_%H%M%S").log

    Note:

    While Installing on the First Site ignore OCCNE_DB_ARGS configuration parameter which Provides Mate Site DB Replication Service Load Balancer IP. Provide the Mate Site DB Replication Service Load Balancer IP and MySQL Cluster identifier while Installing MYSQL NDB Cluster on second site.
  5. Change MySQL root user password. Refer to Change MySQL root user password
  6. Update the Bastion KVM Host repo file:
    Since db-2 was not part of the final OS install and cluster deploy, it's /var/occne/yum.repos.d/*.repo file is not pointing to the Bastion Host as its YUM repo. That file on RMS2/db-2 must be updated so that it now points to the Bastion Host as the repo. After the Bastion Host was created, the .repo file that was copied onto the Bastion Host has the correct settings. That file can just be copied back to RMS2/db-2.
    1. From the bastion Host, login to the Bastion Host KVM Host, using the occne private key and the internal host IP address for that node (extracted from the hosts.ini file):
      $ ssh -i /var/occne/cluster/<cluster_short_name>/.ssh/occne_id_rsa admusr@<bastion_ip_address>
    2. Remove the existing .repo files:
      $ sudo rm /var/occne/yum.repos.d/*.repo
    3. Copy the Bastion specific .repo file from the Bastion Host to the Bastion KVM Host. Execute this command from the Bastion KVM Host:
      scp <bastion_short_name>:/var/occne/yum.repos.d/*.repo /var/occne/yum.repos.d/ 
Example:
scp bastion-2:/var/occne/yum.repos.d/*.repo /var/occne/yum.repos.d/

Virtualized CNE Installation

This procedure details the steps necessary to configure and install an OCCNE cluster in an OpenStack Environment.

Prerequisites

  1. The user must have access to an existing OpenStack Environment including the OpenStack Desktop.
  2. The OpenStack Environment is configured with appropriate resource flavors and network resources for resource allocation to the VMs created using this procedure.
  3. Octavia Load Balancing plugin must be installed on the OpenStack Environment.
  4. Users must have a public SSH key that can be configured for logging into the Bootstrap Host. This key should be placed into the customer OpenStack Environment prior to running this procedure using the following:

    Use the Import Key tab on the Launch Instance→Key Pair dialog or via the Compute→Access and Security screen.

Limitation/Expectations

  1. It is expected that the user is familiar with the use of OpenStack as a virtualized provider and the OpenStack client.

    Note:

    All OpenStack client commands listed in this procedure are executed from the Bootstrap Host after it has been instantiated.
  2. All necessary images, binaries, etc, have been downloaded from Oracle OSDC prior to executing this procedure and these resources are available for use in this procedure.
  3. The customer has made available a central repository for all images, binaries, helm charts, etc, prior to executing this procedure.
  4. All necessary networking (whether using fixed IPs or floating IPs) has been defined for use on the Openstack provider.

Note:

The OpenStack commands in the procedures below are from a specific version of the OpenStack Desktop. The desktop used at the customer site may be slightly different depending on the version installed. The operations should be compatible.

Download the OL7 Image

This section describes procedure on how to download the OL7 image from OSDC.
  1. Go to https://edelivery.oracle.com and login using appropriate credentials.
  2. Enter “Oracle Linux Virtual Machine Image for Openstack” in search field and click on Search.
  3. Select DLP: Oracle Linux 7 Virtual Machine Image for Openstack 7.0.0.0.0 and click on Add to Cart.
  4. With defaults selected, accept the license agreement. Click on Continue.
  5. Click on link for: V979992-01.zip Oracle Linux 7.5 Virtual Machine Image for OpenStack, 458.4 MB
  6. Click on Download.

    Note:

    1. Optionally use the link for 7.6 instead: V981347-01.zip Oracle Linux 7.6 Virtual Machine Image for OpenStack, 487.9 MB
    2. These zip archives files contains the following respective files, OracleLinux-7.5-x86_64.qcow2 and OracleLinux-7.6-x86_64.qcow2.

Upload an Image to an OpenStack Environment

This is the process of uploading the qcow2 image. The image is provided at OSDC.

Note:

This procedure is executed from the OpenStack Desktop.
  1. Login to the Openstack desktop using the specific credentials.
  2. Select Compute and then Images from the left navigation pane.
  3. Click on +Create Image button. The Create Image window is displayed.
  4. Enter a name for the image. Use the same name of the image which was used to create and download the image (ex: occne_bootstrap-1.5.0).
  5. Under Image Source, select File. This will enable a File* button.
  6. Click on Browse to bring up a Windows Explorer dialog.
  7. From the Windows dialog, select the image that was downloaded from the Release Artifacts as mentioned above. This will add the image name into the OpenStack Create Image dialog. Choose the Format from the drop-down menu.

    Note:

    This should be set to QCOW2 - QEMU Emulator. If not set, use the pulldown menu to select that format.
  8. Select the Create Image button at the bottom right of the dialog. This will start the image upload process. It will take a while so be patient. You will not be able to actually see the image being uploaded even if you log into another OpenStack instance.
  9. When the process is complete, the image should be listed in the Images screen.

Bootstrap Host Creation

The Bootstrap Host is provisioned to drive the creation of the virtualized cluster using Terraform, the OpenStack Client, and Ansible Playbook(s). A qcow2 image was provided as part of the OSDC download and should be available on the users OpenStack Environment as per the previous section of this document.

Note:

The examples below are for reference only. While the steps are correct the actual values used will be different. The following steps are to be performed manually on the customer specific Openstack Environment Desktop.
  1. Login to the OpenStack Environment using your OpenStack credentials, the appropriate domain and project name.
  2. Select Compute and then Instances.
  3. Select the Launch Instance tab on the upper right. A dialog will appear to configure a VM instance.
  4. Enter an Instance Name (for example: occne-<name>). Leave the Availability Zone and Count set as is.
  5. Select Source on the left hand side of the dialog. A new dialog appears (Note: there might be a long list of available images to choose from).
  6. Make sure the Select Boot Source drop-down is set to Image.
  7. Enter occne-bootstrap in the Available filter. This will display the occne-bootstrap-<x.y.z> image uploaded in the previous sections of this procedure.
  8. Select the OCCNE Bootstrap Host image by selecting the "↑" on the right side of the image listing. This adds the image as the source for this VM.
  9. Select Flavor on the left hand side.
  10. Enter a string (not case sensitive) which best describes the flavor being used for this customer specific OpenStack Environment in the Available search filter. This shortens the list of possible choices.
  11. Select the appropriate customer specific flavor (for example: OCCNE-Bootstrap-host) by selecting the "↑" on the right side of the flavor listings. This adds the resources to the Launch Instance dialog.

    Note:

    The BSH image requires a flavor that includes a disk size of 40GB or higher. The RAM size should be 8GB or higher although that is not a restriction.
  12. Select Networks on the left hand side.
  13. Enter the appropriate network name as defined by the customer with the OpenStack Environment (example: ext-net) in the Available search filter. This shortens the list of possible choices.
  14. Select the appropriate network by selecting the "↑" on the right side of the flavor listings. This adds the external network interface to the Launch Instance dialog.
  15. Select Key Pair. This dialog assumes you have already uploaded a public key to OpenStack (see Prerequisites above). Select the appropriate key by selecting the "↑" on the right side of the key pair listings. This adds the public key to the authorized_keys file on the Bootstrap Host.
  16. Select Configuration. This screen allows the user to add configuration data which is used by cloud-init to set on the VM, the initial cloud-user and hostname/FQDN additions to the /etc/hosts file. Use the following configuration. This must be copied into the Customization Script text box. Make sure the fields marked as <instance_name_from_details_screen> are updated with the instance name you used in step 5 above. Leave the other fields on this dialog set to their default setting.
    #cloud-config
   hostname: <instance_name_from_details_screen>
   fqdn: <instance_name_from_details_screen>
   system_info:
     default_user:
       name: cloud-user
       lock_passwd: false
   write_files:
     - content: |
         127.0.0.1  localhost localhost4 localhost4.localdomain4 <instance_name_from_details_screen>
         ::1        localhost localhost6 localhost6.localdomain6 <instance_name_from_details_screen>
       path: /etc/hosts
       owner: root:root
       permissions: '0644'
  17. Select Launch Instance at the lower right side of the initial dialog. This will launch the creation of the VM. This can be observed back at the Instances screen.

Pre-deployment Configuration

The commands in this procedure are executed from the Bootstrap Host. All terraform commands are executed from the /var/terraform directory.

Obtain the TLS Certificate for OpenStack

Depending on the Customer's environment it is very likely that the customer's OpenStack uses certificates for TLS access to the API. Without this certificate, OpenStack commands will not work. Customer's may have to obtain this certificate before using OpenStack client commands.

Note:

This step is only required if the customer Openstack environment requires a TLS certificate to access the controller from the cluster nodes and the bootstrap host
  1. Contact the OpenStack admin to provide the required TLS certificate to access the client commands (example, in an Oracle OpenStack system installed with kolla, the certificate will be located at /etc/kolla/certiifcates/haproxy-ca.crt).
  2. Copy the certificate to the Bootstrap Host at location: /etc/pki/<OpenStack_release_name>/haproxy-ca.crt (ex. /etc/pki/kolla/haproxy-ca.crt) (If /etc/pki/<OpenStack_release_name> does not exist, it can be created using command: mkdir -p /etc/pki/<OpenStack_release_name>.
  3. Set the environment variable OS_CACERT to /var/occne/cacertificates/openstack-cacert.pem using the command: export OS_CACERT=/var/occne/cacertificates/openstack-cacert.pem.

Get the Openstack RC (API v3) File

This file exports a number of environment variables on the Bootstrap Host for the given user which directs the OpenStack Client commands towards the particular OpenStack Environment. It must be copied to the users home directory on the Bootstrap Host so that the OpenStack Client commands can be executed.

Note:

These instructions may slightly vary for OpenStack Desktops.
  1. From the OpenStack Desktop: go to Project and then API Access.
  2. On the right hand side, Download OpenStack RC File pulldown menu and select Openstack RC File (Identity API v3).

    This will download an openrc.sh file prefixed with the OpenStack project name (ex: OCCNE-openrc.sh) to your PC.

  3. SCP this file (that is winSCP) to the Bootstrap Host in the /home/admusr directory as .<project_name>-openrc.sh

    Note:

    In order for SCP/winSCP to work properly, the key mentioned in the Prerequisites above must be used to access the Bootstrap Host. It may also be necessary to add the appropriate Security Group Rules to support SSH (Rule: SSH, Remote: CIDR CIDR: 0.0.0.0/0) under the Network → Security Groups page in the OpenStack Environment. Contact the OpenStack Administrator to get the proper rules added if necessary.
  4. Execute the following command: source .<project_name>-openrc.sh
  5. Execute the following command to verify the OpenStack Client is working: openstack image list

Create SSH Key on Bootstrap Host

Create the keys that will be used to access the other VMs. This command generates the private and public keys that are passed to the Bastion Host and used to communicate with other node from that Bastion Host. Do not supply a passphrase when it asks for one. Just hit enter. Also the private key should be copied to a place for safe keeping should the Bootstrap Host be destroyed. 
$ ssh-keygen -m PEM -t rsa -b 2048

Add Files to /tmp Directory

These files must be copied to the directories listed using scp or some other means (ie. winSCP).

  1. There are three directories on the Bootstrap Host. These three directories are as follows:
    1. /tmp/yum.repos.d
    2. /tmp/db
    3. /tmp/certificates
  2. Within these three directories the user must supply the following mandatory files:
    1. Add a customer specific central repo .repo file to the /tmp/yum.repos.d directory which allows for access to the customer specific repo (ex: winterfell-mirror.repo).
    2. Add a mysql .zip file. (ex: V980756-01.zip) to the /tmp/db directory. This file is used for installing the ndb MySQL cluster and is downloaded from OSDC
    3. Add a docker registry certificate to the /tmp/certificates directory for the central docker registry. This file mus tbe copied using the following format:<central_repo_hostname>:<central_repo_port>.crt(ex: winterfell:5000.crt).
  3. Make sure the permissions of each file is at least readable (using 0644) using the sudo chmod 644 <filename> command.

Updating cluster.tfvars File

The cluster.tfvars file should exist on the Bootstrap Host after it has been deployed in /var/terraform/occne_example/cluster.tfvars. The user must create a directory called occne-<user>-<name> with a copy of the example tfvars file included in the /var/terraform directory.

The fields in the cluster.tfvars file must be configured to adapt to the current customer Openstack Environment. The steps below detail how to collect and set the fields that must be changed. These instructions may vary depending on whether the user is deploying the cluster using floating IPs or using fixed IP. Differences are noted below when needed.

The cluster.tfvars file includes a parameter named: use_floating_ip. This field defaults to true and should remain as set if the user is configuring their system to use floating IPs allocated from existing DHCP enabled networks defined within their Openstack Provider. If the user wishes to use fixed IPs, this field must be set to false.

Note: All fields in the cluster.tfvars file are unique and should not be duplicated (even using comments - #) as this can cause possible parsing errors.

Common Configuration

This procedure applies to both floating IPs or Fixed IPs.

  1. From the /var/terraform directory, copy directory occne_example and its contents cluster.tfvars, using the command below to create a new directory. Change the name of the new directory to include your name or user name to distinguish it from the occne_example directory. 
    $ cp -R occne_example occne_<user>
    Example:
    $ cp -R occne-abc-xyz
  2. Use the following commands to retrieve the information necessary to configure the cluster.tfvars
    1. The different flavor settings should be set according to the recommendations from vCNE VM Sizing in this document. An Admin user of the customer specific OpenStack Environment must add the flavors and provide the name of those flavors for configuration into the cluster.tfvars file. The name of each specific flavor that is used must be added as the value field of the key/value fields in the cluster.tfvars file.
    2. Once flavors have been added to the OpenStack Environment, the flavor name can be retrieved via the following OpenStack Client command from the Bootstrap Host shell or the Openstack jumpbox: 
      $ openstack flavor list | grep <flavor_name>
 
Example:
 
$ openstack flavor list | grep OCCNE
| 740c9472-db2c-4015-be5d-04eea1e9f647 | OCCNE-Bastion-Host    |   3840 |  100 | 0 | 2 | True |
| 337263ef-f69e-43b5-9c6f-ddfc7bb8237e | OCCNE-Bootstrap-host  |   8192 |   40 | 0 | 2 | True |
| 2de71982-3a27-45bf-9232-27a3e03c685f | OCCNE-DB-Data-Large   |  65536 |   60 | 0 | 8 | True |
| 27bdc547-a180-45a3-a349-7c574638ba27 | OCCNE-DB-Data-Medium  |  32768 |   40 | 0 | 4 | True |
| f1f1fce6-fc7a-4e33-9711-ea47d476a843 | OCCNE-DB-Data-Small   |  16384 |   20 | 0 | 2 | True |
| 4fbe0ac8-4e63-439b-ac6e-c54f40d6e116 | OCCNE-DB-Mgmt-Large   |  15360 |   60 | 0 | 4 | True |
| 252d96c6-956c-4e99-bae1-c1ff8c09cc8e | OCCNE-DB-Mgmt-Medium  |   7680 |   40 | 0 | 2 | True |
| 1c42a100-9923-4a82-b56c-579ac5060ff9 | OCCNE-DB-Mgmt-Small   |   3840 |   20 | 0 | 1 | True |
| 0ac5719e-0aa6-480e-9835-7ea091e53949 | OCCNE-DB-Sql-Large    |  32768 |   60 | 0 | 8 | True |
| adbe5fa8-09e5-41c1-a6db-91a3650bea95 | OCCNE-DB-Sql-Medium   |  16384 |   40 | 0 | 4 | True |
| 7c40962d-5992-4a8a-9668-f6d0dc0107c4 | OCCNE-DB-Sql-Small    |   8192 |   20 | 0 | 2 | True |
| b219a120-3031-4558-8214-00f5184fc4b4 | OCCNE-K8s-Master-Large|  15360 |   80 | 0 | 4 | True |
| 0d37d92c-8b06-4052-a1cc-94368d63f898 | OCCNE-K8s-Master-Medium | 7680 |   40 | 0 | 2 | True |
| 99883b4d-8532-4c72-acd8-00955a772b9d | OCCNE-K8s-Master-Small  | 3840 |   20 | 0 | 1 | True |
| facc50fa-e784-4259-aac4-731ce0cbf3ce | OCCNE-K8s-Worker-Large  |  32768 | 60 | 0 | 8 | True |
| 7307d7bf-bc78-43fd-a5a7-b6e59eeace81 | OCCNE-K8s-Worker-Medium |  16384 | 40 | 0 | 4 | True |
| 17d58f3d-3f62-4acd-bb2b-b8b29fbb76a4 | OCCNE-K8s-Worker-Small  |   8192 | 20 | 0 | 2 | True |
| f407e26d-c677-47b3-b5ed-22951ad77e49 | OCCNE-K8s-Worker-XLarge |  65536 | 80 | 0 |16 | True |
    3. Use the following command to retrieve the external_net UUID.
      $ openstack network list
+--------------------------------------+-------------------+--------------------------------------+
| ID                                   | Name              | Subnets                              |
+--------------------------------------+-------------------+--------------------------------------+
| 1d25d5ea-77ca-4f56-b364-f53b09292e7b | ext-net2          | f5c5ee71-8688-466d-a79f-4306e2bf3f6a |
| 668bc488-5307-49ad-9332-24fb0767bb39 | test-network      | 9432b2d5-99c0-43ee-8f8c-4709f38b68d9 |
| 903155c7-c3ff-4283-bc2b-f34e8b6e76b0 | occne-ebadger-tc1 | ecbadd3e-e239-4830-b8c1-5ff94fa64c3a |
| 90c160aa-2ef7-47d3-a212-e1790d56c971 | ext-net-ipv6      | 4c0b844f-1557-4454-b561-88fa31f657f3 |
| c4a7569b-5448-4add-8c4e-006bbdd984ef | cluster1          | 4cf62be3-05e9-4a5b-b2a9-6aceee3c860f |
| e4351e3e-81e3-4a83-bdc1-dde1296690e3 | ext-net           | c0e0c185-ed65-4a53-a7a3-418277fb9a20 |
| fc36d63f-b30b-4c7f-979f-9b52b614bbd7 | occne-mkingre     | 7631612f-5d22-49be-975c-6e0a9329339b |
+--------------------------------------+-------------------+--------------------------------------+
  3. Once the necessary data has been collected, navigate to the occne_<user> directory and edit the contents of the cluster.tfvars file in the newly created directory: 
    $ vi occne_<user>/cluster.tfvars
  4. The following miscellaneous fields should remain as set in the example cluster.tfvars file.
    • public_key_path
    • image
    • ssh_user
  5. The fields which define the number of each node type have defaulted values. For a standard deployment the defaults should be used. The user can update these values if their deployment requires additional nodes. Note these fields are integer values and do not require double quotes.
    • number_of_bastionsnumber_of_k8s_nodes
    • number_of_k8s_masters_no_floating_ip

      WARNING: The number of master nodes must be set to an odd number. The recommended value for number_of_k8s_masters_no_floating_ip is 3.

    • number_of_db_tier_management_nodes
    • number_of_db_tier_data_nodes
    • number_of_db_tier_sql_nodes
  6. The corresponding flavors for each node type must be set to a unique flavor name. Flavors are provided from the Openstack Provider administrator.
    • flavor_bastion
    • flavor_k8s_master
    • flavor_k8s_node
    • flavor_db_tier_management_node
    • flavor_db_tier_data_node
    • flavor_db_tier_sql_node
  7. The cluster_name and network_name should be set as the same value.

    Note: This field is used to create each of the nodes contained within the cluster. Kubespray does not allow upper case alphanumeric characters to be used in the node hostname. Do not use any uppercase characters when defining this cluster-name field. 

    # Kubernetes short
 cluster namecluster_name = "<cluster-name>" 
# networking
network_name = "<cluster-name>"
  8. The subnet_cidr defines the tenant side network ip address range. This field should remain set to the default value.
  9. The field bastion_allowed_remote_ips defines the configuration for the bastion networking security group. This field should remain set to the default value.
  10. For setting the ntp_server value in the cluster.tfvars file, use the IP Address of your cloud URL. One way of obtaining this is using the ping command on your Bootstrap Host. (For example: ping thundercloud.us.oracle.com) 
    $ ping <openstack_provider_url>
$ ping thundercloud.us.oracle.com
PING srv-10-75-171-2.us.oracle.com (10.75.171.2) 56(84) bytes of data.
64 bytes from pc1011601.labs.nc.tekelec.com (10.75.171.2): icmp_seq=1 ttl=63 time=0.283 ms
  11. If your deployment requires a specific Availability Zone other than the default availability zone called nova, make the following changes in the cluster.tfvars file. This value can be added just after the cluster_name field. If you wish to use nova then do not add these changes. 
    az_list = ["<your_availability_zone_name>"]
 
Example:
# Authorizing_Zone
az_list = ["foobar"]
  12. If your deployment requires additional external dns_nameservers to be configured, make the following changes in the cluster.tfvars file. There can be multiple ipv4 addresses in this list. This value can be added at the end of the file after wait_for_floatingip. This is an optional value and the default is empty list []. The sample configuration format is as below:
    dns_nameservers = ["<ipv4_address>",...] 
Example:
dns_nameservers = ["10.75.xxx.xxx",...]
  13. If your deployment requires the delivery of metadata and user data through a config drive to each instance, add the following changes in the cluster.tfvars file. This is an optional value. The default is occne_config_drive = "false" which indicates using a metadata server instead of a config drive for OpenStack.

    Note:

    Make sure the OpenStack administrator did not set force_config_drive=true in the /etc/nova/nova.conf file, otherwise it will use config drive in either case. The sample configuration format is as below:
    occne_config_drive = "<true/false>" 
Example:
occne_config_drive = "true"
Configuring Floating IPs

This procedure defines configuration that apply to floating IPs only.

  1. Use the following command to retrieve the floatingip_pool name. This would be one of the existing provider networks.
    $ openstack network list
+--------------------------------------+---------------+--------------------------------------+
| ID                                   | Name          | Subnets                              |
+--------------------------------------+---------------+--------------------------------------+
| 1d25d5ea-77ca-4f56-b364-f53b09292e7b | ext-net2      | f5c5ee71-8688-466d-a79f-4306e2bf3f6a |
| 90c160aa-2ef7-47d3-a212-e1790d56c971 | ext-net-ipv6  | 4c0b844f-1557-4454-b561-88fa31f657f3 |
| de033f1f-1e32-4b1f-b69a-dbf1a7a129f5 | ext-net3      | c4c5375c-4a28-4775-8e42-b2b806514e87 |
| e4351e3e-81e3-4a83-bdc1-dde1296690e3 | ext-net       | c0e0c185-ed65-4a53-a7a3-418277fb9a20 |
+--------------------------------------+---------------+--------------------------------------+
  2. Assign the floatingip_pool field as follows:
    floatip_pool = "<floating_ip_pool_name>" 
Example:
floatingip_pool = ext-net2
  3. wait_for_floatingip provides the ability for the Terraform deployment to poll the instance until the floating IP has been associated. It is set to true by default in the vCNE deployment. There is no need to change this.

    Note:

    This field is a string and requires double quotes around it.
Configuring Fixed IPs

This procedure defines the configuration that applies to fixed IPs only.

Note: The fields below also includes the openstack command used to retrieve the necessary information. It is possible for a network to be defined that does not include a Name value assigned to it. It is important that the network and subnet include a Name value assigned to it.

Note:

These fields can be left undefined if using the Floating IP configuration.
  1. Determine and configure the network name (fixedip_network_name) used to provide the IPs configured in step 2 below. 
    $ openstack network list
+--------------------------------------+---------------+--------------------------------------+
| ID                                   | Name          | Subnets                              |
+--------------------------------------+---------------+--------------------------------------+
| 1d25d5ea-77ca-4f56-b364-f53b09292e7b | ext-net2      | f5c5ee71-8688-466d-a79f-4306e2bf3f6a |
| 90c160aa-2ef7-47d3-a212-e1790d56c971 | ext-net-ipv6  | 4c0b844f-1557-4454-b561-88fa31f657f3 |
| de033f1f-1e32-4b1f-b69a-dbf1a7a129f5 | ext-net3      | c4c5375c-4a28-4775-8e42-b2b806514e87 |
| e4351e3e-81e3-4a83-bdc1-dde1296690e3 | ext-net       | c0e0c185-ed65-4a53-a7a3-418277fb9a20 |
+--------------------------------------+---------------+--------------------------------------+
 
fixedip_network_name = "<fixedip_network_name>"
 
Example:
fixed_ip_network_name = "ext-net3"
  2. Determine the set of fixed IPs that are to be used for the Bastion Host, K8s Nodes and DB Tier SQL Nodes.

    Note:

    Each IP must be in double quotes and separated by commas. The number of IPs MUST match the number of the specific node type.
    1. Use the following command to retrieve the list of networks defined on the Openstack Provider.

      Note:

      In this example, network ext-net3 is the fixed IP network selected. 
      $ openstack network list
+--------------------------------------+--------------+--------------------------------------+
| ID                                   | Name         | Subnets                              |
+--------------------------------------+--------------+--------------------------------------+
| 1d25d5ea-77ca-4f56-b364-f53b09292e7b | ext-net2     | f5c5ee71-8688-466d-a79f-4306e2bf3f6a |
| 90c160aa-2ef7-47d3-a212-e1790d56c971 | ext-net-ipv6 | 4c0b844f-1557-4454-b561-88fa31f657f3 |
| de033f1f-1e32-4b1f-b69a-dbf1a7a129f5 | ext-net3     | c4c5375c-4a28-4775-8e42-b2b806514e87 |
| e4351e3e-81e3-4a83-bdc1-dde1296690e3 | ext-net      | c0e0c185-ed65-4a53-a7a3-418277fb9a20 |
+--------------------------------------+--------------+--------------------------------------+
    2. The user must determine the list of available IPs to use in the fixed IP lists given below. Contact your administrator for available IPs.
      The following command can be used by a user to retrieve the range of ports that are available in the fixed IP network. Use the subnet name from the fixed IP network to display the range.
      To retrieve the number of ips in a subnet, use the following command:
 
$ openstack subnet list --network ext-net3
+--------------------------------------+-----------------+--------------------------------------+----------------+
| ID                                   | Name            | Network                              | Subnet         |
+--------------------------------------+-----------------+--------------------------------------+----------------+
| c4c5375c-4a28-4775-8e42-b2b806514e87 | ext-net3-subnet | de033f1f-1e32-4b1f-b69a-dbf1a7a129f5 | 10.75.235.0/25 |
+--------------------------------------+-----------------+--------------------------------------+----------------+
    3. Select the port IPs that are to be used for the Bastion Host depending on the number of Bastion Hosts as defined in the cluster.tfvars file by field number_of_bastions.
    4. Select the port IPs that are to be used for the K8s Nodes depending on the number of Worker Nodes as defined in the cluster.tfvars file by field number_of_k8s_nodes.
    5. Select the port IPs that are to be used for the DB Tier SQL Nodes depending on the number of DB SQL Nodes as defined in the cluster.tfvars file by field number_of_db_tier_sql_nodes.
    6. Assign the IPs to the field bastions_fixed_ip_list in the following format (example assumes number_of_bastions = 1):
      bastions_fixed_ip_list = ["<fixed_ip_0>...<fixed_ip_(number_of_bastions-1)>"] 
Example:
bastions_fixed_ip_list = ["10.75.10.20"]
    7. Assign the IPs to the field k8s_nodes_fixed_ip_list in the following format (example assumes number_of_k8s_nodes = 4):
      k8s_nodes_fixed_ip_list = ["<fixed_ip_0>...<fixed_ip_(number_of_k8s_nodes-1)>"]
 
Example:
k8s_nodes_fixed_ip_list = ["10.75.10.21","10.75.10.22","10.75.10.23","10.75.10.24"]
    8. Assign the IPs to the field db_tier_sql_nodes_fixed_ip_list in the following format (example assumes the number_of_db_tier_sql_nodes = 2):
      db_tier_sql_nodes_fixed_ip_list = ["<fixed_ip_0>...<fixed_ip_(number_of_db_tier_sql_nodes-1)>"]
 
Example:
db_tier_sql_nodes_fixed_ip_list = ["10.75.10.25","10.75.10.26"]

Obtain Mate Site DB Replication Service Load Balancer IP

While installing MYSQL NDB on the second site we need to provide the Mate Site DB Replication Service Load Balancer IP as the configuration parameter for the geo-replication process to start.

Note:

If this is a single deploy and or a mated site with this being the first site deployed, this step can be skipped.
  1. In order to obtain the Mate Site DB Replication Service Load Balancer IP, login to the Bastion Host of first site and execute the following command to retrieve the DB Replication Service Load Balancer IP.
  2. Fetch DB Replication Service Load Balancer IP of Mate Site MYSQL NDB. 
    [cloud-user@bastion ~]$ kubectl get svc --namespace=occne-infra | grep replication
occne-db-replication-svc     LoadBalancer   10.233.3.117    10.75.182.88     80:32496/TCP   2m8s
    In the above example, "10.75.182.88" is the Mate Site DB Replication Service Load Balancer IP.

Deploy the OCCNE Virtualized Cluster

The execution of the following command does all the work to deploy the VMs in the OpenStack Environment, configure the Bastion Host, and deploy and configure the Kubernetes clusters.

Deploying Using Fixed IPs

If the field use_floating_ip = false (as in using fixed IPs), the deploy.sh script performs a validation of the settings for fixed IP solution in the tfvars file. If the script detects any incorrect configurations within the fixed IP specific fields, it will produce an error explaining why and stop execution before the VMs are instantiated on the Openstack Provider using terraform. If use_floating_ip = true the fixed ip validation is not executed.

The following errors are currently detected by the verification:

  • If the use_floating_ip setting is true or false. Anything else and that verification will fail.
  • Invalid fixed IP network name (network does not exist on the Openstack Provider).
  • Number of nodes of a given type does not match with the number of IP addresses in the IP list for that node type (for example, if number_of_bastion = 1 and the fixed IP list for bastions includes 2 IPs).
  • Invalid IP address (for example, 10.75.10.999)
  • IP address used is currently in use.
Example verification output (assumes the IPs assigned in the cluster.tfvars file are already in use): 
Example (failed case):
...Verifying use_floating_ip setting
...Verifying Fixed IP network info
.....Verifying network name: ext-net3
...Verifying Fixed IP lists
.....Scanning ip list: bastions_fixed_ip_list
.......ip: 10.75.235.4
       ERROR: IPv4: 10.75.235.4 from Network: ext-net3 is already in use.
.....Scanning ip list: k8s_nodes_fixed_ip_list
.......ip: 10.75.235.23
       ERROR: IPv4: 10.75.235.23 from Network: ext-net3 is already in use.
.......ip: 10.75.235.21
       ERROR: IPv4: 10.75.235.21 from Network: ext-net3 is already in use.
.......ip: 10.75.235.19
       ERROR: IPv4: 10.75.235.19 from Network: ext-net3 is already in use.
.......ip: 10.75.235.27
       ERROR: IPv4: 10.75.235.27 from Network: ext-net3 is already in use.
.....Scanning ip list: db_tier_sql_nodes_fixed_ip_list
...occne1-John-Doe/cluster.tfvars verification completed.
   5 errors detected. Please address before continuing.
 
Example (pass case):
Verifying tfvars file...
...Verifying use_floating_ip setting
...Verifying Fixed IP network info
.....Verifying network name: ext-net3
...Verifying Fixed IP lists
.....Scanning ip list: bastions_fixed_ip_list
.......ip: 10.75.235.23
.....Scanning ip list: k8s_nodes_fixed_ip_list
.......ip: 10.75.235.21
.......ip: 10.75.235.15
.......ip: 10.75.235.14
.......ip: 10.75.235.10
.....Scanning ip list: db_tier_sql_nodes_fixed_ip_list
.......ip: 10.75.235.4
.......ip: 10.75.235.31
...occne1-John-Doe/cluster.tfvars verification successful.

Deploy Command Execution

Environmental Variables describes the list of possible environment variables that can be combined with the deploy.sh command.
  1. For openstack certificate authentication environment only

    Set the environment variable ENCODED_CACERT to base64 encoded string of openstack certificate using the command: export ENCODED_CACERT="<base64-encoded-cacert-string>"

    Use the link https://www.base64encode.org/ to generate base64 encoded string of openstack certificate.

    Run the below sed command to change the openstack certificate path in deploy.sh: 

    sed -i 's/\/host\/openstack-cacert.pem/${ENCODED_CACERT}/g' deploy.sh
  2. Execute the following command from the /var/terraform directory on the Bootstrap Host. This command may take a while to run (can be up to 2 hours depending on the machines its run on): 
    First Site:
$ OCCNE_TFVARS_DIR=occne_<user> CENTRAL_REPO=<central_repo_hostname> CENTRAL_REPO_IP=<central_repo_ipv4_address> ./deploy.sh
 
Replica Site:
$ OCCNE_TFVARS_DIR=occne_<user> CENTRAL_REPO=<central_repo_hostname> CENTRAL_REPO_IP=<central_repo_ipv4_address> OCCNE_DB_REPLICATION_MATE_IP=<db_replication_service_load_balancer_ip> OCCNE_DB_REPLICATION_CLUSTER_ID=<mysqlndb_cluster_identifier> ./deploy.sh
    Examples:
    
$ OCCNE_TFVARS_DIR=occne_john_doe CENTRAL_REPO=winterfell CENTRAL_REPO_IP=10.75.216.10 ./deploy.sh
$ OCCNE_TFVARS_DIR=occne_john_doe CENTRAL_REPO=winterfell CENTRAL_REPO_IP=10.75.216.10 OCCNE_DB_REPLICATION_MATE_IP=10.75.182.88 OCCNE_DB_REPLICATION_CLUSTER_ID=2 ./deploy.sh

Note:

While Installing on the First Site ignore OCCNE_DB_REPLICATION configuration parameters which Provides Mate Site DB Replication Service Load Balancer IP and ID. Provide the Mate Site DB Replication Service Load Balancer IP and MySQL Cluster identifier while Installing MYSQL NDB on second site.

Change MySQL root user password

Refer to Change MySQL root user password.