2 Installation Procedures
The installation procedures in this document provision and configure an Oracle Communications Signaling, Network Function Cloud Native Environment (OCCNE). OCCNE offers the choice of deployment platform; the CNE can be deployed directly onto dedicated hardware, (referred to as a bare metal CNE), or deployed onto OpenStack-hosted VMs. (referred to as a virtualized CNE).
Regardless of which deployment platform is selected, OCCNE installation is highly automated. A collection of container-based utilities are used to automate the provisioning, installation, and configuration of OCCNE. These utilities are based on the following automation tools:
- PXE helps reliably automate provisioning the hosts with a minimal operating system.
- Terraform is used to create the virtual resources that the virtualized CNE is hosted on.
- Kubespray helps reliably install a base Kubernetes cluster, including all dependencies (like etcd), using the Ansible provisioning tool.
- Ansible is used to orchestrate the overall deployment.
- Helm is used to deploy and configure common services such as Prometheus, Grafana, ElasticSearch and Kibana.
Note:
Make sure that the shell is configured with Keepalive to avoid unexpected timeout.Bare Metal Installation
This section describes the procedure to install OCCNE onto dedicated bare metal hardware.
OCCNE Installation Overview
Frame and Component Overview
The initial release of the OCCNE system provides support for on-prem deployment to a very specific target environment consisting of a frame holding switches and servers. This section describes the layout of the frame and describes the roles performed by the racked equipment.
Note:
In the installation process, some of the roles of servers change as the installation procedure proceeds.Frame Overview
The physical frame is comprised of HP c-Class enclosure (BL460c blade servers), 5 DL380 rack mount servers, and 2 Top of Rack (ToR) Cisco switches. The frame components are added from the bottom up, thus designations found in the next section number from the bottom of the frame to the top of the frame.
Figure 2-1 Frame Overview
Host Designations
Each physical server has a specific role designation within the CNE solution.
Figure 2-2 Host Designations
Node Roles
Along with the primary role of each host, a secondary role may be assigned. The secondary role may be software related, or, in the case of the Bootstrap Host, hardware related, as there are unique OOB connections to the ToR switches.
Figure 2-3 Node Roles
Transient Roles
Transient role is unique in that it has OOB connections to the ToR switches, which includes the designation of Bootstrap Host. This role is only relevant during initial switch configuration and disaster recovery of the switch. RMS1 also has a transient role as the Installer Bootstrap Host, which is only relevant during initial install of the frame, and subsequent to getting an official install on RMS2, this host is re-paved to its Storage Host role.
Figure 2-4 Transient Roles
Create OCCNE Instance
This section describes the steps and procedures required to create an OCCNE instance at a customer site. The following diagrams shows the installation context:
Figure 2-5 OCCNE Installation Overview
The following is an overview or basic install flow for reference to understand the overall effort contained within these procedures:
- Check that the hardware is on-site and properly cabled and powered up.
- Pre-assemble the basic
ingredients needed to perform a successful install:
-
Identify
- Download and stage software and other configuration files using provided manifests. Refer to Artifact Acquisition and Hosting for manifests information.
- Identify the layer 2 (MAC) and layer 3 (IP) addresses for the equipment in the target frame
- Identify the addresses of key external network services (for example, NTP, DNS, etc.)
- Verify / Set all of the credentials for the target frame hardware to known settings
-
Prepare
- Software Repositories: Load the various SW repositories (YUM, Helm, Docker, etc.) using the downloaded software and configuration
- Configuration Files: Populate the hosts inventory file with credentials and layer 2 and layer 3 network information, switch configuration files with assigned IP addresses, and yaml files with appropriate information.
-
- Bootstrap the System:
- Manually configure a Minimal Bootstrapping Environment (MBE); perform the minimal set of manual operations to enable networking and initial loading of a single Rack Mount Server - RMS1 - the transient Installer Bootstrap Host. In this procedure, a minimal set of packages needed to configure switches, iLOs, PXE boot environment, and provision RMS2 as an OCCNE Storage Host are installed.
- Using the newly constructed MBE, automatically create the first (complete) Management VM on RMS2. This freshly installed Storage Host will include a virtual machine for hosting the Bastion Host.
- Using the newly constructed Bastion Host on RMS2, automatically deploy and configure the OCCNE on the other servers in the frame
- Final Steps
- Perform post installation checks
- Perform recommended security hardening steps
Cluster Bootstrapping Overview
This install procedure is targeted at installing OCCNE onto a new hardware absent of any networking configurations to switches, or operating systems provisioned. Therefore, the initial step in the installation process is to provision RMS1 (see Installation Procedures) as a temporary Installer Bootstrap Host. The Bootstrap Host is configured with a minimal set of packages needed to configure switches, iLOs, PXE boot environment, and provision RMS2 as an OCCNE Storage Host. A virtual Bastion Host is also provisioned on RMS2. The Bastion Host is then used to provision (and in the case of the Bootstrap Host, re-provision) the remaining OCCNE hosts and install Kubernetes, Database services, and Common Services running within the Kubernetes cluster.
Installation Prerequisites
Obtain Mate Site DB Replication Service Load Balancer IP
Complete the procedures outlined in this section before moving on to the Install Procedures section. OCCNE installation procedures require certain artifacts and information to be made available prior to executing installation procedures. Refer to Configure Artifact Acquisition and Hosting for the prerequisites.
While installing MYSQL NDB on the second site the Mate Site DB Replication Service Load Balancer IP must be provided as the configuration parameter for the geo-replication process to start.
- Login to Bastion Host of the first site and execute the following command to retrieve DB Replication Service Load Balancer IP
- Fetch DB Replication
Service Load Balancer IP of Mate Site MYSQL NDB.
In the above example IPv4: 10.75.182.88 is the Mate Site DB Replication Service Load Balancer IP.$ kubectl get svc --namespace=occne-infra | grep replication Example: $ kubectl get svc --namespace=occne-infra | grep replication occne-db-replication-svc LoadBalancer 10.233.3.117 10.75.182.88 80:32496/TCP 2m8s
Configure Artifact Acquisition and Hosting
OCCNE requires artifacts from Oracle eDelivery and certain open-source projects. OCCNE deployment environments are not expected to have direct internet access. Thus, customer-provided intermediate repositories are necessary for the OCCNE installation process. These repositories will need OCCNE dependencies to be loaded into them. This section will address the artifacts list needed to be in these repositories.
Oracle eDelivery Artifact Acquisition
The OCCNE artifacts are posted on Oracle Software delivery Cloud (OSDC) and/or OHC.
Third Party Artifacts
OCCNE dependencies that come from open-source software must be available in repositories reachable by the OCCNE installation tools. For an accounting of third party artifacts needed for this installation, refer to the Artifact Acquisition and Hosting.
Populate the MetalLB Configuration
Introduction
The metalLB configMap file (mb_configmap.yaml) contains the manifest for the metalLB configMap, this defines the BGP peers and address pools for metalLB. This file (mb_configmap.yaml) must be placed in the same directory (/var/occne/<cluster_name>) as the hosts.ini file.
- Add BGP peers and address groups: Referring to the data collected in the Preflight Checklist, add BGP peers (ToRswitchA_Platform_IP, ToRswitchB_Platform_IP) and address groups for each address pool. Address-pools list the IP addresses that metalLB is allowed to allocate.
- Edit the mb_configmap.yaml file with the site-specific values found in the
Preflight Checklist
Note:
The name "signaling" is prone to different spellings (UK vs US), therefore pay special attention to how this signaling pool is referenced.configInline: peers: - peer-address: <ToRswitchA_Platform_IP> peer-asn: 64501 my-asn: 64512 - peer-address: <ToRswitchB_Platform_IP> peer-asn: 64501 my-asn: 64512 address-pools: - name: signaling protocol: bgp auto-assign: false addresses: - '<MetalLB_Signal_Subnet_IP_Range>' - name: oam protocol: bgp auto-assign: false addresses: - '<MetalLB_OAM_Subnet_IP_Range>'
Install Backup Bastion Host
Introduction
This procedure details the steps necessary to install the Backup Bastion Host on the Storage Host db-1/RMS1 and backing up the data from the active Bastion Host on db-2/RMS2 to the Backup Bastion Host.Prerequisites
- Bastion Host is already created on Storage Host db-2/RMS2.
- Storage Host db-2/RMS2 and the Backup Bastion Host
are defined in the Customer hosts.ini file as defined in procedure: Inventory File Preparation.
Note:
If the initial bootstrap host is RMS1 then the bastion host is created on the RMS2 and backup bastion host is created on the RMS1. - Host names and IP Address, network information assigned to Backup Management VM are captured in the Installation PreFlight Checklist.
- All the Network information should be configured in Inventory File Preparation.
Expectations
- Bastion Host VM on Storage Host db-1/RMS1 is created as a backup for Bastion Host VM on Storage Host db-2/RMS2.
- All the required config files and data configured in the Backup Bastion Host on Storage Host db-1/RMS1 are copied from the active Bastion Host on Storage Host db-2/RMS2.
Procedure
Create the Backup Bastion Host on Storage Host db-1/RMS1
All commands are executed from the active Bastion Host on db-2/RMS2.
- Login to the active Bastion Host (VM on RMS2) using the admusr/****** credentials.
- Execute the deploy.sh script from the
/var/occne/
directory with the required parameters set.$ export CENTRAL_REPO=<customer specific repo name> $ export CENTRAL_REPO_IP=<customer_specific_repo_ipv4> $ export OCCNE_CLUSTER=<cluster_name> $ export OCCNE_BASTION=<bastion_full_name> $ ./deploy.sh Customer Example: $ export CENTRAL_REPO=central-repo $ export CENTRAL_REPO_IP=10.10.10.10 $ export OCCNE_CLUSTER=rainbow $ export OCCNE_BASTION=bastion-1.rainbow.lab.us.oracle.com $ ./deploy.sh Note: The above example can be executed like the following: CENTRAL_REPO=central-repo CENTRAL_REPO_IP=10.10.10.10 OCCNE_CLUSTER=rainbow OCCNE_BASTION=bastion-1.rainbow.lab.us.oracle.com ./deploy.sh
- Verify installation of the Backup Bastion Host.
Note:
The IP of the backup Bastion Host can be derived from the hosts.ini file under the group host_kernel_virtual for db-1 ansible_host IP.$ ssh -i ~/.ssh/id_rsa admusr@<backup_bastion_ip_address>
Initial Configuration - Prepare a Minimal Boot Strapping Environment
In the first step of the installation, a minimal bootstrapping environment is established that is to support the automated installation of the CNE environment. The steps in this section provide the details necessary to establish this minimal bootstrap environment on the Installer Bootstrap Host using a Keyboard, Video, Mouse (KVM) connection.
Installation of Oracle Linux 7.5 on Bootstrap Host
This procedure outlines the installation steps for installing the OL7 onto the OCCNE Installer Bootstrap Host. This host is used to configure the networking throughout the system and install OL7 onto RMS1. It is re-paved as a Database Host in a later procedure.
Prerequisites
- USB drive of sufficient size to hold the ISO (approximately 5Gb)
- Oracle Linux 7.5 iso
- YUM repository file
- Keyboard, Video, Mouse (KVM)
Limitations and Expectations
- The configuration of the Installer Bootstrap Host is meant to be quick and easy, without a lot of care on appropriate OS configuration. The Installer Bootstrap Host is re-paved with the appropriate OS configuration for cluster and DB operation at a later stage of installation. The Installer Bootstrap Host needs a Linux OS and some basic network to get the installation process started.
- All steps in this procedure are performed using Keyboard, Video, Mouse (KVM).
References
- Oracle Linux 7 Installation guide: https://docs.oracle.com/cd/E52668_01/E54695/html/index.html
- HPE Proliant DL380 Gen10 Server User Guide
Bootstrap Install Procedure
- Create Bootable USB Media:
- Download the Oracle Linux
7.5.
On the installer's notebook, download the OL ISO from the customer's repository. Since Installer notebook may be Windows or Linux OS, and the Customer repository location may vary so the user executing this procedure determines the appropriate detail to execute this task.
- Push the OL ISO image onto the
USB Flash Drive.
Since the installer's notebook may be Windows or Linux OS-based, the user executing this procedure determines the appropriate detail to execute this task. For a Linux based notebook, insert a USB Flash Drive of the appropriate size into a Laptop (or some other linux host where the iso can be copied to), and run the
dd
command to create a bootable USB drive with the Oracle Linux 7 iso.$ dd -if=<path to ISO> -of=<USB device path> -bs=1m
- Download the Oracle Linux
7.5.
- Install OL7 on the Installer Bootstrap Host:
- Connect a Keyboard, Video, and Mouse (KVM) into the Installer Bootstrap Host's monitor and USB ports.
- Plug the USB flash drive containing the bootable iso into an available USB port on the Bootstrap host (usually in the front panel).
- Reboot the host by momentarily
pressing the power button on the host's front panel. The button will go
yellow. If it holds at yellow, press the button again. The host should
auto-boot to the USB flash drive.
Note:
If the host was previously configured and the USB is not a bootable path in the boot order, it may not boot successfully. - If the host does not boot to the USB, repeat step 3, and interrupt the boot process by pressing F11 which brings up the Boot Menu. If the host has been recently booted with an OL, the Boot Menu will display Oracle Linux at the top of the list. Select Generic USB Boot as the first boot device and proceed.
- The host attempts to boot from the USB. The following menu is displayed on the screen. Select Test this media & install Oracle Linux 7.x and click ENTER. This begins the verification of the media and the boot process. After the verification reaches 100%, the Welcome screen is displayed. When prompted for the language to use, select the default setting: English (United States) and click Continue in the lower left corner.
- The INSTALLATION
SUMMARY page, is displayed. The following settings are expected.
If any of these are not set correctly then please select that menu item
and make the appropriate changes.
- LANGUAGE SUPPORT: English (United States)
- KEYBOARD: English (US)
- INSTALLATION SOURCE: Local Media
- SOFTWARE SELECTION: Minimal Install
INSTALLATION DESTINATION should display
No disks selected
. Select INSTALLATION DESTINATION to indicate the drive to install the OS on.Select the first HDD drive ( in this case that would be the first one listed or the 1.6 TB disk) and select DONE in the upper right corner.
If the server has already been installed a red banner at the bottom of the page may indicate there is an error condition. Selecting that banner causes a dialog to appear indicating there is not enough free space (which might mean an OS has already been installed). In the dialog it may show both 1.6 TB HDDs as claimed or just the one.
If only one HDD is displayed (or it could be both 1.6 TB drives selected, select the Reclaim space button. Another dialog appears. Select the Delete all button and the Reclaim space button again. Select DONE to return to the INSTALLATION SUMMARY screen.If the disk selection dialog appears (after selecting the red banner at the bottom of the page), this implies a full installation of the RMS has already been performed (usually this is because the procedure had to be restarted after it was successfully completed). In this case select the Modify Disk Selection. This will return to the disk selection page. Select both HDDs and hit done. The red banner should now indicate the space must be reclaimed. The same steps to reclaim the space can be performed.
- Select DONE. This returns to the INSTALLATION SUMMARY page.
- At the INSTALLATION SUMMARY screen, select Begin Installation. The CONFIGURATION screen is displayed.
- At the CONFIGURATION
screen, select ROOT
PASSWORD.
Enter a root password appropriate for this installation. It is good practice to use a customer provided secure password to minimize the host being compromised during installation.
- At the conclusion of the install, remove the USB and select Reboot to complete the install and boot to the OS on the host. At the end of the boot, the login prompt appears.
Install Additional Packages
- dnsmasq
- dhcp
- xinetd
- tftp-server
- dos2unix
- nfs-utils
- Login with the root user and password configured above.
- Create the mount directory:
$ mkdir /media/usb
- Insert the USB into an available USB port (usually the front USB port) of the Installer Bootstrap Host.
- Find and mount the USB partition.
Typically the USB device is enumerated as
/dev/sda
but that is not always the case. Use thelsblk
command to find the USB device. An examplelsblk
output is below. The capacity of the USB drive is expected to be approximately 30GiB, therefore the USB drive is enumerated as device/dev/sda
in the example below:$ lsblk sdd 8:48 0 894.3G 0 disk sde 8:64 0 1.7T 0 disk sdc 8:32 0 894.3G 0 disk ├─sdc2 8:34 0 1G 0 part /boot ├─sdc3 8:35 0 893.1G 0 part │ ├─ol-swap 252:1 0 4G 0 lvm [SWAP] │ ├─ol-home 252:2 0 839.1G 0 lvm /home │ └─ol-root 252:0 0 50G 0 lvm / └─sdc1 8:33 0 200M 0 part /boot/efi sda 8:0 1 29.3G 0 disk ├─sda2 8:2 1 8.5M 0 part └─sda1 8:1 1 4.3G 0 part
The
dmesg
command also provides information about how the operating system enumerates devices. In the example below, thedmesg
output indicates the USB drive is enumerated as device /dev/sda.Note: The output is shortened here for display purposes.
$ dmesg ... [8850.211757] usb-storage 2-6:1.0: USB Mass Storage device detected [8850.212078] scsi host1: usb-storage 2-6:1.0 [8851.231690] scsi 1:0:0:0: Direct-Access SanDisk Cruzer Glide 1.00 PQ: 0 ANSI: 6 [8851.232524] sd 1:0:0:0: Attached scsi generic sg0 type 0 [8851.232978] sd 1:0:0:0: [sda] 61341696 512-byte logical blocks: (31.4 GB/29.3 GiB) [8851.234598] sd 1:0:0:0: [sda] Write Protect is off [8851.234600] sd 1:0:0:0: [sda] Mode Sense: 43 00 00 00 [8851.234862] sd 1:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [8851.255300] sda: sda1 sda2 ...
The USB device should contain at least two partitions. One is the boot partition and the other is the install media. The install media is the larger of the two partitions. To find information about the partitions use the
fdisk
command to list the filesystems on the USB device. Use the device name discovered via the steps outlined above. In the examples above, the USB device is/dev/sda
.$ fdisk -l /dev/sda Disk /dev/sda: 31.4 GB, 31406948352 bytes, 61341696 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x137202cf Device Boot Start End Blocks Id System /dev/sda1 * 0 8929279 4464640 0 Empty /dev/sda2 3076 20503 8714 ef EFI (FAT-12/16/32)
In the example output above, the
/dev/sda2
partition is the EFI boot partition. Therefore the install media files are on/dev/sda1
. Use themount
command to mount the install media file system. The same command without any options is used to verify the device is mounted to/media/usb
.$ mount /dev/sda1 /media/usb $ mount ... /dev/sda1 on /media/usb type iso9660 (ro,relatime,nojoliet,check=s,map=n,blocksize=2048)
- Create a
yum config
file to install packages from local install media.Create a repo file
/etc/yum.repos.d/Media.repo
with the following information:[ol7_base_media] name=Oracle Linux 7 Base Media baseurl=file:///media/usb gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle gpgcheck=1 enabled=1
-
Disable the default public yum repo. This is done by renaming the current
.repo
file to end with something other than.repo
. Adding.disabled
to the end of the file name is standard.Note: This can be left in this state as the Installer Bootstrap Host is re-paved in a later procedure.
$ mv /etc/yum.repos.d/public-yum-ol7.repo /etc/yum.repos.d/public-yum-ol7.repo.disabled
- Use the
yum repolist
command to check the repository configuration.The output of
yum repolist
should look like the example below. Verify there no errors regarding un-reachable yum repos.$ yum repolist Loaded plugins: langpacks, ulninfo repo id repo name status ol7_base_media Oracle Linux 7 Base Media 5,134 repolist: 5,134
- Use yum to install the additional
packages from the USB repo.
$ yum install dnsmasq $ yum install dhcp $ yum install xinetd $ yum install tftp-server $ yum install dos2unix $ yum install nfs-utils
-
Verify installation of dhcp, xinetd, and tftp-server.
Note: Currently
dnsmasq
is not being used. The verification of tftp makes sure the tftp file is included in the/etc/xinetd.d
directory. Installation/Verification does not include actually starting any of the services. Service configuration/starting is performed in a later procedure.Verify dhcp is installed: ------------------------- $ cd /etc/dhcp $ ls dhclient.d dhclient-exit-hooks.d dhcpd6.conf dhcpd.conf scripts Verify xinetd is installed: --------------------------- $ cd /etc/xinetd.d $ ls chargen-dgram chargen-stream daytime-dgram daytime-stream discard-dgram discard-stream echo-dgram echo-stream tcpmux-server time-dgram time-stream Verify tftp is installed: ------------------------- $ cd /etc/xinetd.d $ ls chargen-dgram chargen-stream daytime-dgram daytime-stream discard-dgram discard-stream echo-dgram echo-stream tcpmux-server tftp time-dgram time-stream
- Unmount the USB and remove the USB
from the host. The mount command can be used to verify the USB is no longer
mounted to
/media/usb
.$ umount /media/usb $ mount Verify that /dev/sda1 is no longer shown as mounted to /media/usb.
Configure the Installer Bootstrap Host BIOS
Introduction
These procedures define the steps necessary to set up the Legacy BIOS changes on the Bootstrap host using the KVM. Some of the procedures in this document require a reboot of the system and are indicated in the procedure.
Prerequisites
Procedure OCCNE Installation of Oracle Linux 7.5 on Bootstrap Host is complete.
Limitations and Expectations
- Applies to HP Gen10 iLO 5 only.
- The procedures listed here applies to the Bootstrap host only.
Steps to OCCNE Configure the Installer Bootstrap Host BIOS
- Expose the System Configuration Utility: This step details how to expose the HP
iLO 5 System Configuration Utility main page from the KVM. It does not provide
instructions on how to connect the console as these may be different on each
installation.
- After making the proper connections for the KVM on the back of the Bootstrap host to have access to the console, the user should reboot the host by momentarily pressing the power button on the front of the Bootstrap host.
- Expose the HP Proliant DL380
Gen10 System Utilities.
Once the remote console has been exposed, the system must be reset to force it through the restart process. When the initial window is displayed, hit the F9 key repeatedly. Once the F9 is highlighted at the lower left corner of the remote console, it should eventually bring up the main System Utility.
- The System Utilities screen is exposed in the remote console.
- Change over from UEFI Booting Mode to Legacy BIOS Booting Mode: The System
Utility must default the booting mode to UEFI or has been changed to UEFI, it
will be necessary to switch the booting mode to Legacy.
- Expose the System Configuration Utility by following Step 1.
- Select System Configuration.
- Select BIOS/Platform Configuration (RBSU).
- Select Boot Options.
If the Boot Mode is set to UEFI Mode then this procedure should be used to change it to Legacy BIOS Mode.
Note: The server reset must go through an attempt to boot before the changes will actually apply.
- The user is prompted to select the Reboot Required popup dialog. This will drop back into the boot process. The boot must go into the process of actually attempting to boot from the boot order. This should fail since the disks have not been installed at this point. The System Utility can be accessed again.
- After the reboot and the user re-enters the System Utility, the Boot Options page should appear.
- Select F10: Save if it's desired to save and stay in the utility or select the F12: Save and Exit if its desired to save and exit to complete the current boot process.
- Adding a New User Account: This step provides the steps required to add a new
user account to the server iLO 5 interface.
Note:
This user must match the pxe_install_lights_out_usrfields as provided in the hosts inventory files created using the template: OCCNE Inventory File Preparation.- Expose the System Utility by following Step 1.
- Select System Configuration.
- Select iLO 5 Configuration Utility.
- Select User Management → Add User .
- Select the appropriate
permissions. For the root user set all permissions to YES. Enter root as New User Name and
Login Name
fields, and enter
<password>
in the Password field. - Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit, to complete the current boot process.
- Force PXE to boot from the first Embedded FlexibleLOM HPE Ethernet 10Gb 2-port
Adapter. During host PXE, the DHCP DISCOVER requests from the hosts must be
broadcast over the 10Gb port. This step provides the steps necessary to
configure the broadcast to use the 10Gb ports before it attempts to use the 1Gb
ports. Moving the 10Gb port up on the search order helps to speed up the
response from the host servicing the DHCP DISCOVER. Enclosure blades have 2 10GE
NICs which default to being configured for PXE booting. The RMS are
re-configured to use the PCI NICs using this step.
- Expose the System Utility by following Step 1.
- Select System Configuration.
- Select BIOS/Platform Configuration (RBSU).
- Select Boot Options.
This menu defines the boot mode which should be set to Legacy BIOS Mode, the UEFI Optimized Boot which should be disabled, and the Boot Order Policy which should be set to Retry Boot Order Indefinitely (this means it will keep trying to boot without ever going to disk). In this screen select Legacy BIOS Boot Order. If not in Legacy BIOS Mode, please follow procedure 2.2 Change over from UEFI Booting Mode to Legacy BIOS Booting Mode to set the Configuration Utility to Legacy BIOS Mode.
- Select Legacy BIOS Boot Order
This page defines the legacy BIOS boot order. This includes the list of devices from which the server will listen for the DHCP OFFER (includes the reserved IPv4) after the PXE DHCP DISCOVER message is broadcast out from the server.
In the default view, the 10Gb Embedded FlexibleLOM 1 Port 1 is at the bottom of the list. When the server begins the scan for the response, it scans down this list until it receives the response. Each NIC will take a finite amount of time before the server gives up on that NIC and attempts another in the list. Moving the 10Gb port up on this list should decrease the time that is required to finally process the DHCP OFFER.
To move an entry, select that entry, hold down the first mouse button and move the entry up in the list below the entry it must reside under.
- Move the 10 Gb Embedded FlexibleLOM 1 Port 1 entry up above the 1Gb Embedded LOM 1 Port 1 entry.
- Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit, to complete the current boot process.
- Enabling Virtualization: This step provides the steps required to enable
virtualization on a given Bare Metal Server. Virtualization can be configured
using the default settings or via the Workload Profiles.
- Verifying Default Settings
- Expose the System Configuration Utility by following Step 1.
- Select System Configuration.
- Select BIOS/Platform Configuration (RBSU)
- Select Virtualization
Options
This screen displays the settings for the Intel(R) Virtualization Technology (IntelVT), Intel(R) VT-d, and SR-IOV options (Enabled or Disabled). The default values for each option is Enabled.
- Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit, to complete the current boot process.
- Verifying Default Settings
- Disable RAID Configurations:
- Expose the System Configuration Utility by following Step 1.
- Select System Configuration.
- Select Embedded RAID 1 : HPE Smart Array P408i-a SR Gen 10.
- Select Array Configuration.
- Select Manage Arrays.
- Select Array A (or any designated Array Configuration if there are more than one).
- Select Delete Array.
- Select Submit Changes.
- Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit, to complete the current boot process.
- Enable the Primary Boot Device: This step provides the steps necessary to
configure the primary bootable device for a given Gen10 Server. In this case the
RMS would include two devices as Hard Drives (HDDs). Some configurations may
also include two Solid State Drives (SSDs). The SSDs are not to be selected for
this configuration. Only the primary bootable device is set in this procedure
since RAID is being disabled. The secondary bootable device remains as Not
Set.
- Expose the System Configuration Utility by following Step 1.
- Select System Configuration.
- Select Embedded RAID 1 : HPE Smart Array P408i-a SR Gen 10.
- Select Set Bootable Device(s) for Legacy Boot Mode. If the boot devices are not set then it will display Not Set for the primary and secondary devices.
- Select Select Bootable Physical Drive.
- Select Port 1| Box:3 Bay:1
Size:1.8 TB SAS HP EG00100JWJNR.
Note: This example includes two HDDs and two SSDs. The actual configuration may be different.
- Select Set as Primary Bootable Device.
- Select Back to Main Menu.
This will return to the HPE Smart Array P408i-a SR Gen10 menu. The secondary bootable device is left as Not Set.
- Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit, to complete the current boot process.
- Configure the iLO 5 Static IP Address: When configuring the Bootstrap host, the
static IP address for the iLO 5 must be configured.
Note:
This step requires a reboot after completion.- Expose the System Configuration Utility by following Step 1.
- Select System Configuration.
- Select iLO 5 Configuration Utility.
- Select Network Options.
- Enter the IP Address, Subnet Mask, and Gateway IP Address fields provided in Installation PreFlight Checklist.
- Select F12: Save and Exit to complete the current boot process. A reboot is required when setting the static IP for the iLO 5. A warning appears indicating that the user must wait 30 seconds for the iLO to reset and then a reboot is required. A prompt appears requesting a reboot. Select Reboot.
- Once the reboot is complete, the user can re-enter the System Utility and verify the settings if necessary.
Configure Top of Rack 93180YC-EX Switches
Introduction
This procedure provides the steps required to initialize and configure Cisco 93180YC-EX switches as per the topology defined in Physical Network Topology Design.Note:
All instructions in this procedure are executed from the Bootstrap Host.Prerequisites
- Procedure OCCNE Installation of Oracle Linux 7.5 on Bootstrap Host has been completed.
- The switches are in factory default state.
- The switches are connected as per Installation PreFlight Checklist. Customer uplinks are not active before outside traffic is necessary.
- DHCP, XINETD, and TFTP are already installed on the Bootstrap host but are not configured.
- The Utility USB is available containing the necessary files as per: Installation PreFlight checklist: Create Utility USB.
Limitations/Expectations
All steps are executed from a Keyboard, Video, Mouse (KVM) connection.
References
Configuration Procedure
Following is the procedure to configure Top of Rack 93180YC-EX Switches:
- Login to the Bootstrap host as root.
Note:
All instructions in this step are executed from the Bootstrap Host. - Insert and mount the Utility USB that contains the configuration and script
files. Verify the files are listed in the USB using the ls /media/usb command.
Note:
Instructions for mounting the USB can be found in: Installation of Oracle Linux 7.5 on Bootstrap Server : Install Additional Packages. Only steps 2 and 3 need to be followed in that procedure. - Create bridge interface: Create bridge interface to connect both management
ports and setup the management bridge to support switch initialization.
Note:
<CNE_Management_IP_With_Prefix> is from Installation PreFlight Checklist : Complete Site Survey Host IP Table. Row 1 CNE Management IP Addresess (VLAN 4) column.<ToRSwitch_CNEManagementNet_VIP> is from Installation PreFlight Checklist : Complete OA and Switch IP Table.
$ nmcli con add con-name mgmtBridge type bridge ifname mgmtBridge $ nmcli con add type bridge-slave ifname eno2 master mgmtBridge $ nmcli con add type bridge-slave ifname eno3 master mgmtBridge $ nmcli con mod mgmtBridge ipv4.method manual ipv4.addresses 192.168.2.11/24 $ nmcli con up mgmtBridge $ nmcli con add type team con-name team0 ifname team0 team.runner lacp $ nmcli con add type team-slave con-name team0-slave-1 ifname eno5 master team0 $ nmcli con add type team-slave con-name team0-slave-2 ifname eno6 master team0
The following commands are related to the vlan and ip address for this bootstrap server, the <mgmt_vlan_id> is same as in hosts.ini, <bootstrap team0 address> is same as ansible_host ip for this bootstrap server:nmcli con mod team0 ipv4.method manual ipv4.addresses <bootstrap team0 address> nmcli con add con-name team0.<mgmt_vlan_id> type vlan id <mgmt_vlan_id> dev team0 nmcli con mod team0.<mgmt_vlan_id> ipv4.method manual ipv4.addresses <CNE_Management_IP_Address_With_Prefix> ipv4.gateway <ToRswitch_CNEManagementNet_VIP>nmcli con up team0.<mgmt_vlan_id>
Example:nmcli con mod team0 ipv4.method manual ipv4.addresses 172.16.3.4/24 nmcli con add con-name team0.4 type vlan id 4 dev team0 nmcli con mod team0.4 ipv4.method manual ipv4.addresses <CNE_Management_IP_Address_With_Prefix> ipv4.gateway <ToRswitch_CNEManagementNet_VIP> nmcli con up team0.4
- Edit the /etc/xinetd.d/tftp file to enable TFTP service. Change the disable option to
no, if it is set to yes.
$ vi /etc/xinetd.d/tftp # default: off # description: The tftp server serves files using the trivial file transfer \ # protocol. The tftp protocol is often used to boot diskless \ # workstations, download configuration files to network-aware printers, \ # and to start the installation process for some operating systems. service tftp { socket_type = dgram protocol = udp wait = yes user = root server = /usr/sbin/in.tftpd server_args = -s /var/lib/tftpboot disable = no per_source = 11 cps = 100 2 flags = IPv4 }
- Enable tftp on the Bootstrap
host:
$ systemctl start tftp $ systemctl enable tftp Verify tftp is active and enabled: $ systemctl status tftp $ ps -elf | grep tftp
- Copy the dhcpd.conf file from the Utility USB in Installation PreFlight checklist : Create the dhcpd.conf
File to the /etc/dhcp/ directory.
$ cp /media/usb/dhcpd.conf /etc/dhcp/
- Restart and enable dhcpd
service.
# /bin/systemctl restart dhcpd # /bin/systemctl enable dhcpd Use the systemctl status dhcpd command to verify active and enabled. # systemctl status dhcpd
- Copy the switch configuration and script files from the Utility USB to directory
/var/lib/tftpboot/.
$ cp /media/usb/93180_switchA.cfg /var/lib/tftpboot/. $ cp /media/usb/93180_switchB.cfg /var/lib/tftpboot/. $ cp /media/usb/poap_nexus_script.py /var/lib/tftpboot/.
- Modify POAP script File. Change Username and password credentials used to login
to the Bootstrap host.
# vi /var/lib/tftpboot/poap_nexus_script.py # Host name and user credentials options = { "username": "<username>", "password": "<password>", "hostname": "192.168.2.11", "transfer_protocol": "scp", "mode": "serial_number", "target_system_image": "nxos.9.2.3.bin", } Note: The version nxos.9.2.3.bin is used by default. If different version is to be used, modify the "target_system_image" with new version.
- Modify POAP script file md5sum by executing the md5Poap.sh script from the
Utility USB created from Installation PreFlight checklist: Create the md5Poap
Bash Script.
# cd /var/lib/tftpboot/ # /bin/bash md5Poap.sh
- Create the files necessary to configure the ToR switches using the serial number
from the switch. The serial number is located on a pullout card on the back of
the switch in the left most power supply of the switch.
Note:
The serial number is located on a pullout card on the back of the switch in the left most power supply of the switch. Be careful in interpreting the exact letters. If the switches are preconfigured then you can even verify the serial numbers using 'show license host-id
' command. - Copy the /var/lib/tftpboot/93180_switchA.cfg into a file called
/var/lib/tftpboot/conf.<switchA serial number> Modify the switch specific
values in the /var/lib/tftpboot/conf.<switchA serial number> file,
including all the values in the curly braces as following code block.
These values are contained at Installation PreFlight checklist : ToR and Enclosure Switches Variables Table (Switch Specific) and Installation PreFlight Checklist : Complete OA and Switch IP Table. Modify these values with the following sed commands, or use an editor such as vi etc.
# sed -i 's/{switchname}/<switch_name>/' conf.<switchA serial number> # sed -i 's/{admin_password}/<admin_password>/' conf.<switchA serial number> # sed -i 's/{user_name}/<user_name>/' conf.<switchA serial number> # sed -i 's/{user_password}/<user_password>/' conf.<switchA serial number> # sed -i 's/{ospf_md5_key}/<ospf_md5_key>/' conf.<switchA serial number> # sed -i 's/{OSPF_AREA_ID}/<ospf_area_id>/' conf.<switchA serial number> # sed -i 's/{NTPSERVER1}/<NTP_server_1>/' conf.<switchA serial number> # sed -i 's/{NTPSERVER2}/<NTP_server_2>/' conf.<switchA serial number> # sed -i 's/{NTPSERVER3}/<NTP_server_3>/' conf.<switchA serial number> # sed -i 's/{NTPSERVER4}/<NTP_server_4>/' conf.<switchA serial number> # sed -i 's/{NTPSERVER5}/<NTP_server_5>/' conf.<switchA serial number> # Note: If less than 5 ntp servers available, delete the extra ntp server lines such as command: # sed -i 's/{NTPSERVER5}/d' conf.<switchA serial number> Note: different delimiter is used in next two commands due to '/' sign in the variables # sed -i 's#{ALLOW_5G_XSI_LIST_WITH_PREFIX_LEN}#<MetalLB_Signal_Subnet_With_Prefix>#g' conf.<switchA serial number> # sed -i 's#{CNE_Management_SwA_Address}#<ToRswitchA_CNEManagementNet_IP>#g' conf.<switchA serial number> # sed -i 's#{CNE_Management_SwB_Address}#<ToRswitchB_CNEManagementNet_IP>#g' conf.<switchA serial number> # sed -i 's#{CNE_Management_Prefix}#<CNEManagementNet_Prefix>#g' conf.<switchA serial number> # sed -i 's#{SQL_replication_SwA_Address}#<ToRswitchA_SQLreplicationNet_IP>#g' conf.<switchA serial number> # sed -i 's#{SQL_replication_SwB_Address}#<ToRswitchB_SQLreplicationNet_IP>#g' conf.<switchA serial number> # sed -i 's#{SQL_replication_Prefix}#<SQLreplicationNet_Prefix>#g' conf.<switchA serial number> # ipcalc -n <ToRswitchA_SQLreplicationNet_IP/<SQLreplicationNet_Prefix> | awk -F'=' '{print $2}' # sed -i 's/{SQL_replication_Subnet}/<output from ipcalc command as SQL_replication_Subnet>/' conf.<switchA serial number> # sed -i 's/{CNE_Management_VIP}/<ToRswitch_CNEManagementNet_VIP>/g' conf.<switchA serial number> # sed -i 's/{SQL_replication_VIP}/<ToRswitch_SQLreplicationNet_VIP>/g' conf.<switchA serial number> # sed -i 's/{OAM_UPLINK_CUSTOMER_ADDRESS}/<ToRswitchA_oam_uplink_customer_IP>/' conf.<switchA serial number> # sed -i 's/{OAM_UPLINK_SwA_ADDRESS}/<ToRswitchA_oam_uplink_IP>/g' conf.<switchA serial number> # sed -i 's/{SIGNAL_UPLINK_SwA_ADDRESS}/<ToRswitchA_signaling_uplink_IP>/g' conf.<switchA serial number> # sed -i 's/{OAM_UPLINK_SwB_ADDRESS}/<ToRswitchB_oam_uplink_IP>/g' conf.<switchA serial number> # sed -i 's/{SIGNAL_UPLINK_SwB_ADDRESS}/<ToRswitchB_signaling_uplink_IP>/g' conf.<switchA serial number> # ipcalc -n <ToRswitchA_signaling_uplink_IP>/30 | awk -F'=' '{print $2}' # sed -i 's/{SIGNAL_UPLINK_SUBNET}/<output from ipcalc command as signal_uplink_subnet>/' conf.<switchA serial number> # ipcalc -n <ToRswitchA_SQLreplicationNet_IP> | awk -F'=' '{print $2}' # sed -i 's/{MySQL_Replication_SUBNET}/<output from the above ipcalc command appended with prefix >/' conf.<switchA serial number> Note: The version nxos.9.2.3.bin is used by default and hard-coded in the conf files. If different version is to be used, run the following command: # sed -i 's/nxos.9.2.3.bin/<nxos_version>/' conf.<switchA serial number> Note: access-list Restrict_Access_ToR # The following line allow one access server to access the switch management and SQL vlan addresses while other accesses are denied. If no need, delete this line. If need more servers, add similar line. # sed -i 's/{Allow_Access_Server}/<Allow_Access_Server>/' conf.<switchA serial number>
- Copy the /var/lib/tftpboot/93180_switchB.cfg into a file called
/var/lib/tftpboot/conf.<switchB serial number>
Modify the switch specific values in the /var/lib/tftpboot/conf.<switchA serial number> file, including: hostname, username/password, oam_uplink IP address, signaling_uplink IP address, access-list ALLOW_5G_XSI_LIST permit address, prefix-list ALLOW_5G_XSI.
These values are contained at Installation PreFlight checklist : ToR and Enclosure Switches Variables Table and Installation PreFlight Checklist : Complete OA and Switch IP Table.
# sed -i 's/{switchname}/<switch_name>/' conf.<switchB serial number> # sed -i 's/{admin_password}/<admin_password>/' conf.<switchB serial number> # sed -i 's/{user_name}/<user_name>/' conf.<switchB serial number> # sed -i 's/{user_password}/<user_password>/' conf.<switchB serial number> # sed -i 's/{ospf_md5_key}/<ospf_md5_key>/' conf.<switchB serial number> # sed -i 's/{OSPF_AREA_ID}/<ospf_area_id>/' conf.<switchB serial number> # sed -i 's/{NTPSERVER1}/<NTP_server_1>/' conf.<switchB serial number> # sed -i 's/{NTPSERVER2}/<NTP_server_2>/' conf.<switchB serial number> # sed -i 's/{NTPSERVER3}/<NTP_server_3>/' conf.<switchB serial number> # sed -i 's/{NTPSERVER4}/<NTP_server_4>/' conf.<switchB serial number> # sed -i 's/{NTPSERVER5}/<NTP_server_5>/' conf.<switchB serial number> # Note: If less than 5 ntp servers available, delete the extra ntp server lines such as command: # sed -i 's/{NTPSERVER5}/d' conf.<switchB serial number> Note: different delimiter is used in next two commands due to '/' sign in in the variables # sed -i 's#{ALLOW_5G_XSI_LIST_WITH_PREFIX_LEN}#<MetalLB_Signal_Subnet_With_Prefix>#g' conf.<switchB serial number> # sed -i 's#{CNE_Management_SwA_Address}#<ToRswitchA_CNEManagementNet_IP>#g' conf.<switchB serial number> # sed -i 's#{CNE_Management_SwB_Address}#<ToRswitchB_CNEManagementNet_IP>#g' conf.<switchB serial number> # sed -i 's#{CNE_Management_Prefix}#<CNEManagementNet_Prefix>#g' conf.<switchB serial number> # sed -i 's#{SQL_replication_SwA_Address}#<ToRswitchA_SQLreplicationNet_IP>#g' conf.<switchB serial number> # sed -i 's#{SQL_replication_SwB_Address}#<ToRswitchB_SQLreplicationNet_IP>#g' conf.<switchB serial number> # sed -i 's#{SQL_replication_Prefix}#<SQLreplicationNet_Prefix>#g' conf.<switchB serial number> # ipcalc -n <ToRswitchB_SQLreplicationNet_IP/<SQLreplicationNet_Prefix> | awk -F'=' '{print $2}' # sed -i 's/{SQL_replication_Subnet}/<output from ipcalc command as SQL_replication_Subnet>/' conf.<switchB serial number> # sed -i 's/{CNE_Management_VIP}/<ToRswitch_CNEManagementNet_VIP>/' conf.<switchB serial number> # sed -i 's/{SQL_replication_VIP}/<ToRswitch_SQLreplicationNet_VIP>/' conf.<switchB serial number> # sed -i 's/{OAM_UPLINK_CUSTOMER_ADDRESS}/<ToRswitchB_oam_uplink_customer_IP>/' conf.<switchB serial number> # sed -i 's/{OAM_UPLINK_SwA_ADDRESS}/<ToRswitchA_oam_uplink_IP>/g' conf.<switchB serial number> # sed -i 's/{SIGNAL_UPLINK_SwA_ADDRESS}/<ToRswitchA_signaling_uplink_IP>/g' conf.<switchB serial number> # sed -i 's/{OAM_UPLINK_SwB_ADDRESS}/<ToRswitchB_oam_uplink_IP>/g' conf.<switchB serial number> # sed -i 's/{SIGNAL_UPLINK_SwB_ADDRESS}/<ToRswitchB_signaling_uplink_IP>/g' conf.<switchB serial number> # ipcalc -n <ToRswitchB_signaling_uplink_IP>/30 | awk -F'=' '{print $2}' # sed -i 's/{SIGNAL_UPLINK_SUBNET}/<output from ipcalc command as signal_uplink_subnet>/' conf.<switchB serial number> Note: The version nxos.9.2.3.bin is used by default and hard-coded in the conf files. If different version is to be used, run the following command: # sed -i 's/nxos.9.2.3.bin/<nxos_version>/' conf.<switchB serial number> Note: access-list Restrict_Access_ToR # The following line allow one access server to access the switch management and SQL vlan addresses while other accesses are denied. If no need, delete this line. If need more servers, add similar line. # sed -i 's/{Allow_Access_Server}/<Allow_Access_Server>/' conf.<switchB serial number>
- Generate the md5 checksum for each conf file in /var/lib/tftpboot and
copy that into a new file called conf.<switchA/B serial
number>.md5.
$ md5sum conf.<switchA serial number> > conf.<switchA serial number>.md5 $ md5sum conf.<switchB serial number> > conf.<switchB serial number>.md5
- Verify the
/var/lib/tftpboot directory has the correct files.
Make sure the file permissions are set as given below.
Note: The ToR switches are constantly attempting to find and execute the poap_nexus_script.py script which uses tftp to load and install the configuration files.
# ls -l /var/lib/tftpboot/ total 1305096 -rw-r--r--. 1 root root 7161 Mar 25 15:31 conf.<switchA serial number> -rw-r--r--. 1 root root 51 Mar 25 15:31 conf.<switchA serial number>.md5 -rw-r--r--. 1 root root 7161 Mar 25 15:31 conf.<switchB serial number> -rw-r--r--. 1 root root 51 Mar 25 15:31 conf.<switchB serial number>.md5 -rwxr-xr-x. 1 root root 75856 Mar 25 15:32 poap_nexus_script.py
- Disable firewalld.
$ systemctl stop firewalld $ systemctl disable firewalld To verify: $ systemctl status firewalld
Once this is complete, the ToR Switches will attempt to boot from the tftpboot files automatically. Eventually the verification steps can be executed below. It may take about 5 minutes for this to complete.
- Un-mount the Utility USB and remove it:
umount /media/usb
Verification
- After the ToR switches configured, ping the switches from
bootstrap server. The switches mgmt0 interfaces are configured with the IP
addresses which are in the conf files. Note: Wait till the
device responds.
# ping 192.168.2.1 PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data. 64 bytes from 192.168.2.1: icmp_seq=1 ttl=255 time=0.419 ms 64 bytes from 192.168.2.1: icmp_seq=2 ttl=255 time=0.496 ms 64 bytes from 192.168.2.1: icmp_seq=3 ttl=255 time=0.573 ms 64 bytes from 192.168.2.1: icmp_seq=4 ttl=255 time=0.535 ms ^C --- 192.168.2.1 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3000ms rtt min/avg/max/mdev = 0.419/0.505/0.573/0.063 ms # ping 192.168.2.2 PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data. 64 bytes from 192.168.2.2: icmp_seq=1 ttl=255 time=0.572 ms 64 bytes from 192.168.2.2: icmp_seq=2 ttl=255 time=0.582 ms 64 bytes from 192.168.2.2: icmp_seq=3 ttl=255 time=0.466 ms 64 bytes from 192.168.2.2: icmp_seq=4 ttl=255 time=0.554 ms ^C --- 192.168.2.2 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3001ms rtt min/avg/max/mdev = 0.466/0.543/0.582/0.051 ms
- Attempt to ssh to the switches with the username/password
provided in the conf
files.
# ssh plat@192.168.2.1 The authenticity of host '192.168.2.1 (192.168.2.1)' can't be established. RSA key fingerprint is SHA256:jEPSMHRNg9vejiLcEvw5qprjgt+4ua9jucUBhktH520. RSA key fingerprint is MD5:02:66:3a:c6:81:65:20:2c:6e:cb:08:35:06:c6:72:ac. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.2.1' (RSA) to the list of known hosts. User Access Verification Password: Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (C) 2002-2019, Cisco and/or its affiliates. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under their own licenses, such as open source. This software is provided "as is," and unless otherwise stated, there is no warranty, express or implied, including but not limited to warranties of merchantability and fitness for a particular purpose. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or GNU General Public License (GPL) version 3.0 or the GNU Lesser General Public License (LGPL) Version 2.1 or Lesser General Public License (LGPL) Version 2.0. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://opensource.org/licenses/gpl-3.0.html and http://www.opensource.org/licenses/lgpl-2.1.php and http://www.gnu.org/licenses/old-licenses/library.txt. #
- Verify the running-config has all expected configurations in the
conf file using the show
running-config command.
# show running-config !Command: show running-config !Running configuration last done at: Mon Apr 8 17:39:38 2019 !Time: Mon Apr 8 18:30:17 2019 version 9.2(3) Bios:version 07.64 hostname 12006-93108A vdc 12006-93108A id 1 limit-resource vlan minimum 16 maximum 4094 limit-resource vrf minimum 2 maximum 4096 limit-resource port-channel minimum 0 maximum 511 limit-resource u4route-mem minimum 248 maximum 248 limit-resource u6route-mem minimum 96 maximum 96 limit-resource m4route-mem minimum 58 maximum 58 limit-resource m6route-mem minimum 8 maximum 8 feature scp-server feature sftp-server cfs eth distribute feature ospf feature bgp feature interface-vlan feature lacp feature vpc feature bfd feature vrrpv3 .... ....
- Verify license on the switches. In case some of the above
features are missing, verify license on the switches and at least
NXOS_ADVANTAGE level license is "In use". If license not installed or too
low level, contact vendor for correct license key file, following Licensing
document mentioned in reference section to install license key. Then run
"write erase" and "reload" to set back to factory default. The switches will
go to POAP configuration again.
# show license Example output: # show license MDS20190215085542979.lic: SERVER this_host ANY VENDOR cisco INCREMENT NXOS_ADVANTAGE_XF cisco 1.0 permanent uncounted \ VENDOR_STRING=<LIC_SOURCE>MDS_SWIFT</LIC_SOURCE><SKU>NXOS-AD-XF</SKU> \ HOSTID=VDH=FDO22412J2F \ NOTICE="<LicFileID>20190215085542979</LicFileID><LicLineID>1</LicLineID> \ <PAK></PAK>" SIGN=8CC8807E6918 # show license usage Example output: # show license usage Feature Ins Lic Status Expiry Date Comments Count -------------------------------------------------------------------------------- ... NXOS_ADVANTAGE_M4 No - Unused - NXOS_ADVANTAGE_XF Yes - In use never - NXOS_ESSENTIALS_GF No - Unused - ... #
- Verify the RMS1 can ping the CNE_Management VIP.
# ping <ToRSwitch_CNEManagementNet_VIP> PING <ToRSwitch_CNEManagementNet_VIP> (<ToRSwitch_CNEManagementNet_VIP>) 56(84) bytes of data. 64 bytes from <ToRSwitch_CNEManagementNet_VIP>: icmp_seq=2 ttl=255 time=1.15 ms 64 bytes from <ToRSwitch_CNEManagementNet_VIP>: icmp_seq=3 ttl=255 time=1.11 ms 64 bytes from <ToRSwitch_CNEManagementNet_VIP>: icmp_seq=4 ttl=255 time=1.23 ms ^C --- 10.75.207.129 ping statistics --- 4 packets transmitted, 3 received, 25% packet loss, time 3019ms rtt min/avg/max/mdev = 1.115/1.168/1.237/0.051 ms
- Enable customer uplink.
- Verify the RMS1 can be accessed from laptop. Use application
such as putty etc to ssh to RMS1.
$ ssh root@<CNE_Management_IP_Address> Using username "root". root@<CNE_Management_IP_Address>'s password:<root password> Last login: Mon May 6 10:02:01 2019 from 10.75.9.171 [root@RMS1 ~]#
SNMP Trap Configuration
- SNMPv2c Configuration.
When SNMPv2c configuration is needed, ssh to the two switches, run the following commands:
These values <SNMP_Trap_Receiver_Address>and <SNMP_Community_String> are from Installation Preflight Checklist.
[root@RMS1 ~]# ssh <user_name>@<ToRswitchA_CNEManagementNet_IP> # configure terminal (config)# snmp-server host <SNMP_Trap_Receiver_Address> traps version 2c <SNMP_Community_String> (config)# snmp-server host <SNMP_Trap_Receiver_Address> use-vrf default (config)# snmp-server host <SNMP_Trap_Receiver_Address> source-interface Ethernet1/51 (config)# snmp-server enable traps (config)# snmp-server community <SNMP_Community_String> group network-admin
- Restrict direct access to ToR switches. In order to restrict direct access
to ToR switches, IP access list is created and applied on the uplink
interfaces, the following commands are needed on ToR switches:
[root@RMS1 ~]# ssh <user_name>@<ToRswitchA_CNEManagementNet_IP> # configure terminal (config)# ip access-list Restrict_Access_ToR permit ip {Allow_Access_Server}/32 any permit ip {NTPSERVER1}/32 {OAM_UPLINK_SwA_ADDRESS}/32 permit ip {NTPSERVER2}/32 {OAM_UPLINK_SwA_ADDRESS}/32 permit ip {NTPSERVER3}/32 {OAM_UPLINK_SwA_ADDRESS}/32 permit ip {NTPSERVER4}/32 {OAM_UPLINK_SwA_ADDRESS}/32 permit ip {NTPSERVER5}/32 {OAM_UPLINK_SwA_ADDRESS}/32 deny ip any {CNE_Management_VIP}/32 deny ip any {CNE_Management_SwA_Address}/32 deny ip any {CNE_Management_SwB_Address}/32 deny ip any {SQL_replication_VIP}/32 deny ip any {SQL_replication_SwA_Address}/32 deny ip any {SQL_replication_SwB_Address}/32 deny ip any {OAM_UPLINK_SwA_ADDRESS}/32 deny ip any {OAM_UPLINK_SwB_ADDRESS}/32 deny ip any {SIGNAL_UPLINK_SwA_ADDRESS}/32 deny ip any {SIGNAL_UPLINK_SwB_ADDRESS}/32 permit ip any any interface Ethernet1/51 ip access-group Restrict_Access_ToR in interface Ethernet1/52 ip access-group Restrict_Access_ToR in
- Traffic egress out of cluster, including snmptrap traffic to SNMP trap
receiver, and traffic goes to signal server:
[root@RMS1 ~]# ssh <user_name>@<ToRswitchA_CNEManagementNet_IP> # configure terminal (config)# feature nat ip access-list host-snmptrap 10 permit udp 172.16.3.0/24 <snmp trap receiver>/32 eq snmptrap log ip access-list host-sigserver 10 permit ip 172.16.3.0/24 <signal server>/32 ip nat pool sig-pool 10.75.207.211 10.75.207.222 prefix-length 27 ip nat inside source list host-sigserver pool sig-pool overload add-route ip nat inside source list host-snmptrap interface Ethernet1/51 overload interface Vlan3 ip nat inside interface Ethernet1/51 ip nat outside interface Ethernet1/52 ip nat outside Run the same commands on ToR switchB
Configure Addresses for RMS iLOs, OA, EBIPA
Introduction
This procedure is used to configure RMS iLO addresses and add a new user account for each RMS other than the Bootstrap Host. When the RMSs are shipped and out of box after hardware installation and powerup, the RMSs are in a factory default state with the iLO in DHCP mode waiting for DHCP service. DHCP is used to configure the ToR switches, OAs, Enclosure switches, and blade server iLOs, so DHCP can be used to configure RMS iLOs as well.
Prerequisites
Procedure Configure Top of Rack 93180YC-EX Switches has been completed.
Limitations/Expectations
All steps are executed from the ssh session of the Bootstrap server.
Procedure
Following is the procedure to configure Addresses for RMS iLOs, OA, EBIPA:
- Setup the vlan interface to access ilo subnet. The ilo_vlan_id and
ilo_subnet_cidr are the same value as in
hosts.ini:
$ nmcli con add con-name team0.<ilo_vlan_id> type vlan id <ilo_vlan_id> dev team0 $ nmcli con mod team0.<ilo_vlan_id> ipv4.method manual ipv4.addresses <unique ip in ilo subnet>/<ilo_subnet_cidr> $ nmcli con up team0.<ilo_vlan_id>
Example:
$ nmcli con add con-name team0.2 type vlan id 2 dev team0 $ nmcli con mod team0.2 ipv4.method manual ipv4.addresses 192.168.20.11/24 $ nmcli con up team0.2
- Subnet and conf file address.
The /etc/dhcp/dhcpd.conf file should already have been configured in OCCNE Configure Top of Rack 93180YC-EX Switches procedure.
Configure Top of Rack 93180YC-EX Switches and dhcp started/enabled on the bootstrap server. The second subnet 192.168.20.0 is used to assign addresses for OA and RMS iLOs. The "next-server 192.168.20.11" option is same as the server team0.2 IP address. - Display the dhcpd leases file at
/var/lib/dhcpd/dhcpd.leases
. The DHCPD lease file will display the DHCP addresses for all RMS iLOs, Enclosure OAs.# cat /var/lib/dhcpd/dhcpd.leases # The format of this file is documented in the dhcpd.leases(5) manual page. # This lease file was written by isc-dhcp-4.2.5 lease 192.168.20.101 { starts 4 2019/03/28 22:05:26; ends 4 2019/03/28 22:07:26; tstp 4 2019/03/28 22:07:26; cltt 4 2019/03/28 22:05:26; binding state free; hardware ethernet 48:df:37:7a:41:60; } lease 192.168.20.103 { starts 4 2019/03/28 22:05:28; ends 4 2019/03/28 22:07:28; tstp 4 2019/03/28 22:07:28; cltt 4 2019/03/28 22:05:28; binding state free; hardware ethernet 48:df:37:7a:2f:70; } lease 192.168.20.102 { starts 4 2019/03/28 22:05:16; ends 4 2019/03/28 23:03:29; tstp 4 2019/03/28 23:03:29; cltt 4 2019/03/28 22:05:16; binding state free; hardware ethernet 48:df:37:7a:40:40; } lease 192.168.20.106 { starts 5 2019/03/29 11:14:04; ends 5 2019/03/29 14:14:04; tstp 5 2019/03/29 14:14:04; cltt 5 2019/03/29 11:14:04; binding state free; hardware ethernet b8:83:03:47:5f:14; uid "\000\270\203\003G_\024\000\000\000"; } lease 192.168.20.105 { starts 5 2019/03/29 12:56:23; ends 5 2019/03/29 15:56:23; tstp 5 2019/03/29 15:56:23; cltt 5 2019/03/29 12:56:23; binding state free; hardware ethernet b8:83:03:47:5e:54; uid "\000\270\203\003G^T\000\000\000"; } lease 192.168.20.104 { starts 5 2019/03/29 13:08:21; ends 5 2019/03/29 16:08:21; tstp 5 2019/03/29 16:08:21; cltt 5 2019/03/29 13:08:21; binding state free; hardware ethernet b8:83:03:47:64:9c; uid "\000\270\203\003Gd\234\000\000\000"; } lease 192.168.20.108 { starts 5 2019/03/29 09:57:02; ends 5 2019/03/29 21:57:02; tstp 5 2019/03/29 21:57:02; cltt 5 2019/03/29 09:57:02; binding state active; next binding state free; rewind binding state free; hardware ethernet fc:15:b4:1a:ea:05; uid "\001\374\025\264\032\352\005"; client-hostname "OA-FC15B41AEA05"; } lease 192.168.20.107 { starts 5 2019/03/29 12:02:50; ends 6 2019/03/30 00:02:50; tstp 6 2019/03/30 00:02:50; cltt 5 2019/03/29 12:02:50; binding state active; next binding state free; rewind binding state free; hardware ethernet 9c:b6:54:80:d7:d7; uid "\001\234\266T\200\327\327"; client-hostname "SA-9CB65480D7D7"; } server-duid "\000\001\000\001$#\364\344\270\203\003Gim"; lease 192.168.20.107 { starts 5 2019/03/29 18:09:47; ends 6 2019/03/30 06:09:47; cltt 5 2019/03/29 18:09:47; binding state active; next binding state free; rewind binding state free; hardware ethernet 9c:b6:54:80:d7:d7; uid "\001\234\266T\200\327\327"; client-hostname "SA-9CB65480D7D7"; } lease 192.168.20.108 { starts 5 2019/03/29 18:09:54; ends 6 2019/03/30 06:09:54; cltt 5 2019/03/29 18:09:54; binding state active; next binding state free; rewind binding state free; hardware ethernet fc:15:b4:1a:ea:05; uid "\001\374\025\264\032\352\005"; client-hostname "OA-FC15B41AEA05"; } lease 192.168.20.106 { starts 5 2019/03/29 18:10:04; ends 5 2019/03/29 21:10:04; cltt 5 2019/03/29 18:10:04; binding state active; next binding state free; rewind binding state free; hardware ethernet b8:83:03:47:5f:14; uid "\000\270\203\003G_\024\000\000\000"; client-hostname "ILO2M2909004B"; } lease 192.168.20.104 { starts 5 2019/03/29 18:10:35; ends 5 2019/03/29 21:10:35; cltt 5 2019/03/29 18:10:35; binding state active; next binding state free; rewind binding state free; hardware ethernet b8:83:03:47:64:9c; uid "\000\270\203\003Gd\234\000\000\000"; client-hostname "ILO2M2909004F"; } lease 192.168.20.105 { starts 5 2019/03/29 18:10:40; ends 5 2019/03/29 21:10:40; cltt 5 2019/03/29 18:10:40; binding state active; next binding state free; rewind binding state free; hardware ethernet b8:83:03:47:5e:54; uid "\000\270\203\003G^T\000\000\000"; client-hostname "ILO2M29090048";
- Access RMS iLO from the DHCP address with default Administrator password. From
the above
dhcpd.leases
file, find the IP address for the iLO name, the default username is Administrator, the password is on the label which can be pulled out from front of server.Note:
The DNS Name on the pull-out label. The DNS Name on the pull-out label should be used to match the physical machine with the iLO IP since the same default DNS Name from the pull-out label is displayed upon logging in to the iLO command line interface, as shown in the example below.# ssh Administrator@192.168.20.104 Administrator@192.168.20.104's password: User:Administrator logged-in to ILO2M2909004F.labs.nc.tekelec.com(192.168.20.104 / FE80::BA83:3FF:FE47:649C) iLO Standard 1.37 at Oct 25 2018 Server Name: Server Power: On
- Create RMS iLO new user. Create new user with customized username and password.
</>hpiLO-> create /map1/accounts1 username=root password=TklcRoot group=admin,config,oemHP_rc,oemHP_power,oemHP_vm status=0 status_tag=COMMAND COMPLETED Tue Apr 2 20:08:30 2019 User added successfully.
- Disable the DHCP before able to setup static IP. Setup static failed before DHCP
is
disabled.
</>hpiLO-> set /map1/dhcpendpt1 EnabledState=NO status=0 status_tag=COMMAND COMPLETED Tue Apr 2 20:04:53 2019 Network settings change applied. Settings change applied, iLO 5 will now be reset. Logged Out: It may take several minutes before you can log back in. CLI session stopped packet_write_wait: Connection to 192.168.20.104 port 22: Broken pipe
- Setup RMS iLO static IP address. After a while after previous step, can login
back with the same address (which is static IP now) and new username/password.
If don't want to use the same address, go to next step to change the IP address.
# ssh <new username>@192.168.20.104 <new username>@192.168.20.104's password: <new password> User: logged-in to ILO2M2909004F.labs.nc.tekelec.com(192.168.20.104 / FE80::BA83:3FF:FE47:649C) iLO Standard 1.37 at Oct 25 2018 Server Name: Server Power: On </>hpiLO-> set /map1/enetport1/lanendpt1/ipendpt1 IPv4Address=192.168.20.122 SubnetMask=255.255.255.0 status=0 status_tag=COMMAND COMPLETED Tue Apr 2 20:22:23 2019 Network settings change applied. Settings change applied, iLO 5 will now be reset. Logged Out: It may take several minutes before you can log back in. CLI session stopped packet_write_wait: Connection to 192.168.20.104 port 22: Broken pipe #
- Set EBIPA addresses for InterConnect Bays (Enclosure Switches). From bootstrap
server, login to OA, set EBIPA addressed for the two enclosure switches. The
addresses have to be in the subnet with server team0.2 address in order for TFTP
to work.
Set address for each enclosure switch, note the last number 1 or 2 is the interconnect bay number. OA-FC15B41AEA05> set ebipa interconnect 192.168.20.133 255.255.255.0 1 Entering anything other than 'YES' will result in the command not executing. It may take each interconnect several minutes to acquire the new settings. Are you sure you want to change the IP address for the specified interconnect bays? yes Successfully set 255.255.255.0 as the netmask for interconnect bays. Successfully set interconnect bay # 1 to IP address 192.168.20.133 For the IP addresses to be assigned EBIPA must be enabled. OA-FC15B41AEA05> set ebipa interconnect 192.168.20.134 255.255.255.0 2 Entering anything other than 'YES' will result in the command not executing. It may take each interconnect several minutes to acquire the new settings. Are you sure you want to change the IP address for the specified interconnect bays? yes Successfully set 255.255.255.0 as the netmask for interconnect bays. Successfully set interconnect bay # 2 to IP address 192.168.20.134 For the IP addresses to be assigned EBIPA must be enabled.
- Set EBIPA addresses for Blade Servers. Set EBIPA addressed for all the blade
servers. The addresses are in the same subnet with first server team0.2 address
and enclosure switches.
OA-FC15B41AEA05> set ebipa server 192.168.20.141 255.255.255.0 1-16 Entering anything other than 'YES' will result in the command not executing. Changing the IP address for device (iLO) bays that are enabled causes the iLOs in those bays to be reset. Are you sure you want to change the IP address for the specified device (iLO) bays? YES Successfully set 255.255.255.0 as the netmask for device (iLO) bays. Successfully set device (iLO) bay # 1 to IP address 192.168.20.141 Successfully set device (iLO) bay # 2 to IP address 192.168.20.142 Successfully set device (iLO) bay # 3 to IP address 192.168.20.143 Successfully set device (iLO) bay # 4 to IP address 192.168.20.144 Successfully set device (iLO) bay # 5 to IP address 192.168.20.145 Successfully set device (iLO) bay # 6 to IP address 192.168.20.146 Successfully set device (iLO) bay # 7 to IP address 192.168.20.147 Successfully set device (iLO) bay # 8 to IP address 192.168.20.148 Successfully set device (iLO) bay # 9 to IP address 192.168.20.149 Successfully set device (iLO) bay #10 to IP address 192.168.20.150 Successfully set device (iLO) bay #11 to IP address 192.168.20.151 Successfully set device (iLO) bay #12 to IP address 192.168.20.152 Successfully set device (iLO) bay #13 to IP address 192.168.20.153 Successfully set device (iLO) bay #14 to IP address 192.168.20.154 Successfully set device (iLO) bay #15 to IP address 192.168.20.155 Successfully set device (iLO) bay #16 to IP address 192.168.20.156 For the IP addresses to be assigned EBIPA must be enabled. OA-FC15B41AEA05>
- Add New User for OA.
Create new user, set access level as ADMINISTRATOR, and assign access to all blades, all enclosure switches and OAs. After that, the username and password can be used to access OAs.
OA-FC15B41AEA05> ADD USER <username> New Password: ******** Confirm : ******** User "<username>" created. You may set user privileges with the 'SET USER ACCESS' and 'ASSIGN' commands. OA-FC15B41AEA05> set user access <username> ADMINISTRATOR "<username>" has been given administrator level privileges. OA-FC15B41AEA05> ASSIGN SERVER ALL <username> <username> has been granted access to the valid requested bay(s) OA-FC15B41AEA05> ASSIGN INTERCONNECT ALL <username> <username> has been granted access to the valid requested bay(s) OA-FC15B41AEA05> ASSIGN OA <username> <username> has been granted access to the OA.
- From OA, go to each blade with "connect server <bay number>", add New User
for each blade.
OA-FC15B41AEA05> connect server 4 Connecting to bay 4 ... User:OAtmp-root-5CBF2E61 logged-in to ILO2M290605KP.(192.168.20.144 / FE80::AF1:EAFF:FE89:460) iLO Standard Blade Edition 1.37 at Oct 25 2018 Server Name: Server Power: On </>hpiLO-> </>hpiLO-> create /map1/accounts1 username=root password=TklcRoot group=admin,config,oemHPE_rc,oemHPE_power,oemHPE_vm status=2 status_tag=COMMAND PROCESSING FAILED error_tag=COMMAND SYNTAX ERROR Tue Apr 23 16:18:58 2019 User added successfully.
- Change to static IP on OA. In order not reply on DHCP and make the OA address
stable, change to static IP.
Note:
After the following change, on the active OA (could be the bay1 OA or bay2 OA), the OA session will be stuck due to the address change, make another server session ready to ssh with the new IP address and new root user. The change on the standby OA will not stuck the OA session.OA-FC15B41AEA05> SET IPCONFIG STATIC 1 192.168.20.131 255.255.255.0 Static IP settings successfully updated. These setting changes will take effect immediately. OA-FC15B41AEA05> SET IPCONFIG STATIC 2 192.168.20.132 255.255.255.0 Static IP settings successfully updated. These setting changes will take effect immediately. OA-FC15B41AEA05>
Configure Legacy BIOS on Remaining Hosts
Note:
The procedures in this document apply to the HP iLO console accessed via KVM. Each procedure is executed in the order listed.Prerequisites
Procedure OCCNE Configure Addresses for RMS iLOs, OA, EBIPA is complete.
Limitations and Expectations
- Applies to HP iLO 5 only.
- Should the System Utility indicate (or defaults to) UEFI booting, then the user must go through the steps to reset booting back to the Legacy BIOS mode by following step: Change over from UEFI Booting Mode to Legacy BIOS Booting Mode.
- The procedures listed here apply to both Gen10 DL380 RMSs and Gen10 BL460c Blades in a C7000 enclosure.
- Access to the enclosure blades in these procedures is via the Bootstrap host using SSH on the KVM. This is possible because the prerequisites are complete. If the prerequisites are not completed before executing this procedure, the enclosure blades are only accessible via the KVM connected directly to the active OA. In this case the mouse is not usable and screen manipulations are performed using the keyboard ESC and directional keys.
- This procedure does NOT apply to the Bootstrap Host.
References
Procedure
- Expose the System Configuration Utility on a RMS Host on the KVM. This
procedure does not provide instructions on how to connect the KVM as this
may be different on each installation.
- Once the remote console has been exposed, the system must be reset by manually pressing the power button on the front of the RMS host to force it through the restart process. When the initial window is displayed, hit the F9 key repeatedly. Once the F9 is highlighted at the lower left corner of the remote console, it should eventually bring up the main System Utility.
- The System Utilities screen is exposed in the remote console.
- Expose the System Utility for an Enclosure Blade.
- The blades are maintained
via the OAs in the enclosure. Because each blade iLO has already
been assigned an IP address from the prerequisites, the blades can
each be reached using SSH from the Bootstrap host login shell on the
KVM.
- SSH to the blade
using the iLO IP address and the root user and password.
This brings up the HP iLO prompt.
$ ssh root@<blade_ilo_ip_address> Using username "root". Last login: Fri Apr 19 12:24:56 2019 from 10.39.204.17 [root@localhost ~]# ssh root@192.168.20.141 root@192.168.20.141's password: User:root logged-in to ILO2M290605KM.(192.168.20.141 / FE80::AF1:EAFF:FE89:35E) iLO Standard Blade Edition 1.37 at Oct 25 2018 Server Name: Server Power: On </>hpiLO->
- Use VSP to connect
to the blade remote console.
</>hpiLO->vsp
-
Power cycle the blade to bring up the System Utility for that blade.
Note:
The System Utility is a text based version of that exposed on the RMS via the KVM. The user must use the directional (arrow) keys to manipulate between selections, ENTER key to select, and ESC to go back from the current selection. - Access the System Utility by hitting ESC 9.
- SSH to the blade
using the iLO IP address and the root user and password.
This brings up the HP iLO prompt.
- Enabling Virtualization
This procedure provides the steps required to enable virtualization on a given Bare Metal Server. Virtualization can be configured using the default settings or via the default Workload Profiles.
Verifying Default Settings- Expose the System Utility by following step 1 or 2 depending on the hardware being configured.
- Select System Configuration
- Select BIOS/Platform Configuration (RBSU)
- Select Virtualization Options
This view displays the settings for the Intel(R) Virtualization Technology (IntelVT), Intel(R) VT-d, and SR-IOV options (Enabled or Disabled). The default values for each option is Enabled.
- Select F10 if it is desired to save and stay in the utility or select the F12 if it is desired to save and exit to continue the current boot process.
- The blades are maintained
via the OAs in the enclosure. Because each blade iLO has already
been assigned an IP address from the prerequisites, the blades can
each be reached using SSH from the Bootstrap host login shell on the
KVM.
- Change over from UEFI Booting Mode to Legacy BIOS Booting Mode:
- Expose the System Utility by following step 1 or 2 depending on the hardware being configured.
- Select System Configuration
- Select BIOS/Platform Configuration (RBSU)
- Select Boot Options.
This menu defines the boot mode.
If the Boot Mode is set to UEFI Mode then continue this procedure. Otherwise there is no need to make any of the changes below. - Select Boot Mode
This generates a warning indicating the following:Hit the ENTER key and two selections appear: UEFI Mode(highlighted) and Legacy BIOS Mode
Boot Mode changes require a system reboot in order to take effect. Changing the Boot Mode can impact the ability of the server to boot the installed operating system. An operating system is installed in the same mode as the platform during the installation. If the Boot Mode does not match the operating system installation, the system cannot boot. The following features require that the server be configured for UEFI Mode: Secure Boot, IPv6 PXE Boot, Boot > 2.2 TB Disks in AHCI SATA Mode, and Smart Array SW RAID.
- Use the down arrow key to
select Legacy BIOS
Mode and hit the ENTER. The screen indicates:
A reboot is required for the Boot Mode changes.
- Hit F12. This
displays the following:
Changes are pending. Do you want to save changes? Press 'Y" to save and exit, 'N' to discard and stay, or 'ESC' to cancel.
- Hit the y key and an
additional warning appears indicating:
System configuration changed. A system reboot is required. Press ENTER to reboot the system.
-
- Hit ENTER
to force a reboot.
Note:
The boot must go into the process of actually trying to boot from the boot devices using the boot order (not just go back through initialization and access the System Utility again). The boot should fail and the System Utility can be accessed again to continue any further changes needed. - After the reboot, hit the ESC 9key sequence to re-enter the System Utility. Selecting System Configuration->BIOS/Platform Configuration (RBSU)->Boot Options. Verify the Boot Mode is set to Legacy Boot Mode UEFI Optimized Boot is set to Disabled
- Hit ENTER
to force a reboot.
- Select F10 if it is desired to save and stay in the utility or select the F12 if it is desired to save and exit to complete the current boot process.
- Force PXE to boot from the first Embedded FlexibleLOM HPE Ethernet 10Gb
2-port Adapter.
- Expose the System Utility by following step 1 or 2 depending on the hardware being configured.
- Select System Configuration.
- Select BIOS/Platform Configuration (RBSU) .
- Select Boot Options. This menu defines the boot mode.
- Confirm the following settings: Boot Mode Legacy BIOS Mode UEFI Optimized Boot, and Boot Order Policy Retry Boot Order Indefinitely(this means it keeps trying to boot without ever going to disk). If not in Legacy BIOS Mode, follow procedure Change over from UEFI Booting Mode to Legacy BIOS Booting Mode.
- Select Legacy BIOS Boot Order In the default view, the 10Gb Embedded FlexibleLOM 1 Port 1 is at the bottom of the list.
- Move the 10 Gb Embedded FlexibleLOM 1 Port 1 entry up above the 1Gb Embedded LOM 1 Port 1 entry. To move an entry press the '+' key to move an entry higher in the boot list and the '-' key to move an entry lower in the boot list. Use the arrow keys to navigate through the Boot Order list.
- Select F10 if it is desired to save and stay in the utility or select the F12 it is desired to save and exit to continue the current boot process.
- Enabling Virtualization:
This step provides the steps required to enable virtualization on a given Bare Metal Server. Virtualization can be configured using the default settings or via the Workload Profiles.
Verifying Default Settings- Expose the System Utility by following step 1 or 2 depending on the hardware being configured.
- Select System Configuration
- Select BIOS/Platform Configuration (RBSU)
- Select Virtualization Options
This view displays the settings for the Intel(R) Virtualization Technology (IntelVT), Intel(R) VT-d, and SR-IOV options (Enabled or Disabled). The default values for each option is Enabled.
- Select F10 if it is desired to save and stay in the utility or select the F12 if it is desired to save and exit to continue the current boot process.
- Disable RAID Configurations:
OCCNE does not currently support any RAID configuration. Follow this step to disable RAID settings if the default settings of the System Utility include any RAID configuration(s).
Note: There may be more than one RAID Array set up. This procedure should be repeated for any RAID configuration.
- Expose the System Utility by following step 1 or 2 depending on the hardware being configured.
- Select System Configuration.
- Select Embedded RAID 1: HPE Smart Array P408i-a SR Gen 10.
- Select Array Configuration.
- Select Manage Arrays.
- Select Array A (or any designated Array Configuration if there are more than one).
- Select Delete
Array. A warning is displayed indicating the following:
Deletes an Array. All the data on the logical drives that are part of deleted array will be lost. Also if the deleted array is the only one on the controller, the controller settings will be erased and its default configuration is restored.
- Hit ENTER,
the changes are submitted and
Delete Array Successful
is displayed. - Hit ENTER to go back to the main menu for the HPE Smart Array.
- Select F10 if it is desired to save and stay in the utility or select the F12 it is desired to save and exit to continue the current boot process.
- Enable the Primary and Secondary Boot Devices:
This steps provide necessary to configure the primary and secondary bootable devices for a Gen10 Server.
Note:
There can be multiple configurations of hardware drives on the server that include both Hard Drives (HDD) and Solid State Hard Drives (SSD). SSDs are indicated by SATA-SSD ATA in the drive description. The commands below include two HDDs and two SSDs. The SSDs are not to be selected for this configuration. The actual selections may be different based on the hardware being updated.- Expose the System Utility by following step 1 or 2 depending on the hardware being configured.
- Select System Configuration.
- Select Embedded RAID 1 : HPE Smart Array P408i-a SR Gen 10.
- Select Set Bootable
Device(s) for Legacy Boot Mode.
If the boot devices are not set then Not Set is displayed for the primary and secondary devices.
- Examine the list of
available hardware drives. If one or more HDDs are available,
continue with this step.
Note:
A single drive can be set as both the primary and secondary boot device but that is not part of this configuration. - Select Bootable Physical Drive
- Select Port 1| Box:3 Bay:1 Size:1.8 TB SAS HP EG00100JWJNR. Note: This example includes two HDDs and two SSDs. The actual configuration may be different.
- Select Set as Primary Bootable Device.
- Hit ENTER.
Note:
There is no need to set the secondary boot device. Leave it as Not Set. - Hit the ESC key to back out to the System Utilitiesmenu.
- Select F10 if it Is desired to save and stay in the utility or select the F12 if it Is desired to save and exit to continue the current boot process.
Configure Enclosure Switches
Introduction
This procedure is used to configure the 6127XLG enclosure switches.
Prerequisites
- Procedure Configure Top of Rack 93180YC-EX Switches has been completed.
- Procedure Configure Addresses for RMS iLOs, OA, EBIPA has been completed.
- The Utility USB is available containing the necessary files as per: Installation PreFlight checklist: Create Utility USB.
Limitations/Expectations
All steps are executed from a Keyboard, Video, Mouse (KVM) connection.
Procedure
- Copy the 6127XLG configuration file from the Utility USB (See Installation PreFlight checklist: Create the OA 6127XLG
Switch Configuration File) to the /var/lib/tftpboot directory on
the Installer Bootstrap Host and verify it exists and the permissions.
$ cp /media/usb/6127xlg_irf.cfg /var/lib/tftpboot/6127xlg_irf.cfg $ ls -l /var/lib/tftpboot/ total 1305096 -rw-r--r--. 1 root root 311 Mar 25 08:41 6127xlg_irf.cfg
- Modify the switch specific values in the /var/lib/tftpboot/6127xlg_irf.cfg
file.
These values are contained at Installation PreFlight checklist : Create the OA 6127XLG Switch Configuration File from column Enclosure_Switch.
$ cd /var/lib/tftpboot $ sed -i 's/{switchname}/<switch_name>/' 6127xlg_irf.cfg $ sed -i 's/{admin_password}/<admin_password>/' 6127xlg_irf.cfg $ sed -i 's/{user_name}/<user_name>/' 6127xlg_irf.cfg $ sed -i 's/{user_password}/<user_password>/' 6127xlg_irf.cfg
- Access the InterConnect Bay1 6127XLG switch to configure the IRF
(Intelligent Resilient Framework).
Note:
On a new switch the user is presented with the following when connecting to the console and must type CTRL_C or CTRL_D to break out of the loop.When trying to save the config, the following prompt is received: [HPE] [HPE] save The current configuration will be written to the device. Are you sure? [Y/N]: Before pressing ENTER you must choose 'YES' or 'NO'[Y/N]:y Please input the file name(*.cfg)[flash:/startup.cfg] (To leave the existing filename unchanged, press the enter key): User can leave this default startup.cfg unchanged, or change to another name. The cfg file will be used for next reboot.
$ ssh <oa username>@<oa address> If it shows standby, ssh to the other OA address. OA-FC15B41AEA05> connect interconnect 1 .... <HPE>system-view System View: return to User View with Ctrl+Z. (Note: Run the following commands:) irf member 1 priority 32 interface range Ten-GigabitEthernet 1/0/17 to Ten-GigabitEthernet 1/0/20 shutdown quit irf-port 1/1 port group interface Ten-GigabitEthernet1/0/17 port group interface Ten-GigabitEthernet1/0/18 port group interface Ten-GigabitEthernet1/0/19 port group interface Ten-GigabitEthernet1/0/20 quit interface range Ten-GigabitEthernet 1/0/17 to Ten-GigabitEthernet 1/0/20 undo shutdown quit save irf-port-configuration active
- Access the InterConnect Bay2 6127XLG switch to re-number to IRF 2.
OA-FC15B41AEA05> connect interconnect 2 .... <HPE>system-view System View: return to User View with Ctrl+Z. [HPE] irf member 1 renumber 2 Renumbering the member ID may result in configuration change or loss. Continue?[Y/N]Y [HPE]save The current configuration will be written to the device. Are you sure? [Y/N]:Y Please input the file name(*.cfg)[flash:/startup.cfg] (To leave the existing filename unchanged, press the enter key): Validating file. Please wait... Saved the current configuration to mainboard device successfully. [HPE]quit <HPE>reboot Start to check configuration with next startup configuration file, please wait.........DONE! This command will reboot the device. Continue? [Y/N]:Y Now rebooting, please wait... System is starting...
- Configure the IRF on Bay2 6127XLG switch
After rebooting, the interfaces will begin with number 2 such as Ten-GigabitEthernet2/0/17, Ten-GigabitEthernet2/1/5. Run the following commands:
system-view interface range Ten-GigabitEthernet 2/0/17 to Ten-GigabitEthernet 2/0/20 shutdown quit irf-port 2/2 port group interface Ten-GigabitEthernet2/0/17 port group interface Ten-GigabitEthernet2/0/18 port group interface Ten-GigabitEthernet2/0/19 port group interface Ten-GigabitEthernet2/0/20 quit interface range Ten-GigabitEthernet 2/0/17 to Ten-GigabitEthernet 2/0/20 undo shutdown quit save irf-port-configuration active
- Run "reboot" command on both
switches.
<HPE>reboot Start to check configuration with next startup configuration file, please wait.........DONE! This command will reboot the device. Continue? [Y/N]:Y Now rebooting, please wait... System is starting...
- Verify the IRF for the 6127XLG switches. When reboot is finished, verify IRF
is working with both member and ports from previous two switches, which form
IRF to act as one switch now.
<HPE>system-view System View: return to User View with Ctrl+Z. [HPE]display irf configuration MemberID NewID IRF-Port1 IRF-Port2 1 1 Ten-GigabitEthernet1/0/17 disable Ten-GigabitEthernet1/0/18 Ten-GigabitEthernet1/0/19 Ten-GigabitEthernet1/0/20 2 2 disable Ten-GigabitEthernet2/0/17 Ten-GigabitEthernet2/0/18 Ten-GigabitEthernet2/0/19 Ten-GigabitEthernet2/0/20 [HPE]
- Configure the IRF switch with predefined configuration
file.
<HPE>tftp 192.168.20.11 get 6127xlg_irf.cfg startup.cfg startup.cfg already exists. Overwrite it? [Y/N]:Y Press CTRL+C to abort. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 9116 100 9116 0 0 167k 0 --:--:-- --:--:-- --:--:-- 178k <HPE>system-view System View: return to User View with Ctrl+Z. [HPE]configuration replace file flash:/startup.cfg Current configuration will be lost, save current configuration? [Y/N]:N Now replacing the current configuration. Please wait ... Succeeded in replacing current configuration with the file flash:/startup.cfg. [<switch_name>]save flash:/startup.cfg The current configuration will be saved to flash:/startup.cfg. Continue? [Y/N]:Y flash:/startup.cfg exists, overwrite? [Y/N]:Y Now saving current configuration to the device. Saving configuration flash:/startup.cfg.Please wait... Configuration is saved to device successfully. [<switch_name>]
Bastion Host Installation
This section outlines the use of the Installer Bootstrap Host to provision db-2/RMS2 with an operating system and configure it to fulfill the role of Database Host. After the Bastion Host is created, it is used to complete the installation of OCCNE.
Provision Second Database Host (RMS2) from Installer Bootstrap Host (RMS1)
Table 2-1 Terminology used in Procedure
Name | Description |
---|---|
bastion_full_name | This is the full name of the Bastion Host as
defined in the hosts.ini file.
Example: bastion-2.rainbow.us.labs.oracle.com |
bastion_kvm_host_full_name | This is the full name of the KVM server (usually
RMS2/db-2) that hosts the Bastion Host VM.
Example: db-2.rainbow.us.labs.oracle.com |
bastion_kvm_host_ip_address |
This is the IPv4 ansible_host IP address of the server (usually RMS2/db-2) that hosts the Bastion Host VM. Example: 172.16.3.5 |
bastion_short_name | This is the name of the Bastion Host derived
from the bastion_full_name up to the first ".".
Example: bastion-2 |
bastion_external_ip_address | This is the external address for the Bastion
Host
Example : 10.75.148.5 for bastion-2 |
bastion_ip_address |
This is the internal IPv4 "ansible_host" address of the Bastion Host as defined within the hosts.ini file. Example: 172.16.3.100 for bastion-2 |
cluster_full_name | This is the name of the cluster as defined in
the hosts.ini file field: occne_cluster_name.
Example: rainbow.us.labs.oracle.com |
cluster_short_name | This is the short name of the cluster derived
from the cluster_full_name up to the the first ".".
Example: rainbow |
Note:
The Bootstrap Host must be setup to use root/<customer_specific_root_password> as the credentials to access it. Setting that user/password is part of the instructions at: Installation of Oracle Linux 7.x on Bootstrap Host.Procedure
- Copy the necessary files from the Utility USB to support the OS
Install:
This procedure is used to provide the steps for copying all supporting files from the Utility USB to the appropriate directories so that the Provision Container successfully installs OL7 onto RMS2.
- Login to the Bootstrap Host using the root credentials configured during OS installation of the Bootstrap Host.
- Create the directories
needed on the Installer Bootstrap Host.
$ mkdir -p /var/occne/cluster/<cluster_short_name>/yum.repos.d
- Mount the Utility USB.
Note:
Follow the instructions for mounting a USB in Linux are at: Installation of Oracle Linux 7.x on Bootstrap Host. - Copy the hosts.ini file
(created using procedure: OCCNE Inventory File Preparation) into the
/var/occne/cluster/<cluster_short_name>/ directory.
This hosts.ini file defines the Bastion KVM Host to the Provision Container running the provision image downloaded from the repo.
$ cp /<path_to_usb>/hosts.ini /var/occne/cluster/<cluster_short_name>/hosts.ini Example: $ cp /media/usb/hosts.ini /var/occne/cluster/rainbow/hosts.ini
- Edit the
/var/occne/cluster/<cluster_short_name>/hosts.ini file to
include the ToR host_net (vlan3) VIP for NTP clock synchronization.
Use the ToR VIP address (ToRswitch_Platform_VIP ) as defined in
procedure: Installation PreFlight Checklist : Complete OA and
Switch IP SwitchTable as the NTP source. Update the
ntp_server field with the VIP address. Update the
occne_repo_host_address to point to this Bootstrap Host internal
address. This is being used for PXE booting and accessing the NFS
share on the Installer Bootstrap Host (db-1/RMS1).
Example (from hosts.sample.ini): ntp_server='172.16.3.1' occne_repo_host_address='172.16.3.4'
- Follow procedure Populate the MetalLB Configuration File to
copy the MetalLB configuration file into the
/var/occne/cluster/<cluster_short_name>/ directory and
configure it.
$ cp /<path_to_usb>/mb_configmap.yaml /var/occne/cluster/<cluster_short_name>/mb_configmap.yaml Example: $ cp /media/usb/mb_configmap.yaml /var/occne/cluster/rainbow/mb_configmap.yaml
- Copy the customer specific
.repo file from the Utility USB to the Installer Bootstrap Host.
This is the .repo file created by the customer that provides access to the onsite (within their network) yum repositories needed to complete the full deployment of OCCNE onto the Installer Bootstrap Host. It needs to be in two places, one for the local system, and one for the target systems.
$ cp /<path_to_usb>/<customer_specific_repo>.repo /var/occne/cluster/<cluster_short_name>/yum.repos.d/. $ cp -r /var/occne/cluster/<cluster_short_name>/yum.repos.d /var/occne/. $ echo "reposdir=/var/occne/yum.repos.d" >> /etc/yum.conf Example: $ cp /media/usb/ol7-mirror.repo /var/occne/cluster/rainbow/yum.repos.d/ $ cp -r /var/occne/cluster/rainbow/yum.repos.d /var/occne/ $ echo "reposdir=/var/occne/yum.repos.d" >> /etc/yum.conf
- Set up the /etc/hosts file for the Central Repo and Verify Access:
- Add an entry to the
/etc/hosts file on the Installer Bootstrap Host to provide a name
mapping for the Customer Central Repository.
$ vi /etc/hosts Example: 10.75.200.217 rainbow-reg To Verify: $ ping <central_repo_name> Example: # ping rainbow-reg PING reg-1 (10.75.200.217) 56(84) bytes of data. 64 bytes from reg-1 (10.75.200.217): icmp_seq=1 ttl=61 time=0.248 ms 64 bytes from reg-1 (10.75.200.217): icmp_seq=2 ttl=61 time=0.221 ms 64 bytes from reg-1 (10.75.200.217): icmp_seq=3 ttl=61 time=0.239 ms
- Verify the repo access
execute the following command:
$ yum repolist Example: $ yum repolist Loaded plugins: ulninfo repo id repo name status !UEKR5/x86_64 Unbreakable Enterprise Kernel Release 5 for Oracle Linux 7 (x86_64) 80 !addons/x86_64 Oracle Linux 7 Addons (x86_64) 91 !developer/x86_64 Packages for creating test and development environments for Oracle Linux 7 226 !developer_EPEL/x86_64 Packages for creating test and development environments for Oracle Linux 7 13,246 !ksplice/x86_64 Ksplice for Oracle Linux 7 (x86_64) 393 !latest/x86_64 Oracle Linux 7 Latest (x86_64) 5,401 repolist: 19,437
- Add an entry to the
/etc/hosts file on the Installer Bootstrap Host to provide a name
mapping for the Customer Central Repository.
- Copy the OL7 ISO to the Installer Bootstrap Host:
The iso file must be accessible from a Customer Site Specific repository. This file should be accessible because the ToR switch configurations were completed in procedure: Configure Top of Rack 93180YC-EX Switches
Copy the OL7 ISO file to the /var/occne directory. The example below uses OracleLinux-7.5-x86_64-disc1.iso. If this file was copied to the Utility USB, it can be copied from there into the same directory on the Bootstrap Host.
Note:
If the user copies this ISO from their laptop then they must use an application like WinSCP pointing to the Management Interface IP.$ scp <usr>@<site_specific_address>:/<path_to_iso>/OracleLinux-7.5-x86_64-disc1.iso /var/occne/OracleLinux-7.5-x86_64-disc1.iso
- Install Packages onto the Installer Bootstrap Host. Use YUM to install
necessary packages onto the installer Bootstrap Host.
$ yum install docker-engine nfs-utils ansible
- Set up access to the Docker Registry on the Installer Bootstrap Host:
- Copy the docker registry
certificate to two places on the Bootstrap Host.
Note:
How to obtain the docker registry certificate <source> is not necessarily covered in the procedure. The user can use the instructions at reference 1 to understand this more thoroughly.$ mkdir -p /var/occne/certificates $ cp <source>.crt /var/occne/certificates/<occne_private_registry>:<occne_private_registry_port>.crt $ mkdir -p /etc/docker/certs.d/<occne_private_registry>:<occne_private_registry_port> $ cp <source>.crt /etc/docker/certs.d/<occne_private_registry>:<occne_private_registry_port>/ca.crt Example: $ mkdir -p /var/occne/certificates $ cp <source>.crt /var/occne/certificates/rainbow_reg:5000.crt $ mkdir -p /etc/docker/certs.d/rainbow_reg:5000 $ cp <source>.crt /etc/docker/certs.d/rainbow_reg:5000/ca.crt
- Start the docker daemon.
$ systemctl daemon-reload $ systemctl restart docker $ systemctl enable docker Verify docker is running: $ ps -elf | grep docker $ systemctl status docker
- Copy the docker registry
certificate to two places on the Bootstrap Host.
- Setup NFS on the Installer Bootstrap Host:
Run the following commands using sudo (nfs-utils has already been installed in previous steps).
Note:
The IP address used in the echo command is the Platform VLAN IP Address (VLAN 3)of the Bootstrap Host (RMS 1) as given in: Installation PreFlight Checklist : Site Survey Host Table.$ echo '/var/occne 172.16.3.4/24(ro,no_root_squash)' >> /etc/exports $ systemctl start nfs-server $ systemctl enable nfs-server Verify nfs is running: $ ps -elf | grep nfs $ systemctl status nfs-server
- Set up the Boot Loader on the Installer Bootstrap
Host:
$ mkdir -p /var/occne/pxelinux $ mount -t iso9660 -o loop /var/occne/OracleLinux-7.5-x86_64-disc1.iso /mnt $ cp /mnt/isolinux/initrd.img /var/occne/pxelinux $ cp /mnt/isolinux/vmlinuz /var/occne/pxelinux
- Verify and Set the PXE Configuration File Permissions on the Installer
Bootstrap Host:
Note:
Each file configured in the step above must be open for read and write permissions.$ chmod -R 777 /var/occne/pxelinux
- Disable DHCP and TFTP on the Installer Bootstrap Host. The TFTP and DHCP
services running on the Installer Bootstrap Host may still be running. These
services must be disabled.
$ systemctl stop dhcpd $ systemctl disable dhcpd $ systemctl stop tftp $ systemctl disable tftp
- Disable SELINUX: Set SELINUX to permissive mode. In order to successfully
set the SELINUX mode, a reboot of the system is required. The getenforce
command is used to determine the status of SELINUX.
$ getenforce active If the output of this command displays "active", change it to "permissive" by editing the /etc/selinux/config file. $ vi /etc/selinux/config Change the SELINUX variable to passive: SELINUX=permissive save the file Reboot the system: reboot
- Generate the SSH private and public keys on Bootstrap Host.
This command generates a private and public key for the cluster. These keys are passed to the Bastion Host and used to communicate to other nodes from that Bastion Host. The public key is passed to each node on OS install. Do not supply a passphrase when it asks for one. Click Enter.
Note:
The private key (occne_id_rsa) must be copied to a server that going to access the Bastion Host because the Bootstrap Host is repaved. This key is used later in the procedure to access the Bastion Host after it has been created.$ mkdir -m 0700 /var/occne/cluster/<cluster_short_name>/.ssh $ ssh-keygen -b 4096 -t rsa -C "occne installer key" -f "/var/occne/cluster/<cluster_short_name>/.ssh/occne_id_rsa" -q -N ""
- Execute the OS Install and Bastion VM Creation on Bastion KVM Host (RMS2)
from the Installer Bootstrap Host:
- Run the docker commands
below to perform the OS install and Bastion Host VM creation on the
Bastion KVM Host (RMS2):
$ docker run --rm --network host --cap-add=NET_ADMIN -v /var/occne/cluster/<cluster_short_name>/:/host -v /var/occne/:/var/occne:rw -e "OCCNEARGS=--limit=<bastion_full_name>,<bastion_kvm_host_full_name>,localhost" <image_name>:<image_tag> Example: $ docker run -it --rm --network host --cap-add=NET_ADMIN -v /var/occne/cluster/rainbow/:/host -v /var/occne/:/var/occne:rw -e "OCCNEARGS=--limit=bastion-2.rainbow.lab.us.oracle.com,db-2.rainbow.lab.us.oracle.com,localhost" winterfell:5000/occne/provision:1.3.0
- Verify that Bastion Host VM
is installed by logging into RMS2/db-2 and issuing the following
command. The <ansible_host> field (which is an IPv4 address)
is derived from the hosts.ini file db-2 entry for host_hp_gen_x
groups.
Note:
This command is optional. Had a failure actually occurred, the docker run command would have experienced failures.ssh -i /var/occne/cluster/<cluster_short_name>/.ssh/occne_id_rsa admusr@<oam_host> $ sudo virsh list Example: ssh -i /var/occne/cluster/rainbow/.ssh/occne_id_rsa admusr@10.75.148.6 $ sudo virsh list Id Name State ---------------------------------------------------- 11 bastion-2.rainbow.lab.us.oracle.com running
- Login to Bastion Host from
the Boostrap Host as admusr using the generated key from the
/var/occne/cluster/<cluster_short_name> directory to confirm
the VM is set up correctly. The <oam_host> field (which is an
IPv4 address) is derived from the hosts.ini file bastion-2 entry for
the host_kernel_virtual group.
Note:
This command is optional. Had a failure actually occurred, the docker run command would have experienced failures.ssh -i /var/occne/cluster/<cluster_short_name>/.ssh/occne_id_rsa admusr@<oam_host> Example: ssh -i /var/occne/cluster/rainbow/.ssh/occne_id_rsa admusr@10.75.148.5
- Run the docker commands
below to perform the OS install and Bastion Host VM creation on the
Bastion KVM Host (RMS2):
Automated Installation
This section details the steps required to execute the automated configuration of the Bastion Host VM. This consists of two main section:
- Setting up and executing the deploy.sh script on the Bootstrap Host.
- Accessing the Bastion Host and executing the final commands to execute the pipeline.sh script to complete the Bastion Host configuration and deploy the OCCNE cluster.
- Setting up for and executing the deploy.sh script on the Bootstrap Host:
The deploy.sh script performs the initial pre-configuration of the Bastion host. This includes installing ansible, executing the ansible playbook configBastionHost.yaml to setup the initial files and staging directories on the Bastion Host and executing the pipeline to setup the artifacts directory. The script is executed on the Bootstrap Host using a set of environment variables that can be initialized on the command line along with the deploy.sh script. These variables include the following:
Table 2-2 Environmental Variables
Name Comment Example usage CENTRAL_REPO Defines the customer specific repository host name. Note: This would be used in conjunction with CENTRAL_REPO_IP and CENTRAL_REPO_DOCKER_PORT. CENTRAL_REPO=customer_repo \ CENTRAL_REPO_IP=<IP_Address> \ CENTRAL_REPO_DOCKER_PORT=5000 CENTRAL_REPO_IP Defines the customer specific repository IPv4 address. See above. CENTRAL_REPO_DOCKER_PORT Defines the customer specific repository docker port number. Defaults to 5000. See above. OCCNE_CLUSTER Defines the cluster short name. OCCNE_CLUSTER=rainbow OCCNE_BASTION Bastion Host full name OCCNE_BASTION=bastion-2.rainbow.us.labs.oracle.com OCCNE_VERSION The version tag of the image releases OCCNE_VERSION=1.6.0 - Copy necessary files from Utility USB to the Bootstrap Host staging
directory:
- The MySQL .zip file (ex:
V980756-01.zip) must be copied to the staging directory /var/occne
directory. This file should be provided from the Utility USB. This
file is used for installing the ndb MySQL nodes.
$ cp /<usb_dev>/db/*.zip /var/occne/*.zip
- Copy the
configBastionHost.yaml file from the Customer Utility USB to the
staging directory /var/occne/.
$ cp /<usb_dev>/configBastionHost.yaml /var/occne/.
- Copy the deploy.sh script from the
Customer Utility USB to the staging directory at /var/occne/ and set
the file to executable
mode.
$ cp /<usb_dev>/deploy.sh /var/occne/. $ chmod +x /var/occne/deploy.sh
- The MySQL .zip file (ex:
V980756-01.zip) must be copied to the staging directory /var/occne
directory. This file should be provided from the Utility USB. This
file is used for installing the ndb MySQL nodes.
- Execute the deploy.sh script from the /var/occne/ directory with the
required parameters set.
$ export CENTRAL_REPO=<customer_specific_repo_name> $ export CENTRAL_REPO_IP=<customer_specific_repo_ipv4> $ export OCCNE_CLUSTER=<cluster_short_name> $ export OCCNE_BASTION=<bastion_full_name> $ ./deploy.sh Customer Example: $ export CENTRAL_REPO=central-repo $ export CENTRAL_REPO_IP=10.10.10.10 $ export OCCNE_CLUSTER=rainbow $ export OCCNE_BASTION=bastion-2.rainbow.us.labs.oracle.com $ ./deploy.sh The command above can be executed like the following: $ CENTRAL_REPO=central-repo CENTRAL_REPO_IP=10.10.10.10 OCCNE_CLUSTER=rainbow OCCNE_BASTION=bastion-2.rainbow.us.labs.oracle.com ./deploy.sh
- Execute the final deploy on Bastion Host:
The following commands are executed from the Bastion Host to complete the Bastion Host configuration and deploy OCCNE on the Bare Metal system.
Note:
The Bootstrap Host cannot be used to access the Bastion Host as it will be re-paved from execution of this command.- Login to the Bastion
Host as admusr. The private key that was saved earlier
should be used to access the Bastion Host from a server other
than the Bootstrap Host using the ssh command. This private key
should be copied to the /home/<user>/.ssh directory on
that server as id_rsa using scp (or winSCP from a desktop PC).
The permissions of the key must be set to 0600 using the
command:
chmod 0600 ~/.ssh/id_rsa
$ ssh -i ~/.ssh/id_rsa admusr@<bastion_external_ip_address>
- Execute the following
command to complete the deployment of OCCNE from the Bastion
Host (excluding re-install on the Bastion Host and its KVM host,
which are already setup). This action will re-pave the Bootstrap
Host RMS.
$ export OCCNE_PROV_DEPLOY_ARGS='--limit=!<bastion_full_name>,!<bastion_kvm_host_full_name>' $ export OCCNE_DB_ARGS='--extra-vars=mate_site_db_replication_ip=<remote_site_db_replication_service_ip>,occne_mysqlndb_cluster_id=<mysqlndb_cluster_identifier>' $ /var/occne/cluster/<cluster_short_name>/artifacts/pipeline.sh Customer Example: $ export OCCNE_PROV_DEPLOY_ARGS='--limit=!bastion-2.rainbow.lab.us.oracle.com,!db-2.rainbow.lab.us.oracle.com' $ export OCCNE_DB_ARGS='--extra-vars=mate_site_db_replication_ip=10.75.182.88,occne_mysqlndb_cluster_id=2' $ /var/occne/cluster/rainbow/artifacts/pipeline.sh To save the output from the pipeline.sh script the command can be written as: $ /var/occne/cluster/rainbow/artifacts/pipeline.sh | tee pipeline$(date +"%F_%H%M%S").log
Note:
While Installing on the First Site ignore OCCNE_DB_ARGS configuration parameter which Provides Mate Site DB Replication Service Load Balancer IP. Provide the Mate Site DB Replication Service Load Balancer IP and MySQL Cluster identifier while Installing MYSQL NDB Cluster on second site. - Login to the Bastion
Host as admusr. The private key that was saved earlier
should be used to access the Bastion Host from a server other
than the Bootstrap Host using the ssh command. This private key
should be copied to the /home/<user>/.ssh directory on
that server as id_rsa using scp (or winSCP from a desktop PC).
The permissions of the key must be set to 0600 using the
command:
- Change MySQL root user password. Refer to Change MySQL root user password
- Update the Bastion KVM Host repo file:
Since db-2 was not part of the final OS install and cluster deploy, it's /var/occne/yum.repos.d/*.repo file is not pointing to the Bastion Host as its YUM repo. That file on RMS2/db-2 must be updated so that it now points to the Bastion Host as the repo. After the Bastion Host was created, the .repo file that was copied onto the Bastion Host has the correct settings. That file can just be copied back to RMS2/db-2.
- From the bastion Host, login to the Bastion Host KVM Host, using
the occne private key and the internal host IP address for that
node (extracted from the hosts.ini
file):
$ ssh -i /var/occne/cluster/<cluster_short_name>/.ssh/occne_id_rsa admusr@<bastion_ip_address>
- Remove the existing .repo
files:
$ sudo rm /var/occne/yum.repos.d/*.repo
- Copy the Bastion specific .repo file from the Bastion Host to
the Bastion KVM Host. Execute this command from the Bastion KVM
Host:
scp <bastion_short_name>:/var/occne/yum.repos.d/*.repo /var/occne/yum.repos.d/ Example: scp bastion-2:/var/occne/yum.repos.d/*.repo /var/occne/yum.repos.d/
- From the bastion Host, login to the Bastion Host KVM Host, using
the occne private key and the internal host IP address for that
node (extracted from the hosts.ini
file):
Virtualized CNE Installation
This procedure details the steps necessary to configure and install an OCCNE cluster in an OpenStack Environment.
Prerequisites
- The user must have access to an existing OpenStack Environment including the OpenStack Desktop.
- The OpenStack Environment is configured with appropriate resource flavors and network resources for resource allocation to the VMs created using this procedure.
- Octavia Load Balancing plugin must be installed on the OpenStack Environment.
- Users must have a public SSH key that can be
configured for logging into the Bootstrap Host. This key should be placed
into the customer OpenStack Environment prior to running this procedure
using the following:
Use the Import Key tab on the Launch Instance→Key Pair dialog or via the Compute→Access and Security screen.
Limitation/Expectations
- It is expected that the user is familiar with the
use of OpenStack as a virtualized provider and the OpenStack client.
Note:
All OpenStack client commands listed in this procedure are executed from the Bootstrap Host after it has been instantiated. - All necessary images, binaries, etc, have been downloaded from Oracle OSDC prior to executing this procedure and these resources are available for use in this procedure.
- The customer has made available a central repository for all images, binaries, helm charts, etc, prior to executing this procedure.
- All necessary networking (whether using fixed IPs or floating IPs) has been defined for use on the Openstack provider.
Note:
The OpenStack commands in the procedures below are from a specific version of the OpenStack Desktop. The desktop used at the customer site may be slightly different depending on the version installed. The operations should be compatible.Download the OL7 Image
- Go to https://edelivery.oracle.com and login using appropriate credentials.
- Enter “Oracle Linux Virtual Machine Image for Openstack” in search field and click on Search.
- Select DLP: Oracle Linux 7 Virtual Machine Image for Openstack 7.0.0.0.0 and click on Add to Cart.
- With defaults selected, accept the license agreement. Click on Continue.
- Click on link for: V979992-01.zip Oracle Linux 7.5 Virtual Machine Image for OpenStack, 458.4 MB
- Click on Download.
Note:
- Optionally use the link for 7.6 instead: V981347-01.zip Oracle Linux 7.6 Virtual Machine Image for OpenStack, 487.9 MB
- These zip archives files contains the following respective files, OracleLinux-7.5-x86_64.qcow2 and OracleLinux-7.6-x86_64.qcow2.
Upload an Image to an OpenStack Environment
This is the process of uploading the qcow2 image. The image is provided at OSDC.
Note:
This procedure is executed from the OpenStack Desktop.- Login to the Openstack desktop using the specific credentials.
- Select Compute and then Images from the left navigation pane.
- Click on +Create Image button. The Create Image window is displayed.
- Enter a name for the image. Use the same name of the image which was used to create and download the image (ex: occne_bootstrap-1.5.0).
- Under Image Source, select File. This will enable a File* button.
- Click on Browse to bring up a Windows Explorer dialog.
- From the Windows dialog, select the image that was downloaded from
the Release Artifacts as mentioned above. This will add the image name into the
OpenStack Create Image dialog. Choose the Format from the drop-down menu.
Note:
This should be set to QCOW2 - QEMU Emulator. If not set, use the pulldown menu to select that format. - Select the Create Image button at the bottom right of the dialog. This will start the image upload process. It will take a while so be patient. You will not be able to actually see the image being uploaded even if you log into another OpenStack instance.
- When the process is complete, the image should be listed in the Images screen.
Bootstrap Host Creation
The Bootstrap Host is provisioned to drive the creation of the virtualized cluster using Terraform, the OpenStack Client, and Ansible Playbook(s). A qcow2 image was provided as part of the OSDC download and should be available on the users OpenStack Environment as per the previous section of this document.
Note:
The examples below are for reference only. While the steps are correct the actual values used will be different. The following steps are to be performed manually on the customer specific Openstack Environment Desktop.- Login to the OpenStack Environment using your OpenStack credentials, the appropriate domain and project name.
- Select Compute and then Instances.
- Select the Launch Instance tab on the upper right. A dialog will appear to configure a VM instance.
- Enter an Instance Name (for example: occne-<name>). Leave the Availability Zone and Count set as is.
- Select Source on the left hand side of the dialog. A new dialog appears (Note: there might be a long list of available images to choose from).
- Make sure the Select Boot Source drop-down is set to Image.
- Enter occne-bootstrap in the Available filter. This will display the occne-bootstrap-<x.y.z> image uploaded in the previous sections of this procedure.
- Select the OCCNE Bootstrap Host image by selecting the "↑" on the right side of the image listing. This adds the image as the source for this VM.
- Select Flavor on the left hand side.
- Enter a string (not case sensitive) which best describes the flavor being used for this customer specific OpenStack Environment in the Available search filter. This shortens the list of possible choices.
- Select the appropriate customer specific flavor (for example:
OCCNE-Bootstrap-host) by selecting the "↑" on the right side of the flavor
listings. This adds the resources to the Launch Instance dialog.
Note:
The BSH image requires a flavor that includes a disk size of 40GB or higher. The RAM size should be 8GB or higher although that is not a restriction. - Select Networks on the left hand side.
- Enter the appropriate network name as defined by the customer with the OpenStack Environment (example: ext-net) in the Available search filter. This shortens the list of possible choices.
- Select the appropriate network by selecting the "↑" on the right side of the flavor listings. This adds the external network interface to the Launch Instance dialog.
- Select Key Pair. This dialog assumes you have already uploaded a public key to OpenStack (see Prerequisites above). Select the appropriate key by selecting the "↑" on the right side of the key pair listings. This adds the public key to the authorized_keys file on the Bootstrap Host.
- Select Configuration. This screen allows the user to add
configuration data which is used by cloud-init to set on the VM, the initial
cloud-user and hostname/FQDN additions to the /etc/hosts file. Use the following
configuration. This must be copied into the Customization Script text
box. Make sure the fields marked as <instance_name_from_details_screen>
are updated with the instance name you used in step 5 above. Leave the other
fields on this dialog set to their default
setting.
#cloud-config hostname: <instance_name_from_details_screen> fqdn: <instance_name_from_details_screen> system_info: default_user: name: cloud-user lock_passwd: false write_files: - content: | 127.0.0.1 localhost localhost4 localhost4.localdomain4 <instance_name_from_details_screen> ::1 localhost localhost6 localhost6.localdomain6 <instance_name_from_details_screen> path: /etc/hosts owner: root:root permissions: '0644'
- Select Launch Instance at the lower right side of the initial dialog. This will launch the creation of the VM. This can be observed back at the Instances screen.
Pre-deployment Configuration
Obtain the TLS Certificate for OpenStack
Note:
This step is only required if the customer Openstack environment requires a TLS certificate to access the controller from the cluster nodes and the bootstrap host- Contact the OpenStack admin to provide the required TLS certificate to access the client commands (example, in an Oracle OpenStack system installed with kolla, the certificate will be located at /etc/kolla/certiifcates/haproxy-ca.crt).
- Copy the certificate to the Bootstrap Host at location:
/etc/pki/<OpenStack_release_name>/haproxy-ca.crt (ex.
/etc/pki/kolla/haproxy-ca.crt) (If /etc/pki/<OpenStack_release_name>
does not exist, it can be created using command:
mkdir -p /etc/pki/<OpenStack_release_name>
. - Set the environment variable OS_CACERT to /var/occne/cacertificates/openstack-cacert.pem using the command: export OS_CACERT=/var/occne/cacertificates/openstack-cacert.pem.
Get the Openstack RC (API v3) File
This file exports a number of environment variables on the Bootstrap Host for the given user which directs the OpenStack Client commands towards the particular OpenStack Environment. It must be copied to the users home directory on the Bootstrap Host so that the OpenStack Client commands can be executed.
Note:
These instructions may slightly vary for OpenStack Desktops.- From the OpenStack Desktop: go to Project and then API Access.
- On the right hand side, Download OpenStack RC File pulldown menu and
select Openstack RC File (Identity API v3).
This will download an openrc.sh file prefixed with the OpenStack project name (ex: OCCNE-openrc.sh) to your PC.
- SCP this file (that is winSCP) to the Bootstrap Host in the
/home/admusr directory as
.<project_name>-openrc.sh
Note:
In order for SCP/winSCP to work properly, the key mentioned in the Prerequisites above must be used to access the Bootstrap Host. It may also be necessary to add the appropriate Security Group Rules to support SSH (Rule: SSH, Remote: CIDR CIDR: 0.0.0.0/0) under the Network → Security Groups page in the OpenStack Environment. Contact the OpenStack Administrator to get the proper rules added if necessary. - Execute the following command:
source .<project_name>-openrc.sh
- Execute the following command to verify the OpenStack Client is working:
openstack image list
Create SSH Key on Bootstrap Host
Create the keys that will be used to access the other VMs. This command generates the private and public keys that are passed to the Bastion Host and used to communicate with other node from that Bastion Host. Do not supply a passphrase when it asks for one. Just hit enter. Also the private key should be copied to a place for safe keeping should the Bootstrap Host be destroyed.$ ssh-keygen -m PEM -t rsa -b 2048
Add Files to /tmp Directory
These files must be copied to the directories listed using scp or some other means (ie. winSCP).
- There are three directories on the Bootstrap Host. These three directories
are as follows:
- /tmp/yum.repos.d
- /tmp/db
- /tmp/certificates
- Within these three directories the user must supply the following mandatory
files:
- Add a customer specific central repo .repo file to the /tmp/yum.repos.d directory which allows for access to the customer specific repo (ex: winterfell-mirror.repo).
- Add a mysql .zip file. (ex: V980756-01.zip) to the /tmp/db directory. This file is used for installing the ndb MySQL cluster and is downloaded from OSDC
- Add a docker registry certificate to the /tmp/certificates directory for the central docker registry. This file mus tbe copied using the following format:<central_repo_hostname>:<central_repo_port>.crt(ex: winterfell:5000.crt).
- Make sure the permissions of each file is at least readable
(using 0644) using the
sudo chmod 644 <filename>
command.
Updating cluster.tfvars File
The cluster.tfvars file should exist on the Bootstrap Host after it has been deployed in /var/terraform/occne_example/cluster.tfvars. The user must create a directory called occne-<user>-<name> with a copy of the example tfvars file included in the /var/terraform directory.
The fields in the cluster.tfvars file must be configured to adapt to the current customer Openstack Environment. The steps below detail how to collect and set the fields that must be changed. These instructions may vary depending on whether the user is deploying the cluster using floating IPs or using fixed IP. Differences are noted below when needed.
The cluster.tfvars file includes a parameter named: use_floating_ip. This field defaults to true and should remain as set if the user is configuring their system to use floating IPs allocated from existing DHCP enabled networks defined within their Openstack Provider. If the user wishes to use fixed IPs, this field must be set to false.
Note: All fields in the cluster.tfvars file are unique and should not be duplicated (even using comments - #) as this can cause possible parsing errors.
This procedure applies to both floating IPs or Fixed IPs.
- From the /var/terraform directory, copy directory occne_example and its
contents cluster.tfvars, using the command below to create a new
directory. Change the name of the new directory to include your name or user
name to distinguish it from the occne_example directory.
$ cp -R occne_example occne_<user>
Example:$ cp -R occne-abc-xyz
- Use the following commands to retrieve the information necessary to
configure the cluster.tfvars
- The different flavor settings should be set according to the recommendations from vCNE VM Sizing in this document. An Admin user of the customer specific OpenStack Environment must add the flavors and provide the name of those flavors for configuration into the cluster.tfvars file. The name of each specific flavor that is used must be added as the value field of the key/value fields in the cluster.tfvars file.
- Once flavors have been added to the OpenStack Environment, the
flavor name can be retrieved via the following OpenStack Client
command from the Bootstrap Host shell or the Openstack jumpbox:
$ openstack flavor list | grep <flavor_name> Example: $ openstack flavor list | grep OCCNE | 740c9472-db2c-4015-be5d-04eea1e9f647 | OCCNE-Bastion-Host | 3840 | 100 | 0 | 2 | True | | 337263ef-f69e-43b5-9c6f-ddfc7bb8237e | OCCNE-Bootstrap-host | 8192 | 40 | 0 | 2 | True | | 2de71982-3a27-45bf-9232-27a3e03c685f | OCCNE-DB-Data-Large | 65536 | 60 | 0 | 8 | True | | 27bdc547-a180-45a3-a349-7c574638ba27 | OCCNE-DB-Data-Medium | 32768 | 40 | 0 | 4 | True | | f1f1fce6-fc7a-4e33-9711-ea47d476a843 | OCCNE-DB-Data-Small | 16384 | 20 | 0 | 2 | True | | 4fbe0ac8-4e63-439b-ac6e-c54f40d6e116 | OCCNE-DB-Mgmt-Large | 15360 | 60 | 0 | 4 | True | | 252d96c6-956c-4e99-bae1-c1ff8c09cc8e | OCCNE-DB-Mgmt-Medium | 7680 | 40 | 0 | 2 | True | | 1c42a100-9923-4a82-b56c-579ac5060ff9 | OCCNE-DB-Mgmt-Small | 3840 | 20 | 0 | 1 | True | | 0ac5719e-0aa6-480e-9835-7ea091e53949 | OCCNE-DB-Sql-Large | 32768 | 60 | 0 | 8 | True | | adbe5fa8-09e5-41c1-a6db-91a3650bea95 | OCCNE-DB-Sql-Medium | 16384 | 40 | 0 | 4 | True | | 7c40962d-5992-4a8a-9668-f6d0dc0107c4 | OCCNE-DB-Sql-Small | 8192 | 20 | 0 | 2 | True | | b219a120-3031-4558-8214-00f5184fc4b4 | OCCNE-K8s-Master-Large| 15360 | 80 | 0 | 4 | True | | 0d37d92c-8b06-4052-a1cc-94368d63f898 | OCCNE-K8s-Master-Medium | 7680 | 40 | 0 | 2 | True | | 99883b4d-8532-4c72-acd8-00955a772b9d | OCCNE-K8s-Master-Small | 3840 | 20 | 0 | 1 | True | | facc50fa-e784-4259-aac4-731ce0cbf3ce | OCCNE-K8s-Worker-Large | 32768 | 60 | 0 | 8 | True | | 7307d7bf-bc78-43fd-a5a7-b6e59eeace81 | OCCNE-K8s-Worker-Medium | 16384 | 40 | 0 | 4 | True | | 17d58f3d-3f62-4acd-bb2b-b8b29fbb76a4 | OCCNE-K8s-Worker-Small | 8192 | 20 | 0 | 2 | True | | f407e26d-c677-47b3-b5ed-22951ad77e49 | OCCNE-K8s-Worker-XLarge | 65536 | 80 | 0 |16 | True |
- Use the following command to retrieve the external_net
UUID.
$ openstack network list +--------------------------------------+-------------------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+-------------------+--------------------------------------+ | 1d25d5ea-77ca-4f56-b364-f53b09292e7b | ext-net2 | f5c5ee71-8688-466d-a79f-4306e2bf3f6a | | 668bc488-5307-49ad-9332-24fb0767bb39 | test-network | 9432b2d5-99c0-43ee-8f8c-4709f38b68d9 | | 903155c7-c3ff-4283-bc2b-f34e8b6e76b0 | occne-ebadger-tc1 | ecbadd3e-e239-4830-b8c1-5ff94fa64c3a | | 90c160aa-2ef7-47d3-a212-e1790d56c971 | ext-net-ipv6 | 4c0b844f-1557-4454-b561-88fa31f657f3 | | c4a7569b-5448-4add-8c4e-006bbdd984ef | cluster1 | 4cf62be3-05e9-4a5b-b2a9-6aceee3c860f | | e4351e3e-81e3-4a83-bdc1-dde1296690e3 | ext-net | c0e0c185-ed65-4a53-a7a3-418277fb9a20 | | fc36d63f-b30b-4c7f-979f-9b52b614bbd7 | occne-mkingre | 7631612f-5d22-49be-975c-6e0a9329339b | +--------------------------------------+-------------------+--------------------------------------+
- Once the necessary data has been collected, navigate to the
occne_<user> directory and edit the contents of the
cluster.tfvars file in the newly created directory:
$ vi occne_<user>/cluster.tfvars
- The following miscellaneous fields should remain as set in the example
cluster.tfvars file.
- public_key_path
- image
- ssh_user
- The fields which define the number of each node type have
defaulted values. For a standard deployment the defaults should be used. The
user can update these values if their deployment requires additional nodes.
Note these fields are integer values and do not require double quotes.
- number_of_bastionsnumber_of_k8s_nodes
- number_of_k8s_masters_no_floating_ip
WARNING: The number of master nodes must be set to an odd number. The recommended value for number_of_k8s_masters_no_floating_ip is 3.
- number_of_db_tier_management_nodes
- number_of_db_tier_data_nodes
- number_of_db_tier_sql_nodes
- The corresponding flavors for each node type must be set to a
unique flavor name. Flavors are provided from the Openstack Provider
administrator.
- flavor_bastion
- flavor_k8s_master
- flavor_k8s_node
- flavor_db_tier_management_node
- flavor_db_tier_data_node
- flavor_db_tier_sql_node
- The cluster_name and network_name should be set as
the same value.
Note: This field is used to create each of the nodes contained within the cluster. Kubespray does not allow upper case alphanumeric characters to be used in the node hostname. Do not use any uppercase characters when defining this cluster-name field.
# Kubernetes short cluster namecluster_name = "<cluster-name>" # networking network_name = "<cluster-name>"
- The subnet_cidr defines the tenant side network ip address range. This field should remain set to the default value.
- The field bastion_allowed_remote_ips defines the configuration for the bastion networking security group. This field should remain set to the default value.
- For setting the ntp_server value in the cluster.tfvars file,
use the IP Address of your cloud URL. One way of obtaining this is using the
ping command on your Bootstrap Host. (For example:
ping thundercloud.us.oracle.com
)$ ping <openstack_provider_url> $ ping thundercloud.us.oracle.com PING srv-10-75-171-2.us.oracle.com (10.75.171.2) 56(84) bytes of data. 64 bytes from pc1011601.labs.nc.tekelec.com (10.75.171.2): icmp_seq=1 ttl=63 time=0.283 ms
- If your deployment requires a specific Availability Zone other than
the default availability zone called nova, make the following changes in the
cluster.tfvars file. This value can be added just after the
cluster_name field. If you wish to use nova then do not add these changes.
az_list = ["<your_availability_zone_name>"] Example: # Authorizing_Zone az_list = ["foobar"]
- If your deployment requires additional external
dns_nameservers to be configured, make the following changes in
the cluster.tfvars file. There can be multiple ipv4 addresses in this
list. This value can be added at the end of the file after
wait_for_floatingip. This is an optional value and the default is
empty list []. The sample configuration format is as
below:
dns_nameservers = ["<ipv4_address>",...] Example: dns_nameservers = ["10.75.xxx.xxx",...]
- If your deployment requires the delivery of metadata and user
data through a config drive to each instance, add the following changes in
the cluster.tfvars file. This is an optional value. The default is
occne_config_drive = "false" which indicates using a metadata server
instead of a config drive for OpenStack.
Note:
Make sure the OpenStack administrator did not set force_config_drive=true in the /etc/nova/nova.conf file, otherwise it will use config drive in either case. The sample configuration format is as below:occne_config_drive = "<true/false>" Example: occne_config_drive = "true"
This procedure defines configuration that apply to floating IPs only.
- Use the following command to retrieve the floatingip_pool name. This
would be one of the existing provider
networks.
$ openstack network list +--------------------------------------+---------------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+---------------+--------------------------------------+ | 1d25d5ea-77ca-4f56-b364-f53b09292e7b | ext-net2 | f5c5ee71-8688-466d-a79f-4306e2bf3f6a | | 90c160aa-2ef7-47d3-a212-e1790d56c971 | ext-net-ipv6 | 4c0b844f-1557-4454-b561-88fa31f657f3 | | de033f1f-1e32-4b1f-b69a-dbf1a7a129f5 | ext-net3 | c4c5375c-4a28-4775-8e42-b2b806514e87 | | e4351e3e-81e3-4a83-bdc1-dde1296690e3 | ext-net | c0e0c185-ed65-4a53-a7a3-418277fb9a20 | +--------------------------------------+---------------+--------------------------------------+
- Assign the floatingip_pool field as
follows:
floatip_pool = "<floating_ip_pool_name>" Example: floatingip_pool = ext-net2
- wait_for_floatingip provides the ability for the Terraform
deployment to poll the instance until the floating IP has been associated. It is
set to true by default in the vCNE deployment. There is no need to change this.
Note:
This field is a string and requires double quotes around it.
This procedure defines the configuration that applies to fixed IPs only.
Note: The fields below also includes the openstack command used to retrieve the necessary information. It is possible for a network to be defined that does not include a Name value assigned to it. It is important that the network and subnet include a Name value assigned to it.
Note:
These fields can be left undefined if using the Floating IP configuration.- Determine and configure the network name
(fixedip_network_name) used to provide the IPs configured in step 2
below.
$ openstack network list +--------------------------------------+---------------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+---------------+--------------------------------------+ | 1d25d5ea-77ca-4f56-b364-f53b09292e7b | ext-net2 | f5c5ee71-8688-466d-a79f-4306e2bf3f6a | | 90c160aa-2ef7-47d3-a212-e1790d56c971 | ext-net-ipv6 | 4c0b844f-1557-4454-b561-88fa31f657f3 | | de033f1f-1e32-4b1f-b69a-dbf1a7a129f5 | ext-net3 | c4c5375c-4a28-4775-8e42-b2b806514e87 | | e4351e3e-81e3-4a83-bdc1-dde1296690e3 | ext-net | c0e0c185-ed65-4a53-a7a3-418277fb9a20 | +--------------------------------------+---------------+--------------------------------------+ fixedip_network_name = "<fixedip_network_name>" Example: fixed_ip_network_name = "ext-net3"
- Determine the set of fixed IPs that are to be used for the Bastion
Host, K8s Nodes and DB Tier SQL Nodes.
Note:
Each IP must be in double quotes and separated by commas. The number of IPs MUST match the number of the specific node type.- Use the following command to retrieve the list of networks
defined on the Openstack Provider.
Note:
In this example, network ext-net3 is the fixed IP network selected.$ openstack network list +--------------------------------------+--------------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+--------------+--------------------------------------+ | 1d25d5ea-77ca-4f56-b364-f53b09292e7b | ext-net2 | f5c5ee71-8688-466d-a79f-4306e2bf3f6a | | 90c160aa-2ef7-47d3-a212-e1790d56c971 | ext-net-ipv6 | 4c0b844f-1557-4454-b561-88fa31f657f3 | | de033f1f-1e32-4b1f-b69a-dbf1a7a129f5 | ext-net3 | c4c5375c-4a28-4775-8e42-b2b806514e87 | | e4351e3e-81e3-4a83-bdc1-dde1296690e3 | ext-net | c0e0c185-ed65-4a53-a7a3-418277fb9a20 | +--------------------------------------+--------------+--------------------------------------+
- The user must determine the list of available IPs to use in
the fixed IP lists given below. Contact your administrator for available
IPs.
The following command can be used by a user to retrieve the range of ports that are available in the fixed IP network. Use the subnet name from the fixed IP network to display the range.
To retrieve the number of ips in a subnet, use the following command: $ openstack subnet list --network ext-net3 +--------------------------------------+-----------------+--------------------------------------+----------------+ | ID | Name | Network | Subnet | +--------------------------------------+-----------------+--------------------------------------+----------------+ | c4c5375c-4a28-4775-8e42-b2b806514e87 | ext-net3-subnet | de033f1f-1e32-4b1f-b69a-dbf1a7a129f5 | 10.75.235.0/25 | +--------------------------------------+-----------------+--------------------------------------+----------------+
- Select the port IPs that are to be used for the Bastion Host depending on the number of Bastion Hosts as defined in the cluster.tfvars file by field number_of_bastions.
- Select the port IPs that are to be used for the K8s Nodes depending on the number of Worker Nodes as defined in the cluster.tfvars file by field number_of_k8s_nodes.
- Select the port IPs that are to be used for the DB Tier SQL Nodes depending on the number of DB SQL Nodes as defined in the cluster.tfvars file by field number_of_db_tier_sql_nodes.
- Assign the IPs to the field bastions_fixed_ip_list in the
following format (example assumes number_of_bastions =
1):
bastions_fixed_ip_list = ["<fixed_ip_0>...<fixed_ip_(number_of_bastions-1)>"] Example: bastions_fixed_ip_list = ["10.75.10.20"]
- Assign the IPs to the field k8s_nodes_fixed_ip_list
in the following format (example assumes number_of_k8s_nodes =
4):
k8s_nodes_fixed_ip_list = ["<fixed_ip_0>...<fixed_ip_(number_of_k8s_nodes-1)>"] Example: k8s_nodes_fixed_ip_list = ["10.75.10.21","10.75.10.22","10.75.10.23","10.75.10.24"]
- Assign the IPs to the field
db_tier_sql_nodes_fixed_ip_list in the following format
(example assumes the number_of_db_tier_sql_nodes =
2):
db_tier_sql_nodes_fixed_ip_list = ["<fixed_ip_0>...<fixed_ip_(number_of_db_tier_sql_nodes-1)>"] Example: db_tier_sql_nodes_fixed_ip_list = ["10.75.10.25","10.75.10.26"]
- Use the following command to retrieve the list of networks
defined on the Openstack Provider.
Obtain Mate Site DB Replication Service Load Balancer IP
While installing MYSQL NDB on the second site we need to provide the Mate Site DB Replication Service Load Balancer IP as the configuration parameter for the geo-replication process to start.
Note:
If this is a single deploy and or a mated site with this being the first site deployed, this step can be skipped.- In order to obtain the Mate Site DB Replication Service Load Balancer IP, login to the Bastion Host of first site and execute the following command to retrieve the DB Replication Service Load Balancer IP.
- Fetch DB Replication Service Load Balancer IP of Mate Site MYSQL NDB.
In the above example, "10.75.182.88" is the Mate Site DB Replication Service Load Balancer IP.[cloud-user@bastion ~]$ kubectl get svc --namespace=occne-infra | grep replication occne-db-replication-svc LoadBalancer 10.233.3.117 10.75.182.88 80:32496/TCP 2m8s
Deploy the OCCNE Virtualized Cluster
The execution of the following command does all the work to deploy the VMs in the OpenStack Environment, configure the Bastion Host, and deploy and configure the Kubernetes clusters.
Deploying Using Fixed IPs
If the field use_floating_ip = false (as in using fixed IPs), the deploy.sh script performs a validation of the settings for fixed IP solution in the tfvars file. If the script detects any incorrect configurations within the fixed IP specific fields, it will produce an error explaining why and stop execution before the VMs are instantiated on the Openstack Provider using terraform. If use_floating_ip = true the fixed ip validation is not executed.
The following errors are currently detected by the verification:
- If the use_floating_ip setting is true or false. Anything else and that verification will fail.
- Invalid fixed IP network name (network does not exist on the Openstack Provider).
- Number of nodes of a given type does not match with the number of IP addresses in the IP list for that node type (for example, if number_of_bastion = 1 and the fixed IP list for bastions includes 2 IPs).
- Invalid IP address (for example, 10.75.10.999)
- IP address used is currently in use.
Example (failed case):
...Verifying use_floating_ip setting
...Verifying Fixed IP network info
.....Verifying network name: ext-net3
...Verifying Fixed IP lists
.....Scanning ip list: bastions_fixed_ip_list
.......ip: 10.75.235.4
ERROR: IPv4: 10.75.235.4 from Network: ext-net3 is already in use.
.....Scanning ip list: k8s_nodes_fixed_ip_list
.......ip: 10.75.235.23
ERROR: IPv4: 10.75.235.23 from Network: ext-net3 is already in use.
.......ip: 10.75.235.21
ERROR: IPv4: 10.75.235.21 from Network: ext-net3 is already in use.
.......ip: 10.75.235.19
ERROR: IPv4: 10.75.235.19 from Network: ext-net3 is already in use.
.......ip: 10.75.235.27
ERROR: IPv4: 10.75.235.27 from Network: ext-net3 is already in use.
.....Scanning ip list: db_tier_sql_nodes_fixed_ip_list
...occne1-John-Doe/cluster.tfvars verification completed.
5 errors detected. Please address before continuing.
Example (pass case):
Verifying tfvars file...
...Verifying use_floating_ip setting
...Verifying Fixed IP network info
.....Verifying network name: ext-net3
...Verifying Fixed IP lists
.....Scanning ip list: bastions_fixed_ip_list
.......ip: 10.75.235.23
.....Scanning ip list: k8s_nodes_fixed_ip_list
.......ip: 10.75.235.21
.......ip: 10.75.235.15
.......ip: 10.75.235.14
.......ip: 10.75.235.10
.....Scanning ip list: db_tier_sql_nodes_fixed_ip_list
.......ip: 10.75.235.4
.......ip: 10.75.235.31
...occne1-John-Doe/cluster.tfvars verification successful.
Deploy Command Execution
Environmental Variables describes the list of possible environment variables that can be combined with the deploy.sh command.- For openstack certificate authentication environment only
Set the environment variable ENCODED_CACERT to base64 encoded string of openstack certificate using the command: export ENCODED_CACERT="<base64-encoded-cacert-string>"
Use the link https://www.base64encode.org/ to generate base64 encoded string of openstack certificate.
Run the below sed command to change the openstack certificate path in deploy.sh:
sed -i 's/\/host\/openstack-cacert.pem/${ENCODED_CACERT}/g' deploy.sh
- Execute the following command from the /var/terraform directory on the
Bootstrap Host. This command may take a while to run (can be up to 2 hours
depending on the machines its run on):
First Site: $ OCCNE_TFVARS_DIR=occne_<user> CENTRAL_REPO=<central_repo_hostname> CENTRAL_REPO_IP=<central_repo_ipv4_address> ./deploy.sh Replica Site: $ OCCNE_TFVARS_DIR=occne_<user> CENTRAL_REPO=<central_repo_hostname> CENTRAL_REPO_IP=<central_repo_ipv4_address> OCCNE_DB_REPLICATION_MATE_IP=<db_replication_service_load_balancer_ip> OCCNE_DB_REPLICATION_CLUSTER_ID=<mysqlndb_cluster_identifier> ./deploy.sh
Examples:$ OCCNE_TFVARS_DIR=occne_john_doe CENTRAL_REPO=winterfell CENTRAL_REPO_IP=10.75.216.10 ./deploy.sh $ OCCNE_TFVARS_DIR=occne_john_doe CENTRAL_REPO=winterfell CENTRAL_REPO_IP=10.75.216.10 OCCNE_DB_REPLICATION_MATE_IP=10.75.182.88 OCCNE_DB_REPLICATION_CLUSTER_ID=2 ./deploy.sh
Note:
While Installing on the First Site ignore OCCNE_DB_REPLICATION configuration parameters which Provides Mate Site DB Replication Service Load Balancer IP and ID. Provide the Mate Site DB Replication Service Load Balancer IP and MySQL Cluster identifier while Installing MYSQL NDB on second site.Change MySQL root user password
Refer to Change MySQL root user password.