2 Installing CNE

This chapter provides information about installing Oracle Communications Cloud Native Core, Cloud Native Environment (CNE). CNE can be deployed onto dedicated hardware, referred to as a baremetal CNE, or deployed onto Virtual Machines, referred to as a virtualized CNE.

Regardless of which deployment platform is selected, CNE installation is highly automated. A collection of container-based utilities automate the provisioning, installation, and configuration of CNE. These utilities are based on the following automation tools:

• PXE helps to reliably automate the process of provisioning the hosts with a minimal operating system.
• Terraform creates the virtual resources that are used to host the virtualized CNE.
• Kubespray helps reliably install a base Kubernetes cluster, including all dependencies such as etcd, using the Ansible provisioning tool.
• Ansible is used to deploy and manage a collection of operational tools (Common Services) that are:
  • provided by open source third party products such as Prometheus and Grafana
  • built from source and packaged as part of CNE releases such as Oracle OpenSearch and Oracle OpenSearch Dashboard
• Kyverno Policy management is used to enforce security posture in CNE.
• Helm is used to deploy and configure common services such as Prometheus, Grafana, and OpenSearch.

Note:

• Ensure that the shell is configured with Keepalive to avoid unexpected timeout.
• The 'X' in Oracle Linux X or OLX in the installation procedures indicates the latest version of Oracle Linux supported by CNE.

2.1 Preinstallation Tasks

This section describes the procedure to run before installing an Oracle Communications Cloud Native Environment also known in these installation procedures as CNE.

2.1.1 Sizing Kubernetes Cluster

CNE deploys a Kubernetes cluster to host application workloads and common services. The minimum node counts as per table Table 2-1 are required for production deployments of CNE.

Note:

Deployments that do not meet the minimum sizing requirements may not operate correctly when maintenance operations (including upgrades) are performed on the CNE.

Table 2-1 Kubernetes Cluster Sizing

Node Type	Minimum Required	Maximum Allowed
Kubernetes controller node	3	3
Kubernetes worker node	6	100

2.1.2 Sizing Prometheus Persistent Storage

Prometheus stores metrics in Kubernetes persistent storage. Use the following calculations to reserve the correct amount of persistent storage during installation so that Prometheus can store metrics for the desired retention period.

Calculating metrics storage requirements

Prometheus creates a Kubernetes Persistent Volume Claim (PVC) to reserve persistent storage for metrics. To reserve the correct amount of persistent storage, the size of the metrics stored by Prometheus each day needs to be determined. This value varies based on the deployed NFs. To know each NF value on daily metrics storage requirements for various ingress traffic rates, see the related NF documents. Record the daily metrics storage requirement (at the expected traffic rate) for each NF deployment in the CNE instance. This value, plus the daily CNE metrics storage requirement (given below) provides the expected metrics storage growth per day. To determine the total amount of storage that needs to be reserved for Prometheus, use the formula:

total_metrics_storage = (nf_metrics_daily_growth + occne_metrics_daily_growth) * metrics_retention_period * 1.2

Note:

• An extra 20% storage is reserved to allow for a future increase in metrics growth.
• It is recommended to maintain the metrics_retention_period at the default value of 14 days.
• If the resulting storage size as per the above formula is greater than 500GB, then the retention period must be reduced until the resulting value is less than 500GB.
• Ensure that the retention period is set to more than 3 days.
• The default value for Prometheus persistent storage is 8GB.
• If the total_metrics_storage value calculated as per the above fomula is less than 8GB, then use the default value.

Example:

One NF is installed on the OCCNE instance. From the NFs documentation, we learn that it generates 150 MB of metrics data per day at the expected ingress signaling traffic rate. Using the formula above:

metrics_retention_period = 14 days

nf_metrics_daily_growth = 150 MB/day (from NF documentation)

occne_metrics_daily_growth = 144 MB/day (from calculation below)

(0.15 GB/day + 0.144 GB/day) * 14 days * 1.2 = 5 GB (rounded up)

Since this is less than the default value of 8 GB, use 8 GB as the total_metrics_storage value.

Note:

After determining the required metrics storage, record the total_metrics_storage value for later use in the installation procedures.

Calculating CNE metrics daily storage growth requirements

CNE stores varying amounts of metrics data each day depending on the size of the Kubernetes cluster deployed in the CNE instance. To determine the correct occne_metrics_daily_growth value for the CNE instance, use the formula:

occne_metrics_daily_growth = 36 MB * num_kubernetes_worker_nodes

2.1.3 Sizing Oracle OpenSearch Persistent Storage

Oracle OpenSearch stores logs and traces in Kubernetes persistent storage. Use the following calculations to reserve the correct amount of persistent storage during installation so that Oracle OpenSearch can store logs and traces for the desired retention period.

Calculating log+trace storage requirements

Oracle OpenSearch creates a Kubernetes Persistent Volume Claim (PVC) to reserve persistent storage for logs and traces. To reserve the correct amount of persistent storage, the size of the logs and traces stored by Oracle OpenSearch each day must be determined. This value varies based on the deployed NFs and the ingress signaling traffic rate. To know each NF value on daily log storage requirements for various ingress traffic rates, see the related NF documents. Record the daily logs storage requirement (at the expected traffic rate) for each NF deployment in the CNE instance. This value, plus the daily CNE logs storage requirement (given below) provides the expected log storage growth per day. The amount of trace storage also needs to be calculated (see below). To determine the amount of storage that needs to be reserved for Oracle OpenSearch, use the formula:

log_trace_daily_growth = (nf_logs_daily_growth + nf_trace_daily_growth + occne_logs_daily_growth)

Every day, CNE creates a new set of indices for CNE logs, NF logs, and traces. Data for these active indices are stored on hot data nodes. When the new indices are created, the daily logs from the previous day are made inactive and are moved to warm data nodes. Oracle OpenSearch warm and hot data nodes each use their own PVC to store indices. Use the formulas below to determine the storage requirements for active and inactive index storage.

log_trace_active_storage = log_trace_daily_growth * 2 * 1.2

log_trace_inactive_storage = log_trace_daily_growth * (log_trace_retention_period-1) * 1.2

Note:

• An extra day's worth of storage is allocated on the hot data nodes that gets used when deactivating the old daily indices.
• An extra 20% storage is reserved to allow for a future increase in logging growth.
• It is recommended to maintain the log_trace_retention_period at the default value of 7 days.
• If the resulting storage size as per the above formula is greater than 500GB, then the retention period must be reduced until the resulting value is less than 500GB.
• Ensure that the retention period is set to more than 3 days.
• The default values for Oracle OpenSearch persistent storage for log_trace_active_storage is 10GB and for log_trace_inactive_storage is 30GB
• If the log_trace_active_storage and log_trace_inactive_storage values calculated as per the above fomula is less than 10GB and 30GB, then use the default value.

Example:

One NF will be installed on the OCCNE instance. From the NFs documentation, we learn that it generates 150 MB of logs data per day at the expected ingress signaling traffic rate. Using the formula above:

log_trace_retention_period = 7

nf_mps_rate = 200 msgs/sec

nf_logs_daily_growth = 150 MB/day (from NF documentation)

nf_trace_daily_growth = 500 MB/day (from section below)

occne_logs_daily_growth = 100 MB/day (from section below)

log_trace_daily_growth = (0.15 GB/day + 0.50 GB/day + 0.10 GB/day) = 0.75 GB/day

log_trace_active_storage = 0.75 * 2 * 1.2 = 1.8 GB

log_trace_inactive_storage = 0.75 * 6 * 1.2 = 5.4 GB

Note:

After determining the required logs and trace storage, record the log_trace_active_storage and log_trace_inactive_storage values for later use in the installation procedures.

Calculating NF trace data daily storage growth requirements

NFs store varying amounts of trace data each day, depending on the ingress traffic rate, the trace sampling rate, and the error rate for handling ingress traffic. The default trace sampling rate is .01%. Space is reserved for 10M trace records per NF per day (an amount equivalent to a 1% trace sampling rate) and uses 50 bytes as the average record size (as measured during testing). 1% is used instead of .01% to account for the capture of error scenarios and overhead.

nf_trace_daily_num_records = 10M records/day

nf_trace_avg_record_size = 50 bytes/record

nf_trace_daily_growth = 10M records/day * 50 bytes/record = 500 MB/day

Record the value log_trace_daily_growth for later use in the installation procedures.

Note:

Ensure that trace sampling rate is set to less than 1% under normal circumstances. Collecting a higher percentage of traces causes the Oracle OpenSearch to respond more slowly, and impacts the performance of the CNE instance. If you want to collect a higher percentage of traces, contact Oracle customer service.

Calculating CNE logs daily storage growth requirements

CNE stores varying amounts of logs data each day depending on the size of the Kubernetes cluster deployed in the CNE instance. To determine the correct occne_logs_daily_growth value for the CNE instance, use the formula:

occne_logs_daily_growth = 25 MB * num_kubernetes_worker_nodes

Note:

CNE does not generate any traces, so no additional storage is reserved for CNE trace data.

2.1.4 Generating Root CA Certificate

To use an intermediate Certificate Authority (CA) as an issuer for Istio service mesh mTLS certificates, a signing certificate and key from an external CA must be generated. The generated certificate and key values must be base64 encoded.

Check the Certificate Authority documentation for generating the required certificate and key.

2.2 Baremetal Installation

This section provides information about installing CNE in a dedicated bare metal hardware.

Note:

Before installing CNE in bare metal, you must complete the preinstallation tasks.

2.2.1 Overview

This section provides information about frames, components, and creating CNE instances.

2.2.1.1 Frame and Component

The initial release of the CNE system provides support for on-prem deployment to a very specific target environment consisting of a frame holding switches and servers. This section describes the layout of the frame and the roles performed by the racked equipment.

2.2.1.1.1 Frame

The physical frame is comprised of DL380 or DL360 rack mount servers, and 2 Top of Rack (ToR) Cisco switches. The frame components are added from the bottom up, thus designations found in the next section number from the bottom of the frame to the top of the frame.

Figure 2-1 Frame Overview

2.2.1.1.2 Host Designations

Each physical server has a specific role designation within the CNE solution.

Figure 2-2 Host Designations

2.2.1.1.3 Node Roles

Along with the primary role of each host, you can assign a secondary role. The secondary role can be software related or in the case of the Bootstrap Host, hardware related, as there are unique out-of-band (OOB) connection to the ToR switches.

Figure 2-3 Node Roles

2.2.1.1.4 Transient Roles

RMS1 has unique out-of-band (OOB) connections to the ToR switches, which brings the designation of management host. This role is only relevant during initial switch configuration and fault recovery of the switch. RMS1 also has a transient role as the Installer Bootstrap Host, which is applicable only during initial installation of the frame, and subsequently to get an official install on RMS2. Later, this host is re-paved to its K8s Master role.

Figure 2-4 Transient Roles

2.2.1.2 Creating CNE Instance

This section describes the procedures to create the CNE instance at a customer site. The following diagram shows the installation context:

Figure 2-5 CNE Installation Overview

Following is the basic installation flow to understand the overall effort:

1. Check that the hardware is on-site, properly cabled, and powered up.
2. Pre-assemble the basic equipments needed to perform a successful install:
   1. Identify
      1. Download and stage software and other configuration files using the manifests.
      2. Identify the layer 2 (MAC) and layer 3 (IP) addresses for the equipment in the target frame.
      3. Identify the addresses of key external network services, for example, NTP, DNS, and so on.
      4. Verify or set all of the credentials for the target frame hardware to known settings.
   2. Prepare
      1. Software Repositories: Load the various SW repositories (YUM, Helm, Docker, and so on) using the downloaded software and configuration files.
      2. Configuration Files: Populate the host's inventory file with credentials, layer 2 and layer 3 network information, switch configuration files with assigned IP addresses, and yaml files with appropriate information.
3. Bootstrap the System:
   1. Manually configure a Minimal Bootstrapping Environment (MBE): perform the minimal set of manual operations to enable networking and initial loading of a single Rack Mount Server (RMS1), the transient Installer Bootstrap Host. In this procedure, a minimal set of packages needed to configure switches, iLOs, PXE boot environment, and provision RMS2 as an CNE Storage Host are installed.
   2. Using the newly constructed MBE, automatically create the first Bastion Host on RMS2.
   3. Using the newly constructed Bastion Host on RMS2, automatically deploy and configure the CNE on the other servers in the frame.
4. Final Steps:
   1. Perform postinstallation checks.
   2. Perform recommended security hardening steps.

Cluster Bootstrapping Overview

The following install procedure describes how to install the CNE onto a new hardware that does not contain any networking configurations to switches or provisioned operating systems. Therefore, the initial step in the installation process is to provision RMS1 (see Installing CNE) as a temporary Installer Bootstrap Host. The Bootstrap Host is configured with a minimal set of packages to configure switches, iLOs, and Boot Firmware. From the Bootstrap host, a virtual Bastion Host is provisioned on RMS2. The Bastion Host is then used to provision (and in the case of the Bootstrap Host, re-provision) the remaining CNE hosts, install Kubernetes, and Common Services running within the Kubernetes cluster.

2.2.2 Prerequisites

Before installing and configuring CNE on Bare Metal, ensure that the following prerequisites are met.

Note:

Ensure that Oracle X8-2 or X9-2 Server's Integrated Lights Out Manager (ILOM) firmware is up to date. The ILOM firmware is crucial for seamless functioning of CNE and is essential for optimal performance, security, and compatibility. To update the ILOM firmware, perform the steps outlined in the Oracle documentation or contact the system administrator.

2.2.2.1 Configuring Artifact Acquisition and Hosting

CNE requires artifacts from Oracle Software Delivery Cloud (OSDC), Oracle Support (MOS), the Oracle YUM repository, and certain open-source projects. CNE deployment environments are not expected to have direct internet access. Thus, customer-provided intermediate repositories are necessary for the CNE installation process. These intermediate repositories need CNE dependencies to be loaded into them. This section provides the list of artifacts needed in the repositories.

2.2.2.1.1 Oracle eDelivery Artifact Acquisition

The CNE artifacts are posted on Oracle Software delivery Cloud (OSDC) or MOS.

Table 2-2 CNE Artifacts

Artifact	Description	File Type	Destination repository
occne-images-23.4.x.tgz	CNE Installers (Docker images) from OSDC/MOS	Tar GZ	Docker Registry
Templates	OSDC/MOS Switch config files metallb config file snmp mib files sample hosts.ini file vCNE deploy script configuration files from MOS	Config files (.conf, .ini, .yaml, .mib, .sh, .txt)	Local media

2.2.2.1.2 Third Party Artifacts

CNE dependencies needed from open-source software must be available in repositories that are reachable by the CNE installation tools. For an accounting of third party artifacts needed for this installation, see the Artifact Acquisition and Hosting chapter.

2.2.2.1.3 Populating MetalLB Configuration

Introduction

The metalLB resources file (mb_resources.yaml) defines the BGP peers and address pools for metalLB. The mb_resources.yaml file must be placed in the same directory (/var/occne/<cluster_name>) as the hosts.ini file.

Limitations

The IP addresses listed below can have three possible formats. Each peer address pool can use a different format from the others if desired.

• IP List

  A list of IPs (each on a single line) in single quotes in the following format: 'xxx.xxx.xxx.xxx/32'. The IPs do not have to be sequential. The number of IPs must cover the number of IPs as needed by the application.

• IP Range

  A range of IPs, separated by a dash in the following format: 'xxx.xxx.xxx.xxx - xxx.xxx.xxx.xxx'. The range must cover the number of IPs as needed by the application.

• CIDR (IP-slash notation)

  A single subnet defined in the following format: 'xxx.xxx.xxx.xxx/nn'. The CIDR must cover the number of IPs as needed by the application.

• The peer-address IP must be a different subnet from the IP subnets used to define the IPs for each peer address pool.

There is a limit of 10 peer address pools that can be managed within the mb_resources.yaml file.

Configuring MetalLB Pools and Peers

Following is the procedure to configure MetalLB pools and peers:

1. Add BGP peers and address groups: Referring to the data collected in the Installation Preflight Checklist, add BGP peers (ToRswitchA_Platform_IP, ToRswitchB_Platform_IP) and address groups for each address pool. The Address-pools list the IP addresses that metalLB is allowed to allocate.
2. Edit the mb_resources.yaml file with the site-specific values found in the Installation Preflight Checklist

   Note:

   The oam peer address pool is required for defining the IPs. Other pools are application specific and can be named to best fit the application it applies to. The following examples show how oam and signaling are used to define the IPs, each using a different method.

   Example for oam:

   apiVersion: metallb.io/v1beta2
   kind: BGPPeer
   metadata:
     creationTimestamp: null
     name: peer1
     namespace: occne-infra
   spec:
     holdTime: 1m30s
     keepaliveTime: 0s
     myASN: 64512
     passwordSecret: {}
     peerASN: 64501
     peerAddress: <ToRswitchA_Platform_IP>
   status: {}
   ---
   apiVersion: metallb.io/v1beta2
   kind: BGPPeer
   metadata:
     creationTimestamp: null
     name: peer2
     namespace: occne-infra
   spec:
     holdTime: 1m30s
     keepaliveTime: 0s
     myASN: 64512
     passwordSecret: {}
     peerASN: 64501
     peerAddress: <ToRswitchB_Platform_IP>
   status: {}
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     creationTimestamp: null
     name: oam
     namespace: occne-infra
   spec:
     addresses:
     - '<MetalLB_oam_Subnet_IPs>'
     autoAssign: false
   status: {}
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     creationTimestamp: null
     name: <application_specific_peer_address_pool_name>
     namespace: occne-infra
   spec:
     addresses:
     - '<MetalLB_app_Subnet_IPs>'
     autoAssign: false
   status: {}
   ---
   apiVersion: metallb.io/v1beta1
   kind: BGPAdvertisement
   metadata:
     creationTimestamp: null
     name: bgpadvertisement1
     namespace: occne-infra
   spec:
     ipAddressPools:
     - oam
     - <application_specific_peer_address_pool_name>
   status: {}

   Example for signaling:

   apiVersion: metallb.io/v1beta2
   kind: BGPPeer
   metadata:
     creationTimestamp: null
     name: peer1
     namespace: occne-infra
   spec:
     holdTime: 1m30s
     keepaliveTime: 0s
     myASN: 64512
     passwordSecret: {}
     peerASN: 64501
     peerAddress: 172.16.2.3
   status: {}
   ---
   apiVersion: metallb.io/v1beta2
   kind: BGPPeer
   metadata:
     creationTimestamp: null
     name: peer2
     namespace: occne-infra
   spec:
     holdTime: 1m30s
     keepaliveTime: 0s
     myASN: 64512
     passwordSecret: {}
     peerASN: 64501
     peerAddress: 172.16.2.2
   status: {}
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     creationTimestamp: null
     name: oam
     namespace: occne-infra
   spec:
     addresses:
       - '10.75.200.22/32'
       - '10.75.200.23/32'
       - '10.75.200.24/32'
       - '10.75.200.25/32'
       - '10.75.200.26/32'
      autoAssign: false
   status: {}
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     creationTimestamp: null
     name: signalling
     namespace: occne-infra
   spec:
     addresses:
       - '10.75.200.30 - 10.75.200.40'
     autoAssign: false
   status: {}
   ---
   apiVersion: metallb.io/v1beta1
   kind: BGPAdvertisement
   metadata:
     creationTimestamp: null
     name: bgpadvertisement1
     namespace: occne-infra
   spec:
     ipAddressPools:
     - oam
     - signalling
   status: {}

2.2.3 Predeployment Configuration - Preparing a Minimal Boot Strapping Environment

The steps in this section provide the details to establish a minimal bootstrap environment (to support the automated installation of the CNE environment) on the Installer Bootstrap Host using a Keyboard, Video, Mouse (KVM) connection.

2.2.3.1 Installing Oracle Linux X.x on Bootstrap Host

This procedure defines the steps to install Oracle Linux X.x onto the CNE Installer Bootstrap Host. This host is used to configure the networking throughout the system and install OLX. After OLX installation, the host is repaved as a Kubernetes Master Host in a later procedure.

Prerequisites

1. USB drive of sufficient size to contain the ISO (approximately 5Gb)
2. Oracle Linux X.x iso (For example: Oracle Linux 9.x iso) is available
3. YUM repository file is available
4. Keyboard, Video, Mouse (KVM) are available

Limitations and Expectations

1. The configuration of the Installer Bootstrap Host has to be quick and easy. The Installer Bootstrap Host is re-paved with the appropriate OS configuration for cluster and DB operation at a later installation stage. The Installer Bootstrap Host needs a Linux OS and some basic network to start the installation process.
2. All steps in this procedure are performed using Keyboard, Video, Mouse (KVM).

References

1. Oracle Linux 9 Installation Guide
2. HPE Proliant DL380 Gen10 Server User Guide

Bootstrap Install Procedure

1. Create Bootable USB Media:
   1. On the installer's notebook, download the OLX ISO from the customer's repository.
   2. Push the OLX ISO image onto the USB Flash Drive.

      Since the installer's notebook can be Windows or Linux OS, you must determine the appropriate details to run this task. For a Linux based notebook, insert a USB Flash Drive of the appropriate size into a Laptop (or some other Linux host on which you can copy the ISO), and run the dd command to create a bootable USB drive with the Oracle Linux X ISO (For example: Oracle Linux 9 ISO).

      $ dd -if=<path to ISO> -of=<USB device path> -bs=1m

2. Install Oracle Linux on the Installer Bootstrap Host:

   Note:

   The following procedure considers installing OL9 and provides the options and commands accordingly. The procedure vary for other versions.
   1. Connect a Keyboard, Video, and Mouse (KVM) into the Installer Bootstrap Host's monitor and USB ports.
   2. Plug the USB flash drive containing the bootable ISO into an available USB port on the Bootstrap host (usually in the front panel).
   3. Reboot the host by momentarily pressing the power button on the host's front panel. The button turns yellow. If the button stays yellow, press the button again. The host automatically boots onto the USB flash drive.

      Note:

      If the host was configured previously, and the USB is unavailable in the bootable path as per the boot order, the booting process will be unsuccessful.
   4. If the host is unable to boot onto the USB, repeat step 2c, and interrupt the boot process by pressing F11 button which displays the Boot Menu.

      If the host has been recently booted with an OL, the Boot Menu displays Oracle Linux at the top of the list. Select Generic USB Boot as the first boot device and proceed.

   5. The host attempts to boot from the USB. The Boot Menu is displayed on the screen. Select Install Oracle Linux 9.x.y and click ENTER. This begins the boot process and the system displays the Welcome screen.

      When prompted for the language to use, select the default setting: English (United States) and click Continue in the lower left corner.

      Note:

      You can also select the second option Test this media & install Oracle Linux 9.x.y. This option first runs the media verification process. This process takes some time to complete.
   6. The options on the Welcome screen are displayed with a list of languages on the left and a list of locales on the right. Select the appropriate language from the search box below the list of language.
   7. The system displays the INSTALLATION SUMMARY page. The system expects the following settings on the page. If any of these are not set correctly, then select that menu item and make the appropriate changes.
      1. LANGUAGE SUPPORT: English (United States)
      2. KEYBOARD: English (US)
      3. INSTALLATION SOURCE: Local Media
      4. SOFTWARE SELECTION: Minimal Install
      5. INSTALLATION DESTINATION must display No disks selected. Select INSTALLATION DESTINATION to indicate on which drive to install the OS.
   8. Select the first HDD drive (in this case, it is the first one on the list or the 1.6 TB disk) and click DONE on the upper right corner.

      If the server has already been installed, a red banner at the bottom of the page indicates an error condition. Selecting that banner causes a dialog to appear indicating there is not enough free space (which can also indicate an OS has already been installed). In the dialog it can show both 1.6 TB HDDs, as claimed, or just the one.

      If only one HDD is displayed (or it can be both 1.6 TB drives selected), select the Reclaim space button. Another dialog appears. Select the Delete all button and the Reclaim space button again. Select DONE to return to the INSTALLATION SUMMARY screen.

      If the disk selection dialog appears (after selecting the red banner at the bottom of the page), this implies a full installation of the RMS has already been performed (usually this is because the procedure had to be restarted after it was successfully completed). In this case, select the Modify Disk Selection. This will return to the disk selection page. Select both HDDs and click DONE. The red banner must indicate to reclaim the space. Perform step 2h to reclaim the space again.

   9. Select DONE. This returns to the INSTALLATION SUMMARY screen.
   10. At the INSTALLATION SUMMARY screen, select ROOT PASSWORD.

       Enter a root password appropriate for this installation.

       It is recommended to use the secure password that the customer provides. This helps to minimize the host from being compromised during installation.

   11. At the INSTALLATION SUMMARY screen, select Begin Installation. The INSTRALLATION PROGRESS screen is displayed.
   12. After completing the installation process, remove the USB and select Reboot System to complete the installation and booting to the OS on the Bootstrap Host. At the end of the boot process, the Log in prompt appears.

2.2.3.2 Configuring Host BIOS

Introduction

The following procedure defines the steps to set up the Basic Input Output System (BIOS) changes on the following server types:

• Bootstrap host uses the KVM. If you are using a previously configured Bootstrap host that can be accessed through the remote HTML5 console, follow the procedure according to the remaining servers.
• All the remaining servers use remote HTML5 console.

The steps can vary based on the server type. Follow the appropriate steps specific to configured server. Some of the steps require a system reboot and are indicated in the procedure.

Prerequisites

• For Bootstrap host, the procedure Installing of Oracle Linux X.X on Bootstrap Host is complete.
• For all other servers, the procedure Configure Top of Rack 93180YC-EX Switches and Configuring Addresses for RMS iLOs are complete.

Limitations and Expectations

1. Applies to HP Gen10 iLO 5 and Netra X8-2 server only.
2. Procedures listed here apply to both Bootstrap Host and other servers unless indicated explicitly.

Procedure for Netra X8-2 server

By default, BIOS of the Netra X8-2 server is set to the factory settings with predefined default values. Do not change BIOS of a new X8-2 server. If any issue occurs in the new X8-2 server, reset BIOS to the default factory settings.

To reset the BIOS to factory setting, do the following:

1. Log in to https://<netra ilom address>.
2. Navigate to System Management, select BIOS.
3. Set the value to Factory from the drop-down list for Reset to Defaults under Settings.
4. Click Save.

Exposing the System Configuration Utility on a RMS Host

Steps to Configure the Installer Bootstrap Host BIOS

Perform the following steps to launch the HP iLO 5 System Configuration Utility main page from the KVM. It does not provide instructions to connect the console, as it can differ on each installation.

1. After providing connections for the KVM to access the console, you must reboot the host by momentarily pressing the power button on the front of the Bootstrap host.
2. Navigate to the HP ProLiant DL380 Gen10 System Utilities.

   Once the remote console has been exposed, reset the system to force it through a restart. When the initial window is displayed, keep pressing the F9 key repeatedly. Once the F9 key is highlighted at the lower left corner of the remote console, it eventually brings up the main System Utilities screen.

3. The System Utilities screen is displayed in the remote console.

Launching the System Utility for other RMS servers

Each RMS iLO is assigned an IP address from the installation prerequisites process. Each server can be reached using SSH from the Bootstrap Host login shell on the KVM.

Note:

As CNE 23.4.0 upgraded Oracle Linux to version 9, some of the OS capabilities are removed for security reasons. This includes the removal of older insecure cryptographic policies as well as shorter RSA lengths that are no longer supported. For more information, see Step c.

1. Perform the following steps to launch the system utility for other RMS servers:
   1. SSH to the RMS using the iLO IP address, and the root user and password previously assigned at the Installation PreFlight Checklist. This displays the HP iLO prompt.

      ssh root@<rms_ilo_ip_address>
      Using username "root".
      Last login: Fri Apr 19 12:24:56 2019 from 10.39.204.17
      [root@localhost ~]# ssh root@192.168.20.141
      root@192.168.20.141's password:
      User:root logged-in to ILO2M2909004F.labs.nc.tekelec.com(192.168.20.141 / FE80::BA83:3FF:FE47:649C)
      Integrated Lights-Out 5
      iLO Advanced 2.30 at  Aug 24 2020
      Server Name:
      Server Power: On
      </>hpiLO->

   2. Use Virtual Serial Port (VSP) to connect to the blade remote console:

      </>hpiLO->vsp

   3. Power cycle the blade to bring up the System Utilities for that blade.

      Note:

      The System Utility is a text based version of that exposed on the RMS via the KVM. You must use the directional (arrow) keys to manipulate between selections, ENTER key to select, and ESC to return from the current selection.
   4. Access the System Utility by pressing ESC 9.
2. [Optional]: If you are using OL9, depending on the host that is used to connect the RMS, you may encounter the following error messages when you connect to iLOM. These errors are encountered in OL9 due to the change in security:
   Error in Oracle X8-2 or X9-2 server when RSA key length is too short:

   $ ssh root@172.10.10.10
   Bad server host key: Invalid key length

   Error in HP server when legacy crypto policy is not enabled:

   $ ssh root@172.11.11.10
   Unable to negotiate with 172.11.11.10 port 22: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

   Perform the following steps to resolve these iLOM connectivity issues:

   Note:

   Run the following command on the connecting host only if the host experiences the aforementioned errors while connecting to HP iLO or Oracle iLOM through SSH.
   1. Perform the following steps to reenable Legacy crypto policies to connect to HP iLO using SSH:
      1. Run the following command to enable Legacy crypto policies:

         $ sudo update-crypto-policies --set LEGACY

      2. Run the following command to revert the policies to default:

         $ sudo update-crypto-policies --set DEFAULT

   2. Run the following command to allow short short RSA length while connecting to Oracle X8-2 or X9-2 server using SSH:

      $ ssh -o RSAMinSize=1024 root@172.10.10.10

Changing from UEFI Booting Mode to Legacy BIOS Booting Mode

1. Navigate to the System Utility as per step 1.
2. Select System Configuration.
3. Select BIOS/Platform Configuration (RBSU).
4. Select Boot Options: If the Boot Mode is currently UEFI Mode and you decide to use BIOS Mode, use this procedure to change to Legacy BIOS Mode.

   Note:

   The server reset must go through an attempt to boot before the changes apply.
5. Select the Reboot Required dialog window to drop back into the boot process. The boot must go into the process of actually attempting to boot from the boot order. Attempting to boot fails as disks are not installed at this point. The System Utility can be accessed again.
6. After the reboot and you re-enter the System Utility, the Boot Options page appears.
7. Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit to complete the current boot process.

Changing the System Utility Mode from Legacy BIOS Booting Mode to UEFI Booting Mode

The system supports both Legacy BIOS Booting Mode and UEFI Booting Mode. If you plan to use UEFI Booting mode and the current System Utility is Legacy BIOS Booting Mode, then use the following procedure to switch the booting mode to UEFI Booting Mode:

1. Navigate to the System Utility as per step 1.
2. Select System Configuration.
3. Select BIOS/Platform Configuration (RBSU).
4. Select Boot Options. Click drop-down the Warning prompt appears, click OK.

   Note:

   The server reset must go through an attempt to boot before the changes apply.
5. Select the Reboot Required dialog window. Click OK for the warning reboot window.
6. After the reboot and you re-enter the System Utility, the Boot Options page appears.

   The Boot Mode is changed to UEFI Mode and the UEFI Optimized Boot has changed to Enabled automatically.

7. Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit to complete the current boot process.

Adding a New User Account to the Server iLO 5 Interface

Note:

Ensure the pxe_install_lights_out_usr and pxe_install_lights_out_passwd fields match as provided in the hosts inventory files created using the template: CNE Inventory File Preparation.

1. Navigate to the System Utility as per step 1.
2. Select System Configuration.
3. Select iLO 5 Configuration Utility.
4. Select User Management → Add User.
5. Select the appropriate permissions. For the root user set all permissions to YES. Enter root as New User Name and Login Name fields, and select Password field, press Enter key to type <password> twice.
6. Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit to complete the current boot process.

Forcing PXE to Boot from the First Embedded FlexibleLOM HPE Ethernet 10Gb 2-port Adapter

During host PXE, the DHCP DISCOVER requests from the hosts must be broadcast over the 10Gb port. This step provides details to configure the broadcast to use the 10Gb ports before using the 1Gb ports. Moving the 10Gb port up in the search order speeds up the response from the host servicing the DHCP DISCOVER. Enclosure blades have 2 10GE NICs which default to being configured for PXE booting. The RMS is re-configured to use the PCI NICs using this step.

1. Navigate to the System Utility as per step 1.
2. Select System Configuration.
3. Select BIOS/Platform Configuration (RBSU).
4. Select Boot Options for Legacy Mode:

   This menu defines the boot mode that must be set to Legacy BIOS Mode, the UEFI Optimized Boot which must be disabled, and the Boot Order Policy which must be set to Retry Boot Order Indefinitely (this means it will keep trying to boot without ever going to disk). In this screen select Legacy BIOS Boot Order. If not in Legacy BIOS Mode, follow procedure Change over from UEFI Booting Mode to Legacy BIOS Booting Mode to set the Configuration Utility to Legacy BIOS Mode.

5. Select Boot Option for UEFI Mode, click UEFI Boot Settings. Click UEFI Boot Order.
6. Select Legacy BIOS Boot Order.

   This window defines the legacy BIOS boot order. It includes the list of devices from which the server will listen for the DHCP OFFER, including the reserved IPv4 after the PXE DHCP DISCOVER message is broadcast out from the server.

   The 10Gb Embedded FlexibleLOM 1 Port 1 is at the bottom of the list in the default view. When the server begins the scan for the response, it scans down this list until it receives the response. Each NIC will take a finite amount of time before the server gives up on that NIC and attempts another in the list. Moving the 10Gb port up on this list decreases the time required to process the DHCP OFFER.

   To move an entry, select that entry, hold down the left mouse button and move the entry up in the list under the entry it must reside. The UEFI Mode has a similar list.

7. Move the 10 Gb Embedded FlexibleLOM 1 Port 1 entry up above the 1Gb Embedded LOM 1 Port 1 entry.
8. Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit to complete the current boot process.

Enabling Virtualization on a Given BareMetal Server

You can configure virtualization using the default settings or via the Workload Profiles.

1. Verifying Default Settings
   1. Navigate to the System Configuration Utility as per step 1.
   2. Select System Configuration.
   3. Select BIOS/Platform Configuration (RBSU).
   4. Select Virtualization Options.

      This screen displays the settings for the Intel(R) Virtualization Technology (IntelVT), Intel(R) VT-d, and SR-IOV options (Enabled or Disabled). The default value for each option is Enabled.

   5. Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit to complete the current boot process.

Disabling RAID Configurations

1. Navigate to the System Configuration Utility as per step 1.
2. Select System Configuration.
3. Select Embedded RAID 1 : HPE Smart Array P408i-a SR Gen10.
4. Select Array Configuration.
5. Select Manage Arrays.
6. Select Array A (or any designated Array Configuration if there are more than one).
7. Select Delete Array.
8. Select Submit Changes.
9. Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit to complete the current boot process.

Enabling Primary Boot Device

This section provides details to configure the primary bootable device for a given Gen10 Server. In this case, the RMS includes two devices as Hard Drives (HDDs). Some configurations can also have two Solid State Drives (SSDs). Do not select SSDs for this configuration. Only the primary bootable device is set in this procedure since RAID is disabled. The secondary bootable device remains as Not Set.

1. Navigate to the System Configuration Utility as per step 1.
2. Select System Configuration.
3. Select Embedded RAID 1 : HPE Smart Array P408i-a SR Gen10.
4. Select Set Bootable Device(s) for Legacy Boot Mode. If the boot devices are not set then it will display Not Set for the primary and secondary devices.
5. Select Select Bootable Physical Drive.
6. Select Port 1| Box:3 Bay:1 Size:1.8 TB SAS HP EG00100JWJNR.

   Note:

   This example includes two HDDs and two SSDs. The actual configuration can be different.
7. Select Set as Primary Bootable Device.
8. Select Back to Main Menu.

   This returns to the HPE Smart Array P408i-a SR Gen10 menu. The secondary bootable device is left as Not Set.

9. Select F10: Save to save and stay in the utility or select the F12: Save and Exit to save and exit to complete the current boot process.

Configuring the iLO 5 Static IP Address

Note:

This step requires a reboot after completion.

1. Navigate to the System Configuration Utility as per step 1.
2. Select System Configuration.
3. Select iLO 5 Configuration Utility.
4. Select Network Options.
5. Enter the IP Address, Subnet Mask, and Gateway IP Address fields provided in Installation PreFlight Checklist.
6. Select F12: Save and Exit to complete the current boot process. A reboot is required when setting the static IP for the iLO 5. A warning appears indicating that you must wait 30 seconds for the iLO to reset. A prompt requesting a reboot appears. Select Reboot.
7. Once the reboot is complete, you can re-enter the System Utility and verify the settings if necessary.

2.2.3.3 Configuring Top of Rack 93180YC-EX Switches

Introduction

The following procedure provides the steps to initialize and configure Cisco 93180YC-EX switches as per the topology design.

Note:

All instructions in this procedure are run from the Bootstrap Host.

Prerequisites

Ensure that the following tasks are performed before configuration:

1. Procedure Installation of Oracle Linux X.X on Bootstrap Host is complete.
2. The switches are in a factory default state. If the switches are out of the box or preconfigured, run write erase and reload to factory default.
3. The switches are connected as per the Installation PreFlight Checklist. The customer uplinks are not active before outside traffic is necessary.
4. DHCP, XINETD, and TFTP are already installed on the Bootstrap host but not configured.
5. Available Utility USB contains all the necessary files according to the Installation PreFlight checklist: Create Utility USB.

Limitations/Expectations

All steps are run from a Keyboard, Video, Mouse (KVM) connection.

References

• https://github.com/datacenter/nexus9000/blob/master/nx-os/poap/poap.py
• https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli/CLIConfigurationGuide/Licensing.html
• https://computingforgeeks.com/how-to-setup-a-tftp-server-on-centos-rhel-8/

Configuration Procedure

Following is the procedure to configure Top of Rack 93180YC-EX Switches:

1. Log in to the Bootstrap host as the root user.

   Note:

   All instructions in this step are run from the Bootstrap Host.
2. Insert and mount the Utility USB that contains the configuration and script files. Verify the files are listed in the USB using the ls /media/usb command.

   Note:

   Instructions for mounting the USB can be found in: Installation of Oracle Linux X.X on Bootstrap Host. Perform only steps 2 and 3 in that procedure.
3. Create a bridge interface as follows:
   1. If it is a HP Gen10 RMS server, create bridge interface to connect both management ports and setup the management bridge to support switch initialization.

      Note:

      <CNE_Management_IP_With_Prefix> is from Installation PreFlight Checklist: Complete Site Survey Host IP Table. Row 1 CNE Management IP Addresess (VLAN 4) column.

      <ToRSwitch_CNEManagementNet_VIP> is from Installation PreFlight Checklist: Complete OA and Switch IP Table.

      nmcli con add con-name mgmtBridge type bridge ifname mgmtBridge
      nmcli con add type bridge-slave ifname eno2 master mgmtBridge
      nmcli con add type bridge-slave ifname eno3 master mgmtBridge
      nmcli con mod mgmtBridge ipv4.method manual ipv4.addresses 192.168.2.11/24
      nmcli con up mgmtBridge
       
       
      nmcli con add type bond con-name bond0 ifname bond0 mode 802.3ad
      nmcli con add type bond-slave con-name bond0-slave-1 ifname  eno5 master bond0
      nmcli con add type bond-slave con-name bond0-slave-2 ifname  eno6 master bond0

      The following commands are related to the VLAN and IP address for this bootstrap server, the <mgmt_vlan_id> is the same as in hosts.ini and the <bootstrap bond0 address> is same as ansible_host IP for this bootstrap server:

      nmcli con mod bond0 ipv4.method manual ipv4.addresses <bootstrap bond0 address>
      nmcli con add con-name bond0.<mgmt_vlan_id> type vlan id <mgmt_vlan_id> dev bond0
      nmcli con mod bond0.<mgmt_vlan_id> ipv4.method manual ipv4.addresses <CNE_Management_IP_Address_With_Prefix> ipv4.gateway <ToRswitch_CNEManagementNet_VIP>
      nmcli con up bond0.<mgmt_vlan_id>

      Example:

      nmcli con mod bond0 ipv4.method manual ipv4.addresses 172.16.3.4/24
      nmcli con add con-name bond0.4 type vlan id 4 dev bond0
      nmcli con mod bond0.4 ipv4.method manual ipv4.addresses <CNE_Management_IP_Address_With_Prefix> ipv4.gateway <ToRswitch_CNEManagementNet_VIP>
      nmcli con up bond0.4

   2. For Netra X8-2 server, due to limitation of Ethernet ports, only three out of five ports can be enabled. So there are not enough ports to configure mgmtBridge and bond0 at the same time. Connect NIC1 or NIC2 to ToR switch mgmt ports in this step to configure ToR switches. After that, reconnect the ports to Ethernet ports on the ToR switches.

      nmcli con add con-name mgmtBridge type bridge ifname mgmtBridge
      nmcli con add type bridge-slave ifname eno2np0 master mgmtBridge
      nmcli con add type bridge-slave ifname eno3np1 master mgmtBridge
      nmcli con mod mgmtBridge ipv4.method manual ipv4.addresses 192.168.2.11/24
      nmcli con up mgmtBridge

4. Copy the customer Oracle Linux repo file (for example: ol9.repo) from USB to /etc/yum.repo.d directory. Move the origin to backup file (For exmaple: oracle-linux-ol9.repo, uek-ol9.repo, and virt-ol9.repo):

   # cd /etc/yum.repos.d
   # mv oracle-linux-ol9.repo oracle-linux-ol9.repo.bkp
   # mv virt-ol9.repo virt-ol9.repo.bkp
   # mv uek-ol9.repo uek-ol9.repo.bkp
   # cp /media/usb/winterfell-ol9.repo ./

5. Install and set up tftp server on bootstrap host:

   dnf install -y tftp-server tftp
   cp /usr/lib/systemd/system/tftp.service /etc/systemd/system/tftp-server.service
   cp /usr/lib/systemd/system/tftp.socket /etc/systemd/system/tftp-server.socket
    
   tee /etc/systemd/system/tftp-server.service<<'EOF'
   [Unit]
   Description=Tftp Server
   Requires=tftp-server.socket
   Documentation=man:in.tftpd
    
   [Service]
   ExecStart=/usr/sbin/in.tftpd -c -p -s /var/lib/tftpboot
   StandardInput=socket
    
   [Install]
   WantedBy=multi-user.target
   Also=tftp-server.socket
   EOF

6. Enable tftp on the Bootstrap host and verify if tftp is active and enabled:

   systemctl daemon-reload
   systemctl enable --now tftp-server
   systemctl status tftp-server
   ps -elf | grep tftp

   Verify tftp is active and enabled:

   $ systemctl status tftp
   $ ps -elf | grep tftp
   

7. Install and setup dhcp server on bootstrap host:

   dnf -y install dhcp-server

8. Copy the dhcpd.conf file from the Utility USB in Installation PreFlight checklist : Create the dhcpd.conf File to the /etc/dhcp/ directory.

   $ cp /media/usb/dhcpd.conf /etc/dhcp/

9. Restart and enable dhcpd service.

   #  systemctl enable --now dhcpd

   Use the following command to verify the active and enabled state:

   # systemctl status dhcpd

10. Copy the switch configuration and script files from the Utility USB to /var/lib/tftpboot/ directory as follows:

    cp /media/usb/93180_switchA.cfg /var/lib/tftpboot/
    cp /media/usb/93180_switchB.cfg /var/lib/tftpboot/
    cp /media/usb/poap_nexus_script.py /var/lib/tftpboot/

11. To modify the POAP script file change username and password credentials used to log in to the Bootstrap host.

    
    # vi /var/lib/tftpboot/poap_nexus_script.py
    # Host name and user credentials
    options = {
       "username": "<username>",
       "password": "<password>",
       "hostname": "192.168.2.11",
       "transfer_protocol": "scp",
       "mode": "serial_number",
       "target_system_image": "nxos.9.2.3.bin",
    }
    

    Note:

    The version nxos.9.2.3.bin is used by default. If different version is to be used, modify target_system_image with new version.
12. Run the md5Poap.sh script from the Utility USB created from Installation PreFlight checklist: Create the md5Poap Bash Script, to modify the POAP script file md5sum as follows:

    # cd /var/lib/tftpboot/
    # /bin/bash md5Poap.sh
    

13. Create the files necessary to configure the ToR switches using the serial number from the switch.

    
    
    

    Note:

    The serial number is located on a pullout card on the back of the switch in the left most power supply of the switch. Be careful in interpreting the exact letters. If the switches are preconfigured, then you can even verify the serial numbers using show license host-id command.
14. Copy the /var/lib/tftpboot/93180_switchA.cfg into a file called /var/lib/tftpboot/conf.<switchA serial number>.

    Modify the switch specific values in the /var/lib/tftpboot/conf.<switchA serial number> file, including all the values in the curly braces as shown in the following code block.

    These values are contained at Installation PreFlight checklist : ToR and Enclosure Switches Variables Table (Switch Specific) and Installation PreFlight Checklist : Complete OA and Switch IP Table. Modify these values with the following sed commands or use an editor such as vi to modify the commands.

    Note:

    The template supports 12 RMS servers. If there are less than 12 servers, then the extra configurations may not work without physical connections and will not affect the first number of servers. If there are more than 12 servers, simulate the pattern to add for more servers.

    # sed -i 's/{switchname}/<switch_name>/' conf.<switchA serial number>
    # sed -i 's/{admin_password}/<admin_password>/' conf.<switchA serial number>
    # sed -i 's/{user_name}/<user_name>/' conf.<switchA serial number>
    # sed -i 's/{user_password}/<user_password>/' conf.<switchA serial number>
    # sed -i 's/{ospf_md5_key}/<ospf_md5_key>/' conf.<switchA serial number>
    # sed -i 's/{OSPF_AREA_ID}/<ospf_area_id>/' conf.<switchA serial number>
     
    # sed -i 's/{NTPSERVER1}/<NTP_server_1>/' conf.<switchA serial number>
    # sed -i 's/{NTPSERVER2}/<NTP_server_2>/' conf.<switchA serial number>
    # sed -i 's/{NTPSERVER3}/<NTP_server_3>/' conf.<switchA serial number>
    # sed -i 's/{NTPSERVER4}/<NTP_server_4>/' conf.<switchA serial number>
    # sed -i 's/{NTPSERVER5}/<NTP_server_5>/' conf.<switchA serial number>
     
    # Note: If less than 5 ntp servers available, delete the extra ntp server lines such as command:
    # sed -i 's/{NTPSERVER5}/d' conf.<switchA serial number>
     
     Note: different delimiter is used in next two commands due to '/' sign in the variables
    # sed -i 's#{ALLOW_5G_XSI_LIST_WITH_PREFIX_LEN}#<MetalLB_Signal_Subnet_With_Prefix>#g' conf.<switchA serial number>
    # sed -i 's#{CNE_Management_SwA_Address}#<ToRswitchA_CNEManagementNet_IP>#g' conf.<switchA serial number>
    # sed -i 's#{CNE_Management_SwB_Address}#<ToRswitchB_CNEManagementNet_IP>#g' conf.<switchA serial number>
    # sed -i 's#{CNE_Management_Prefix}#<CNEManagementNet_Prefix>#g' conf.<switchA serial number>
    # sed -i 's#{SQL_replication_SwA_Address}#<ToRswitchA_SQLreplicationNet_IP>#g' conf.<switchA serial number>
    # sed -i 's#{SQL_replication_SwB_Address}#<ToRswitchB_SQLreplicationNet_IP>#g' conf.<switchA serial number>
    # sed -i 's#{SQL_replication_Prefix}#<SQLreplicationNet_Prefix>#g' conf.<switchA serial number>
    # ipcalc -n  <ToRswitchA_SQLreplicationNet_IP/<SQLreplicationNet_Prefix> | awk -F'=' '{print $2}'
    # sed -i 's/{SQL_replication_Subnet}/<output from ipcalc command as SQL_replication_Subnet>/' conf.<switchA serial number>
     
    # sed -i 's/{CNE_Management_VIP}/<ToRswitch_CNEManagementNet_VIP>/g' conf.<switchA serial number>
    # sed -i 's/{SQL_replication_VIP}/<ToRswitch_SQLreplicationNet_VIP>/g' conf.<switchA serial number>
    # sed -i 's/{OAM_UPLINK_CUSTOMER_ADDRESS}/<ToRswitchA_oam_uplink_customer_IP>/' conf.<switchA serial number>
     
    # sed -i 's/{OAM_UPLINK_SwA_ADDRESS}/<ToRswitchA_oam_uplink_IP>/g' conf.<switchA serial number>
    # sed -i 's/{SIGNAL_UPLINK_SwA_ADDRESS}/<ToRswitchA_signaling_uplink_IP>/g' conf.<switchA serial number>
    # sed -i 's/{OAM_UPLINK_SwB_ADDRESS}/<ToRswitchB_oam_uplink_IP>/g' conf.<switchA serial number>
    # sed -i 's/{SIGNAL_UPLINK_SwB_ADDRESS}/<ToRswitchB_signaling_uplink_IP>/g' conf.<switchA serial number>
    # ipcalc -n  <ToRswitchA_signaling_uplink_IP>/30 | awk -F'=' '{print $2}' 
    # sed -i 's/{SIGNAL_UPLINK_SUBNET}/<output from ipcalc command as signal_uplink_subnet>/' conf.<switchA serial number>
     
    # ipcalc -n  <ToRswitchA_SQLreplicationNet_IP> | awk -F'=' '{print $2}'
    # sed -i 's/{MySQL_Replication_SUBNET}/<output from the above ipcalc command appended with prefix >/' conf.<switchA serial number>
     
    Note: The version nxos.9.2.3.bin is used by default and hard-coded in the conf files. If different version is to be used, run the following command: 
    # sed -i 's/nxos.9.2.3.bin/<nxos_version>/' conf.<switchA serial number>
     
    Note: access-list Restrict_Access_ToR
    # The following line allow one access server to access the switch management and SQL vlan addresses while other accesses are denied. If no need, delete this line. If need more servers, add similar line. 
    # sed -i 's/{Allow_Access_Server}/<Allow_Access_Server>/' conf.<switchA serial number>

15. Copy the /var/lib/tftpboot/93180_switchB.cfg into a /var/lib/tftpboot/conf.<switchB serial number> file:

    Modify the switch specific values in the /var/lib/tftpboot/conf.<switchA serial number> file, including: hostname, username/password, oam_uplink IP address, signaling_uplink IP address, access-list ALLOW_5G_XSI_LIST permit address, prefix-list ALLOW_5G_XSI.

    These values are contained at Installation PreFlight checklist : ToR and Enclosure Switches Variables Table and Installation PreFlight Checklist : Complete OA and Switch IP Table.

    # sed -i 's/{switchname}/<switch_name>/' conf.<switchB serial number>
    # sed -i 's/{admin_password}/<admin_password>/' conf.<switchB serial number>
    # sed -i 's/{user_name}/<user_name>/' conf.<switchB serial number>
    # sed -i 's/{user_password}/<user_password>/' conf.<switchB serial number>
    # sed -i 's/{ospf_md5_key}/<ospf_md5_key>/' conf.<switchB serial number>
    # sed -i 's/{OSPF_AREA_ID}/<ospf_area_id>/' conf.<switchB serial number>
     
    # sed -i 's/{NTPSERVER1}/<NTP_server_1>/' conf.<switchB serial number>
    # sed -i 's/{NTPSERVER2}/<NTP_server_2>/' conf.<switchB serial number>
    # sed -i 's/{NTPSERVER3}/<NTP_server_3>/' conf.<switchB serial number>
    # sed -i 's/{NTPSERVER4}/<NTP_server_4>/' conf.<switchB serial number>
    # sed -i 's/{NTPSERVER5}/<NTP_server_5>/' conf.<switchB serial number>
     
    # Note: If less than 5 ntp servers available, delete the extra ntp server lines such as command:
    # sed -i 's/{NTPSERVER5}/d' conf.<switchB serial number>
     
    Note: different delimiter is used in next two commands due to '/' sign in in the variables
    # sed -i 's#{ALLOW_5G_XSI_LIST_WITH_PREFIX_LEN}#<MetalLB_Signal_Subnet_With_Prefix>#g' conf.<switchB serial number>
    # sed -i 's#{CNE_Management_SwA_Address}#<ToRswitchA_CNEManagementNet_IP>#g' conf.<switchB serial number>
    # sed -i 's#{CNE_Management_SwB_Address}#<ToRswitchB_CNEManagementNet_IP>#g' conf.<switchB serial number>
    # sed -i 's#{CNE_Management_Prefix}#<CNEManagementNet_Prefix>#g' conf.<switchB serial number>
    # sed -i 's#{SQL_replication_SwA_Address}#<ToRswitchA_SQLreplicationNet_IP>#g' conf.<switchB serial number>
    # sed -i 's#{SQL_replication_SwB_Address}#<ToRswitchB_SQLreplicationNet_IP>#g' conf.<switchB serial number>
    # sed -i 's#{SQL_replication_Prefix}#<SQLreplicationNet_Prefix>#g' conf.<switchB serial number>
    # ipcalc -n  <ToRswitchB_SQLreplicationNet_IP/<SQLreplicationNet_Prefix> | awk -F'=' '{print $2}'
    # sed -i 's/{SQL_replication_Subnet}/<output from ipcalc command as SQL_replication_Subnet>/' conf.<switchB serial number>
     
    # sed -i 's/{CNE_Management_VIP}/<ToRswitch_CNEManagementNet_VIP>/' conf.<switchB serial number>
    # sed -i 's/{SQL_replication_VIP}/<ToRswitch_SQLreplicationNet_VIP>/' conf.<switchB serial number>
    # sed -i 's/{OAM_UPLINK_CUSTOMER_ADDRESS}/<ToRswitchB_oam_uplink_customer_IP>/' conf.<switchB serial number>
     
    # sed -i 's/{OAM_UPLINK_SwA_ADDRESS}/<ToRswitchA_oam_uplink_IP>/g' conf.<switchB serial number>
    # sed -i 's/{SIGNAL_UPLINK_SwA_ADDRESS}/<ToRswitchA_signaling_uplink_IP>/g' conf.<switchB serial number>
    # sed -i 's/{OAM_UPLINK_SwB_ADDRESS}/<ToRswitchB_oam_uplink_IP>/g' conf.<switchB serial number>
    # sed -i 's/{SIGNAL_UPLINK_SwB_ADDRESS}/<ToRswitchB_signaling_uplink_IP>/g' conf.<switchB serial number>
    # ipcalc -n  <ToRswitchB_signaling_uplink_IP>/30 | awk -F'=' '{print $2}'
    # sed -i 's/{SIGNAL_UPLINK_SUBNET}/<output from ipcalc command as signal_uplink_subnet>/' conf.<switchB serial number>
     
    Note: The version nxos.9.2.3.bin is used by default and hard-coded in the conf files. If different version is to be used, run the following command: 
    # sed -i 's/nxos.9.2.3.bin/<nxos_version>/' conf.<switchB serial number>
     
    Note: access-list Restrict_Access_ToR
    # The following line allow one access server to access the switch management and SQL vlan addresses while other accesses are denied. If no need, delete this line. If need more servers, add similar line. 
    # sed -i 's/{Allow_Access_Server}/<Allow_Access_Server>/' conf.<switchB serial number>

16. Generate the md5 checksum for each conf file in /var/lib/tftpboot and copy that into a new file called conf.<switchA/B serial number>.md5.

    
    $ md5sum conf.<switchA serial number> > conf.<switchA serial number>.md5
    $ md5sum conf.<switchB serial number> > conf.<switchB serial number>.md5
    

17. Verify that the /var/lib/tftpboot directory has the correct files.
    Ensure that the file permissions are set as follows:

    Note:

    The ToR switches are constantly attempting to find and run the poap_nexus_script.py script which uses tftp to load and install the configuration files.

    
    # ls -l /var/lib/tftpboot/
    total 1305096
    -rw-r--r--. 1 root root       7161 Mar 25 15:31 conf.<switchA serial number>
    -rw-r--r--. 1 root root         51 Mar 25 15:31 conf.<switchA serial number>.md5
    -rw-r--r--. 1 root root       7161 Mar 25 15:31 conf.<switchB serial number>
    -rw-r--r--. 1 root root         51 Mar 25 15:31 conf.<switchB serial number>.md5
    -rwxr-xr-x. 1 root root      75856 Mar 25 15:32 poap_nexus_script.py
    

18. Enable tftp-server again and verify the tftp-server status.

    Note:

    The tftp-server status stays as active for only 15 minutes after enable. So you must enable the tftp-server again before configuring the switches.

    # systemctl enable --now tftp-server
    

    Verify tftp is active and enabled:

    # systemctl status tftp-server
    # ps -elf | grep tftp

19. Disable firewalId and verify the firewalId service status:

    systemctl stop firewalld
    systemctl disable firewalld
    systemctl status firewalld

    After completing the above steps, the ToR Switches will attempt to boot from the tftpboot files automatically.

20. Unmount the Utility USB and remove it as follows: umount /media/usb

Verification

Following is the procedure to verify Top of Rack 93180YC-EX Switches:

1. After the configuration of ToR switches, ping the switches from bootstrap server. The switches mgmt0 interfaces are configured with the IP addresses that are in the conf files.

   Note:

   Wait for the device to respond.

   # ping 192.168.2.1
   PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
   64 bytes from 192.168.2.1: icmp_seq=1 ttl=255 time=0.419 ms
   64 bytes from 192.168.2.1: icmp_seq=2 ttl=255 time=0.496 ms
   64 bytes from 192.168.2.1: icmp_seq=3 ttl=255 time=0.573 ms
   64 bytes from 192.168.2.1: icmp_seq=4 ttl=255 time=0.535 ms
   ^C
   --- 192.168.2.1 ping statistics ---
   4 packets transmitted, 4 received, 0% packet loss, time 3000ms
   rtt min/avg/max/mdev = 0.419/0.505/0.573/0.063 ms
   # ping 192.168.2.2
   PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
   64 bytes from 192.168.2.2: icmp_seq=1 ttl=255 time=0.572 ms
   64 bytes from 192.168.2.2: icmp_seq=2 ttl=255 time=0.582 ms
   64 bytes from 192.168.2.2: icmp_seq=3 ttl=255 time=0.466 ms
   64 bytes from 192.168.2.2: icmp_seq=4 ttl=255 time=0.554 ms
   ^C
   --- 192.168.2.2 ping statistics ---
   4 packets transmitted, 4 received, 0% packet loss, time 3001ms
   rtt min/avg/max/mdev = 0.466/0.543/0.582/0.051 ms
   

2. Attempt to ssh to the switches with the username and password provided in the conf files.

   # ssh plat@192.168.2.1
   The authenticity of host '192.168.2.1 (192.168.2.1)' can't be established.
   RSA key fingerprint is SHA256:jEPSMHRNg9vejiLcEvw5qprjgt+4ua9jucUBhktH520.
   RSA key fingerprint is MD5:02:66:3a:c6:81:65:20:2c:6e:cb:08:35:06:c6:72:ac.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added '192.168.2.1' (RSA) to the list of known hosts.
   User Access Verification
   Password:
    
   Cisco Nexus Operating System (NX-OS) Software
   TAC support: http://www.cisco.com/tac
   Copyright (C) 2002-2019, Cisco and/or its affiliates.
   All rights reserved.
   The copyrights to certain works contained in this software are
   owned by other third parties and used and distributed under their own
   licenses, such as open source.  This software is provided "as is," and unless
   otherwise stated, there is no warranty, express or implied, including but not
   limited to warranties of merchantability and fitness for a particular purpose.
   Certain components of this software are licensed under
   the GNU General Public License (GPL) version 2.0 or
   GNU General Public License (GPL) version 3.0  or the GNU
   Lesser General Public License (LGPL) Version 2.1 or
   Lesser General Public License (LGPL) Version 2.0.
   A copy of each such license is available at
   http://www.opensource.org/licenses/gpl-2.0.php and
   http://opensource.org/licenses/gpl-3.0.html and
   http://www.opensource.org/licenses/lgpl-2.1.php and
   http://www.gnu.org/licenses/old-licenses/library.txt.
   #

3. Verify that the running-config contains all expected configurations in the conf file using the show running-config command as follows:

   # show running-config
   !Command: show running-config
   !Running configuration last done at: Mon Apr  8 17:39:38 2019
   !Time: Mon Apr  8 18:30:17 2019
   version 9.2(3) Bios:version 07.64
   hostname 12006-93108A
   vdc 12006-93108A id 1
     limit-resource vlan minimum 16 maximum 4094
     limit-resource vrf minimum 2 maximum 4096
     limit-resource port-channel minimum 0 maximum 511
     limit-resource u4route-mem minimum 248 maximum 248
     limit-resource u6route-mem minimum 96 maximum 96
     limit-resource m4route-mem minimum 58 maximum 58
     limit-resource m6route-mem minimum 8 maximum 8
   feature scp-server
   feature sftp-server
   cfs eth distribute
   feature ospf
   feature bgp
   feature interface-vlan
   feature lacp
   feature vpc
   feature bfd
   feature vrrpv3
   ....
   ....
   

4. In case some of the above features are missing, verify license on the switches and at least NXOS_ADVANTAGE level license is in use. If the license is not installed or too low level, contact the vendor for correct license key file. Then run write erase and reload to set back to factory default. The switches will go to POAP configuration again.

   # show license

   Example output:

   
   # show license
   MDS20190215085542979.lic:
   SERVER this_host ANY
   VENDOR cisco
   INCREMENT NXOS_ADVANTAGE_XF cisco 1.0 permanent uncounted \
           VENDOR_STRING=<LIC_SOURCE>MDS_SWIFT</LIC_SOURCE><SKU>NXOS-AD-XF</SKU> \
           HOSTID=VDH=FDO22412J2F \
           NOTICE="<LicFileID>20190215085542979</LicFileID><LicLineID>1</LicLineID> \
           <PAK></PAK>" SIGN=8CC8807E6918

   # show license usage

   Example output:

   
   # show license usage
   Feature                      Ins  Lic   Status Expiry Date Comments
                                    Count
   --------------------------------------------------------------------------------
   ...
   NXOS_ADVANTAGE_M4             No    -   Unused             -
   NXOS_ADVANTAGE_XF             Yes   -   In use never       -
   NXOS_ESSENTIALS_GF            No    -   Unused             -
   ...
   #

5. For Netra X8-2 server, reconnect the cable on mgmt ports to the Ethernet ports for RMS1, delete mgmtBridge, and configure bond0 and management vlan interface:

   nmcli con delete con-name mgmtBridge
     
   nmcli con add type bond con-name bond0 ifname bond0 mode 802.3ad
   nmcli con add type bond-slave con-name bond0-slave-1 ifname  eno2np0 master bond0
   nmcli con add type bond-slave con-name bond0-slave-2 ifname  eno3np1 master bond0

   The following commands are related to the VLAN and IP address for this bootstrap server, the <mgmt_vlan_id> is the same as in hosts.ini and the <bootstrap bond0 address> is same as ansible_host IP for this bootstrap server:

   nmcli con mod bond0 ipv4.method manual ipv4.addresses <bootstrap bond0 address>
   nmcli con add con-name bond0.<mgmt_vlan_id> type vlan id <mgmt_vlan_id> dev bond0
   nmcli con mod bond0.<mgmt_vlan_id> ipv4.method manual ipv4.addresses <CNE_Management_IP_Address_With_Prefix> ipv4.gateway <ToRswitch_CNEManagementNet_VIP>
   nmcli con up bond0.<mgmt_vlan_id>

   For example:

   nmcli con mod bond0 ipv4.method manual ipv4.addresses 172.16.3.4/24
   nmcli con add con-name bond0.4 type vlan id 4 dev bond0
   nmcli con mod bond0.4 ipv4.method manual ipv4.addresses <CNE_Management_IP_Address_With_Prefix> ipv4.gateway <ToRswitch_CNEManagementNet_VIP>
   nmcli con up bond0.4

6. Verify if the RMS1 can ping the CNE_Management VIP.

   # ping <ToRSwitch_CNEManagementNet_VIP>
   PING <ToRSwitch_CNEManagementNet_VIP> (<ToRSwitch_CNEManagementNet_VIP>) 56(84) bytes of data.
   64 bytes from <ToRSwitch_CNEManagementNet_VIP>: icmp_seq=2 ttl=255 time=1.15 ms
   64 bytes from <ToRSwitch_CNEManagementNet_VIP>: icmp_seq=3 ttl=255 time=1.11 ms
   64 bytes from <ToRSwitch_CNEManagementNet_VIP>: icmp_seq=4 ttl=255 time=1.23 ms
   ^C
   --- 10.75.207.129 ping statistics ---
   4 packets transmitted, 3 received, 25% packet loss, time 3019ms
   rtt min/avg/max/mdev = 1.115/1.168/1.237/0.051 ms

7. Connect or enable customer uplink.
8. Verify if the RMS1 can be accessed from laptop. Use application such as Putty to ssh to RMS1.

   
   $ ssh root@<CNE_Management_IP_Address>
   Using username "root".
   root@<CNE_Management_IP_Address>'s password:<root password>
   Last login: Mon May  6 10:02:01 2019 from 10.75.9.171
   [root@RMS1 ~]#
   

SNMP Trap Configuration

The following procedure details the steps to configure SNMP Trap:

1. SNMPv2c Configuration.

   When SNMPv2c configuration is needed, ssh to the two switches and run the following commands:

   These values <SNMP_Trap_Receiver_Address> and <SNMP_Community_String> are from Installation Preflight Checklist.

   [root@RMS1 ~]# ssh <user_name>@<ToRswitchA_CNEManagementNet_IP>
   # configure terminal
   (config)# snmp-server host <SNMP_Trap_Receiver_Address> traps version 2c <SNMP_Community_String>
   (config)# snmp-server host <SNMP_Trap_Receiver_Address> use-vrf default
   (config)# snmp-server host <SNMP_Trap_Receiver_Address> source-interface Ethernet1/51
   (config)# snmp-server enable traps
   (config)# snmp-server community <SNMP_Community_String> group network-admin

2. To restrict the direct access to ToR switches, create IP access list and apply on the uplink interfaces. Use the following commands on ToR switches:

   [root@RMS1 ~]# ssh <user_name>@<ToRswitchA_CNEManagementNet_IP>
   # configure terminal
   (config)#
   ip access-list Restrict_Access_ToR
     permit ip {Allow_Access_Server}/32 any
     permit ip {NTPSERVER1}/32 {OAM_UPLINK_SwA_ADDRESS}/32
     permit ip {NTPSERVER2}/32 {OAM_UPLINK_SwA_ADDRESS}/32
     permit ip {NTPSERVER3}/32 {OAM_UPLINK_SwA_ADDRESS}/32
     permit ip {NTPSERVER4}/32 {OAM_UPLINK_SwA_ADDRESS}/32
     permit ip {NTPSERVER5}/32 {OAM_UPLINK_SwA_ADDRESS}/32
     deny ip any {CNE_Management_VIP}/32
     deny ip any {CNE_Management_SwA_Address}/32
     deny ip any {CNE_Management_SwB_Address}/32
     deny ip any {SQL_replication_VIP}/32
     deny ip any {SQL_replication_SwA_Address}/32
     deny ip any {SQL_replication_SwB_Address}/32
     deny ip any {OAM_UPLINK_SwA_ADDRESS}/32
     deny ip any {OAM_UPLINK_SwB_ADDRESS}/32
     deny ip any {SIGNAL_UPLINK_SwA_ADDRESS}/32
     deny ip any {SIGNAL_UPLINK_SwB_ADDRESS}/32
     permit ip any any
    
   interface Ethernet1/51
     ip access-group Restrict_Access_ToR in
    
   interface Ethernet1/52
     ip access-group Restrict_Access_ToR in

3. Traffic egress out of cluster (including snmptrap traffic to SNMP trap receiver) and traffic goes to signal server:

   [root@RMS1 ~]# ssh <user_name>@<ToRswitchA_CNEManagementNet_IP>
   # configure terminal
   (config)#
   feature nat
   ip access-list host-snmptrap
    10 permit udp 172.16.3.0/24 <snmp trap receiver>/32 eq snmptrap log
    
   ip access-list host-sigserver
    10 permit ip 172.16.3.0/24 <signal server>/32
    
   ip nat pool sig-pool 10.75.207.211 10.75.207.222 prefix-length 27
   ip nat inside source list host-sigserver pool sig-pool overload add-route
   ip nat inside source list host-snmptrap interface Ethernet1/51 overload
    
   interface Vlan3
    ip nat inside
    
   interface Ethernet1/51
    ip nat outside
    
   interface Ethernet1/52
    ip nat outside
    
    
   Run the same commands on ToR switchB

2.2.3.4 Configuring Addresses for RMS iLOs

Introduction

This section provides the procedure to configure RMS iLO addresses and add a new user account for each RMS other than the Bootstrap Host. When the RMSs are shipped and out of the box after hardware installation and powerup, the RMSs are in a factory default state with the iLO in Dynamic Host Configuration Protocol (DHCP) mode waiting for DHCP service. DHCP is used to configure the ToR switches, OAs, Enclosure switches, and blade server iLOs. It can also be used to configure the RMS iLOs.

Note:

Skip this procedure if the iLO network is controlled by lab network or customer network that is beyond the ToR switches. The iLO network can be accessed from the bastion host management interface. Perform this procedure only if the iLO network is local on the ToR switches and iLO addresses are not configured on the servers.

Prerequisites

Ensure that the procedure Configure Top of Rack 93180YC-EX Switches has been completed.

Limitations

All steps must be run from the SSH session of the Bootstrap server.

References

HPE iLO 5 User Guide

Procedure

Following is the procedure to configure addresses for RMS iLOs:

Setting up interface on bootstrap server and find iLO DHCP address

1. Setup the VLAN interface to access ILO subnet. The ilo_vlan_id and ilo_subnet_cidr are the same value as in hosts.ini:

   nmcli con add con-name bond0.<ilo_vlan_id> type vlan id <ilo_vlan_id> dev bond0
   nmcli con mod bond0.<ilo_vlan_id> ipv4.method manual ipv4.addresses <unique ip in ilo subnet>/<ilo_subnet_cidr>
   nmcli con up bond0.<ilo_vlan_id>

   Example:

   nmcli con add con-name bond0.2 type vlan id 2 dev bond0
   nmcli con mod bond0.2 ipv4.method manual ipv4.addresses 192.168.20.11/24
   nmcli con up bond0.2

2. Subnet and conf file address.

   The /etc/dhcp/dhcpd.conf file is already configured as per the OCCNE Configure Top of Rack 93180YC-EX Switches procedure and DHCP started or enabled on the bootstrap server. The second subnet 192.168.20.0 is used to assign addresses for OA and RMS iLOs.

3. Display the DHCPD leases file at /var/lib/dhcpd/dhcpd.leases. The DHCPD lease file displays the DHCP addresses for all RMS iLOs:

   # cat /var/lib/dhcpd/dhcpd.leases
   # The format of this file is documented in the dhcpd.leases(5) manual page.
   # This lease file was written by isc-dhcp-4.2.5
   ...
   lease 192.168.20.106 {
     starts 5 2019/03/29 18:10:04;
     ends 5 2019/03/29 21:10:04;
     cltt 5 2019/03/29 18:10:04;
     binding state active;
     next binding state free;
     rewind binding state free;
     hardware ethernet b8:83:03:47:5f:14;
     uid "\000\270\203\003G_\024\000\000\000";
     client-hostname "ILO2M2909004B";
   }
   lease 192.168.20.104 {
     starts 5 2019/03/29 18:10:35;
     ends 5 2019/03/29 21:10:35;
     cltt 5 2019/03/29 18:10:35;
     binding state active;
     next binding state free;
     rewind binding state free;
     hardware ethernet b8:83:03:47:64:9c;
     uid "\000\270\203\003Gd\234\000\000\000";
     client-hostname "ILO2M2909004F";
   }
   lease 192.168.20.105 {
     starts 5 2019/03/29 18:10:40;
     ends 5 2019/03/29 21:10:40;
     cltt 5 2019/03/29 18:10:40;
     binding state active;
     next binding state free;
     rewind binding state free;
     hardware ethernet b8:83:03:47:5e:54;
     uid "\000\270\203\003G^T\000\000\000";
     client-hostname "ILO2M29090048";

HP iLO address setup and user account setup

1. Access RMS iLO from the DHCP address with default Administrator password. From the above dhcpd.leases file. Find the IP address for the iLO name, the default username is Administrator, the password is on the label that can be pulled out from front of server.

   Note:

   The DNS Name is on the pull-out label. Use the DNS Name on the pull-out label to match the physical machine with the iLO IP. The same default DNS Name from the pull-out label is displayed upon logging in to the iLO command line interface, as shown in the following example:

   # ssh Administrator@192.168.20.104
   Administrator@192.168.20.104's password:
   User:Administrator logged-in to ILO2M2909004F.labs.nc.tekelec.com(192.168.20.104 / FE80::BA83:3FF:FE47:649C)
   iLO Standard 1.37 at  Oct 25 2018
   Server Name:
   Server Power: On

2. Create an RMS iLO new user with customized username and password.

   </>hpiLO-> create /map1/accounts1 username=root password=TklcRoot group=admin,config,oemHP_rc,oemHP_power,oemHP_vm
   
   status=0
   status_tag=COMMAND COMPLETED
   Tue Apr  2 20:08:30 2019
   User added successfully.

3. Disable the DHCP before you are able to setup the static IP. The setup of static IP failed before DHCP is disabled.

   </>hpiLO-> set /map1/dhcpendpt1 EnabledState=NO
   status=0
   status_tag=COMMAND COMPLETED
   Tue Apr  2 20:04:53 2019
   Network settings change applied.
   Settings change applied, iLO 5 will now be reset.
   Logged Out: It may take several minutes before you can log back in.
   CLI session stopped
   packet_write_wait: Connection to 192.168.20.104 port 22: Broken pipe

4. Setup RMS iLO static IP address.

   After the previous step, log in back with the same address (which is static IP now), and enter new username and password. Go to next step to change the IP address, if required.

   # ssh <new username>@192.168.20.104
   <new username>@192.168.20.104's password: <new password>
   User: logged-in to ILO2M2909004F.labs.nc.tekelec.com(192.168.20.104 / FE80::BA83:3FF:FE47:649C)
   iLO Standard 1.37 at  Oct 25 2018
   Server Name:
   Server Power: On
    
   </>hpiLO-> set /map1/enetport1/lanendpt1/ipendpt1 IPv4Address=192.168.20.122 SubnetMask=255.255.255.0
    
   status=0
   status_tag=COMMAND COMPLETED
   Tue Apr  2 20:22:23 2019
    
   Network settings change applied.
   Settings change applied, iLO 5 will now be reset.
   Logged Out: It may take several minutes before you can log
   back in.
     
   CLI session stopped
    
   packet_write_wait: Connection to 192.168.20.104 port 22:
   Broken pipe

5. Setup RMS iLO default gateway.

   # ssh <new username>@192.168.20.122
   <new username>@192.168.20.122's password: <new password>
   User: logged-in to ILO2M2909004F.labs.nc.tekelec.com(192.168.20.104 / FE80::BA83:3FF:FE47:649C)
   iLO Standard 1.37 at  Oct 25 2018
   Server Name:
   Server Power: On
   
   </>hpiLO-> set /map1/gateway1 AccessInfo=192.168.20.1
   
   status=0
   status_tag=COMMAND COMPLETED
   Fri Oct  8 16:10:27 2021
   
   Network settings change applied.
   
   
   Settings change applied, iLO will now be reset.
   Logged Out: It may take several minutes before you can log back in.
   
   CLI session stopped
   Received disconnect from 192.168.20.122 port 22:11:  Client Disconnect
   Disconnected from 192.168.20.122 port 22

Netra X8-2 iLO address setup and user account setup

1. Access RMS iLO from the DHCP address with default root password. From the above dhcpd.leases file, find the IP address for the iLO name. The default username is root and the password is changeme. At the same time, note the DNS Name on the pull-out label.

   Note:

   The DNS Name is on the pull-out label. Use the DNS Name on the pull-out label to match the physical machine with the iLO IP. The same default DNS Name from the pull-out label is displayed upon logging in to the iLO command line interface, as shown in the following example:

   Using username "root".
   Using keyboard-interactive authentication.
   Password:
    
   Oracle(R) Integrated Lights Out Manager
    
   Version 5.0.1.28 r140682
    
   Copyright (c) 2021, Oracle and/or its affiliates. All rights reserved.
    
   Warning: password is set to factory default.
    
   Warning: HTTPS certificate is set to factory default.
    
   Hostname: ORACLESP-2117XLB00V

2. Netra server has the default root user. To change the default password, run the following set command:

   -> set /SP/users/root password
   Enter new password: ********
   Enter new password again: ********

3. Create an RMS iLO new user with customized username and password:

   -> create /SP/users/<username>
   Creating user...
   Enter new password: ****
   
   create: Non compliant password. Password length must be between 8 and 16 characters.
   Enter new password: ********
   Enter new password again: ********
   Created /SP/users/<username>

4. Setup RMS iLO static IP address.

   After the previous step, log in with the same address (which is a static IP now), and the new username and password. If not using the same address, go to next step to change the IP address:

   1. Check the current state before change:

      # show /SP/network

   2. Set command to configure:

      # set /SP/network state=enabled|disabled ipdiscovery=static|dhcp
      ipaddress=value ipgateway=value ipnetmask=value

      Example command:

      # set /SP/network state=enabled ipdiscovery=static
      ipaddress=172.16.9.13 ipgateway=172.16.9.1 ipnetmask=255.255.255.0

5. Commit the changes to implement the updates performed:

   # set /SP/network commitpending=true

2.2.3.5 Generating SSH Key on Oracle Servers with Oracle ILOM

This section provides the procedures to generate a new SSH key on an Oracle X8-2 and Oracle X9-2 server using the Oracle Integrated Lights Out Manager (ILOM) web interface. The new SSH key is created at the service level and has a length of 3072 bits. It is automatically managed by the firmware.

Firmware Update for Oracle X8-2 Server:

The following firmwares are updated in Oracle X8-2 server. This update brings enhancements to the management and operation of the system. For more details, contact My Oracle Support.

• Oracle ILOM: Version 5.1.3.20, revision 153596
• BIOS (Basic Input/Output System): Version 51.11.02.00

Firmware Update for Oracle X9-2 Server:

Oracle X9-2 server is compatible with firmware 5.1. For more details, contact My Oracle Support.

Prerequisites

Before craeting a new SSH key, ensure that you have the necessary access permissions to log in to the Oracle ILOM web interface.

Generating SSH Key on Oracle X8-2 Server Using iLOM

1. Open a web browser and access the Oracle ILOM user interface by entering the corresponding IP address in the address bar.
2. Enter your login credentials for Oracle ILOM.
3. Perform the following steps to generate the SSH key:
   1. Navigate to the SSH or security configuration section in the following path: ILOM Administration → Managament Access → SSH Server
   2. Click Generate Key to generate a new SSH key.

      The system generates a new SSH key of 3072 bits length.

   3. Run the following command on the CLI to validate the generated key:

      -> show -d properties /SP/services/ssh/keys/rsa

      Sample output:

      /SP/services/ssh/keys/rsa
          Properties:
              fingerprint = 53:66:65:85:45:ba:4e:63:2d:aa:ab:8b:ef:fa:95:ac:9e:17:8e:92
              fingerprint_algorithm = SHA1
              length = 3072

      Note:

      • The length of the SSH key is managed by the firmware and set to 3072 bits. There are no options to configure it to 1024 or 2048 bits.
      • Ensure that the client's configuration is compatible with 3072 bit SSH keys.
   4. After making the changes to SSH keys or user configuration in the ILOM web interface, log out from Oracle ILOM and then log back in. This applies the changes without having to restart the entire ILOM.
   5. [Optional]: You can also restart ILOM using the ILOM command line interface by running the following command. This command applies any configuration changes that you've made and initiates a restart of the ILOM:

      -> reset /SP

Generating SSH Key on Oracle X9-2 Server Using iLOM

Use the following properties and commands in the Oracle ILOM CLI to configure and manage SSH settings on the X9-2 server. Refer to the specific documentation for Oracle ILOM version 5.1 on the X9-2 server for any updates or changes to the commands (https://docs.oracle.com/en/servers/management/ilom/5.1/admin-guide/modifying-default-management-access-configuration-properties.html#GUID-073D4AA6-E5CC-45B5-9CF4-28D60B56B548).

The following list provides details about the configurable target and user role requirements:

• CLI path: /SP/services/ssh
• Web path: ILOM Administration > Management Access > SSH Server > SSH Server Settings
• User Role: admin(a). Required for all property modifications.

Table 2-3 SSH Configuration Properties

Property	Description
State	Parameter: state= Description: Determines whether the SSH server is enabled or disabled. When enabled, the SSH server uses the server side keys to allow remote clients to securely connect to the Oracle ILOM SP using a command-line interface. On disabling or restarting, the SSH server automatically terminates all connected SP CLI sessions over SSH. Default Value: Enabled CLI Syntax: -> set /SP/services/ssh state=enabled|disabled Note: If you are using a web interface, the changes you made to the SSH Server State in the web interface takes effect in Oracle ILOM only after clicking Save. Restarting the SSH server is not required for this property.
Restart Button	Parameter: restart_sshd_action= Description: This property allows you to restart the SSH server by terminating all connected SP CLI sessions and activating the newly generated server-side keys. Default Value: NA Available Options: True, False CLI Syntax: -> set /SP/services/ssh restart_sshd_action=true
Generate RSA Key Button	Parameter: generate_new_key_type=rsa generate_new_key_action=true Description: This property provides the ability to generate a new RSA SSH key. This action is used for creating a new key pair for SSH authentication. Default Value: NA CLI Syntax: -> set /SP/services/ssh generate_new_key_type=rsa generate_new_key_action=true

Note:

• Periodic firmware updates for Oracle ILOM are crucial. Regularly check for updates to access the new features, improvements, or security enhancements in the Firmware Downloads and Release History for Oracle Systems page.
• Verify that the clients connecting to the Oracle X8-2 server support 3072 bit SSH keys.
• For detailed information about SSH key generation and management in your specific environment, refer to the official Oracle ILOM documentation.

2.2.4 Installing Bastion Host

This section describes the use of Installer Bootstrap Host to provision RMS2 with an operating system and configure it to fulfill the role of Kubernetes master. After the Bastion Host is created, it is used to complete the installation of CNE.

Provision Second Database Host (RMS2) from Installer Bootstrap Host (RMS1)

Table 2-4 Terminology used in Procedure

Name	Description
bastion_full_name	This is the full name of the Bastion Host as defined in the hosts.ini file. Example: bastion-2.rainbow.lab.us.oracle.com
bastion_kvm_host_full_name	This is the full name of the KVM server (usually RMS2/db-2) that hosts the Bastion Host VM. Example: db-2.rainbow.lab.us.oracle.com
bastion_kvm_host_ip_address	This is the IPv4 ansible_host IP address of the server (usually RMS2/db-2) that hosts the Bastion Host VM. Example: 172.16.3.5
bastion_short_name	This is the name of the Bastion Host derived from the bastion_full_name up to the first ".". Example: bastion-2
bastion_external_ip_address	This is the external address for the Bastion Host Example: 10.75.148.5 for bastion-2
bastion_ip_address	This is the internal IPv4 "ansible_host" address of the Bastion Host as defined within the hosts.ini file. Example: 172.16.3.100 for bastion-2
cluster_full_name	This is the name of the cluster as defined in the hosts.ini file field: occne_cluster_name. Example: rainbow.us.lab.us.oracle.com
cluster_short_name	This is the short name of the cluster derived from the cluster_full_name up to the first ".". Note: Following are the specifications for cluster_short_name value: only lowercase letters and numbers are allowed. underscore and spaces are not allowed. Example: rainbow

Note:

• Setup the Bootstrap Host to use root/<customer_specific_root_password> as the credentials to access it. For the procedure to configure the user and password, see Installation of Oracle Linux X.X on Bootstrap Host.
• The commands and examples in this procedure assume that Bastion Host is installed on Oracle Linux 9. The procedure vary for other versions.

Procedure

Following is the procedure to install Bastion Host:

1. Run the following commands to create a admur user and edit the sudoers file.

   groupadd admusr
   useradd -g admusr admusr
   passwd admusr
   <Enter new password twice>    
    
   echo "%admusr  ALL=(ALL)       ALL" | tee -a  /etc/sudoers
   echo "%admusr ALL=(ALL) NOPASSWD: ALL" | tee -a /etc/sudoers

   Log in as admur with the newly created password and perform the following steps in this procedure as a admur.
2. Set the cluster short-name for use in the bootstrap environment, and load it into the current environment:

   echo 'export OCCNE_CLUSTER=<cluster_short_name>' | sudo tee -a /etc/profile.d/occne.sh
   echo 'export CENTRAL_REPO=<central_repo_name>' | sudo tee -a /etc/profile.d/occne.sh
   echo 'export CENTRAL_REPO_IP=<central_repo_ip>' | sudo tee -a /etc/profile.d/occne.sh
   echo 'export CENTRAL_REPO_REGISTRY_PORT=<central_repo_registry_port>' | sudo tee -a /etc/profile.d/occne.sh
   sudo chmod 755 /etc/profile.d/occne.sh
   . /etc/profile.d/occne.sh

   Example:

   echo 'export OCCNE_CLUSTER=rainbow' | sudo tee -a /etc/profile.d/occne.sh
    echo 'export CENTRAL_REPO=winterfell' | sudo tee -a /etc/profile.d/occne.sh
    echo 'export CENTRAL_REPO_IP=128.128.128.128' | sudo tee -a /etc/profile.d/occne.sh
    echo 'export CENTRAL_REPO_REGISTRY_PORT=5000' | sudo tee -a /etc/profile.d/occne.sh
    sudo chmod 755 /etc/profile.d/occne.sh
    . /etc/profile.d/occne.sh

   Note:

   After running this step, the bash variable reference ${OCCNE_CLUSTER} expands to the short-cluster name in this shell and subsequent ones.
3. Copy the necessary files from the Utility USB to support the OS Install:

   This procedure provides the steps to copy all supporting files from the Utility USB to the appropriate directories so that the Provision Container successfully installs OLX onto RMS2.

   1. Log in to the Bootstrap Host using the root credentials configured during OS installation of the Bootstrap Host.
   2. Mount the Utility USB.

      Note:

      For instructions on mounting a USB in Linux, see Installation of Oracle Linux X.X on Bootstrap Host.
   3. Copy the hosts.ini file (created using procedure: CNE Inventory File Preparation) into the /var/occne/cluster/${OCCNE_CLUSTER}/ directory.

      $ cp /<path_to_usb>/hosts.ini /var/occne/cluster/${OCCNE_CLUSTER}/hosts.ini

      Example:

      $ cp /media/usb/hosts.ini /var/occne/cluster/${OCCNE_CLUSTER}/hosts.ini

   4. Verify the occne_repo_host_address in hosts.ini is the bond0 IP address configured in Configuring Top of Rack 93180YC-EX Switches. If not, configure the file:

      occne_repo_host_address = <bootstrap server bond0 address> 

   5. Follow procedure Populate the MetalLB Configuration File to copy the MetalLB configuration file into the /var/occne/cluster/${OCCNE_CLUSTER}/ directory and configure it.

      $ cp /<path_to_usb>/mb_resources.yaml /var/occne/cluster/${OCCNE_CLUSTER}/

      Example:

      $ cp /media/usb/mb_resources.yaml /var/occne/cluster/${OCCNE_CLUSTER}/

   6. Copy the customer specific .repo file from the Utility USB to the Installer Bootstrap Host.

      This is the .repo file created by the customer that provides access to the onsite (within their network) YUM or DNF repositories needed to complete the full deployment of CNE onto the Installer Bootstrap Host.

      $ mkdir -p /var/occne/yum.repos.d
      $ cp /<path_to_usb>/<customer_specific_repo>.repo /var/occne/yum.repos.d/
      $ echo "reposdir=/var/occne/yum.repos.d" >> /etc/yum.conf

      Example:

      $ mkdir -p /var/occne/yum.repos.d
      $ cp /media/usb/ol9-mirror.repo /var/occne/yum.repos.d/
      $ echo "reposdir=/var/occne/yum.repos.d" >> /etc/yum.conf

4. Set up the /etc/hosts file for the central repo and verify access:
   1. Add an entry to the /etc/hosts file on the Installer Bootstrap Host to provide a name mapping for the Customer Central Repository.

      $ vi /etc/hosts

      Example:

      10.75.200.217 rainbow-reg

      To verify, run the following command:

      $ ping <central_repo_name>

      Example:

      # ping rainbow-reg
      PING reg-1 (10.75.200.217) 56(84) bytes of data.
      64 bytes from reg-1 (10.75.200.217): icmp_seq=1 ttl=61 time=0.248 ms
      64 bytes from reg-1 (10.75.200.217): icmp_seq=2 ttl=61 time=0.221 ms
      64 bytes from reg-1 (10.75.200.217): icmp_seq=3 ttl=61 time=0.239 ms

   2. Run the following command to verify the repo access:

      $ dnf repolist

      Example:

      repo id                             repo name
      ol9_UEKR7                           Unbreakable Enterprise Kernel Release 7 for Oracle Linux 9 (x86_64)
      ol9_addons                          Oracle Linux 9 Addons (x86_64)
      ol9_appstream                       Application packages released for Oracle Linux 9 (x86_64)
      ol9_baseos_latest                   Oracle Linux 9 Latest (x86_64)
      ol9_developer                       Packages for creating test and development environments for Oracle Linux 9 (x86_64)
      ol9_developer_EPEL                  EPEL Packages for creating test and development environments for Oracle Linux 9 (x86_64)

5. Copy the OLX ISO to the Installer Bootstrap Host:

   The iso file must be accessible from a Customer Site Specific repository. This file must be accessible because the ToR switch configurations were completed in procedure: Configure Top of Rack 93180YC-EX Switches

   Copy the OLX ISO file to the /var/occne/os/ directory. The example below uses OracleLinux-R9-U2-x86_64-dvd.iso. If this file was copied to the utility USB, it can be copied from there into the same directory on the Bootstrap Host.

   Note:

   If the user copies this ISO from their laptop then they must use an application like WinSCP pointing to the Management Interface IP.

   $ scp <usr>@<site_specific_address>:/<path_to_iso>/OracleLinux-R9-U2-x86_64-dvd.iso /var/occne/os/

6. Install packages onto the Installer Bootstrap Host. Use DNF to install podman, httpd, and sshpass onto the installer Bootstrap Host.

   $ dnf install -y podman httpd sshpass

7. Set up access to the container registry on the Installer Bootstrap Host:
   1. Copy the container registry certificate to two places on the Bootstrap Host.

      Note:

      The procedure to obtain the container registry certificate <source> is not covered in this procedure. For information about obtaining the container registry certificate, refer to the Docker Engine documentation..

      $ mkdir -p /var/occne/certificates
      $ cp <source>.crt /var/occne/certificates/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}.crt
      $ mkdir -p /etc/containers/certs.d/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}
      $ cp <source>.crt /etc/containers/certs.d/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/ca.crt

8. Setup HTTPD on the Installer Bootstrap Host: Run the following commands using sudo (assumes httpd is already installed in step 6 and the OLX ISO file is named OracleLinux-R9-U2-x86_64-dvd.iso):

   $ mkdir -p -m 0750 /var/www/html/occne/pxe
   $ mkdir -p -m 0750 /var/www/html/os/OL8
   $ mount -t iso9660 -o loop /var/occne/os/OracleLinux-R9-U2-x86_64-dvd.iso /var/www/html/os/OL9
   $ systemctl enable --now httpd

9. Disable SELINUX:
   1. Set SELINUX to permissive mode. To successfully set the SELINUX mode, a reboot of the system is required. The getenforce command is used to determine the status of SELINUX.

      $ getenforce
      Enforcing

   2. If the output of this command does not display Permissive or Disabled, change it to Permissive or Disabled by running the following command (This must be redone if the system reboots before the installation process completes):

      $ sudo setenforce 0

10. Generate the SSH private and public keys on Bootstrap Host.

    This command generates a private and public key for the cluster. These keys are passed to the Bastion Host and used to communicate to other nodes from that Bastion Host. The public key is passed to each node on OS install. Do not supply a passphrase when it asks for one. Click Enter.

    Note:

    The private key (occne_id_rsa) must be copied to a server that is going to access the Bastion Host because the Bootstrap Host is repaved. This key is used later in the procedure to access the Bastion Host after it has been created. The user can also use an SSH client like Putty (keyGen and Pagent) to generate their own key pair and place the public key into the Bastion Host authorized_keys file to provide access.
    Run the following commands on the Bootstrap Host:

    $ mkdir -p -m 0700 /var/occne/cluster/${OCCNE_CLUSTER}/.ssh
    $ ssh-keygen -b 4096 -t rsa -C "occne installer key" -f "/var/occne/cluster/${OCCNE_CLUSTER}/.ssh/occne_id_rsa" -q -N ""
    $ mkdir -p -m 0700 ~/.ssh
    $ cp /var/occne/cluster/${OCCNE_CLUSTER}/.ssh/occne_id_rsa ~/.ssh/id_rsa

2.2.5 Performing Automated Installation

This section details the steps to run the automated configuration of the Bastion Host VM.

This procedure involves two stages:

1. Setting up and running the deploy.sh script on the Bootstrap Host.
2. Accessing the Bastion Host and implementing the final commands to run the pipeline.sh script to complete the Bastion Host configuration and deploy the CNE cluster.

Following is the procedure to perform an automated installation of the Bastion Host VM:

1. Set up and run the deploy.sh script on the Bootstrap Host:
   The deploy.sh script performs the initial configuration of the Bastion host. This includes installing the OS on the bastion and its kvm-host, populating the bastion with repositories, and verifying that everything is up to date. The script is run on the Bootstrap Host using a set of environment variables that can be initialized on the command line along with the deploy.sh script. These variables include the following:

   Table 2-5 Environmental Variables

   Name	Comment	Example usage
   OCCNE_CLUSTER	Defines the cluster short name.	OCCNE_CLUSTER=rainbow
   OCCNE_BASTION	Bastion Host full name	OCCNE_BASTION=bastion-2.rainbow.us.labs.oracle.com
   OCCNE_VERSION	The version tag of the image releases	OCCNE_VERSION=23.4.x

2. Copy necessary files from Utility USB to the Bootstrap Host staging directory:
   1. Copy the deploy.sh script from the Customer Utility USB to the staging directory at /var/occne/ and set the file to executable mode.

      $ cp /<usb_dev>/deploy.sh /var/occne/

3. Run the deploy.sh script from the /var/occne/ directory with the required parameters set.

   $ export OCCNE_BASTION=<bastion_full_name>
   $ ./deploy.sh

   Example:

   $ export OCCNE_BASTION=bastion-2.rainbow.us.labs.oracle.com 		
   $ ./deploy.sh

   You can also run the command as follows:

   $ OCCNE_BASTION=bastion-2.rainbow.lab.us.oracle.com ./deploy.sh

   Note:

   The release version defaults to the 23.4.x GA release. If there is a reason to use a different version, it can be specified by exporting or adding to the command line, the OCCNE_VERSION=<release> variable.
4. Run the final deploy on Bastion Host:

   The following commands are run from the Bastion Host to complete the Bastion Host configuration and deploy CNE on the Baremetal system.

   Note:

   The Bootstrap Host cannot be used to access the Bastion Host as it is re-paved from running this command.
   1. Log in to the Bastion Host as admusr. Use the private key that was saved earlier to access the Bastion Host from a server other than the Bootstrap Host using the ssh command. Copy this private key to the /home/<user>/.ssh directory on that server as id_rsa using scp (or winSCP from a desktop PC). The permissions of the key must be set to 0600 using the command: chmod 0600 ~/.ssh/id_rsa

      $ ssh admusr@<bastion_external_ip_address>

   2. Common Installation Configuration lists configuration options for customizing common services and cnDBTier installation.
   3. Run the following command to complete the deployment of CNE from the Bastion Host (excluding re-install on the Bastion Host and its KVM host, which are already setup). This action will repave the Bootstrap Host RMS.

      $ /var/occne/cluster/${OCCNE_CLUSTER}/artifacts/pipeline.sh

   Note:

   The release version defaults to the 23.4.x GA release. If there is a reason to use a different version, it can be specified by exporting the OCCNE_VERSION=<release> variable.

2.3 Virtualized CNE Installation

This section explains how to install CNE in an OpenStack Environment and VMware cloud director Environment.

Note:

Before installing CNE on vCNE, you must complete the preinstallation tasks.

2.3.1 Installing vCNE on OpenStack Environment

This installation procedure details the steps necessary to configure and install an CNE cluster in an OpenStack Environment. Currently, there are two disk configurations that are supported for installing vCNE on OpenStack:

• Custom Configurable Volumes (CCVs) as block devices for each VM resource: The Custom Configurable Volumes (CCVs) allows the customers to configure the volume cinder type on their Openstack cloud. These configuration can be used to create the hard disk for each VM resource.
• Non-Configureable volumes that are created by Openstack using the flavors assigned to each VM resource. This is a standard configuration.

2.3.1.1 Prerequisites

Before installing and configuring CNE on OpenStack, ensure that the following prerequisites are met.

1. You must have access to an existing OpenStack environment and OpenStack Dashboard (web-based user interface).
2. Ensure that the Nova, Neutron, and Cinder modules are configured. The OpenStack environment uses these modules to manage compute, networking, and storage, respectively.
3. Ensure that the OpenStack environment is configured with appropriate resource flavors and network resources for resource allocation to the VMs.
4. The DHCP Enabled value for OpenStack subnet in each MetalLB pool must be set as "Yes".
5. Ensure that all the required images, binaries, and files are downloaded from the Oracle OSDC before running this procedure. Ensure that these resources are available for use in this procedure. For instructions on how to generate the lists of images and binaries, see the Artifact Acquisition and Hosting chapter.
6. You must obtain a public key (that can be configured) for logging into the Bootstrap Host. Before running this procedure, you must place the public key into the customer's OpenStack Environment as follows:

   Use the Import Key tab on the Launch Instance→Key Pair dialog or via the Compute→Access and Security screen.

7. Ensure that there is a default or custom security group which allows the Bootstrap instance to reach the central repository.

2.3.1.2 Expectations

1. You must be familiar with the use of OpenStack as a virtualized provider including the use of the OpenStack Client and OpenStack Dashboard.

   Note:

   The Bootstrap host doesn't provide OpenStack CLI tools. Therefore, you must use OpenStack Dashboard or an external Linux instance with access to OpenStack CLI to fetch the data or values from OpenStack.
2. You must make a central repository available for all resources like images, binaries, and helm charts before running this procedure.
3. The default user on installation is "cloud-user" and this is still recommended. However, starting 23.4.0, it is possible to define a different default user. This user will be used to access the VMs, configure, run tasks, and manage the cluster.

   Note:

   You can change the default user only during the installation. It is not possible to change the default user during the upgrade.
4. You must define all the necessary networking protocols, such as using fixed IPs or floating IPs, for use on the OpenStack provider.
5. If you select Custom Configurable Volumes (CCVs), the size defined for each volume must match the size of the disk as defined in the flavor used for the given VM resource.
6. When using CCV, you must be fully aware of the volume storage used. If there is insufficient volume storage on the cloud on which CNE is deployed, the deployment will fail while applying the terraform.

2.3.1.3 Downloading OLX Image

Download the OLX image by following the procedure in the Downloading Oracle Linux section.

Note:

The letter 'X' in Oracle Linux X or OLX in this procedure indicates the latest version of Oracle Linux supported by CNE.

2.3.1.4 Uploading Oracle Linux X to OpenStack

This procedure describes the process to upload the qcow2 image, that is obtained using the Downloading OLX Image procedure, to an OpenStack environment.

Note:

Run this procedure from the OpenStack Dashboard.

1. Log in to OpenStack Dashboard using your credentials.
2. Select Compute → Images.
3. Click the +Create Image button. This displays the OpenStack Create Image dialog.
4. In the OpenStack Create Image dialog, enter a name for the image.

   Use a name similar to the name of the qcow2 image at the time of download. It's recommended to include at least the OS version and the update version as part of the name. For example: ol9u2.

5. Under Image Source, select File. This enables the File* → Browse button to search for the image file.

   Click the File* → Browse button to display the Windows Explorer dialog.

6. From the Windows dialog, select the qcow2 image that is downloaded in the Downloading OLX Image section. This inserts the file name and set the Format option to QCOW2 - QEMU Emulator automatically.

   Note:

   If Format isn't set automatically, use the drop-down to select QCOW2 - QEMU Emulator.
7. Retain the default values for the other options. However, you can adjust the Visibility and Protected options according to your requirement.
8. Click the Create Image button at the bottom right corner of the dialog. This starts the process to upload image.

   It takes a while for the system to complete uploading the image. During the upload process, the system doesn't display any progress bar or final confirmation.

9. Navigate to Compute → Images to verify the uploaded image.

2.3.1.5 Creating Bootstrap Host Using OpenStack Dashboard

This section describes the procedures to create Bootstrap host using OpenStack Dashboard. There are two ways to create Bootstrap host of which you can select one depending on your requirement:

• Creating Bootstrap Host Using Non-Configurable volumes: These non-configurable volumes are created by Openstack using the flavors assigned to each VM resource (standard configuration).
• Creating Bootstrap Host Using Custom Configurable Volumes (CCV): The CCVs are created as block devices for each VM resource. This allows you to configure the volume cinder type on your Openstack environment with the configuration that is used for creating the hard disk for each VM resource.

Note:

• A separate bootstrap image (qcow2) isn't required and not provided as part of the artifacts. The Bootstrap VM is created as an instance of a regular base OS image, similar to the other VM instances on the cluster.
• Use the following examples for reference only. The actual values differ from the example values.
• Perform the following procedures manually on the customer specific OpenStack environment.

Creating Bootstrap Host Using Nonconfigurable Volumes

The Bootstrap host drives the creation of virtualized cluster using Terraform, OpenStack Client, and Ansible Playbooks.

Note:

These tools are installed as part of the previous step and are not bundled along with the base image.

1. Log in to the OpenStack Dashboard using your credentials.
2. Select Compute→Instances.
3. Select the Launch Instance button on the upper right. A dialog box appears to configure a VM instance.
4. In the dialog box, enter a VM instance name. For example, occne-<cluster-name>-bootstrap. Retain the Availability Zone and Count values as is.
5. Perform the following steps to select Source from the left pane:

   Note:

   There can be a long list of available images to choose from. Ensure that you choose the correct image.
   1. Ensure that the Select Boot Source drop-down is set to Image.
   2. Enter the OLX image name that you created using the Uploading Oracle Linux X to OpenStack procedure. You can also use the Available search filter to search for the required image. For example, ol9u2.
   3. Enter occne-bootstrap in the Available filter. This displays the occne-bootstrap-<x.y.z> image uploaded earlier.

      Note:

      Do not use a Bootstrap image from any earlier versions of CNE.
   4. Select the OLX Image image by clicking "↑" on the right side of the image listing. This adds the image as the source for the current VM.
6. Perform the following steps to select Flavor from the left pane:
   1. Enter a string (not case-sensitive) which best describes the flavor that is used for this customer specific OpenStack Environment in the Available search filter. This reduces the list of possible choices.
   2. Select the appropriate customer specific flavor (for example, OCCNE-Bootstrap-host) by clicking "↑" on the right side of the flavor listings. This adds the resources to the Launch Instance dialog.

      Note:

      The Bootstrap requires a flavor that includes a disk size of 40GB or higher and RAM size must be 8GB or higher.
7. Perform the following steps to select Networks from the left pane:
   1. Enter the appropriate network name as defined by the customer with the OpenStack Environment (for example, ext-net) in the Available search filter. This reduces the list of possible choices.
   2. Select the appropriate network by clicking "↑" on the right side of the network listings. This adds the external network interface to the Launch Instance dialog.
8. Perform the following step to select Key Pair from the left pane. This dialog assumes you have already uploaded a public key to OpenStack. For more information, see Prerequisites:
   • Choose the appropriate key by clicking "↑" on the right side of the key pair listings. This adds the public key to the authorized_keys file on the Bootstrap Host.
9. Select Configuration from the left pane. This screen allows you to add configuration data that is used by cloud-init to set on the VM, the initial username, and hostname or FQDN additions to the /etc/hosts file.
   Copy the following configuration into the Customization Script text box:

   Note:

   • Ensure that the fields marked as <instance_name_from_details_screen> are updated with the instance name provided as per step 4 in this procedure.
   • Ensure that the <user-name> field is updated. The recommended value for this field is "cloud-user"

   #cloud-config
      hostname: <instance_name_from_details_screen>
      fqdn: <instance_name_from_details_screen>
      system_info:
        default_user:
          name: <user-name>
          lock_passwd: false
      write_files:
        - content: |
            127.0.0.1  localhost localhost4 localhost4.localdomain4 <instance_name_from_details_screen>
            ::1        localhost localhost6 localhost6.localdomain6 <instance_name_from_details_screen>
          path: /etc/hosts
          owner: root:root
          permissions: '0644'

10. Select Launch Instance at the bottom right of the Launch Instance window. This initiates the creation of the VM. After the VM creation process is complete, you can see the VM instance on the Compute→Instances screen.

Creating Bootstrap Host Using Custom Configurable Volumes (CCV)

The CCV deployment includes additional steps to create the volume for the Bootstrap host prior to creating the Bootstrap host instance.

Note:

The Bootstrap host drives the creation of virtualized cluster using Terraform, OpenStack Client, and Ansible Playbooks.

To create Bootstrap host using Custom Configurable Volumes (CCV), perform the following steps manually on the customer specific OpenStack environment:

Creating Custom Configurable Volume (CCV)

Note:

Make sure you know the volume size defined by the flavor that is used to create the Bootstrap host. This information is required to create the Bootstrap host volume.

1. Login to the OpenStack Dashboard using your credentials.
2. Select Compute→Volumes.
3. Select the + Create Volume button on the top right.

   The system displays a dialog box to configure the volume.

4. Enter the volume name. Example: occne-boostrap-host.
5. Enter a description. Example: Customer Configurable Volume for the Bootstrap host.
6. From the Volume Source drop-down list, select image.
7. From the OLX image name that you created using the Uploading Oracle Linux X to OpenStack procedure. For example, ol9u2.

   Note:

   Do not use a Bootstrap image from any earlier versions of CNE.
8. From the Type drop-down list, select the image type. Example: nfs or as configured on the cloud.
9. In the Size (GiB) field, enter the size of the image. This size must match the size defined for the Bootstrap host flavor that is used.
10. From the Availability Zone drop-down list, select an availability zone.
11. From the Group drop-down list, select No group.
12. Click Create Volume.

    The system creates the Custom Configurable Volume (CCV) that is used to create the Bootstrap host VM.

Creating Bootstrap Host VM

1. Log in to the OpenStack Environment using your credentials.
2. Select Compute→Instances.
3. Select the Launch Instances tab on the top right. A dialog box appears to configure a VM instance.
4. In the dialog box, enter a VM instance name. For example, occne-<cluster-name>-bootstrap. Retain Availability Zone and Count values as is.
5. Perform the following steps to select Source from the left pane:

   Note:

   There can be a long list of available images to choose from. Ensure that you choose the correct image.
   1. Ensure that the Select Boot Source drop-down is set to Volume.
   2. Ensure that the Delete Volume on Instance Delete is set to Yes.
   3. Enter the volume name in the Available search filter. The system displays the volume that you created in the previous section. For example, occne-boostrap-host.
   4. Select the volume by clicking "↑" on the right side of the volume listing. This adds the volume as the source for the current VM.
6. Perform the following steps to select Flavor from the left pane:
   1. Enter a string (not case-sensitive) which best describes the flavor that is used for this customer specific OpenStack Environment in the Available search filter. This reduces the list of possible choices.
   2. Select the appropriate customer specific flavor (for example, OCCNE-Bootstrap-host) by clicking "↑" on the right side of the flavor listings. This adds the resources to the Launch Instance dialog.

      Note:

      The Bootstrap requires a flavor that includes a disk size of 40GB or higher and the RAM size must be 8GB or higher.
7. Perform the following steps to select Networks from the left pane:
   1. Enter the appropriate network name as defined by the customer with the OpenStack Environment (for example, ext-net) in the Available search filter. This reduces the list of possible choices.
   2. Select the appropriate network by clicking "↑" on the right side of the network listings. This adds the external network interface to the Launch Instance dialog.
8. Perform the following step to select Key Pair from the left pane. This dialog assumes that you have already uploaded a public key to OpenStack. For more information, see Prerequisites:
   • Choose the appropriate key by clicking "↑" on the right side of the key pair listings. This adds the public key to the authorized_keys file on the Bootstrap host.
9. Select Configuration from the left pane. This screen allows you to add configuration data that is used by cloud-init to set on the VM, the initial username, and hostname or FQDN additions to the /etc/hosts file.
   Copy the following configuration into the Customization Script text box:

   Note:

   • Ensure that the fields marked as <instance_name_from_details_screen> are updated with the instance name provided as per step 4 in this procedure.
   • Ensure that the <user-name> field is updated. The recommended value for this field is "cloud-user".

   #cloud-config
      hostname: <instance_name_from_details_screen>
      fqdn: <instance_name_from_details_screen>
      system_info:
        default_user:
          name: <user-name>
          lock_passwd: false
      write_files:
        - content: |
            127.0.0.1  localhost localhost4 localhost4.localdomain4 <instance_name_from_details_screen>
            ::1        localhost localhost6 localhost6.localdomain6 <instance_name_from_details_screen>
          path: /etc/hosts
          owner: root:root
          permissions: '0644'

10. Select Launch Instance at the bottom right of the Launch Instance window. This initiates the creation of the VM. After the VM creation process is complete, you can see the VM instance on the Compute→Instances screen.

2.3.1.6 Predeployment Configuration

Run the commands in this procedure from the Bootstrap host.

Log in to Bootstrap VM

Use SSH to log in to Bootstrap using the private key uploaded to OpenStack. For more information about the private key, see Prerequisites.

For example:

$ ssh -i $BOOTSTRAP_PRIVATE_KEY $USER@$BOOTSTRAP_EXT_IP

The values used in the example are for reference only. You must obtain the Bootstrap external IP from Compute -> Instances on the OpenStack Dashboard. The $USER parameter is the same as <user-name>.

Set Cluster Short-Name and Central Repository Variables

Set the cluster short-name and other necessary variables for use in the Bootstrap environment, and load them into the current environment:

$ echo 'export OCCNE_CLUSTER=<cluster_short_name>' | sudo tee -a  /etc/profile.d/occne.sh
$ echo 'export CENTRAL_REPO=<central_repo_name>' | sudo tee -a /etc/profile.d/occne.sh
$ echo 'export CENTRAL_REPO_IP=<central_repo_ip>' | sudo tee -a /etc/profile.d/occne.sh
$ echo 'export CENTRAL_REPO_REGISTRY_PORT=<central_repo_registry_port>' | sudo tee -a /etc/profile.d/occne.sh
$ echo 'export OCCNE_USER=<user-name>' | sudo tee -a /etc/profile.d/occne.sh
$ . /etc/profile.d/occne.sh

For example:

$ echo 'export OCCNE_CLUSTER=occne1-rainbow' | sudo tee -a  /etc/profile.d/occne.sh
$ echo 'export CENTRAL_REPO=winterfell' | sudo tee -a /etc/profile.d/occne.sh
$ echo 'export CENTRAL_REPO_IP=128.128.128.128' | sudo tee -a /etc/profile.d/occne.sh
$ echo 'export CENTRAL_REPO_REGISTRY_PORT=5000' | sudo tee -a /etc/profile.d/occne.sh
$ echo 'export OCCNE_USER=cloud-user' | sudo tee -a /etc/profile.d/occne.sh
$ . /etc/profile.d/occne.sh

In this step, the Bash variable references, such as ${OCCNE_CLUSTER}, expands to the short-cluster name in this shell, and subsequent ones.

Create Cluster Specific Directories

Create the base occne directory, cluster directory (using the cluster short-name), and YUM local repo directories.

For example:

• Use the following command to create base directory:

  $ sudo mkdir -p -m 0750 /var/occne
  $ sudo chown -R ${USER}:${USER} /var/occne

• Use the following command to create cluster directory:

  $ mkdir -p -m 0750 /var/occne/cluster/${OCCNE_CLUSTER}/

• Use the following command to create YUM local repo directory and certificates directory:

  $ mkdir -p -m 0750 /var/occne/yum.repos.d
  $ mkdir -p -m 0750 /var/occne/certificates

Obtain the TLS Certificate for OpenStack

Depending on your environment, the OpenStack may use a TLS certificate for accessing the OpenStack controller API. In such cases, Terraform cannot communicate with the Openstack environment and fails. Therefore, you must obtain the TLS certificate before using OpenStack.

Note:

Perform this step only if your OpenStack environment requires a TLS certificate to access the controller from the cluster nodes and the Bootstrap host.

1. Contact the OpenStack admin to provide the required TLS certificate to access the client commands. For example, in an Oracle OpenStack system installed with kolla, the certificate is available at /etc/kolla/certificates/openstack-cacert.pem.
2. Copy the entire certificate (including the intermediate and root CA, if provided) to the /var/occne/cluster/${OCCNE_CLUSTER}/openstack-cacert.pem directory on the Bootstrap Host. Run this step after creating cluster specific directory.

   Ensure that the certificate file name is openstack-cacert.pem, when you copy the file to the /var/occne/cluster/${OCCNE_CLUSTER}/ directory.

   If the certificate file name is different, then rename it to openstack-cacert.pem before copying to the /var/occne/cluster/${OCCNE_CLUSTER}/ directory

3. Set the OS_CACERT environment variable to /var/occne/cluster/${OCCNE_CLUSTER}/openstack-cacert.pem using the following command:

   export OS_CACERT=/var/occne/cluster/${OCCNE_CLUSTER}/openstack-cacert.pem

Get the OpenStack RC (API v3) File

The OpenStack RC (API v3) file exports several environment variables on the Bootstrap host. Terraform uses these environment variables to communicate with OpenStack and create different cluster resources.

Note:

The following instructions may slightly vary depending on the version of OpenStack Dashboard you're using.

1. From the OpenStack Dashboard, go to Project - > API Access.
2. From the Download OpenStack RC File drop-down menu on the right side, choose OpenStack RC File (Identity API v3).

   This downloads an openrc.sh file prefixed with the OpenStack project name (for example, OCCNE-openrc.sh) to your local system.

3. Copy the file securely (using SCP or WinSCP) to the Bootstrap host in the /home/${USER} directory as .<project_name>-openrc.sh

   Note:

   In order for SCP or WinSCP to work properly, use the key mentioned in the Prerequisites to access the Bootstrap host. Also, it may be necessary to add the appropriate Security Group Rules to support SSH (Rule: SSH, Remote: CIDR CIDR: 0.0.0.0/0) under the Network → Security Groups page in the OpenStack Environment. If required, contact the OpenStack administrator to add the correct rules.
4. Run the following command to source the OpenStack RC file:

   source .<project_name>-openrc.sh

Create SSH Key on Bootstrap Host

Create the private and public keys to access the other VMs. The following command generates the keys that are passed to the Bastion Host and are used to communicate to other nodes from that Bastion Host.

$ mkdir -p -m 0700 /var/occne/cluster/${OCCNE_CLUSTER}/.ssh
$ ssh-keygen -m PEM -t rsa -b 2048 -f /var/occne/cluster/${OCCNE_CLUSTER}/.ssh/occne_id_rsa -N ""
$ cp /var/occne/cluster/${OCCNE_CLUSTER}/.ssh/occne_id_rsa ~/.ssh/id_rsa

Note:

The Bootstrap host is transient. Therefore, create a backup of the occne_id_rsa private key and copy the private key to a safe location for using it in case of Bootstrap failures.

You can also use an SSH client like Putty (keyGen and Pagent) to generate your own key pair and place the public key into the Bastion Host authorized_keys file to provide access.

Copy Central Repository Files to Bootstrap

Use SCP or WinSCP to copy the necessary files to the listed directories.

1. Update the /etc/hosts file with central repository IP:

   $ echo ${CENTRAL_REPO_IP} ${CENTRAL_REPO} | sudo tee -a  /etc/hosts

2. There are a few empty directories on the Bootstrap host for holding cluster specific files. The directories are as follows:
   1. /var/occne/yum.repos.d/
   2. /var/occne/certificates/
3. Supply the following mandatory files within the directories mentioned in the previous step:
   1. Add a customer specific central repo .repo file to the /var/occne/yum.repos.d/ directory that allows access to the customer-specific central repo (for example, winterfell-ol9.repo).
   2. Add a container registry certificate to the /var/occne/certificates/ directory for the central container registry. Copy the file using the following format:

      <central_repo_hostname>:<central_repo_port>.crt

      For example: winterfell:5000.crt

4. Ensure that the permissions for each file are readable using the sudo chmod 644 <filename> command.
5. On the Bootstrap, create an appropriate directory and copy the registry certificate into it as follows for local use:

   $ sudo mkdir -p /etc/containers/certs.d/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/
   $ sudo cp /var/occne/certificates/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}.crt /etc/containers/certs.d/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/ca.crt

6. Copy the yum repository directory to the cluster-specific directory for use in targeted cluster nodes:

   $ cp -r /var/occne/yum.repos.d/ /var/occne/cluster/${OCCNE_CLUSTER}/

7. Configure dnf config to point to the central repository:

   echo reposdir=/var/occne/yum.repos.d | sudo tee -a /etc/dnf/dnf.conf

Verify Central Repository and Install Required Packages

In previous versions of CNE, the Bootstrap image came built-in with the prerequisite YUM packages and their dependencies. Starting 23.3.1, when you are using Oracle Linux Base OS image, most of the packages and dependencies are handled by the deploy.sh script, and are installed directly from the central repository. However, it's necessary to manually install Podman, downloaded from the central repository.

1. [Optional]: Allow access to central repository (Optional). Ensure that proper security group that allows access to the central repository is created. For more information, see Prerequisites. If necessary, add a security group to the Bootstrap instance by performing the following steps:
   1. Navigate to Compute → Instances → occne-<cluster-name>-bootstrap.
   2. On the right most drop-down menu, select Edit Instance.
   3. Select Security Groups.
   4. Click the plus symbol on the default or custom security group to add it to the Bootstrap image.
   5. The security group may be already allocated depending on the OpenStack environment.
2. Perform the following steps to test the central repository:
   1. Run the following command to perform a simple ping test:

      $ ping -c 3 ${CENTRAL_REPO}

      Sample output:

      PING winterfell (0.0.0.0) 56(84) bytes of data.
      64 bytes from winterfell (128.128.128.128): icmp_seq=1 ttl=60 time=0.448 ms
      64 bytes from winterfell (128.128.128.128): icmp_seq=2 ttl=60 time=0.478 ms
      64 bytes from winterfell (128.128.128.128): icmp_seq=3 ttl=60 time=0.287 ms
       --- winterfell ping statistics ---
      3 packets transmitted, 3 received, 0% packet loss, time 2060ms
      rtt min/avg/max/mdev = 0.287/0.404/0.478/0.085 ms

   2. Run the following command to list the repositories:

      $ dnf repolist

      Sample output:

      repo id                                    repo name
      ol9_UEKR7                                  Unbreakable Enterprise Kernel Release 7 for Oracle Linux 9 (x86_64)
      ol9_addons                                 Oracle Linux 9 Addons (x86_64)
      ol9_appstream                              Application packages released for Oracle Linux 9 (x86_64)
      ol9_baseos_latest                          Oracle Linux 9 Latest (x86_64)
      ol9_developer                              Packages for creating test and development environments for Oracle Linux 9 (x86_64)
      ol9_developer_EPEL                         EPEL Packages for creating test and development environments for Oracle Linux 9 (x86_64)

3. Run the following command to instal Podman:

   $ sudo dnf install -y podman

Copy Necessary Files to Bootstrap

1. Copy the <project_name>-openrc.sh script, created in a previous section, from the user's home directory to the cluster directory:

   cp ~/<project_name>-openrc.sh /var/occne/cluster/${OCCNE_CLUSTER}/openrc.sh

2. The Bootstrap is not preloaded with default settings, including the presetting of OCCNE_VERSION. It is necessary to temporarily set the CNE version before copying or downloading the cluster files. Use the following command to replace the CNE version:

   $ export OCCNE_VERSION=<occne_version>

   For example:

   $ export OCCNE_VERSION=23.4.6

   This value is permanently set by the deploy script in a later step.
3. Perform the following steps to copy the templates to cluster directory and update its ownership:
   1. Copy files from the provisioning container:

      $ podman run --rm -v /var/occne/cluster/${OCCNE_CLUSTER}:/host -v /var/occne:/var/occne:z ${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/occne/provision:${OCCNE_VERSION} cp -r /virtual/scripts /virtual/terraform_openstack/. /host

   2. Ensure that all the copied files have proper ownerships:

      $ sudo chown -R ${USER}:${USER} /var/occne/cluster/${OCCNE_CLUSTER}

Update the cluster.tfvars File

The cluster.tfvars file contains all the variables required by Terraform to implement the cluster. Use the following steps to modify and complete a copy of the occne_example/cluster.tfvars template (copied from the provisioning container as part of the previous step).

Configure the fields in the cluster.tfvars file to adapt to the current customer OpenStack Environment. This requires the user to get some of the fields and information directly from the OpenStack Dashboard or OpenStack CLI (not bundled with Bootstrap). This procedure doesn't detail OpenStack CLI-related examples. The following steps provide details on how to collect and set the fields that must be changed.

Note:

All fields in the cluster.tfvars file are unique. So, ensure that there are no duplicate fields in the cluster.tfvars file. Don't duplicate any fields in the file using comments "#" as this can cause possible parsing errors.

Common Configuration

1. Run the following command to create a copy of the occne_example directory and its content. The command creates a cluster name specific directory and the cluster.tfvars file that is used for configuring terraform. The configuration changes indicated in this procedure are made in the new copy of the cluster.tfvars file.

   $ cp -R /var/occne/cluster/${OCCNE_CLUSTER}/occne_example /var/occne/cluster/${OCCNE_CLUSTER}/${OCCNE_CLUSTER}

   Sample output:

   -rw-r--r--. 1 <user-name> <user-name> 6390 Aug 24 23:18 /var/occne/cluster/occne1-rainbow/occne1-rainbow/cluster.tfvars
   drwxr-xr-x. 2 <user-name> <user-name>   28 Aug 24 23:18 /var/occne/cluster/occne1-rainbow/occne1-rainbow

2. Access the OpenStack Dashboard to retrieve the information needed to configure the cluster.tfvars file:
   1. Set the different flavor settings according to the recommendations from the Sizing vCNE VMs section in this document. An admin user of the customer specific OpenStack Environment must add the flavors and provide the names of the flavors for configuration into the cluster.tfvars file. The name of each specific flavor that is used must be added to the value field of the key/value fields in the cluster.tfvars file.
   2. Once the flavors are added to the OpenStack environment, you can use the OpenStack Dashboard to retrieve the flavor name:

      Note:

      The options may vary depending on the environment that you are using.
      1. On the OpenStack Dashboard, navigate to Compute→Instances→Launch Instance to open a new dialog.
      2. Select Flavor from the left side panel.
      3. Scroll the list of available options or use the Available filter to verify if the flavor is present.
      4. Click Cancel to close the dialog.
   3. Perform the following steps to retrieve the external_net UUID:
      1. On the OpenStack Dashboard, navigate to Network→Networks and search for the appropriate external network.
      2. Click the network name. For example, ext-net.
      3. Get the ID from the Overview tab.
         For example:

         Name ext-net
         ID 2ddd3534-8875-4357-9d67-d8229dae81ff

      4. On the Subnets tab, click the appropriate subnet to retrieve the ID.
         For example:

         Name ext-net-subnet
         ID a3d433fe-d931-4dca-bf6b-123dde4a94de

3. [Optional]: If <user-name> is set to a value other than “cloud-user", uncomment the ssh_user field and set it to the desired <user-name>.

   Note:

   The <user-name> value must be the same as the one used in the previous sections of this procedure. That is, it must be same as same as $OCCNE_USER and $USER.

   ssh_user = "<user-name>"

4. Set the image field to the same value that was used when uploading the image to Openstack:

   For example:

   image = "ol9u2"

   Note:

   This image is the boot image of all VMs in the cluster.
5. The following fields define the number of each node type and contain default values. For a standard deployment, use the default values. You can update these values if the deployment requires additional nodes.

   Note:

   The following fields are integer values and don't require double quotation marks.
   • number_of_bastions
   • number_of_k8s_nodes
   • number_of_k8s_ctrls_no_floating_ip

     WARNING: Set the number of control nodes to an odd number. The recommended value for number_of_k8s_ctrls_no_floating_ip is 3.

6. Set the corresponding flavors for each node type to a unique flavor name. Flavors are provided from the OpenStack Provider administrator.
   • flavor_bastion
   • flavor_k8s_ctrl
   • flavor_k8s_node
   There are four settings for server group affinity, one for each group type (Bastion Hosts, k8s Nodes, k8s Control Nodes, and LBVMs). Currently all of these settings default to anti-affinity. Using this default, forces terraform/openstack to create the VMs for each server group on different hosts within the OpenStack environment. If there are not enough hosts to perform this task, terraform will fail which causes the deploy.sh command to fail. Soft-anti-affinity could also be used to force VMs to be spread across the hosts as much as possible without failing. The only other option available is affinity.

   Note:

   Before trying to use soft-anti-affinity, ensure that your OpenStack release supports the use of that value. Also, ensure that anti-affinity works for your deployment (as in the number of hosts is greater than or equal to the number of worker nodes, Kubernetes nodes, or LBVMs, whichever is greater). The OpenStack instance where CNE will be installed must have at least the following number of physical servers in order to install CNE: the larger of the number of Kubernetes nodes or the number of LB VMs (the number of LB VMs is equal to the number of configured external networks times 2).

   • k8s_ctrl_server_group_policies
   • k8s_node_server_group_policies
   • bastion_server_group_policies
   • lbvm_server_group_policies
7. Set the cluster_name and network_name to contain the same value.

   Note:

   This field is used to create each of the nodes contained within the cluster. Kubespray doesn't allow to use uppercase alphanumeric characters in the node hostname. So, don't use any uppercase characters when defining the cluster-name field.

   # Kubernetes short cluster name
   cluster_name = "<cluster-short-name>"
    
   # networking
   network_name = "<cluster-short-name>"

8. The subnet_cidr defines the tenant side network IP address range. This field must remain in the default value.
9. The field bastion_allowed_remote_ips defines the configuration for the bastion networking security group. This field must remain in the default value.
10. Set the ntp_server value in the cluster.tfvars file. If the OpenStack environment has a NTP service, then use the cloud IP of the OpenStack URL. If not, use a customer-specific NTP server. You can use the ping command to get the IP of the OpenStack environment:

    $ ping <openstack_provider_url>

    For example:

    $ ping openstack-cloud.us.oracle.com

    Sample output:

    PING srv-10-10-10-10.us.oracle.com (10.10.10.10) 56(84) bytes of data.
    64 bytes from srv-10-10-10-10.us.oracle.com (10.10.10.10): icmp_seq=1 ttl=63 time=0.283 ms

11. If the deployment requires a specific Availability Zone other than the default availability zone called nova, make the following changes in the cluster.tfvars file. This value can be added after the cluster_name field. If you want to use nova, then skip this step.

    az_list = ["<availability_zone_name>"]

    For example:

    # Authorizing_Zone
    az_list = ["foobar"]

12. If the deployment requires configuration of additional external dns_nameservers, perform the following changes in the cluster.tfvars file. There can be multiple IPv4 addresses in the list. This value can be added at the end of the file after the wait_for_floatingip. This is an optional value and the default is empty list []. The sample configuration format is as below:

    dns_nameservers = ["<ipv4_address>",...] 

    Example:

    
    dns_nameservers = ["10.10.xxx.xxx",...]

13. If the deployment requires the delivery of metadata and user data through a configuration drive for each instance, add the following changes in the cluster.tfvars file. This is an optional value. The default value is occne_config_drive = "false" that indicates using a metadata server instead of a config drive for OpenStack.

    Note:

    Ensure that the OpenStack administrator did not set the force_config_drive=true in the /etc/nova/nova.conf file, otherwise it uses the config drive in either case. The sample configuration format is as follows:

    occne_config_drive = "<true/false>" 

    Example:

    occne_config_drive = "true"

14. Set the use_floating_ip variable to true. This field defaults to true and must remain as set to configure the system to use floating IPs allocated from existing DHCP enabled networks defined within the OpenStack Provider.
15. Perform the following steps to retrieve the floatingip_pool and external_net ID name from Openstack Dashboard and set the variables on the cluster.tfvars file:
    1. Obtain the value of floatingip_pool and external_net from Openstack Dashboard:
       1. On the OpenStack Dashboard, navigate to Network → Networks.
       2. From the list, select an existing external network and click Overview.
       3. Obtain the value of floatingip_pool from the Name field and external_net from the ID field.
          For example, the following codeblock displays the data displayed by OpenStack Dashboards. This data may vary depending on the OpenStack environment:

          ext-net2
          Overview    /     Subnets     /     Ports
          Name                ext-net2
          ID                  4ebb3784-0192-7482-9d67-a1389c3a8a93
          Project ID          3bf3937f03414845ac09d41e6cb9a8b2
          Status              Active
          Admin State         UP
          Shared              Yes
          External Network    Yes
          MTU                 9050

    2. Assign the floatingip_pool field in the cluster.tfvars file.

       floatingip_pool = "<floating_ip_pool_name>"

       where, <floating_ip_pool_name> is the name of the external network obtained in step a.

       For example:

       floatingip_pool = "ext-net2"

    3. Assign the external_net field in the cluster.tfvars file.

       external_net = "<network UUID>"

       where, <network UUID> is the ID of the external network obtained in step a.

       For example:

       external_net = "4ebb3784-0192-7482-9d67-a1389c3a8a93"

16. The wait_for_floatingip variable provides the ability for the Terraform deployment to poll the instance until the floating IP is associated. By default, the field value is set to true in the vCNE deployment. Don't change the default value.

    Note:

    This field is a string and requires double quotation marks around it.

Configuring Custom Configurable Volumes (CCV)

Custom Configurable Volumes (CCVs) replaces the local hard disk created by OpenStack which is derived from the OpenStack flavor, that are used to define a given resource.

Note:

• This procedure is optional. Perform this procedure only if you choose to use CCV over the standard configuration.
• CCV and the standard hard disk configuration are mutually exclusive. You can choose one of the two.
• If you select CCV, all resources use the custom boot disk configuration. It can't be used on a per VM resource basis.
• The size of the volume created must be equal to the size of the disk as defined in the flavor that is used for the VM.

Perform the following steps to configure Custom Configurable Volumes:

1. Set use_configurable_boot_volume to true.
2. Set the volume size variables to the size of the hard disk as defined in the flavor's Disk field assigned to the given resource type. Flavors can be listed using the OpenStack Dashboard.

   Note:

   The volume size settings must match the disk size defined by the flavor setting for each resource type. The defaults listed in this example are derived from the current flavor settings. Update the setting correctly.
   For example:

   occne_lbvm_boot_volume_size = 40
   occne_bh_boot_volume_size = 100
   occne_k8s_ctrl_boot_volume_size = 40
   occne_k8s_node_boot_volume_size = 60

3. Set occne_boot_volume_image_id to the ID of the image from which the VM is booted (Example: ol9u2). Perform the following steps to obtain the image ID from OpenStack Dashboard:
   1. Select Compute →Images.
   2. From the list, select the appropriate OLX image.
   3. Obtain the value of occne_boot_volume_image_id from the ID field.
      For example:

      → test              Image   Active  Public  No  QCOW2   641.13 MB   Launch
      ↓ ol9u2             Image   Active  Shared  No  QCOW2   708.81 MB   Launch
      ------------------------------------------------------------------------------------
      Name                                     Visibility              Min. Disk
      ol9u2                                    Shared                  0
      ID                                       Protected               Min. RAM
      2c673fc4-9d72-4a60-bb14-bc44f636ee94     No                      0
      ------------------------------------------------------------------------------------
      → test-imag         Image   Active  Public  No  QCOW2   988.75 MB   Launch

4. By default, occne_delete_on_termination is set to true, this means that the volumes are deleted along with the VMs (recommended behavior). You can set this field to false if you want to preserve the volumes while deleting VMs.

Configuring MetalLB in tfvars file

MetalLB replaces Octavia (which is no longer supported) on the vCNE deployment in OpenStack and is the default behavior for the use of Load Balancers in CNE vCNE.

The vCNE deployment uses the following configuration to pre-reserve the ports used for external access to the services created during the configure stage of the deployment. It generates the mb_configmap.yaml file from the pre-reserved ports. The mb_configmap.yaml file is used to configure MetalLB during the configure stage of the deployment.

The ports are pre-reserved in OpenStack and are assigned names using the following format (which can be listed in the OpenStack Dashboard (Network → Network → Network Name → Ports). You can sort the ports using the filter option:

<cluster_name>-mb-<pool_name>-port-n where n = 0CC.(sum of num ports for all pool names included in the tfvars file)

Example: For pool_name: oam, the assigned port names of the oam pool is in the following format: <cluster_name>-mb-oam-port-n or mycne-cluster-mb-oam-port-1

1. MetalLB allows Terraform to automatically create the necessary Load Balancing VMs (LBVMs) to support service load balancing for CNE (and the NFs as they are configured). An LBVM is created per Peer Address Pool name.

   Note:

   The oam pool is required, while other network pools are NF application specific. The VMs are named according to the following format in OpenStack: <cluster_name>-<peer_address_pool>-lbvm-1 or 2 and are visible from the OpenStack Dashboard.
2. Update the occne_metallb_flavor field. Ensure that the field is set as per the Oracle sizing charts Sizing vCNE VMs in Reference 1. Obtain the flavor detail from the OpenStack Dashboard or the OpenStack administrator.

   occne_metallb_flavor="<lbvm_flavor_name>"

   Example:

   occne_metallb_flavor="OCCNE-lbvm-host"

3. Update the occne_metallb_peer_addr_pool_names list variable. It can take any value as address pool name, for example, "oam", "signaling", "random1", and so on. Set this field depending on what you want to configure, for example, ["oam"] or ["oam", "signaling"] or ["oam", "signaling", "random1"].

   Note:

   Don't use any special characters or capital letters in the pool name. The name must only contain [a-z,1-9] character sets. There is no limitation on the number of characters used in a pool name. However it's recommended to keep the name as short as possible, as the name is used to build the LBVM host names. For example, for a LBVM1 with pool name "oam", and cluster name "my-cluster", the LBVM host name build is "my-cluster-oam-lbvm1". Linux has a hostname limitation of 253 ascii(7) and allows [a-z,0-1] characters including the dots, and each section of the hostname can only have 63 characters.

   occne_metallb_peer_addr_pool_names=["oam","poolname"]

   Examples:

   occne_metallb_peer_addr_pool_names=["oam"]
   occne_metallb_peer_addr_pool_names=["oam","signaling"]
   occne_metallb_peer_addr_pool_names=["oam","signaling","nfpoolname1"]

4. Use the following fields to create the MetalLB peer address pools. This part of the MetalLB configuration is used to pre-reserve the ports for MetalLB to use in the generated mb_configmap.yaml file.

   There can be more than one pool added to the tfvars file and each can be configured with a different IP list field as given in the following table.

   When configuring the port objects, use only one of the three input field types to define the IPs for each peer address pool: ip_list, ip_range, or ip_cidr. Delete the other two unused input fields for that peer address pool. Ensure that you follow the above process for all port objects. If you fail to configure any of the fields correctly, then Terraform will fail at the deployment stage before it gets to the configure stage.

   Note:

   If you configure more than one IP field per peer address pool, the first one that is found is selected.

   In occne_metallb_list, the field num_pools is the number of pools mentioned in occne_metallb_peer_addr_pool_names.

   For example, for occne_metallb_peer_addr_pool_names=["oam","signaling","xyz","abc1"] the value of num_pools is 4.

   Each object includes the following common key or value pairs except the type of field it represents:

   • num_ports: specifies the number of ports to reserve for the given peer address pool (use this key without quotation marks).
   • subnet_id: specifies the OpenStack UUID (or ID) for the subnet from which the ports are configured within.
   • network_id: specifies the OpenStack network UUID (or ID) which the subnet is configured within.
   • pool_name: specifies one of the peer address pool names given in occne_metallb_peer_addr_pool_names.
   • egress_ip_addr: specifies the IP address used for LBVM egress interface. Each network pool has only one address.
   • Configuring IP input field: use only one of the three IP address input fields for each peer address pool. Delete the other two unused input fields.

   Table 2-6 MetalLB Fields

   Field Name	Notes
   ip_list	Represents a random list of IPs from the same subnet or network for the peer address pool. Define the ip_list within square brackets [], with each value surrounded by double quotation marks and separated by a comma. The ports (IPs) defined within this list are available within the configured OpenStack network or subnet. Note: The number of IPs provided in the list must be equal to or greater than the setting of num_ports. If the number of IPs in the list is greater than the num_ports, only the first num_ports IPs are used to pre-reserve the ports for MetalLB. All the IPs must be unique. IP used once in one address pool can't be used in any other address pool. You can define multiple pools at a time by adding each pool object to the comma-separated pool_object list. The following codeblock provides a sample template for adding a single pool object. Copy and paste the same template for each pool object that you want to add. { pool_name = "<pool_name>" num_ports = <no_of_ips_needed_for_this_addrs_pool_object> ip_list = ["<ip_0>","<ip_(num_ports-1)>"] subnet_id = "<subnet UUID for the given network>" network_id = "<network UUID>" egress_ip_addr = "<IP address for egress port>" } Example: num_pools is 1 and occne_metallb_peer_addr_pool_names=["oam"] occne_metallb_list = { num_pools = 1 pool_object = [ { pool_name = "oam" num_ports = 6 ip_list = ["10.10.235.1","10.10.235.3","10.10.235.19","10.10.235.6","10.10.235.21","10.10.235.45"] subnet_id = "c3a5381a-3a17-4775-8e42-a1c816414e12" network_id = "cd132f1f-1a31-1a1f-b69a-ade2f7a283f4" egress_ip_addr = "10.10.235.56" } ] }
   ip_range	Represents a range of IPs from the same subnet or network for the peer address pool. Define the ip_range as a single string within double quotation marks and separate the min and max IPs using a dash '-". Note: You can extend the range of IPs beyond the setting of num_ports, but ensure that the range is not less than num_ports. If num_ports is less than the range, only the first num_ports IPs are used to pre-reserve the ports for MetalLB. All the IPs must be unique. IP used once in one address pool can't be used in any other address pool. You can define multiple pools at a time by adding each pool object to the comma-separated pool_object list. The following codeblock provides a sample template for adding a single pool object. Copy and paste the same template for each pool object that you want to add. { pool_name = "<pool_name>" num_ports = <no_of_ips_needed_for_this_addrs_pool_object> ip_range = "<ip_n> - <ip_(n + num_ports - 1)>" subnet_id = "<subnet UUID for the given network>" network_id = "network UUID" } Example: num_pools is 1 and occne_metallb_peer_addr_pool_names=["oam"] occne_metallb_list = { num_pools = 1 pool_object = [ { pool_name = "oam" num_ports = 6 ip_range = "10.10.235.10 - 10.10.235.15" subnet_id = "c3a5381a-3a17-4775-8e42-a1c816414e12" network_id = "cd132f1f-1a31-1a1f-b69a-ade2f7a283f4" egress_ip_addr = "10.10.235.17" } ] }
   cidr	Represents a range of IPs from the same subnet or network for the peer address pool using the cidr slash notation. This must be a single string enclosed within double quotation marks with a starting IP and the netmask as designated by the forward slash "/". Note: You can extend the range of IPs determined by the cidr beyond the setting of num_ports but can't be less than num_ports. If num_ports is less than the range provided by the cidr slash notation, only the first num_ports IPs are used to pre-reserve the ports for MetalLB. You can define multiple pools at a time by adding each pool object to the comma-separated pool_object list. The following codeblock provides a sample template for adding a single pool object. Copy and paste the same template for each pool object that you want to add. { pool_name = "<pool_name>" num_ports = <no_of_ips_needed_for_this_addrs_pool_object> cidr = "<0.0.0.0/29>" subnet_id = "<subnet_id or network UUID for cidr>" network_id = "network_id" egress_ip_addr = "<IP address for egress port>" } Example: num_pools is 1 and occne_metallb_peer_addr_pool_names=["oam"] occne_metallb_list = { num_pools = 1 pool_object = [ { pool_name = "oam" num_ports = 6 cidr = "10.10.235.8/29" subnet_id = "c3a5381a-3a17-4775-8e42-a1c816414e12" network_id = "cd132f1f-1a31-1a1f-b69a-ade2f7a283f4" egress_ip_addr = "10.10.235.50" } ] }

5. If you want to use more than one pool, and for each pool you want to input IPs in a different way, then follow the example given below:
   num_pools is 4 and occne_metallb_peer_addr_pool_names=["oam","signaling","userpool1","userpool2"], here:

   • ip_list, is used for "oam" and assigned with 6 IPs to it
   • ip_range, is used for "userpool1"
   • cidr, is used for "userpool2"
   Note:

   • Ensure that there are no extra lines.
   • Each pool in the pool_object list must contain only five input fields. oam is a required pool and other network pools are application-specific.
   • Ensure that you use only one IP input field per pool. Delete the other two unused input fields.
   • Ensure that the IP is used only once in a pool. Don't use this IP in another pool.
   • Different pools can have various network and subnet IDs.
   • The num_ports field must contain an integer input (without quotation marks).
   Example for more than one address pool:

   occne_metallb_list =  {
     num_pools = 4
     pool_object = [
      {
        pool_name = "oam"
        num_ports = 6
        ip_list = ["10.10.235.1","10.10.235.3","10.10.235.19","10.10.235.6","10.10.235.21","10.10.235.45"]
        subnet_id = "a1f0c182-af23-1a12-a7a3-118223fd9c10"
        network_id = "f1251f4f-81e3-4a23-adc2-def1294490c2"
        egress_ip_addr = "10.10.235.2"
      },
      {
        pool_name = "userpool1"
        num_ports = 4
        ip_range = "10.10.235.34 - 10.10.235.37"
        subnet_id = "a1f0c182-af23-1a12-a7a3-118223fd9c10"
        network_id = "f1251f4f-81e3-4a23-adc2-def1294490c2"
        egress_ip_addr = "10.10.235.38"
      },
      {
        pool_name = "userpool2"
        num_ports = 2
        cidr = "10.10.231.0/29"
        subnet_id = "c3a5381a-3a17-4775-8e42-a1c816414e12"
        network_id = "cd132f1f-1a31-1a1f-b69a-ade2f7a283f4"
        egress_ip_addr = "10.10.231.12"
      }
     ]
   }

2.3.1.7 Deploying CNE Cluster in OpenStack Environment

This section describes the procedure to deploy the VMs in the OpenStack Environment, configure the Bastion Host, and deploy and configure the Kubernetes clusters.

Running Deploy Command

Environmental Variables describes the list of possible environment variables that can be combined with the deploy.sh command.

Common Installation Configuration lists configuration options that can be used for customizing common services.

Run the following command from the /var/occne/cluster/${OCCNE_CLUSTER}/ directory on the Bootstrap Host. This command can take a while to run. It can take up to 2 to 4 hours depending on the machines its running on.

$ ./deploy.sh

Note:

If the deploy.sh command fails during installation, you can troubleshoot as follows:

• Check the configuration parameter values and perform reinstallation.
• Contact Oracle support for assistance.

When the CNE Cluster is deployed successfully in an OpenStack Environment, the system displays a message similar to the following:

...
-POST Post Processing Finished
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Mon Sep  4 21:10:39 UTC 2023
/var/occne/cluster/<cluster-name>/artifacts/pipeline.sh completed successfully
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

2.3.2 Installing vCNE on VMware Environment

The installation procedure details the steps necessary to configure and install CNE cluster in a VMware Environment.

2.3.2.1 Prerequisites

Before installing CNE on a VMware, ensure that the following prerequisites are met.

1. A vSphere vCenter account must be available with the following permissions:
   1. Datastore
      • Low level file operations
   2. Host
      • Configuration
        • Storage partition configuration
   3. Cns
      • Searchable
   4. Profile-driven storage
      • Profile-driven storage view
   5. Virtual machine
      • Change configuration
        • Add existing disk
        • Add or remove a device
        • Advanced configuration
2. Configure the following minimal versions of both vCloud Director and vSphere to allow correct functioning of vCNE:
   • vCloud Director (VCD) = 10.2.1
   • vSphere or vCenter = 7.0
3. A vCloud Director account with "vApp Author" and "Organization Network View" rights must be available.

   Note:

   The solution is to create a role with all the required rights and assign the account to this role. Check the vApp Author default rights in VMware Cloud Director Documentation 10.2.
4. Configure the vCloud Director environment with the backend.cloneBiosUuidOnVmCopy parameter set to 0 to ensure that VMs created from a vApp template have a different BIOS UUID than BIOS UUID of the template.
5. Configure the VMware Environment with appropriate CPU, disk, RAM, and Network required for resources allocation to the VMs.
6. Download all the necessary images, binaries, and files. Also, set up and configure the central and YUM repository. For procedures, see the Artifact Acquisition and Hosting section.
7. Users must be familiar with using VMware as a virtual provider, vCloud Director, and vSphere.
8. All necessary networking must be defined for use on the VMware.

For more information, refer to the following documentations:

• Vmware knowledge base
• vSphere CSI Driver prerequisitesvSphere CSI Driver Prerequisites

2.3.2.2 Downloading Oracle Linux X Image

This section describes the procedure to download the OLX ISO image.

Note:

The 'X' in Oracle Linux X or OLX in this procedure indicates the version of Oracle Linux supported by CNE.

Procedure

1. Open the link page https://yum.oracle.com/oracle-linux-isos.html
2. Select the Full ISO version of the image in Oracle Linux x86_64 ISOs for the release X.X (For example: 9u2) to download the image.
3. Verify the downloaded image by following the procedure in Verify Oracle Linux Downloads.

2.3.2.3 Uploading OLX Image as VMware Media

This section describes the procedure to upload OLX image as VMware media.

1. Log in to VMware GUI using your credentials.
2. Click Libraries on the top navigation bar.
3. From the left panel, select Media & Other.
4. Click Add.
5. Select ISOS from the Select a catalog drop-down.
6. Click the up arrow on Select media to upload to select a file.
7. Select the OL8 image downloaded in the Downloading Oracle Linux X Image section and click OK.

2.3.2.4 Creating a Template

This procedure describes the process to create a template.

Use this template to clone all the virtual machines (VMs) for CNE installation.

Note:

The following procedure considers creating template for OL9 and provides the configurations accordingly. The options vary for other versions.

Procedure

1. Log in to the vCloud Director Environment using your credentials.
2. Select the Virtual DataCenter where you want to create the template.
3. On the left panel, under Compute section, click vApps.
4. Click on the NEW drop-down list and select New vApp.
5. In the newly created vAPP, create a virtual machine with the following configurations:

   Type: select New.
   
   Operation System:
       OS family: Linux
       Operating System: Oracle Linux 9 (64-bit)
       Boot image: OracleLinx-R9-U2-x86_64-dvd.iso
    
   Compute
      Virtual CPUs: 1
      Cores per socket: 1
      Number of sockets: 1
     Memory: 4GB
   
   Storage:
       Size: 32GB (drop down list in 16,32,64,128etc)
   
   ADD NETWORK TO VAPP
         Check Type Direct, Select network to add
   Networking: 
   
        Click on "ADD NETWORK TO VAPP"     
             Select network, click "ADD" button. 
        Select Network from drop down list, Select Static-IP Pool from drop down list
        Click OK

6. Install Oracle Linux by performing the following steps:
   1. Select the newly created VM and click ALL ACTIONS.
   2. Select Media.
   3. Click Insert Media and select OracleLinux-R9-U2-x86_64-dvd.iso.
   4. Click INSERT at the bottom right corner.
   5. Click POWER ON.
   6. Click LAUNCH REMOTE CONSOLE, once active, to open the remote console.
   7. Follow the procedure in Oracle Linux Install Guide to install Oracle Linux on the VM template.
   8. While performing an installation, refer to the following steps for additional details on the specific configurations.
7. To install Oracle Linux X:
   1. Click Installation Source under the Selecting the Source of the Installation Image option.
   2. Select "Auto Detect installation media" under the Selecting the Source of the Installation Image option.
   3. Under Selecting the Software to Install, select Software Selection.
   4. On the Software Selection window, choose Minimal Installation as your Base Environment on the left panel. Do not select any additional software from the right panel.

      Figure 2-6 Software Selection

      
      

      

      
      
   5. For Installation Destination:

      Select Custom under Storage Configuration, and then click Done to open the Manual Partitioning screen, where you can create partitions for the mount points with assigned disk spaces.

      Figure 2-7 Manual Partitioning

      
      

      

      
      
8. Create the root password. Do not create a user account.
9. Click Begin Installation, once all the above instructions are complete.
10. Once the installation is completed, reboot the VM and log in using VMware Remote Console and root user.
11. Make sure one of the network interfaces has set the same IP that was assigned by vCloud Director.
12. From the remote console, run the following commands to configure network. If there is no IP address on the new VM, add the IP address assigned by vCloud Director on the new VM.

    ip address
    nmcli con mod <interface name> ipv4.method manual ipv4.addresses <ip-address/prefix> ipv4.gateway  connection.autoconnect yes

    Get the <interface name> from the ip address command. Get the gateway and prefix from VMware GUI: Networking → <select networking name> → Static IP Pools → Gateway CIDR

    .
13. If reaching yum.oracle.com requires a proxy, add the proxy = parameter to /etc/dnf/dnf.conf file.
14. Check that the nameservers are indicated in /etc/resolv.conf file. If empty, fill in the required values.
15. Run the following command to update all the packages to the latest versions.

    dnf update -y

16. Run the following command to install the packages required for CNE Installation.

    dnf install -y perl-interpreter cloud-utils-growpart

17. Change the following line in /etc/sudoers using visudo command as a root user:
    1. Run the following command to open the file in edit mode:

       chmod 640 /etc/sudoers

    2. Search next line and comment out the following line:

       %wheel ALL=(ALL) ALL

    3. Uncomment the following line:

       # %wheel ALL=(ALL) NOPASSWD: ALL

18. Run the following commands to enable VMware customization tools:

    vmware-toolbox-cmd config set deployPkg enable-customization true
    vmware-toolbox-cmd config set deployPkg enable-custom-scripts true

19. Run the following commands to clean the temporary files:

    dnf clean all
    logrotate -f /etc/logrotate.conf
    find /var/log/ -type f -iname '*gz' -delete
    find /var/log/ -type f -iname *$(date +%Y)* -delete
    for log in $(find /var/log/ -type f -size +0) ; do echo " " > $log ; done
    rm -rf /tmp/* /tmp/.* /var/tmp/* /var/tmp/.*

20. Remove the proxy parameter added (if any) to /etc/dnf/dnf.conf.
21. Unmount the media from the VM.
22. Power off the VM from vCloud Director.
23. Log in to vSphere vCenter and search for the name given to the created VM.
24. Right click on the VM and click Edit Settings...
25. Click the VM Options tab.
26. Expand the drop-down list Advanced.
27. Search for Configuration Parameters and select Edit Configuration...
28. Add the parameter disk.EnableUUID and set it to TRUE.
29. Go back to the vCloud Director GUI, search for the created vApp.
30. Click Actions drop down menu and click Create Template.
31. Add template to a catalog.
32. Enter a name for the template in the Name field.
33. Select Make identical copy and click OK.

    Figure 2-8 Template creation

    
    

    

    
    

A new template is stored in Libraries / vApp Templates.

2.3.2.5 Configuring Bootstrap VM

This procedure describes the process to configure a Bootstrap Virtual Machine (VM).

Procedure

1. Create a new VM from CNE Template:
   1. Log in to the VMware GUI using your credentials.
   2. Click Data Centers on the top of the page.
   3. Select Virtual Machines from the leaft panel.
   4. Click New VM and perform the following steps:
      1. Input the name of the new VM in Name.
      2. Update the name of the computer in Computer Name if you want a different name than the default one.
      3. Select the From Template option if it is not preselected by default.
      4. Click Power on if it is not preselected by default.
      5. Select the template from the available list.
      6. Click OK
2. When the VM is powered on, select the newly created VM and perform the following steps:
   1. From the left pane, select NICs.
   2. Click Edit.
   3. Select the Connected checkbox.
   4. Select the Network dropdown.
   5. From the IP Mode dropdown, select Static - IP Pool.
   6. Click Save.
3. Check if the values of Network, IP Mode are set per the values provided in the previous step. Also, check if the IP Address column displays the valid IP address.
4. Click LAUNCH REMOTE CONSOLE.
5. From the remote console, log in with the root user and password. Run the following commands to configure network. If there is no IP address on the new VM, add the IP address assigned by vCloud Director on the new VM.

   ip address
   nmcli con mod <interface name> ipv4.method manual ipv4.addresses <ip-address/prefix> ipv4.gateway  connection.autoconnect yes

   Get the <interface name> from the output of the ip address command. Get the gateway and prefix from VMware GUI: Networking → <select networking name> → Static IP Pools → Gateway CIDR.

   .
6. If reaching yum.oracle.com requires a proxy, add the proxy = parameter to /etc/dnf/dnf.conf file.
7. Run the following commands to install packages and create an user (<user-name>). The recommended value for <user-name> is cloud-user:

   dnf update -y
   dnf install -y oraclelinux-developer-release-el9 oracle-epel-release-el9
   dnf install -y rsync terraform podman python3-pip
    
    
   groupadd -g 1000 <user-name>
   useradd -g <user-name> <user-name>
   passwd <user-name> <enter new password twice>
    
   su - <user-name>
   mkdir -m700 /home/<user-name>/.ssh

8. Create the /home/<user-name>/.ssh/config file using the touch command:

   touch /home/<user-name>/.ssh/config

   Add the following content:

   ServerAliveInterval 10
   TCPKeepAlive yes
   StrictHostKeyChecking=no
   UserKnownHostsFile=/dev/null

9. Perform the following steps to set the Cluster Short-Name and Central Repository variables:
   1. Set the following cluster variables for the bootstrap environment and load them into the current environment:

      Note:

      The <cluster name> and <user-name> parameters must contain lowercase alphanumeric characters, '.' or '-', and must start with an alphanumeric character.

      echo 'export LANG=en_US.utf-8' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export LC_ALL=en_US.utf-8' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export OCCNE_VERSION=23.4.6' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export OCCNE_PREFIX=' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export OCCNE_CLUSTER=<cluster name>' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export OCCNE_USER=<user-name>' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export OCCNE_vCNE=vcd' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export CENTRAL_REPO=<central repo name>' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export CENTRAL_REPO_IP=<central repo IP address>' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export CENTRAL_REPO_REGISTRY_PORT=<central repo port>' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export OCCNE_TFVARS_DIR=/var/occne/cluster/<cluster name>/<cluster name>' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export VCD_USERNAME=<username>' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export VCD_PASSWORD=<password>' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export VCD_AUTH_URL=https://<vcd IP address>' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export VCD_ORG=<org>' | sudo tee -a  /etc/profile.d/occne.sh
      echo 'export VCD_VDC=<virtual data center>' | sudo tee -a  /etc/profile.d/occne.sh
      source /etc/profile.d/occne.sh

10. Run the following commands to create a directory specific to your cluster (using the cluster short-name):

    sudo mkdir /var/occne/
    sudo chown -R <user-name>:<user-name> /var/occne/
    mkdir -p -m 0750 /var/occne/cluster/${OCCNE_CLUSTER}/

11. Create the private and public keys that are used to access the other VMs. Run the following commands to generate the keys that are passed to the Bastion Host and used to communicate to other nodes from that Bastion Host.

    mkdir -p -m 0700 /var/occne/cluster/${OCCNE_CLUSTER}/.ssh
    ssh-keygen -m PEM -t rsa -b 2048 -f /tmp/occne_id_rsa -N ""
    cp /tmp/occne_id_rsa ~/.ssh/id_rsa
    cp /tmp/occne_id_rsa /var/occne/cluster/${OCCNE_CLUSTER}/.ssh/occne_id_rsa
    sudo chmod 600 /var/occne/cluster/${OCCNE_CLUSTER}/.ssh/occne_id_rsa
    ssh-keygen -f ~/.ssh/id_rsa -y > ~/.ssh/id_rsa.pub
    ssh-keygen -f /var/occne/cluster/${OCCNE_CLUSTER}/.ssh/occne_id_rsa -y > /var/occne/cluster/${OCCNE_CLUSTER}/.ssh/occne_id_rsa.pub

    Note:

    The private key (occne_id_rsa) must be backed-up and copied to a server that is going to be used, longer-term, to access the Bastion Host because the Bootstrap Host is transient. The user can also use an SSH client like Putty (keyGen and Pagent) to generate their own key pair and place the public key into the Bastion Host authorized_keys file to provide access.
12. Perform the following steps to copy necessary files to Bootstrap Host:
    1. Create the following directories:

       mkdir /var/occne/certificates
       mkdir /var/occne/yum.repos.d
       sudo mkdir -p /etc/containers/certs.d/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/
       sudo chown cloud-user:wheel /etc/containers/certs.d/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/
       

    2. Within these directories you must store the following mandatory files:
       1. Add a customer specific central .repo file to the /var/occne/yum.repos.d/ directory which allows for access to the customer specific repo.

          For example: winterfell-ol9.repo

       2. Add a container registry certificate to the /var/occne/certificates/ directory for the central container registry. This file must be copied using the following name format: <central_repo_hostname>:<central_repo_port>.crt.

          For example: winterfell:5000.crt

    3. Make sure the permissions of each file is at least readable (using 0644) using the sudo chmod 644 <filename> command.
    4. Create the proper directory and copy the registry certificate to it, on the bootstrap for local use:

       mkdir /var/occne/certificates
       mkdir /var/occne/yum.repos.d
       sudo mkdir -p /etc/containers/certs.d/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/
       sudo chown <user-name>:<user-name> /etc/containers/certs.d/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/
       sudo cp /var/occne/certificates/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}.crt /etc/containers/certs.d/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/ca.crt
       sudo chown <user-name>:<user-name> /etc/containers/certs.d/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/ca.crt
       chmod 644 /etc/containers/certs.d/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/ca.crt
       cp /etc/containers/certs.d/${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/ca.crt /var/occne/certificates/

    5. Update /etc/hosts file with central repository IP:

       $ echo ${CENTRAL_REPO_IP} ${CENTRAL_REPO} | sudo tee -a /etc/hosts

    6. Get the repo file from the central repository server to /var/occne/yum.repos.d. You can either curl from the server or copy the content to /var/occne/yum.repos.d:

       $ curl http://${CENTRAL_REPO_IP}/occne/repo/${CENTRAL_REPO}-ol9.repo -o /var/occne/yum.repos.d/${CENTRAL_REPO}-ol9.repo

       Copy the yum repository directory to the cluster-specific directory to use in targeted cluster nodes:

       $ echo "reposdir=/var/occne/yum.repos.d" | sudo tee -a /etc/yum.conf
       $ echo "reposdir=/var/occne/yum.repos.d/" | sudo tee -a /etc/dnf/dnf.conf
       $ cp -r /var/occne/yum.repos.d/ /var/occne/cluster/${OCCNE_CLUSTER}/

    7. Copy templates to cluster directory and update the ownership:

       mkdir /var/occne/cluster/${OCCNE_CLUSTER}/scripts
       mkdir /var/occne/cluster/${OCCNE_CLUSTER}/modules
       mkdir /var/occne/cluster/${OCCNE_CLUSTER}/${OCCNE_CLUSTER}
       podman run --rm -v /var/occne/cluster/${OCCNE_CLUSTER}:/host -v /var/occne:/var/occne:z ${CENTRAL_REPO}:${CENTRAL_REPO_REGISTRY_PORT}/occne/provision:${OCCNE_VERSION} cp -r /virtual/scripts /virtual/terraform_vcd/. /host
       

    8. Run the following command to install vcd-cli:

       sudo yum install -y gcc
       pip3 install -U vcd-cli -i http://${CENTRAL_REPO}/occne/python --trusted-host ${CENTRAL_REPO}

    9. Log in to the environment where CNE is going to be installed. Use the variables set in Step 9.

       vcd login -i -w $VCD_AUTH_URL $VCD_ORG $VCD_USERNAME -p $VCD_PASSWORD -v $VCD_VDC

2.3.2.6 Deploying CNE Cluster in VCD Environment

This section describes the procedure to deploy the CNE cluster in a VMware Cloud Director (VCD) Environment.

1. Customize the occne.ini file in the /var/occne/cluster/${OCCNE_CLUSTER}/ directory as shows below. This file contains important information about the vsphere account (different from VCD), needed to allow cluster to run.

   Note:

   The vsphere_version is a mandatory parameter. Take into consideration that external_vsphere_datacenter is not the same as VCD_VDC. Ensure that you retrieve all the information from the VCD administrator, otherwise, the cluster does not work correctly.

   Parameters not shown here can be left blank as they are auto-completed.

   [occne:vars]
   occne_cluster_name= <cluster-name>
     
   ####
   ## Auto filled by deploy.sh from values in TFVARS file
   occne_ntp_server =
   occne_cluster_network =
    
   ## Indicate DNS nameservers, comma separated
   name_server = <name_server1>,<name_server2>
    
   # Specify 'True' (case matters, no quotes) to deploy services as ClusterIP instead of LoadBalancer. Default is 'False'
   # cncc_enabled=False
    
   # Below is the default calico_mtu value. Change if needed.
   # The value should be a number, not a string.
   calico_mtu = 1500
    
   # Below is the default kube_network_node_prefix value. Change if needed.
   # Default value has room for 128 nodes. The value should be a number, not a string.
   # kube_network_node_prefix_value=25
    
   [vcd:vars]
   ## Specify the vSphere information of the External vSphere Controller/CSI Cinder plugin accounts,
   ## account must be provided by OCCNE personel and it is needed for deployment.
   ## All the rest of the information is the same as used before.
   external_vsphere_version = <6.7u3> or <7.0u1> or <7.0u2>
   external_vsphere_vcenter_ip = <Need to be completed>
   external_vsphere_vcenter_port = <Need to be completed>
   external_vsphere_insecure = <Need to be completed>
   external_vsphere_user = <Need to be completed>
   external_vsphere_password = <Need to be completed>
   external_vsphere_datacenter = <Need to be completed>
   external_vsphere_kubernetes_cluster_id = <cluster-name>
    
   # Below versions are required for vSphere CSI Driver v3.0.2
   vsphere_csi_attacher_image_tag = v4.2.0
   vsphere_csi_resizer_tag = v1.7.0
   vsphere_csi_controller = v3.0.2
   vsphere_csi_driver_image_tag = v3.0.2
   vsphere_syncer_image_tag = v3.0.2
   vsphere_csi_liveness_probe_image_tag = v2.9.0
   vsphere_csi_provisioner_image_tag = v3.4.0
   vsphere_csi_node_driver_registrar_image_tag = v2.7.0
   vsphere_csi_snapshotter_image_tag = v6.2.1
    
   # custom Cloud Controller Manager for vSphere (VCD)
   external_vsphere_cloud_controller_image_tag = v1.26.1
    
   # vCloud Director Information required for LB Controller.
   # User must have catalog author permissions + Org Network View
   # User and password must use alphanumeric characters, it can be uppercase or lowercase
   # The password cannot contain "/" or "\", neither contains or be contained in "()"
   vcd_user             = <Need to be completed>
   vcd_passwd           = <Need to be completed>
   org_name             = <Need to be completed>
   org_vdc              = <Need to be completed>
   vcd_url              = <Need to be completed>

2. Complete all the needed parameters for the cluster.tfvars file. This release supports multiple pool addresses for metallb, hence the range of the IP addresses, as well as the ports are needed to be specified in the file, along with the networks to be used, template or catalogs for VM creation, and all the parameters, to allow the process get access to VCD. Some of the parameters that are completed by default are not shown in the following sample:

   Important: Once this file has been set up, move it to /var/occne/cluster/<cluster-name>/<cluster-name>.

   # vCloud Director Information required to create resources.
   # User must have catalog author permissions + Org Network View
   # User and password must use alphanumeric characters, it can be uppercase or lowercase
   # The password cannot contain "/" or "\", neither contains or be contained in "()"
   vcd_user             = "<Need to be completed>"
   vcd_passwd           = "<Need to be completed>"
   org_name             = "<Need to be completed>"
   org_vdc              = "<Need to be completed>"
   vcd_url              = "<Need to be completed>"
   allow_unverified_ssl = true
    .
   cluster_name         = "<customer_specific_short_cluster_name>"
   # <user-name> used to deploy your cluster, should be same as $OCCNE_USER
   # Uncomment the below line only if $OCCNE_USER is NOT "cloud-user"
   # ssh_user= "<user-name>"
    
   #if affinity rules will be created set polarity to "Affinity"  if anti-affinity rules set polarity to "Anti-Affinity"
   polarity             = "<Anti-Affinity or Affinity>"
   # specify if the affinity/anti-affinity rule will be hard or soft  if hard set variable to true if soft set variable to false.
   hard_rule            = <true or false>
   # Network used for cluster communication, this network must have the feature to create SNAT and DNAT rules
   private_net_name     = "<Need to be completed>"
   # Networks used for external network communication, normally used for Bastion and LBVMs
   ext_net1_name        = "<Need to be completed>"
   ext_net2_name        = "<Need to be completed>"
   # Catalog and template name where the vApp template is stored. This template will be used for all the VM's
   catalog_name         = "<Need to be completed>"
   template_name        = "<Need to be completed>"
    
    
   # number of hosts
   number_of_bastions = 2
   number_of_k8s_ctrls_no_floating_ip = 3
   number_of_k8s_nodes = <number of worker nodes>
    
   # Amount of RAM assigned to VM's, expressed in MB
   memory_bastion = "4096"
   memory_k8s_node = "32768"
   memory_k8s_ctrl = "8192"
   memory_lbvm = "2048"# Amount of CPU assigned to VM's
   cpu_bastion = "2"
   cpu_k8s_node = "8"
   cpu_k8s_ctrl = "2"
   cpu_lbvm = "4"
    
   # Amount of cores assigned to VM's. Terraform has a bug related to templates created with a different
   # core number of the ones assigned here, it is suggested to use the same number as the template
   cores_bastion = 1cores_k8s_ctrl = 1
   cores_k8s_node = 1
   cores_lbvm = 1# Disk size, expressed in MB. Minimum disk size is 25600.
   disk_bastion = "102400"
   disk_k8s_node = "40960"
   disk_k8s_ctrl = "40960"
   disk_lbvm = "40960"
   # Update this list with the names of the pools.
   # It can take any value as an address pool name e.g. "oam", "signaling", "random_pool_name_1", etc.
   # This field should be set depending on what peer address pools the user wishes to configure.
   # eg ["oam"]  or ["oam", "signaling"] or ["oam", "signaling", "random_pool_name_1"]
   # Note : "oam" is required while other network pools are application specific.
   #
   occne_metallb_peer_addr_pool_names = ["oam"]
     
   # Use the following for creating the Metallb peer address pool object:
   #
   # (A) num_pools = number of network pools as defined in occne_metallb_peer_addr_pool_names
   #
   # (B) Configuring pool_object list.
   # Each object in this list must have only 4 input fields:
   # 1. pool_name : Its name must match an existing pool defined in occne_metallbaddr_pool_names list
   # 2. num_ports = number of ips needed for this address pool object
   # 3. Configuring 3rd input field : use only one of the three ip address input fields for
   #    each peer address pool. The other two input fields should be commented out or deleted.
   #
   # - .. cidr : A string representing a range of IPs from the same subnet/network.
   # - .. ip_list : A random list of IPs from the same subnet/network. Must be
   #                defined within brackets [].
   # - .. ip_range: A string representing a range of IPs from the same subnet/network.
   #                This range is converted to a list before input to terraform.
   #                IPs will be selected starting at the beginning of the range.
   #
   # WARNING: The cidr/list/range must include the number of IPs equal to or
   #          greater than the number of ports defined for that peer address
   #          pool.
   #
   # NOTE: Below variables are not relevant to vCloud Director, but they must have
   #       a value, for example: nework_id = net-id
   # 4. network_id : this input field specifies network id of current pool object
   # 5. subnet_id  : this input field specifies subnet id of current pool object
   #
     
   # Make sure all fields within the selected
   # input objects are set correctly.
     
     
   occne_metallb_list = {
     num_pools = 1
     pool_object = [
       {
         pool_name  = "<pool_name>"
         num_ports  = <no_of_ips_needed_for_this_addrs_pool_object>
         ip_list    = ["<ip_0>","<ip_(num_ports-1)>"]
         ip_range   = "<ip_n> - <ip_(n + num_ports - 1)>"
         cidr       = "<0.0.0.0/29>"
         subnet_id  = "<subnet UUID for the given network>"
         network_id = "<network_id>"
         egress_ip_addr = "<IP address for egress port>"
        }
     ]
   }

3. At this point, all the required components are set up:
   1. occne.ini file, with the credentials for the vSphere account.
   2. dbparams.ini file with the values needed to run cnDBTier.
   3. occne.sh profile file, with the credentials for the VCD log in datacenter and CNE version.
   4. cluster.tfvars, with the range of IP addresses, ports, VCD log in information, and other parameters needed for the cluster to run.
   5. repositories (repos), to distribute software across the clusters set up by the bootstrap ansible file or task.
   6. all the needed files for the cluster to run must be already in place.
4. Check if the network defined in the cluster.tfvars file are reflecting in the VMware GUI:
   1. From VMware GUI, go to Applications → <new vApp> → Networks.
   2. Check if the networks defined in the cluster.tfvars file (ext_net1_name/ext_net2_name) are present in the vApp.
   3. If the networks are not present, perform the following steps to add the networks:
      1. Click NEW.
      2. Under Type, select Direct.
      3. Select the network.
      4. Click ADD.
5. Run the following command from the /var/occne/cluster/${OCCNE_CLUSTER}/ directory on the Bootstrap Host. This command may take a while to run (can be up to 2 to 4 hours depending on the machines it is run on):

   $ chmod u+x deploy.sh
   $ ./deploy.sh

2.4 Postinstallation Tasks

This section explains the postinstallation tasks for CNE.

2.4.1 Verifying and Configuring Common Services

Introduction

This section describes the steps to verify and configure CNE Common services hosted on the cluster. There are various UI endpoints that are installed with common services, such as OpenSearch Dashboards, Grafana, Prometheus Server, and Alert Manager. The following sub-sections provide information about launching, verifying, and configuring the UI endpoints.

Prerequisites

1. Ensure that all the Common services are installed.
2. Gather the cluster names and version tags that are used during the installation.
3. All the commands in this section must be run on a Bastion Host.
4. Ensure you have an HTML5 compliant web browser with network connectivity to CNE.

Common Services Release Information

On successful installation, CNE generates a list of files on Bastion Host to store the release details of Kubernetes and all the common services. You can refer to the files for release information. These files get updated during the upgrade. You can locate these files in the following directory on Bastion Host:

/var/occne/cluster/${OCCNE_CLUSTER}/artifacts

Kubernetes Release File: K8S_container_images.txt

Common Services Release File: CFG_container_images.txt

Verify if OpenSearch Dashboards are Running and Accessible

1. Run the commands to get the LoadBalancer IP address and port number for occne-opensearch-dashboards Web Interface:
   1. To retrieve the LoadBalancer IP address of the occne-opensearch-dashboards service:

      $ export OSD_LOADBALANCER_IP=$(kubectl get services occne-opensearch-dashboards --namespace occne-infra -o jsonpath="{.status.loadBalancer.ingress[*].ip}")
      

   2. To retrieve the LoadBalancer port number of the occne-opensearch-dashboards service:

      $ export OSD_LOADBALANCER_PORT=$(kubectl get services occne-opensearch-dashboards --namespace occne-infra -o jsonpath="{.spec.ports[*].port}")

   3. To print the complete URL for accessing occne-opensearch-dashboards in an external browser:

      $ echo http://${OSD_LOADBALANCER_IP}:${OSD_LOADBALANCER_PORT}/${OCCNE_CLUSTER}/dashboard

      Sample output:

      http://10.75.226.78:80/my-cluster/dashboard

2. Launch the browser and navigate to OpenSearch Dashboards at http://$OSD_LOADBALANCER_IP:$OSD_LOADBALANCER_PORT/$OCCNE_CLUSTER/dashboard/app/home# (for example, http://10.75.226.78:80/my-cluster/dashboard/app/home#).
3. From the welcome screen that appears, choose OpenSearch Dashboards to navigate to OpenSearch's home screen.

Create an Index Pattern Using OpenSearch Dashboards

1. On OpenSearch Dashboards GUI, click the Hamburger icon on the top left corner to open the sidebar menu.
2. Expand the Management section and select Stack Management.
3. Select Index Patterns and click Create index pattern.
4. Enter the name of index pattern in the Index pattern name field.
5. Verify that you get a "Your index pattern matches <n> sources" message and index patterns in the occne-logstash-YYYY.MM.DD format as seen in the following image:

   Figure 2-9 Verify Index Pattern

   
   

   

   
   
6. Click Next step.
7. Select I don't want to use the time filter and click Create index pattern.
8. Ensure that the web page containing the indices appears on the main viewer frame.
9. Click the Hamburger icon on the top left corner to open the sidebar menu and select Discover.
10. Select your index from the drop-down.

    The system displays the raw log records.

11. Repeat Steps 3 to 8 using logstash* as the index patter name instead of occne-logstash* to ensure that the data is stored in OpenSearch.

Configure OpenSearch Dashboards to Save Queries

Perform the ofllowing steps to configure OpenSearch Dashboards to save search or filter queries:

1. Open the OpenSearch Dashboard GUI on a browser.
2. Navigate to Dev Tools.
3. Run the following command on the console window:

   PUT .kibana_1/_settings
   {
     "settings": {
         "index.blocks.write": false
      }
   }

   Sample output:

   "{ "acknowledged" : true }"

Verify OpenSearch Dashboards Cluster Health

1. On the OpenSearch Dashboards' home page, click Interact with the OpenSearch API to navigate to Dev Tools.
2. Enter the command GET _cluster/health and send the request by clicking the Play icon.
3. On the right panel, verify that the value of Status is green.

Verify if Prometheus Alert Manager is Accessible

1. Run the following commands to get the LoadBalancer IP address and port number for Prometheus Alertmanager web interface:
   1. Run the following command to retrieve the LoadBalancer IP address of the Alertmanager service:

      
      $ export ALERTMANAGER_LOADBALANCER_IP=$(kubectl get services occne-kube-prom-stack-kube-alertmanager --namespace occne-infra -o jsonpath="{.status.loadBalancer.ingress[*].ip}")

   2. Run the following command to retrieve the LoadBalancer port number of the Alertmanager service:

      $ export ALERTMANAGER_LOADBALANCER_PORT=$(kubectl get services occne-kube-prom-stack-kube-alertmanager --namespace occne-infra -o jsonpath="{.spec.ports[*].port}")

   3. Run the following command to print the complete URL for accessing Alertmanager in an external browser:

      $ echo http://$ALERTMANAGER_LOADBALANCER_IP:$ALERTMANAGER_LOADBALANCER_PORT/$OCCNE_CLUSTER/alertmanager

      Sample output:

      http://10.75.225.175:80/mycne-cluster/alertmanager

2. Launch the Browser and navigate to http://$ALERTMANAGER_LOADBALANCER_IP:$ALERTMANAGER_LOADBALANCER_PORT/$OCCNE_CLUSTER/alertmanager (for example, http://10.75.225.175:80/mycne-cluster/alertmanager) received in the output of the above commands. Ensure that the Alertmanager GUI is accessible.

Verify if Metrics are Scraped and Stored in Prometheus

1. Run the following commands to get the LoadBalancer IP address and port number for Prometheus Server Web Interface:
   1. Run the following command to retrieve the LoadBalancer IP address of the Prometheus service:

      $ export PROMETHEUS_LOADBALANCER_PORT=$(kubectl get services occne-kube-prom-stack-kube-prometheus --namespace occne-infra -o jsonpath="{.spec.ports[*].port}")

   2. Run the following command to retrieve the LoadBalancer port number of the Prometheus service:

      $ export PROMETHEUS_LOADBALANCER_IP=$(kubectl get services occne-kube-prom-stack-kube-prometheus --namespace occne-infra -o jsonpath="{.status.loadBalancer.ingress[*].ip}")

   3. Run the following command to print the complete URL for accessing Prometheus in an external browser:

      $ echo http://$PROMETHEUS_LOADBALANCER_IP:$PROMETHEUS_LOADBALANCER_PORT/$OCCNE_CLUSTER/prometheus

      Sample output:

      http://10.75.226.37:80/mycne-cluster/prometheus

2. Launch the browser and navigate to http://$PROMETHEUS_LOADBALANCER_IP:$PROMETHEUS_LOADBALANCER_PORT/$OCCNE_CLUSTER/prometheus (for example, http://10.75.226.37:80/mycne-cluster/prometheus) received in the output of the above commands. Ensure that the Prometheus server GUI is accessible.
3. Type up inside the Expression search query box and click "Execute. This will display a table of scrape endpoints.
4. The rightmost column corresponds to the status of the endpoints (1 indicates UP, 0 indicates DOWN). Ensure that all the scrape endpoints have the value 1, indicating that they are up and running.

Verify if Alerts are Configured Properly

1. Navigate to alerts tab of the Prometheus server GUI.

   Alternatively, you can access the Prometheus Alerts tab using the http://$PROMETHEUS_LOADBALANCER_IP:$PROMETHEUS_LOADBALANCER_PORT/$OCCNE_CLUSTER/prometheus/alerts URL. For <PROMETHEUS_LOADBALANCER_IP>

   and <PROMETHEUS_LOADBALANCER_PORT> values, refer to Step 1 of the Verify metrics are scraped and stored in Prometheus section.
2. To verify if the Alerts are configured properly, check if the following alerts are displayed in the Alerts tab of the Prometheus GUI.

   /etc/prometheus/rules/prometheus-occne-kube-prom-stack-kube-prometheus-rulefiles-0/occne-infra-occne-alerting-rules-191d7608-cd07-432c-93de-95b2ebfb0e90.yaml > BASTION_HOST_ALERTS
   BASTION_HOST_FAILED (0 active)
   ALL_BASTION_HOSTS_FAILED (0 active)
   /etc/prometheus/rules/prometheus-occne-kube-prom-stack-kube-prometheus-rulefiles-0/occne-infra-occne-alerting-rules-191d7608-cd07-432c-93de-95b2ebfb0e90.yaml > COMMON_SERVICES_STATUS_ALERTS
   PROMETHEUS_NODE_EXPORTER_NOT_RUNNING (0 active)
   OPENSEARCH_CLUSTER_HEALTH_RED (0 active)
   OPENSEARCH_CLUSTER_HEALTH_YELLOW (0 active)
   OPENSEARCH_TOO_FEW_DATA_NODES_RUNNING (0 active)
   OPENSEARCH_DOWN (0 active)
   PROMETHEUS_DOWN (0 active)
   ALERT_MANAGER_DOWN (0 active)
   SNMP_NOTIFIER_DOWN (0 active)
   JAEGER_DOWN (0 active)
   METALLB_SPEAKER_DOWN (0 active)
   METALLB_CONTROLLER_DOWN (0 active)
   GRAFANA_DOWN (0 active)
   PROMETHEUS_NO_HA (0 active)
   ALERT_MANAGER_NO_HA (0 active)
   PROMXY_METRICS_AGGREGATOR_DOWN (0 active)
   VCNE_LB_CONTROLLER_FAILED (0 active)
   VSPHERE_CSI_CONTROLLER_FAILED (0 active)
   /etc/prometheus/rules/prometheus-occne-kube-prom-stack-kube-prometheus-rulefiles-0/occne-infra-occne-alerting-rules-191d7608-cd07-432c-93de-95b2ebfb0e90.yaml > HOST_ALERTS
   DISK_SPACE_LOW (0 active)
   CPU_LOAD_HIGH (0 active)
   LOW_MEMORY (0 active)
   OUT_OF_MEMORY (0 active)
   NTP_SANITY_CHECK_FAILED (0 active)
   NETWORK_INTERFACE_FAILED (2 active)
   PVC_NEARLY_FULL (0 active)
   PVC_FULL (0 active)
   NODE_UNAVAILABLE (0 active)
   ETCD_NODE_DOWN (0 active)
   APISERVER_CERTIFICATE_EXPIRATION_90D (0 active)
   APISERVER_CERTIFICATE_EXPIRATION_30D (0 active)
   APISERVER_CERTIFICATE_EXPIRATION_7D (0 active)
   CEPH_OSD_NEARLY_FULL (0 active)
   CEPH_OSD_FULL (0 active)
   CEPH_OSD_DOWN (0 active)
   /etc/prometheus/rules/prometheus-occne-kube-prom-stack-kube-prometheus-rulefiles-0/occne-infra-occne-alerting-rules-191d7608-cd07-432c-93de-95b2ebfb0e90.yaml > LOAD_BALANCER_ALERTS
   LOAD_BALANCER_NO_HA (0 active)
   LOAD_BALANCER_NO_SERVICE (0 active)
   LOAD_BALANCER_FAILED (0 active)
   EGRESS_CONTROLLER_NOT_AVAILABLE (0 active)
   OPENSEARCH_DASHBOARD_DOWN (0 active)
   FLUENTD_OPENSEARCH_NOT_AVAILABLE (0 active)
   OPENSEARCH_DATA_PVC_NEARLY_FULL (0 active)

   If no alerts are configured, you can manually configure the alerts by creating the Prometheusrule CRD:

   $ cd /var/occne/cluster/$OCCNE_CLUSTER/artifacts/alerts
   $ kubectl apply -f occne-alerts.yaml --namespace occne-infra

Verify if Grafana is Accessible

1. Run the following commands to get the load-balancer IP address and port number for Grafana Web Interface:
   1. Retrieve the LoadBalancer IP address of the Grafana service:

      $ export GRAFANA_LOADBALANCER_IP=$(kubectl get services occne-kube-prom-stack-grafana --namespace occne-infra -o jsonpath="{.status.loadBalancer.ingress[*].ip}")

   2. Retrieve the LoadBalancer port number of the Grafana service:

      $ export GRAFANA_LOADBALANCER_PORT=$(kubectl get services occne-kube-prom-stack-grafana --namespace occne-infra -o jsonpath="{.spec.ports[*].port}")

   3. Run the following command to print the complete URL for accessing Grafana in an external browser:

      $ echo http://$GRAFANA_LOADBALANCER_IP:$GRAFANA_LOADBALANCER_PORT/$OCCNE_CLUSTER/grafana

      Sample output:

      http://10.75.225.60:80/mycne-cluster/grafana

2. Launch the browser and navigate to the http://$GRAFANA_LOADBALANCER_IP:$GRAFANA_LOADBALANCER_PORT/$OCCNE_CLUSTER/grafana URL (for example, http://10.75.225.60:80/mycne-cluster/grafana) retrieved in the output of the previous step. Ensure that the Prometheus server GUI is accessible. The default username and password is admin/admin for the first time access.
3. Log in to Grafana.
4. Click Create your first dashboard and choose + Add visualization.
5. On the Query tab, use the Data source drop-down to select Promxy as data source. Promxy is the default metrics aggregator for Prometheus time series database.

   Note:

   If Promxy is down, then use Prometheus data source temporarily to obtain metric information.
6. Select Code and enter the following query in the Enter a PromQL query... textbox:

   round((1 - (sum by(kubernetes_node, instance) (node_cpu_seconds_total{mode="idle"}) / sum by(kubernetes_node, instance)  (node_cpu_seconds_total))) * 100, 0.01)
   

7. Click Run queries.

   The susyem displays the CPU usage of all the Kubernetes nodes.

2.4.2 Performing Security Hardening

Introduction

After installation, perform an audit of the CNE system security stance before placing the system into service. The audit primarily consists of changing credentials and sequestering SSH keys to the trusted servers. The following table lists all the credentials to be checked, changed, or retained:

Note:

Refer to this section if you are performing bare metal installation.

Table 2-7 Credentials

Credential Name	Type	Associated Resource	Initial Setting	Credential Rotation
TOR Switch	username/password	Cisco Top of Rack Switch	username/password from PreFlight Checklist	Reset post-install
HP ILO Admin	username/password	HP Integrated Lights Out Manger	username/password from PreFlight Checklist	Reset post-install
Oracle ILOM user	username/password	Oracle Integrated Lights-out Manager	username/password from PreFlight Checklist	Reset post-install
Server Super User (root)	username/password	Server Super User	Set to well-known Oracle default during server installation	Reset post-install
Server Admin User (admusr)	username/password	Server Admin User	Set to well-known Oracle default during server installation	Reset post-install
Server Admin User SSH	SSH Key Pair	Server Admin User	Key Pair generated at install time	Can rotate keys at any time; key distribution manual procedure

If a factory or Oracle defaults were used for any of these credentials, they must be changed before placing the system into operation. You must then store these credentials in a safe a secure way, off site. It is recommended to plan a regular schedule for updating (rotating) these credentials.

Prerequisites

This procedure is performed after the site has been deployed and prior to placing the site into service.

Limitations and Expectations

The focus of this procedure is to secure the various credentials used or created during the install procedure. There are additional security audits that the CNE operator must perform, such as scanning repositories for vulnerabilities, monitoring the system for anomalies, regularly checking security logs. These audits are outside the scope of this post-installation procedure.

References

1. Nexus commands to configure Top of Rack switch username and password: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/security/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide_chapter_01001.html
2. See ToR switch procedure for initial username and password configuration: Configure Top of Rack 93180YC-EX Switches
3. See procedure to configure initial iLO/OA username and password: Configure Addresses for RMS iLOs

Procedure

1. Reset credentials on the TOR Switch:
   1. From the Bastion Host, log in to the switch with username and password from the procedure:

      [bastion host]# ssh <username>@<switch IP address>
      User Access Verification
      Password: <password>
       
      Cisco Nexus Operating System (NX-OS) Software
      TAC support: http://www.cisco.com/tac
      ...
      ...
      <switch name>#

   2. Change the password for current username:

      # configure
      Enter configuration commands, one per line. End with CNTL/Z.
      (config)# username <username> password <newpassword>
      (config)#exit
      #

   3. Create a new username:

      # configure
      Enter configuration commands, one per line. End with CNTL/Z.
      (config)# username <newusername> password <newpassword> role [network-operator|network-admin|vdc-admin|vdc-operator]
      (config)#exit
      #

   4. Exit from the switch and log in with the new username and password to verify if the new credentials are working:

      # exit
      Connection to <switch IP address> closed.
      [bastion host]#
       
      [some server]# ssh <newusername>@<switch IP address>
      User Access Verification
      Password: <newpassword>
       
      Cisco Nexus Operating System (NX-OS) Software
      TAC support: http://www.cisco.com/tac
      ...
      ...
      <switch name>#

   5. Delete the previous old username if it is not needed:

      # configure
      Enter configuration commands, one per line. End with CNTL/Z.
      (config)# no username <username>
      (config)#exit
      #

   6. Change the enable secret when needed:

      #
      (config)# enable secret <newenablepassword>
      (config)# exit
      #

   7. Save the above configuration:

      # copy running-config startup-config
      [########################################] 100%
      Copy complete, now saving to disk (please wait)...
      Copy complete.
      #
      

2. Reset credentials for the ILO Admin Console:
   1. From the Bastion Host, log in to the iLO with username and password from the procedure:

      [root@winterfell ~]# ssh <username>@<iLO address>
      <username>@<iLO address>'s password: <password>
      User:<username> logged-in to ...(<iLO address> / <ipv6 address>)
       
      iLO Advanced 2.61 at  Jul 27 2018
      Server Name: <server name>
      Server Power: On
       
      </>hpiLO->

   2. Change the password for current username:

      </>hpiLO-> set /map1/accounts1/<username> password=<newpassword>
       
      status=0
      status_tag=COMMAND COMPLETED
      Tue Aug 20 13:27:08 2019
       
      </>hpiLO->

   3. Create a new username:

      </>hpiLO-> create /map1/accounts1 username=<newusername> password=<newpassword> group=admin,config,oemHP_rc,oemHP_power,oemHP_vm
      status=0
      status_tag=COMMAND COMPLETED
      Tue Aug 20 13:47:56 2019
       
      User added successfully.

   4. Exit from the iLOM and log in with the new username and password to verify if the new change works:

      </>hpiLO-> exit
       
      status=0
      status_tag=COMMAND COMPLETED
      Tue Aug 20 13:30:52 2019
       
       
       
       
      CLI session stopped
      Received disconnect from <iLO address> port 22:11:  Client Disconnect
      Disconnected from <iLO address> port 22
       
      [bastion host]# ssh <newusername>@<iLO address>
      <newusername>@<iLO address>'s password: <newpassword>
      User:<newusername> logged-in to ...(<iLO address> / <ipv6 address>)
       
      iLO Advanced 2.61 at  Jul 27 2018
      Server Name: <server name>
      Server Power: On
       
      </>hpiLO->

   5. Delete the previous old username if it is not needed:

      </>hpiLO-> delete /map1/accounts1/<username>
       
      status=0
      status_tag=COMMAND COMPLETED
      Tue Aug 20 13:59:04 2019
       
      User deleted successfully.

3. Reset credentials for the Oracle ILOM user console:
   1. From the Bastion Host, log in to the iLOM with username and password:

      [admusr@bastion-2 icemark]$ ssh <username>@<iLOM address>
      Password:
       
      Oracle(R) Integrated Lights Out Manager
       
      Version 4.0.4.51 r134837
       
      Copyright (c) 2020, Oracle and/or its affiliates. All rights reserved.
       
      Warning: password is set to factory default.
       
      Warning: HTTPS certificate is set to factory default.
       
      Hostname: ORACLESP-2114XLB026
       
      ->

   2. Change current password:

      -> set /SP/users/<currentuser> password
      Enter new password: ********
      Enter new password again: ********

   3. Create a new user:

      -> create /SP/users/<username>
      Creating user...
      Enter new password: ****
      create: Non compliant password. Password length must be between 8 and 16 characters.
      Enter new password: ********
      Enter new password again: ********
      Created /SP/users/<username>

   4. Exit from the iLO and log in with the new username and password to verify if the new change works:

      -> exit
      Connection to <iLOM address> closed.
      [admusr@bastion-2 ~]$ ssh <newusername>@<iLOM address>
      Password:
       
      Oracle(R) Integrated Lights Out Manager
       
      Version 4.0.4.51 r134837
       
      Copyright (c) 2020, Oracle and/or its affiliates. All rights reserved.
       
      Warning: password is set to factory default.
       
      Warning: HTTPS certificate is set to factory default.
       
      Hostname: ORACLESP-2114XLB026

   5. Delete the previous username if not needed: (log in with different user to delete the unrequired user)

      -> delete /SP/users/<non-needed-user>
      Are you sure you want to delete /SP/users/<non-needed-user> (y/n)? y
      Deleted /SP/users/<non-needed-user>

4. Reset credentials for the root account on each server:

   Log in to each server in the cluster (ssh admusr@cluster_host) and perform the following command:

   $ sudo passwd root

5. Reset (or Delete) credentials for the admusr account on each server:

   Log in to each server in the cluster (ssh <user>@cluster_host ) and run the following command:

   $ sudo passwd -l admusr
   $ sudo passwd -l cloud-user

6. Regenerate or Redistribute SSH Keys Credentials for the admusr/cloud-user account:

   Log in to the Bastion Host VM and generate a new cluster-wide keypair as follows:

   ssh-keygen -b 4096 -t rsa -C "New SSH Key" -f .ssh/new_occne_id_rsa -q -N ""
   

   Now, for each server in the cluster, perform these actions:
   1. Copy the public key to the node:

      $ scp .ssh/new_occne_id_rsa.pub <user>@cluster_host:/var/occne/cluster/<cluster_name>/.ssh/

   2. Install the key:

      $ ssh <user>@cluster_host "cat /var/occne/cluster/<cluster_name>/.ssh/new_occne_id_rsa.pub >> /var/occne/cluster/<cluster_name>/.ssh/authorized_keys"
      

   At this point, the new key must be usable. Switch from using the old key to the new key, and confirm that each and every cluster host is still reachable. On the Bastion Host VM, perform these actions:
   1. Remove the old keys from the agent (assuming you are using an agent):

      ssh-add -D

   2. Add the new key to the agent:

      ssh-add /var/occne/cluster/<cluster_name>/.ssh/new_occne_id_rsa

   3. Confirm access to the cluster hosts and remove the old key:

      $ ssh <user>@cluster_host "sed -i '/ occne installer key$/d' /var/occne/cluster/<cluster_name>/.ssh/authorized_keys"

   4. Copy the new private key to $home/.ssh directory:

      $ cp /var/occne/cluster/<cluster_name>/.ssh/new_occne_id_rsa ~/.ssh/id_rsa

   Copy the new private key (new_occne_id_rsa) to any secondary Bastion Host VM, and definitely copied off site and securely escrowed.

2.4.3 Activating Velero

This section describes the procedure to activate Velero to enable the functionality to take on-demand backup and restore of CNE. Perform this procedure after installing CNE 23.1.x or upgrading to CNE 23.1.x to address Velero requirements. Velero gets installed as a common service in CNE. CNE configures Velero to perform all operations of backup and restore in an on-demand basis. That is, Velero doesn't perform any scheduled or triggered backups or restores.

Prerequisites

Before activating Velero, ensure that you meet the following prerequisites:

• Perfrom this procedure from the active Bastion only.
• CNE cluster must have connectivity to s3 object storage location.
• The following table provides the list of required dependencies that the system must meet before activating Velero. Failing which, the Velero activation can fail.

  Table 2-8 Required Dependencies

  Dependency	Description	CNE Variable Name
  backupStorageLocation. name	Name of the backup storage location where backups must be stored.	occne_velero_bucket_provider_name
  backupStorageLocation. provider	Name for the backup storage location provider.	occne_velero_bucket_provider
  backupStorageLocation.bucket	Name of the bucket in which the backups are stored. It's recommended to use this bucket name as your cluster name, unless you use a single bucket for several clusters.	occne_velero_bucket_name
  backupStorageLocation.config.region	Region where the bucket cloud is located. For example, if the region is minio, then set the dependency to "minio".	occne_velero_bucket_region
  backupStorageLocation.config.s3Url	URL of the Bucket s3 API.	occne_velero_bucket_s3url
  backupStorageLocation.config.publicUrl	Public URL of the Bucket s3 API. In some cases, the public URL is same as the URL.	occne_velero_bucket_public_s3url
  volumeSnapshotLocation.name	Name of the volume snapshot location where snapshots are stored.	occne_velero_volume_snapshot_location_name
  credentials.name	Name given to the created secret that contains the bucket credentials.	occne_velero_bucket_credentials_name
  credentials.secreContents.access_keyid	Key ID that is used for authentication.	occne_velero_bucket_credentials_access_key_id
  credentials.secreContents.access_accesskey	Key secret that is passed for authentication purposes.	occne_velero_bucket_credentials_secret_access_key

  Sample Dependency Values

  Refer to the following sample /var/occne/cluster/ ${OCCNE_CLUSTER}/artifacts/backup/velero.ini file and replace the values in the sample with your s3 compatible storage values:

  [occne:vars]
  occne_velero_bucket_provider_name=minio
  occne_velero_bucket_provider=aws
  occne_velero_bucket_name=default-bucket
  occne_velero_bucket_region=minio
  occne_velero_bucket_s3url=http://10.75.216.10:9000
  occne_velero_bucket_public_s3url=http://10.75.216.10:9000
  occne_velero_volume_snapshot_location_name=aws-minio
  occne_velero_bucket_credentials_name=bucket-credentials
  occne_velero_bucket_credentials_access_key_id=default-user
  occne_velero_bucket_credentials_secret_access_key=default-password

  Note:

  • Once the velero.ini file is filled with the actual values, it must be appended to hosts.ini for Bare Metal clusters or to occne.ini for VMWare or OpenStack clusters. The occne.ini or hosts.ini files are located in the /var/occne/cluster/${OCCNE_CLUSTER}/ directory.
  • Connectivity to s3 bucket is tested only on HTTP.
  • Velero takes backup restore of CNE components only.

Procedure

1. Perform the following steps to fill up the values in velero.ini:
   1. Navigate to the /var/occne/cluster/${OCCNE_CLUSTER}/artifacts/backup directory:

      cd /var/occne/cluster/${OCCNE_CLUSTER}/artifacts/backup

   2. Fill up the values for the Velero variables in velero.ini. For sample Velero variables, see Sample Dependency Values.
2. Append the velero.ini file to hosts.ini or occne.ini depending on your deployment. The occne.ini or hosts.ini file is located in the /var/occne/cluster/ ${OCCNE_CLUSTER}/ directory:
   • For bare metal clusters, run the following command to append the velero.ini file to hosts.ini:

     cat /var/occne/cluster/${OCCNE_CLUSTER}/artifacts/backup/velero.ini >> /var/occne/cluster/${OCCNE_CLUSTER}/hosts.ini

   • For OpenStack or VMware clusters, run the following command to append the velero.ini file to occne.ini:

     cat /var/occne/cluster/${OCCNE_CLUSTER}/artifacts/backup/velero.ini >> /var/occne/cluster/${OCCNE_CLUSTER}/occne.ini

3. CNE requires Boto3 python library to upload Bastion backup into S3 object storage. Not installing the library results in a cluster backup failure. Run the following command to install boto3 python library:

   $ pip3 install boto3

   If you encounter any network unreachable warning, perform the following steps to add your proxy for downloading the library and unset the setting after installation the library:

   1. Run the following commands to set your proxy:

      export http_proxy=YOUR_PROXY
      export https_proxy=$http_proxy
      export HTTP_PROXY=$http_proxy
      export HTTPS_PROXY=$http_proxy  

   2. Run the following command to install boto3 python library:

      pip3 install boto3

   3. Run the following commands to unset your proxy:

      unset HTTP_PROXY
      unset https_proxy
      unset http_proxy
      unset HTTPS_PROXY

4. Navigate to the /var/occne/cluster/ ${OCCNE_CLUSTER}/artifacts/backup/ directory and run the install_velero.sh script:

   $ cd /var/occne/cluster/${OCCNE_CLUSTER}/artifacts/backup/
   $ ./install_velero.sh

2.4.4 Activating Local DNS

This section provides information about activating Local DNS post installation.

The Local DNS feature is a reconfiguration of core DNS (CoreDNS) to support external hostname resolution. When Local DNS is enabled, CNE routes the connection to external hosts through core DNS rather than the nameservers on the Bastion Hosts. For information about activating this feature post installation, see the "Activating Local DNS" section in Oracle Communications Cloud Native Core, Cloud Native Environment User Guide.

To stop DNS forwarding to Bastion DNS, you must define the DNS details through 'A' records and SRV records. A records and SRV records are added to CNE cluster using Local DNS API calls. For more information about adding and deleting DNS records, see the "Adding and Removing DNS Records" section in Oracle Communications Cloud Native Core, Cloud Native Environment User Guide.

2.4.5 Verifying LBVM HTTP Server

This section provides information about verifying the ports in LBVM HTTP server.

CNE runs an http server service (lbvm_http_server.service) on port 8887 of each LBVM. Ensure that you don’t deploy any LoadBalancer service using the TCP port 8887 on the LB as lbvm_http_server.service listens on this port.