| Oracle® Communications OC-CNE Installation Guide Release 1.0 F16979-01 |
|
 Previous |
 Next |
Introduction
This procedure provides the steps required to initialize and configure Cisco 93180YC-EX switches as per the topology defined in Physical Network Topology Design.Note:
All instructions in this procedure are executed from the Bootstrap Host.All steps are executed from a Keyboard, Video, Mouse (KVM) connection.
Procedures
Table 3-3 Procedure to configure Top of Rack 93180YC-EX Switches
| Step # | Procedure | Description |
|---|---|---|
| 1.
|
Login to the Bootstrap host as root. | Using the KVM, login to the Bootstrap host as
root.
Note: All instructions in this procedure are executed from the Bootstrap Host. |
| 2.
|
Insert and mount the Utility USB | Insert and mount the Utility USB that
contains the configuration and script files. Verify the files are listed in the
USB using the ls /media/usb command.
Note: Instructions for mounting the USB can be found in: OCCNE Installation of Oracle Linux 7.5 on Bootstrap Server : Install Additional Packages. Only steps 2 and 3 need to be followed in that procedure. |
| 3.
|
Create bridge interface | Create bridge interface to connect both
management ports and setup the management bridge to support switch
initialization.
Note: <CNE_Management_IP_With_Prefix> is from OCCNE 1.0 Installation PreFlight Checklist : Complete Site Survey Host IP Table. Row 1 CNE Management IP Addresess (VLAN 4) column. <ToRSwitch_CNEManagementNet_VIP> is from OCCNE 1.0 Installation PreFlight Checklist : Complete OA and Switch IP Table. $ nmcli con add con-name mgmtBridge type bridge ifname mgmtBridge $ nmcli con add type bridge-slave ifname eno2 master mgmtBridge $ nmcli con add type bridge-slave ifname eno3 master mgmtBridge $ nmcli con mod mgmtBridge ipv4.method manual ipv4.addresses 192.168.2.11/24 $ nmcli con up mgmtBridge $ nmcli con add type team con-name team0 ifname team0 team.runner lacp $ nmcli con add type team-slave con-name team0-slave-1 ifname eno5 master team0 $ nmcli con add type team-slave con-name team0-slave-2 ifname eno6 master team0 $ nmcli con mod team0 ipv4.method manual ipv4.addresses 172.16.3.4/24 $ nmcli con add con-name team0.4 type vlan id 4 dev team0 $ nmcli con mod team0.4 ipv4.method manual ipv4.addresses <CNE_Management_IP_Address_With_Prefix> ipv4.gateway <ToRswitch_CNEManagementNet_VIP> $nmcli con up team0.4 |
| 4.
|
Edit the /etc/xinetd.d/tftp file |
Edit the /etc/xinetd.d/tftp file to enable TFTP service. Change the disable option to no, if it is set to yes. $ vi /etc/xinetd.d/tftp
# default: off
# description: The tftp server serves files using the trivial file transfer \
# protocol. The tftp protocol is often used to boot diskless \
# workstations, download configuration files to network-aware printers, \
# and to start the installation process for some operating systems.
service tftp
{
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -s /var/lib/tftpboot
disable = no
per_source = 11
cps = 100 2
flags = IPv4
}
|
| 5.
|
Enable tftp on the Bootstrap host. |
$ systemctl start tftp $ systemctl enable tftp Verify tftp is active and enabled: $ systemctl status tftp $ ps -elf | grep tftp |
| 6.
|
Copy the dhcpd.conf file |
Copy the dhcpd.conf file from the Utility USB in OCCNE 1.0 Installation PreFlight checklist : Create the dhcpd.conf File to the /etc/dhcp/ directory. $ cp /media/usb/dhcpd.conf /etc/dhcp/ |
| 7.
|
Restart and enable dhcpd service. |
$ /bin/systemctl restart dhcpd.service $ /bin/systemctl enable dhcpd.service Use the systemctl status dhcpd command to verify active and enabled. $ systemctl status dhcpd |
| 8.
|
Copy the switch configuration and script files |
Copy the switch configuration and script files from the Utility USB to directory /var/lib/tftpboot/. $ cp /media/usb/93180_switchA.cfg /var/lib/tftpboot/. $ cp /media/usb/93180_switchB.cfg /var/lib/tftpboot/. $ cp /media/usb/poap_nexus_script.py /var/lib/tftpboot/. |
| 9.
|
Copy the ifcfg template files |
Copy the ifcfg template files to /tmp directory for later use. $ cp /media/usb/ifcfg-vlan /tmp $ cp /media/usb/ifcfg-bridge /tmp |
| 10.
|
Modify POAP script File. |
Modify POAP script File. Make the following change for the first server information: The username and password are the credentials used to login to the Bootstrap host.
$ vi /var/lib/tftpboot/poap_nexus_script.py
Host name and user credentials
options = {
"username": "<username>",
"password": "<password>",
"hostname": "192.168.2.11",
"transfer_protocol": "scp",
"mode": "serial_number",
"target_system_image": "nxos.9.2.3.bin",
}
Note: The version nxos.9.2.3.bin is used by default. If different version is to be used, modify the "target_system_image" with new version.
|
| 11.
|
Modify POAP script file |
Modify POAP script file md5sum by executing the md5Poap.sh script from the Utility USB created from OCCNE 1.0 Installation PreFlight checklist : Create the md5Poap Bash Script. $ cd /var/lib/tftpboot/ $ /bin/bash md5Poap.sh |
| 12.
|
Create the files necessary to configure the ToR switches using the serial number from the switch. | The serial number is located on a pullout
card on the back of the switch in the left most power supply of the switch.
 |
| 13.
|
Copy the /var/lib/tftpboot/93180_switchA.cfg into a file called /var/lib/tftpboot/conf.<switchA serial number> | Modify the switch specific values in the
/var/lib/tftpboot/conf.<switchA serial number> file, including all the
values in the curly braces as following code block.
These values are contained at OCCNE 1.0 Installation PreFlight checklist : ToR and Enclosure Switches Variables Table (Switch Specific) and OCCNE 1.0 Installation PreFlight Checklist : Complete OA and Switch IP Table. Modify these values with the following sed commands, or use an editor such as vi etc. $ sed -i 's/{switchname}/<switch_name>/' conf.<switchA serial number>
$ sed -i 's/{admin_password}/<admin_password>/' conf.<switchA serial number>
$ sed -i 's/{user_name}/<user_name>/' conf.<switchA serial number>
$ sed -i 's/{user_password}/<user_password>/' conf.<switchA serial number>
$ sed -i 's/{ospf_md5_key}/<ospf_md5_key>/' conf.<switchA serial number>
$ sed -i 's/{OSPF_AREA_ID}/<ospf_area_id>/' conf.<switchA serial number>
$ sed -i 's/{NTPSERVER1}/<NTP_server_1>/' conf.<switchA serial number>
$ sed -i 's/{NTPSERVER2}/<NTP_server_2>/' conf.<switchA serial number>
$ sed -i 's/{NTPSERVER3}/<NTP_server_3>/' conf.<switchA serial number>
$ sed -i 's/{NTPSERVER4}/<NTP_server_4>/' conf.<switchA serial number>
$ sed -i 's/{NTPSERVER5}/<NTP_server_5>/' conf.<switchA serial number>
Note: If less than 5 ntp servers available, delete the extra ntp server lines such as command:
$ sed -i 's/{NTPSERVER5}/d' conf.<switchA serial number>
Note: different delimiter is used in next two commands due to '/' sign in the variables
$ sed -i 's#{ALLOW_5G_XSI_LIST_WITH_PREFIX_LEN}#<MetalLB_Signal_Subnet_With_Prefix>#g' conf.<switchA serial number>
$ sed -i 's#{CNE_Management_SwA_Address}#<ToRswitchA_CNEManagementNet_IP>#g' conf.<switchA serial number>
$ sed -i 's#{CNE_Management_SwB_Address}#<ToRswitchB_CNEManagementNet_IP>#g' conf.<switchA serial number>
$ sed -i 's#{CNE_Management_Prefix}#<CNEManagementNet_Prefix>#g' conf.<switchA serial number>
$ sed -i 's#{SQL_replication_SwA_Address}#<ToRswitchA_SQLreplicationNet_IP>#g' conf.<switchA serial number>
$ sed -i 's#{SQL_replication_SwB_Address}#<ToRswitchB_SQLreplicationNet_IP>#g' conf.<switchA serial number>
$ sed -i 's#{SQL_replication_Prefix}#<SQLreplicationNet_Prefix>#g' conf.<switchA serial number>
$ ipcalc -n <ToRswitchA_SQLreplicationNet_IP/<SQLreplicationNet_Prefix> | awk -F'=' '{print $2}'
$ sed -i 's/{SQL_replication_Subnet}/<output from ipcalc command as SQL_replication_Subnet>/' conf.<switchA serial number>
$ sed -i 's/{CNE_Management_VIP}/<ToRswitch_CNEManagementNet_VIP>/g' conf.<switchA serial number>
$ sed -i 's/{SQL_replication_VIP}/<ToRswitch_SQLreplicationNet_VIP>/g' conf.<switchA serial number>
$ sed -i 's/{OAM_UPLINK_CUSTOMER_ADDRESS}/<ToRswitchA_oam_uplink_customer_IP>/' conf.<switchA serial number>
$ sed -i 's/{OAM_UPLINK_SwA_ADDRESS}/<ToRswitchA_oam_uplink_IP>/g' conf.<switchA serial number>
$ sed -i 's/{SIGNAL_UPLINK_SwA_ADDRESS}/<ToRswitchA_signaling_uplink_IP>/g' conf.<switchA serial number>
$ sed -i 's/{OAM_UPLINK_SwB_ADDRESS}/<ToRswitchB_oam_uplink_IP>/g' conf.<switchA serial number>
$ sed -i 's/{SIGNAL_UPLINK_SwB_ADDRESS}/<ToRswitchB_signaling_uplink_IP>/g' conf.<switchA serial number>
$ ipcalc -n <ToRswitchA_signaling_uplink_IP>/30 | awk -F'=' '{print $2}'
$ sed -i 's/{SIGNAL_UPLINK_SUBNET}/<output from ipcalc command as signal_uplink_subnet>/' conf.<switchA serial number>
$ ipcalc -n <ToRswitchA_SQLreplicationNet_IP> | awk -F'=' '{print $2}'
$ sed -i 's/{MySQL_Replication_SUBNET}/<output from the above ipcalc command appended with prefix >/' conf.<switchA serial number>
Note: The version nxos.9.2.3.bin is used by default and hard-coded in the conf files. If different version is to be used, run the following command:
$ sed -i 's/nxos.9.2.3.bin/<nxos_version>/' conf.<switchA serial number>
Note: access-list Restrict_Access_ToR
The following line allow one access server to access the switch management and SQL vlan addresses while other accesses are denied. If no need, delete this line. If need more servers, add similar line.
$ sed -i 's/{Allow_Access_Server}/<Allow_Access_Server>/' conf.<switchA serial number>
|
| 14.
|
Copy the /var/lib/tftpboot/93180_switchB.cfg into a file called /var/lib/tftpboot/conf.<switchB serial number> |
Modify the switch specific values in the /var/lib/tftpboot/conf.<switchA serial number> file, including: hostname, username/password, oam_uplink IP address, signaling_uplink IP address, access-list ALLOW_5G_XSI_LIST permit address, prefix-list ALLOW_5G_XSI. These values are contained at OCCNE 1.0 Installation PreFlight checklist : ToR and Enclosure Switches Variables Table and OCCNE 1.0 Installation PreFlight Checklist : Complete OA and Switch IP Table. $ sed -i 's/{switchname}/<switch_name>/' conf.<switchB serial number>
$ sed -i 's/{admin_password}/<admin_password>/' conf.<switchB serial number>
$ sed -i 's/{user_name}/<user_name>/' conf.<switchB serial number>
$ sed -i 's/{user_password}/<user_password>/' conf.<switchB serial number>
$ sed -i 's/{ospf_md5_key}/<ospf_md5_key>/' conf.<switchB serial number>
$ sed -i 's/{OSPF_AREA_ID}/<ospf_area_id>/' conf.<switchB serial number>
$ sed -i 's/{NTPSERVER1}/<NTP_server_1>/' conf.<switchB serial number>
$ sed -i 's/{NTPSERVER2}/<NTP_server_2>/' conf.<switchB serial number>
$ sed -i 's/{NTPSERVER3}/<NTP_server_3>/' conf.<switchB serial number>
$ sed -i 's/{NTPSERVER4}/<NTP_server_4>/' conf.<switchB serial number>
$ sed -i 's/{NTPSERVER5}/<NTP_server_5>/' conf.<switchB serial number>
Note: If less than 5 ntp servers available, delete the extra ntp server lines such as command:
$ sed -i 's/{NTPSERVER5}/d' conf.<switchB serial number>
Note: different delimiter is used in next two commands due to '/' sign in in the variables
$ sed -i 's#{ALLOW_5G_XSI_LIST_WITH_PREFIX_LEN}#<MetalLB_Signal_Subnet_With_Prefix>#g' conf.<switchB serial number>
$ sed -i 's#{CNE_Management_SwA_Address}#<ToRswitchA_CNEManagementNet_IP>#g' conf.<switchB serial number>
$ sed -i 's#{CNE_Management_SwB_Address}#<ToRswitchB_CNEManagementNet_IP>#g' conf.<switchB serial number>
$ sed -i 's#{CNE_Management_Prefix}#<CNEManagementNet_Prefix>#g' conf.<switchB serial number>
$ sed -i 's#{SQL_replication_SwA_Address}#<ToRswitchA_SQLreplicationNet_IP>#g' conf.<switchB serial number>
$ sed -i 's#{SQL_replication_SwB_Address}#<ToRswitchB_SQLreplicationNet_IP>#g' conf.<switchB serial number>
$ sed -i 's#{SQL_replication_Prefix}#<SQLreplicationNet_Prefix>#g' conf.<switchB serial number>
$ ipcalc -n <ToRswitchB_SQLreplicationNet_IP/<SQLreplicationNet_Prefix> | awk -F'=' '{print $2}'
$ sed -i 's/{SQL_replication_Subnet}/<output from ipcalc command as SQL_replication_Subnet>/' conf.<switchB serial number>
$ sed -i 's/{CNE_Management_VIP}/<ToRswitch_CNEManagementNet_VIP>/' conf.<switchB serial number>
$ sed -i 's/{SQL_replication_VIP}/<ToRswitch_SQLreplicationNet_VIP>/' conf.<switchB serial number>
$ sed -i 's/{OAM_UPLINK_CUSTOMER_ADDRESS}/<ToRswitchB_oam_uplink_customer_IP>/' conf.<switchB serial number>
$ sed -i 's/{OAM_UPLINK_SwA_ADDRESS}/<ToRswitchB_oam_uplink_IP>/g' conf.<switchB serial number>
$ sed -i 's/{SIGNAL_UPLINK_SwA_ADDRESS}/<ToRswitchB_signaling_uplink_IP>/g' conf.<switchB serial number>
$ sed -i 's/{OAM_UPLINK_SwB_ADDRESS}/<ToRswitchB_oam_uplink_IP>/g' conf.<switchB serial number>
$ sed -i 's/{SIGNAL_UPLINK_SwB_ADDRESS}/<ToRswitchB_signaling_uplink_IP>/g' conf.<switchB serial number>
$ ipcalc -n <ToRswitchB_signaling_uplink_IP>/30 | awk -F'=' '{print $2}'
$ sed -i 's/{SIGNAL_UPLINK_SUBNET}/<output from ipcalc command as signal_uplink_subnet>/' conf.<switchB serial number>
Note: The version nxos.9.2.3.bin is used by default and hard-coded in the conf files. If different version is to be used, run the following command:
$ sed -i 's/nxos.9.2.3.bin/<nxos_version>/' conf.<switchB serial number>
Note: access-list Restrict_Access_ToR
The following line allow one access server to access the switch management and SQL vlan addresses while other accesses are denied. If no need, delete this line. If need more servers, add similar line.
$ sed -i 's/{Allow_Access_Server}/<Allow_Access_Server>/' conf.<switchB serial number>
|
| 15.
|
Generate the md5 checksum for each conf file in /var/lib/tftpboot and copy that into a new file called conf.<switchA/B serial number>.md5. |
$ md5sum conf.<switchA serial number> > conf.<switchA serial number>.md5 $ md5sum conf.<switchB serial number> > conf.<switchB serial number>.md5 |
| 16.
|
Verify the /var/lib/tftpboot directory has the correct files. Make sure the file permissions are set as given below. |
Note: The ToR switches are
constantly attempting to find and execute the poap_nexus_script.py script which
uses tftp to load and install the configuration files.
$ ls -l /var/lib/tftpboot/ total 1305096 -rw-r--r--. 1 root root 7161 Mar 25 15:31 conf.<switchA serial number> -rw-r--r--. 1 root root 51 Mar 25 15:31 conf.<switchA serial number>.md5 -rw-r--r--. 1 root root 7161 Mar 25 15:31 conf.<switchB serial number> -rw-r--r--. 1 root root 51 Mar 25 15:31 conf.<switchB serial number>.md5 -rwxr-xr-x. 1 root root 75856 Mar 25 15:32 poap_nexus_script.py |
| 17.
|
Disable firewalld. |
$ systemctl stop firewalld $ systemctl disable firewalld To verify: $ systemctl status firewalld Once this is complete, the ToR Switches will attempt to boot from the tftpboot files automatically. Eventually the verification steps can be executed below. It may take about 5 minutes for this to complete. |
Table 3-4 Procedure to verify Top of Rack 93180YC-EX Switches
| Step # | Procedure | Description |
|---|---|---|
| 1.
|
After the ToR switches configured, ping the switches from bootstrap server. The switches mgmt0 interfaces are configured with the IP addresses which are in the conf files. | Note: Wait till the device
responds.
$ ping 192.168.2.1 PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data. 64 bytes from 192.168.2.1: icmp_seq=1 ttl=255 time=0.419 ms 64 bytes from 192.168.2.1: icmp_seq=2 ttl=255 time=0.496 ms 64 bytes from 192.168.2.1: icmp_seq=3 ttl=255 time=0.573 ms 64 bytes from 192.168.2.1: icmp_seq=4 ttl=255 time=0.535 ms ^C --- 192.168.2.1 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3000ms rtt min/avg/max/mdev = 0.419/0.505/0.573/0.063 ms $ ping 192.168.2.2 PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data. 64 bytes from 192.168.2.2: icmp_seq=1 ttl=255 time=0.572 ms 64 bytes from 192.168.2.2: icmp_seq=2 ttl=255 time=0.582 ms 64 bytes from 192.168.2.2: icmp_seq=3 ttl=255 time=0.466 ms 64 bytes from 192.168.2.2: icmp_seq=4 ttl=255 time=0.554 ms ^C --- 192.168.2.2 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3001ms rtt min/avg/max/mdev = 0.466/0.543/0.582/0.051 ms |
| 2.
|
Attempt to ssh to the switches with the username/password provided in the conf files. |
$ ssh plat@192.168.2.1 The authenticity of host '192.168.2.1 (192.168.2.1)' can't be established. RSA key fingerprint is SHA256:jEPSMHRNg9vejiLcEvw5qprjgt+4ua9jucUBhktH520. RSA key fingerprint is MD5:02:66:3a:c6:81:65:20:2c:6e:cb:08:35:06:c6:72:ac. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.2.1' (RSA) to the list of known hosts. User Access Verification Password: Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (C) 2002-2019, Cisco and/or its affiliates. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under their own licenses, such as open source. This software is provided "as is," and unless otherwise stated, there is no warranty, express or implied, including but not limited to warranties of merchantability and fitness for a particular purpose. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or GNU General Public License (GPL) version 3.0 or the GNU Lesser General Public License (LGPL) Version 2.1 or Lesser General Public License (LGPL) Version 2.0. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://opensource.org/licenses/gpl-3.0.html and http://www.opensource.org/licenses/lgpl-2.1.php and http://www.gnu.org/licenses/old-licenses/library.txt. # |
| 3.
|
Verify the running-config has all expected configurations in the conf file using the show running-config command. |
$ show running-config !Command: show running-config !Running configuration last done at: Mon Apr 8 17:39:38 2019 !Time: Mon Apr 8 18:30:17 2019 version 9.2(3) Bios:version 07.64 hostname 12006-93108A vdc 12006-93108A id 1 limit-resource vlan minimum 16 maximum 4094 limit-resource vrf minimum 2 maximum 4096 limit-resource port-channel minimum 0 maximum 511 limit-resource u4route-mem minimum 248 maximum 248 limit-resource u6route-mem minimum 96 maximum 96 limit-resource m4route-mem minimum 58 maximum 58 limit-resource m6route-mem minimum 8 maximum 8 feature scp-server feature sftp-server cfs eth distribute feature ospf feature bgp feature interface-vlan feature lacp feature vpc feature bfd feature vrrpv3 .... .... |
| 4.
|
Un-mount the Utility USB and remove it: umount /media/usb | Connect or enable customer uplink. |
| 5.
|
Verify the RMS1 can ping the CNE_Management VIP |
$ ping <ToRSwitch_CNEManagementNet_VIP> PING <ToRSwitch_CNEManagementNet_VIP> (<ToRSwitch_CNEManagementNet_VIP>) 56(84) bytes of data. 64 bytes from <ToRSwitch_CNEManagementNet_VIP>: icmp_seq=2 ttl=255 time=1.15 ms 64 bytes from <ToRSwitch_CNEManagementNet_VIP>: icmp_seq=3 ttl=255 time=1.11 ms 64 bytes from <ToRSwitch_CNEManagementNet_VIP>: icmp_seq=4 ttl=255 time=1.23 ms ^C --- 10.75.207.129 ping statistics --- 4 packets transmitted, 3 received, 25% packet loss, time 3019ms rtt min/avg/max/mdev = 1.115/1.168/1.237/0.051 ms |
| 6.
|
Verify the RMS1 can be accessed from laptop. Use application such as putty etc to ssh to RMS1. |
$ ssh root@<CNE_Management_IP_Address> Using username "root". root@<CNE_Management_IP_Address>'s password:<root password> Last login: Mon May 6 10:02:01 2019 from 10.75.9.171 [root@RMS1 ~]# |
