Appendix B Cabling Reference

This appendix provides a reference for the cabling between components of Oracle Private Cloud Appliance.

B.1 Cisco Nexus 9348GC-FXP Switch Management Ethernet Connections

This section lists the cabling connections for the Cisco Nexus 9348GC-FXP Switch. They are Ethernet connections intended for the internal administration network.

Table B.1 Cabling Connections for Cisco Nexus 9348GC-FXP Switch

From Rack Unit

From Component and Port

To Cisco Nexus 9348GC-FXP Switch Rack Unit and Port

U42

compute node: NET0

U21: port 23

U41

compute node: NET0

U21: port 22

U40

compute node: NET0

U21: port 21

U39

compute node: NET0

U21: port 20

U38

compute node: NET0

U21: port 19

U37

compute node: NET0

U21: port 18

U36

compute node: NET0

U21: port 17

U35

compute node: NET0

U21: port 16

U34

compute node: NET0

U21: port 15

U33

compute node: NET0

U21: port 42

U32

compute node: NET0

U21: port 41

U31

compute node: NET0

U21: port 40

U30

compute node: NET0

U21: port 39

U29

compute node: NET0

U21: port 38

U28

compute node: NET0

U21: port 37

U27

compute node: NET0

U21: port 36

U26

compute node: NET0

U21: port 35

U23

Cisco Nexus 9336C-FX2 Switch: NET MGMT

U21: port 14

U22

Cisco Nexus 9336C-FX2 Switch: NET MGMT

U21: port 13

U16

Cisco Nexus 9336C-FX2 Switch: NET MGMT

U21: port 34

U15

Cisco Nexus 9336C-FX2 Switch: NET MGMT

U21: port 33

U14

compute node: NET0

U21: port 12

U13

compute node: NET0

U21: port 11

U12

compute node: NET0

U21: port 10

U11

compute node: NET0

U21: port 9

U10

compute node: NET0

U21: port 8

U09

compute node: NET0

U21: port 7

U08

compute node: NET0

U21: port 6

U07

compute node: NET0

U21: port 5

U06

management node: NET0

U21: port 32

U05

management node: NET0

U21: port 31

U03

ZFS storage appliance controller: NET0

U21: port 28

U03

ZFS storage appliance controller: PCIe 5/0

U21: port 29

U03

ZFS storage appliance controller: PCIe 5/1

U21: port 30

U01

ZFS storage appliance controller: NET0

U21: port 25

U01

ZFS storage appliance controller: PCIe 5/0

U21: port 26

U01

ZFS storage appliance controller: PCIe 5/1

U21: port 27

N/A

PDU-A

U21: port 24

N/A

PDU-B

U21: port 43

N/A

Oracle Support

U21: port 1


B.2 Cisco Nexus 9336C-FX2 Switch Data Network Connections

This section lists the cabling connections for the Cisco Nexus 9336C-FX2 Switch. They are high speed Ethernet connections up to 100Gbit, intended for the internal data network.

Table B.2 Cabling Connections for Cisco Nexus 9336C-FX2 Switch

From Rack Unit

From Component and Port

To Cisco Nexus 9336C-FX2 Switch Rack Unit and Port

U42

compute node: CX-5 HCA port 1

U15: port 31

U42

compute node: CX-5 HCA port 2

U16: port 31

U41

compute node: CX-5 HCA port 1

U15: port 30

U41

compute node: CX-5 HCA port 2

U16: port 30

U40

compute node: CX-5 HCA port 1

U15: port 29

U40

compute node: CX-5 HCA port 2

U16: port 29

U39

compute node: CX-5 HCA port 1

U15: port 28

U39

compute node: CX-5 HCA port 2

U16: port 28

U38

compute node: CX-5 HCA port 1

U15: port 27

U38

compute node: CX-5 HCA port 2

U16: port 27

U37

compute node: CX-5 HCA port 1

U15: port 26

U37

compute node: CX-5 HCA port 2

U16: port 26

U36

compute node: CX-5 HCA port 1

U15: port 25

U36

compute node: CX-5 HCA port 2

U16: port 25

U35

compute node: CX-5 HCA port 1

U15: port 24

U35

compute node: CX-5 HCA port 2

U16: port 24

U34

compute node: CX-5 HCA port 1

U15: port 23

U34

compute node: CX-5 HCA port 2

U16: port 23

U33

compute node: CX-5 HCA port 1

U15: port 22

U33

compute node: CX-5 HCA port 2

U16: port 22

U32

compute node: CX-5 HCA port 1

U15: port 21

U32

compute node: CX-5 HCA port 2

U16: port 21

U31

compute node: CX-5 HCA port 1

U15: port 20

U31

compute node: CX-5 HCA port 2

U16: port 20

U30

compute node: CX-5 HCA port 1

U15: port 19

U30

compute node: CX-5 HCA port 2

U16: port 19

U29

compute node: CX-5 HCA port 1

U15: port 18

U29

compute node: CX-5 HCA port 2

U16: port 18

U28

compute node: CX-5 HCA port 1

U15: port 17

U28

compute node: CX-5 HCA port 2

U16: port 17

U27

compute node: CX-5 HCA port 1

U15: port 16

U27

compute node: CX-5 HCA port 2

U16: port 16

U26

compute node: CX-5 HCA port 1

U15: port 15

U26

compute node: CX-5 HCA port 2

U16: port 15

U14

compute node: CX-5 HCA port 1

U15: port 14

U14

compute node: CX-5 HCA port 2

U16: port 14

U13

compute node: CX-5 HCA port 1

U15: port 13

U13

compute node: CX-5 HCA port 2

U16: port 13

U12

compute node: CX-5 HCA port 1

U15: port 12

U12

compute node: CX-5 HCA port 2

U16: port 12

U11

compute node: CX-5 HCA port 1

U15: port 11

U11

compute node: CX-5 HCA port 2

U16: port 11

U10

compute node: CX-5 HCA port 1

U15: port 10

U10

compute node: CX-5 HCA port 2

U16: port 10

U09

compute node: CX-5 HCA port 1

U15: port 9

U09

compute node: CX-5 HCA port 2

U16: port 9

U08

compute node: CX-5 HCA port 1

U15: port 8

U08

compute node: CX-5 HCA port 2

U16: port 8

U07

compute node: CX-5 HCA port 1

U15: port 7

U07

compute node: CX-5 HCA port 2

U16: port 7

U06

management node: CX-5 HCA port 1

U15: port 6

U06

management node: CX-5 HCA port 2

U16: port 6

U05

management node: CX-5 HCA port 1

U15: port 5

U05

management node: CX-5 HCA port 2

U16: port 5

U04

Oracle ZFS Storage Appliance ZS7-2 controller: slot 4 port 1

U22: port 35

U04

Oracle ZFS Storage Appliance ZS7-2 controller: slot 4 port 2

U23: port 35

U03

Oracle ZFS Storage Appliance ZS7-2 controller: slot 8 port 1

U22: port 36

U03

Oracle ZFS Storage Appliance ZS7-2 controller: slot 8 port 2

U23: port 36

U02

Oracle ZFS Storage Appliance ZS7-2 controller: slot 4 port 1

U22: port 33

U02

Oracle ZFS Storage Appliance ZS7-2 controller: slot 4 port 2

U23: port 33

U01

Oracle ZFS Storage Appliance ZS7-2 controller: slot 8 port 1

U22: port 34

U01

Oracle ZFS Storage Appliance ZS7-2 controller: slot 8 port 2

U23: port 34


B.3 Data and Spine Switch Interconnects

This section lists the cabling connections between the Cisco Nexus 9348GC-FXP Switches performing the role of either data switch or spine switch. They are 100GbE Ethernet connections intended for high bandwidth traffic between the data and spine switches.

B.3.1 Spine Switch to Data Switch Connections

Table B.3 lists the cabling connections between the data switches and spine switches.

Table B.3 Cabling Connections Between Spine and Data Switches

From Rack Unit

From Component and Port

To Rack Unit

To Component and Port

U22

spine switch: port 23

U15

data switch: port 33

U22

spine switch: port 24

U15

data switch: port 34

U22

spine switch: port 25

U16

data switch: port 35

U22

spine switch: port 26

U16

data switch: port 36

U23

spine switch: port 23

U16

data switch: port 33

U23

spine switch: port 24

U16

data switch: port 34

U23

spine switch: port 25

U15

data switch: port 35

U23

spine switch: port 26

U15

data switch: port 36


B.3.2 Data Switch to Data Switch Connections

Table B.4 lists the cabling connections between the data switches.

Table B.4 Cabling Connections Between Data Switches

From Rack Unit

From Component and Port

To Rack Unit

To Component and Port

U15

data switch: port 3

U16

data switch: port 3

U15

data switch: port 4

U16

data switch: port 4


B.3.3 Spine Switch to Spine Switch Connections

Table B.5 lists the cabling connections between the spine switches.

Table B.5 Cabling Connections Between Spine Switches

From Rack Unit

From Component and Port

To Rack Unit

To Component and Port

U22

spine switch: port 28

U23

spine switch: port 28

U22

spine switch: port 29

U23

spine switch: port 29

U22

spine switch: port 30

U23

spine switch: port 30

U22

spine switch: port 31

U23

spine switch: port 31

U22

spine switch: port 32

U23

spine switch: port 32


B.4 Oracle Switch ES1-24 Ethernet Connections

This section lists the cabling connections for Oracle Switch ES1-24.

Table B.6 Cabling Connections for Oracle Switch ES1-24

From

Port

To

Port

Color

Notes

N/A

SER MGT

U21-A

Serial Port

1.8m console cable

7061953 cable accessible for Service only

N/A

SER MGT

U21-B

Serial Port

1.8m console cable

7061953 cable accessible for Service only

U42

NET0

U21-A

19

Blue

U41

NET0

U21-A

18

Blue

U40

NET0

U21-A

17

Blue

U39

NET0

U21-A

16

Blue

U38

NET0

U21-A

15

Blue

U37

NET0

U21-A

14

Blue

U36

NET0

U21-A

13

Blue

U35

NET0

U21-A

12

Blue

U34

NET0

U21-A

11

Blue

U33

NET0

U21-B

18

Blue

U32

NET0

U21-B

17

Blue

U31

NET0

U21-B

16

Blue

U30

NET0

U21-B

15

Blue

U29

NET0

U21-B

14

Blue

U28

NET0

U21-B

13

Blue

U27

NET0

U21-B

12

Blue

U26

NET0

U21-B

11

Blue

U22

U21-A

9

Black

Oracle Fabric Interconnect F1-15

U20

U21-B

10

Black

NM2-36P Sun Datacenter InfiniBand Expansion Switch

U21-A

24

Customer network

Transceiver/Black Cable

U21-B

24

Customer network

Transceiver/Black Cable

U21-A

22

U21-B

22

SFP+

Switch-to-switch connection within rack

U19

U21-A

10

Black

NM2-36P Sun Datacenter InfiniBand Expansion Switch

U15

U21-B

9

Black

Oracle Fabric Interconnect F1-15

U14

NET0

U21-A

8

Blue

U13

NET0

U21-A

7

Blue

U12

NET0

U21-A

6

Blue

U11

NET0

U21-A

5

Blue

U10

NET0

U21-A

4

Blue

U9

NET0

U21-A

3

Blue

U8

NET0

U21-A

2

Blue

U7

NET0

U21-A

1

Blue

U6

NET0

U21-B

8

Blue

U5

NET0

U21-B

7

Blue

U2

NET2

U21-B

6

Blue

U2

NET1

U21-B

5

Blue

U2

NET0

U21-B

4

Blue

U1

NET2

U21-B

3

Blue

U1

NET1

U21-B

2

Blue

U1

NET0

U21-B (Left switch)

1

Blue

PDU-A

NETMGT

U21-A

20

White

PDU-B

NETMGT

U21-B

20

White


B.5 Storage Head Cluster Interconnects for InfiniBand-based Systems

Note

This cabling information applies to the models of the Oracle ZFS Storage Appliance installed in Oracle Private Cloud Appliance systems with an InfiniBand-based network architecture.

This section contains cabling connections for the storage head cluster in Oracle ZFS Storage Appliance ZS5-ES and earlier models.

Table B.7 Cabling Connections for Storage Head Cluster

From

Port

To

Port

Length

Color

Part Number and Notes

U1

PCIe-1 P0

U2

PCIe-1 P2

2.13m

Yellow

530-4527-01

U1

PCIe-1 P1

U2

PCIe-1 P1

2.13m

Black

530-4436-01

U1

PCIe-1 P2

U2

PCIe-1 P0

2.13m

Green

530-4526-01

U2

PCIe-2 P1

U3

IOM1 (top) SAS Port 1

2m

Black

2m SAS

U1

PCIe-2 P1

U3

IOM1 SAS Port 0

2m

Black

2m SAS

U2

PCIe-2 P0

U3

IOM0 (bottom) SAS Port 1

2m

Black

2m SAS

U1

PCIe-2 P0

U3

IOM0 SAS Port 0

2m

Black

2m SAS


B.6 Connections from NM2-36P Sun Datacenter InfiniBand Expansion Switch to Oracle Fabric Interconnect F1-15

This section contains in-rack cabling connections for the NM2-36P Sun Datacenter InfiniBand Expansion Switch to the Oracle Fabric Interconnect F1-15.

Table B.8 Cabling Connections for NM2-36P IB Switch to Oracle Fabric Interconnect F1-15

From

Name

Port

To

Name

Port

NM2 to F1 Interconnect

U19

NM2-36P IB

9B

U22

F1-15

8

Bottom NM2-36P IB to top F1-15 (Blue Velcro)

U19

NM2-36P IB

10B

U22

F1-15

9

Bottom NM2-36P IB to top F1-15 (Blue Velcro)

U19

NM2-36P IB

11B

U15

F1-15

10

Bottom NM2-36P IB to bottom F1-15 (Blue Velcro)

U19

NM2-36P IB

9A

U15

F1-15

8

Bottom NM2-36P IB to bottom F1-15 (Blue Velcro)

U19

NM2-36P IB

10A

U15

F1-15

9

Bottom NM2-36P IB to bottom F1-15 (Blue Velcro)

U19

NM2-36P IB

11A

U22

F1-15

10

Bottom NM2-36P IB to top F1-15 (Blue Velcro)

U20

NM2-36P IB

9B

U15

F1-15

18

Top NM2-36P IB to bottom F1-15 (Blue Velcro)

U20

NM2-36P IB

10B

U15

F1-15

19

Top NM2-36P IB to bottom F1-15 (Blue Velcro)

U20

NM2-36P IB

11B

U22

F1-15

20

Top NM2-36P IB to top F1-15 (Blue Velcro)

U20

NM2-36P IB

9A

U22

F1-15

18

Top NM2-36P IB to top F1-15 (Blue Velcro)

U20

NM2-36P IB

10A

U22

F1-15

19

Top NM2-36P IB to top F1-15 (Blue Velcro)

U20

NM2-36P IB

11A

U15

F1-15

20

Top NM2-36P IB to bottom F1-15 (Blue Velcro)


B.7 Connections from NM2-36P Sun Datacenter InfiniBand Expansion Switches to Compute Nodes

This section contains in-rack cabling connections for the NM2-36P Sun Datacenter InfiniBand Expansion Switches to the compute nodes.

B.7.1 NM2-36P (U19) InfiniBand Switch to Compute Nodes Connections

The following table contains cabling connections for the NM2-36P (U19) InfiniBand switch to the compute nodes.

Table B.9 Cabling Connections for NM2-36P (U19) IB Switch to Compute Nodes

From

Name

Port

To

Name

Port

U19

NM2-36P IB

0A

U42

Compute Node

PCIe-2 P2

U19

NM2-36P IB

0B

U41

Compute Node

PCIe-2 P2

U19

NM2-36P IB

1A

U40

Compute Node

PCIe-2 P2

U19

NM2-36P IB

1B

U39

Compute Node

PCIe-2 P2

U19

NM2-36P IB

2A

U38

Compute Node

PCIe-2 P2

U19

NM2-36P IB

2B

U37

Compute Node

PCIe-2 P2

U19

NM2-36P IB

3A

U36

Compute Node

PCIe-2 P2

U19

NM2-36P IB

3B

U35

Compute Node

PCIe-2 P2

U19

NM2-36P IB

4A

U34

Compute Node

PCIe-2 P2

U19

NM2-36P IB

4B

U33

Compute Node

PCIe-2 P2

U19

NM2-36P IB

5A

U32

Compute Node

PCIe-2 P2

U19

NM2-36P IB

5B

U31

Compute Node

PCIe-2 P2

U19

NM2-36P IB

6A

U30

Compute Node

PCIe-2 P2

U19

NM2-36P IB

6B

U29

Compute Node

PCIe-2 P2

U19

NM2-36P IB

7A

U28

Compute Node

PCIe-2 P1

U19

NM2-36P IB

7B

U27

Compute Node

PCIe-2 P1

U19

NM2-36P IB

8A

U26

Compute Node

PCIe-2 P1

U19

NM2-36P IB

12A

U14

Compute Node

PCIe-2 P1

U19

NM2-36P IB

12B

U13

Compute Node

PCIe-2 P1

U19

NM2-36P IB

13A

U12

Compute Node

PCIe-2 P1

U19

NM2-36P IB

13B

U11

Compute Node

PCIe-2 P1

U19

NM2-36P IB

14A

U10

Compute Node

PCIe-2 P1

U19

NM2-36P IB

14B

U9

Compute Node

PCIe-2 P1

U19

NM2-36P IB

15A

U8

Compute Node

PCIe-2 P1

U19

NM2-36P IB

15B

U7

Compute Node

PCIe-2 P1

U19

NM2-36P IB

16A

U6

Management Node

PCIe-2 P1

U19

NM2-36P IB

16B

U5

Management Node

PCIe-2 P1

U19

NM2-36P IB

17A

U2

Storage Appliance Controller

PCIe-0 P1

U19

NM2-36P IB

17B

U1

Storage Appliance Controller

PCIe-0 P1


B.7.2 NM2-36P (U20) InfiniBand Switch to Compute Nodes Connections

The following table contains cabling connections for the NM2-36P (U20) InfiniBand switch to the compute nodes.

Table B.10 Cabling Connections for NM2-36P (U20) IB Switch to Compute Nodes

From

Name

Port

To

Name

Port

U20

NM2-36P IB

0A

U42

Compute Node

PCIe-2 P1

U20

NM2-36P IB

0B

U41

Compute Node

PCIe-2 P1

U20

NM2-36P IB

1A

U40

Compute Node

PCIe-2 P1

U20

NM2-36P IB

1B

U39

Compute Node

PCIe-2 P1

U20

NM2-36P IB

2A

U38

Compute Node

PCIe-2 P1

U20

NM2-36P IB

2B

U37

Compute Node

PCIe-2 P1

U20

NM2-36P IB

3A

U36

Compute Node

PCIe-2 P1

U20

NM2-36P IB

3B

U35

Compute Node

PCIe-2 P1

U20

NM2-36P IB

4A

U34

Compute Node

PCIe-2 P1

U20

NM2-36P IB

4B

U33

Compute Node

PCIe-2 P1

U20

NM2-36P IB

5A

U32

Compute Node

PCIe-2 P1

U20

NM2-36P IB

5B

U31

Compute Node

PCIe-2 P1

U20

NM2-36P IB

6A

U30

Compute Node

PCIe-2 P1

U20

NM2-36P IB

6B

U29

Compute Node

PCIe-2 P1

U20

NM2-36P IB

7A

U28

Compute Node

PCIe-2 P2

U20

NM2-36P IB

7B

U27

Compute Node

PCIe-2 P2

U20

NM2-36P IB

8A

U26

Compute Node

PCIe-2 P2

U20

NM2-36P IB

12A

U14

Compute Node

PCIe-2 P2

U20

NM2-36P IB

12B

U13

Compute Node

PCIe-2 P2

U20

NM2-36P IB

13A

U12

Compute Node

PCIe-2 P2

U20

NM2-36P IB

13B

U11

Compute Node

PCIe-2 P2

U20

NM2-36P IB

14A

U10

Compute Node

PCIe-2 P2

U20

NM2-36P IB

14B

U9

Compute Node

PCIe-2 P2

U20

NM2-36P IB

15A

U8

Compute Node

PCIe-2 P2

U20

NM2-36P IB

15B

U7

Compute Node

PCIe-2 P2

U20

NM2-36P IB

16A

U6

Management Node

PCIe-2 P2

U20

NM2-36P IB

16B

U5

Management Node

PCIe-2 P2

U20

NM2-36P IB

17A

U2

Storage Appliance Controller

PCIe-0 P2

U20

NM2-36P IB

17B

U1

Storage Appliance Controller

PCIe-0 P2