Mesh Topology with ECMP Static Routing

ECMP Mesh enables Layer 3 network deployment according to industry-proven best practices. This uplink topology is highly recommended.

Configuration Properties

  • Mesh topology – each spine switch is connected to two independent data center switches

  • Static routing – all egress traffic from an uplink goes through a single gateway IP configured on its peer network device in the data center

  • ECMP – bandwidth optimization across multiple redundant links or paths

  • Separate /30 subnets – each uplink connects one spine switch port channel to one data center switch port channel in a /30 subnet

Topology Highlights

  • All uplinks are configured as LACP/active port channels with rate=fast

    • Port channel Po41 represents the first set of links on both spine switches. They connect straight to the corresponding ToR switches.

    • Port channel Po42 represents the second set of links on both spine switches. They cross-connect to the corresponding ToR switches.

  • ToR switch ports connecting to the spine switches must be set up in access mode. Spanning tree protocol must be disabled.

  • Requires 4 unique subnets: a /30 subnet size is recommended, but /31 is possible if the ToR switches support it.

    • Equal cost static routes to both ToR switches are set up automatically.

    • Egress traffic can hash to any of the 4 uplinks.

    • It is NOT possible to isolate specific VCN/VM egress traffic through one particular uplink.


Diagram showing a reference configuration of uplinks in a mesh topology with ECMP static routing.

Detailed Spine Switch Configuration Example

  • Spine Switch 1

    interface port-channel41
  description "customer uplink"
  no switchport
  mtu 9216
  speed 10000
  no negotiate auto
  ip access-group ingress-ports-acl in
  no ip redirects
  ip address 10.25.16.1/30
  ip nat outside

interface port-channel42
  description "customer uplink 2"
  no switchport
  mtu 9216
  speed 10000
  no negotiate auto
  ip access-group ingress-ports-acl in
  no ip redirects
  ip address 10.25.16.9/30
  ip nat outside

ip route 0.0.0.0/0 po41 10.25.16.2 20
ip route 0.0.0.0/0 po42 10.25.16.10 20

    Routes added:

    0.0.0.0/0, ubest/mbest: 2/0
  *via 10.25.16.2, [20/0], 6d08h, static
  *via 10.25.16.10, [20/0], 6d08h, static

  • Spine Switch 2

    interface port-channel41
  description "customer uplink"
  no switchport
  mtu 9216
  speed 10000
  no negotiate auto
  ip access-group ingress-ports-acl in
  no ip redirects
  ip address 10.25.16.5/30
  ip nat outside

interface port-channel42
  description "customer uplink 2"
  no switchport
  mtu 9216
  speed 10000
  no negotiate auto
  ip access-group ingress-ports-acl in
  no ip redirects
  ip address 10.25.16.13/30
  ip nat outside

ip route 0.0.0.0/0 po41 10.25.16.6 20
ip route 0.0.0.0/0 po42 10.25.16.14 20

    Routes added:

    0.0.0.0/0, ubest/mbest: 2/0
  *via 10.25.16.6, [20/0], 6d07h, static
  *via 10.25.16.14, [20/0], 6d07h, static