Mesh Topology with Dynamic Routing

Dynamic Mesh enables Layer 3 network deployment according to industry-proven best practices. This uplink topology is highly recommended.

Configuration Properties

• Mesh topology – each spine switch is connected to two independent data center switches

• Dynamic routing – both peered Autonomous Systems, the appliance and the data center, exchange routing information using eBGP (external Border Gateway Protocol). The best routing path is dynamically adjusted based on network availability information advertized by each AS.

• Separate /30 subnets – each uplink connects one spine switch port channel to one data center switch port channel in a /30 subnet

Topology Highlights

• All uplinks are configured as LACP/active port channels with rate=fast

  • Port channel Po41 represents the first set of links on both spine switches. They connect straight to the corresponding ToR switches.

  • Port channel Po42 represents the second set of links on both spine switches. They cross-connect to the corresponding ToR switches.

• ToR switch ports connecting to the spine switches must be set up in access mode. Spanning tree protocol must be disabled.

• Requires 4 unique subnets: a /30 subnet size is recommended, but /31 is possible if the ToR switches support it.

  • Two eBGP peering sessions are established between each spine and both ToR switches.

  • Egress traffic can hash to any of the 4 uplinks.

  • It is NOT possible to isolate specific VCN/VM egress traffic through one particular uplink.

Detailed Spine Switch Configuration Example

• Spine Switch 1

  interface port-channel41
    description "customer uplink"
    no switchport
    mtu 9216
    speed 10000
    no negotiate auto
    ip access-group ingress-ports-acl in
    no ip redirects
    ip address 10.25.16.1/30
    ip nat outside
  
  interface port-channel42
    description "customer uplink 2"
    no switchport
    mtu 9216
    speed 10000
    no negotiate auto
    ip access-group ingress-ports-acl in
    no ip redirects
    ip address 10.25.16.9/30
    ip nat outside
  
  router bgp 136025
    router-id 10.25.16.1
    neighbor 10.25.16.2
      bfd singlehop
      remote-as 50000
      address-family ipv4 unicast
    neighbor 10.25.16.10
      bfd singlehop
      remote-as 50000
      address-family ipv4 unicast
  
  BGP Sessions:
  ASN 136025
  VRF default, local ASN 136025
  
  Neighbor       ASN    Flaps LastUpDn|LastRead|LastWrit  St  Port(L/R)  Notif(S/R)
  10.25.16.2     50000  0     1w4d    |00:00:50|00:00:20  E   34408/179        0/0
  10.25.16.10    50000  0     1w4d    |00:00:43|00:00:20  E   57322/179        0/0

• Spine Switch 2

  interface port-channel41
    description "customer uplink"
    no switchport
    mtu 9216
    speed 10000
    no negotiate auto
    ip access-group ingress-ports-acl in
    no ip redirects
    ip address 10.25.16.5/30
    ip nat outside
  
  interface port-channel42
    description "customer uplink 2"
    no switchport
    mtu 9216
    speed 10000
    no negotiate auto
    ip access-group ingress-ports-acl in
    no ip redirects
    ip address 10.25.16.13/30
    ip nat outside
  
  router bgp 136025
    router-id 10.25.16.5
    neighbor 10.25.16.6
      bfd singlehop
      remote-as 50000
      address-family ipv4 unicast
    neighbor 10.25.16.14
      bfd singlehop
      remote-as 50000
      address-family ipv4 unicast
  
  BGP Sessions:
  ASN 136025
  VRF default, local ASN 136025
  
  Neighbor       ASN    Flaps LastUpDn|LastRead|LastWrit  St  Port(L/R)  Notif(S/R)
  10.25.16.6     50000  0     1w4d    |00:00:50|00:00:20  E   34408/179        0/0
  10.25.16.14    50000  0     1w4d    |00:00:43|00:00:20  E   57322/179        0/0