Learn About Dynamic Routing Gateway Solutions

Use this reference document to learn about Oracle Cloud Infrastructure (OCI) virtual network routing. It deciphers IPv4 routing in OCI cloud networks, and introduces basic OCI routing functions. It also provides typical use cases in different deployment scenarios, which is helpful if you need to design, operate, or troubleshoot OCI virtual networks.

OCI offers a software-defined virtual network solution. An OCI network consists of virtual cloud networks (VCNs), subnets, network gateways, OCI native or 3rd party L4-7 network service virtual appliances, and so on. Routing is the core function to establish network connectivity among the elements in an OCI network, or between an OCI network and on-premises networks or other cloud networks.

Full network reachability requires network connectivity that is achieved by proper routing and network security policies that are managed through Security Lists or Network Security Groups, or policies on network firewall appliances. This document solely focuses on routing functions and designs, it doesn't discuss management of network security policies.

OCI uses the same routing mechanisms for both IPv4 and IPv6. However, there are unique considerations that you must add into an IPv6 network design. For example, the different scopes of IPv6 addresses and the fact that IPv6 Internet Routing does not go through NATing. While the same theories apply to IPv4 and IPv6 routing, the discussions and examples here focus on IPv4 routing.

About DRG Routing

A dynamic routing gateway (DRG) is a regional virtual router that inter-connects VCNs in a region and connects the VCNs with on-premises networks through Oracle Cloud Infrastructure FastConnect virtual circuits or IPSec VPN tunnels. It also provides network connectivity between regions through a Remote Peering Connection (RPC).

A DRG acts like a central hub to connect the network resources that are attached to it. The network resources can be VCNs, site-to-site IPSec VPN tunnels, OCI FastConnect virtual circuits, or RPC. When a network resource is attached to a DRG, an attachment of the corresponding type is created:

• Virtual Cloud Network Attachment (VCN Attachment): When a VCN attached to the DRG
• Virtual Circuit Attachment (VC Attachment): When an OCI FastConnect virtual circuit is attached to the DRG
• IPSec Tunnel Attachment: When an IPSec tunnel is attached to the DRG
• Remote Peering Connection Attachment (RPC Attachment): When an RPC is attached to the DRG

DRG routes traffic between the attachments using DRG route tables. Each attachment is associated with a DRG route table. Traffic enters DRG from an attachment and is routed to another attachment by the DRG based on the DRG route table associated with the ingress attachment of the traffic.

Routing Tables on DRG

A dynamic routing gateway (DRG) uses DRG route tables to route traffic between its attachments. OCI automatically generates two route tables for each DRG, one for VCN attachments and the second one is for IPSec, OCI FastConnect Virtual Circuit Attachment, and Remote Peering Connection (RPC) attachments. You can create more DRG route tables.

The route rules in a DRG route table contains the following fields:

• Type: The route type can be dynamic or static. Dynamic routes are imported from the DRG attachments. You can use the OCI Console or API to create static routes.
• Destination CIDR: The destination CIDR.
• Next Hop Attachment Type: The next hop of a route rule in a DRG route table is the DRG attachment of the network where the destination resides or in route to the destination. The attachment can be a VCN attachment, a cross-regional RPC attachment, or a cross-tenancy RPC attachment. It cannot be a VPN attachment or OCI FastConnect virtual circuit attachment.
• Next Hop Attachment Name: The name of the attachment.
• Route Status: Status.

The following is an example of the contents of a DRG route table.

Type	Destination CIDR	Next Hop Attachment Type	Next Hop Attachment Name	Route Status
Dynamic	0.0.0.0/0	Virtual Cloud Network (VCN)	Att-DRG-1-VCN-0	Active
Dynamic	10.0.0.0/8	IPSec Tunnel	DRG Attachment for IPSec Tunnel: IPSec Tunnel to On-premises 2	Active
Dynamic	10.0.0.0/16	Virtual Cloud Network	Attachment DRG 1 to VCN 0	Active
Dynamic	21.0.1.0/24	Remote Peering Connection	DRG Attachment for RPC: RPC to SJC-DRG-1 (us-west-1 San Jose-DRG-1)	Active

Each DRG attachment has one DRG route table associated with it. By default, it is the auto-generated DRG route table for the attachment type. You can change it to a user-created DRG route table.

When traffic gets onto a DRG, the DRG performs ingress routing lookup based on the DRG route table associated with the ingress attachment of the traffic. The routing lookup resolves the next-hop attachment (the egress attachment). The DRG sends the traffic onto the egress attachment through which the traffic will get to the next-hop network. There is no routing lookup at the egress attachment on the DRG.

Route Preference in DRG Route Table

It's possible that multiple routes for the identical prefix and mask show up in a DRG route table. The dynamic routing gateway has a built-in mechanism to resolve such route conflicts. The decision is made based on the following route preference and is evaluated in the following order:

1. In a DRG route table, static routes have higher preference than dynamic routes.
2. Among dynamic routes in a DRG route table, routes with shorter AS-path are preferred over routes with a longer AS-path.

   Note:

   Routes with a route source of VCN or STATIC always have an empty AS Path. Routes with a route source of IPSec VPN tunnel or OCI FastConnect virtual circuit will have the AS Paths shown in the following table.

   Route Source	Details of how Oracle prefers the path	Resulting AS path for the route
   OCI FastConnect	OCI prepends no ASNs to the routes. This results in a total AS path length of 1.	Customer ASN
   Site-to-Site VPN with Border Gateway Protocol (BGP) routing	OCI prepends a single private ASN on all the routes that customer edge device advertises over Site-to-Site VPN with BGP, for a total AS path length of 2.	Private ASN, Customer ASN
   Site-to-Site VPN with static routing	OCI advertises those static routes to DRG as BGP dynamic routes. OCI prepends 3 private ASNs on these routes. This results in a total AS path length of 3.	Private ASN, Private ASN, Private ASN

3. The attachment type that imported the route is evaluated according to the following priority based on the attachment type:
   1. VCN
   2. VIRTUAL_CIRCUIT: If Equal-cost multi-path routing (ECMP) is disabled for the DRG route table, then the DRG makes an arbitrary but stable selection. If ECMP is enabled, then all routes are added to the route table and the DRG makes routing choices using ECMP. The maximum supported ECMP width inside a DRG is 8.
   3. IPSEC_TUNNEL: If ECMP is disabled for the DRG route table, the DRG makes an arbitrary but stable selection. If ECMP is enabled, all routes will be added to the route table and the DRG makes routing choices using ECMP. The maximum supported ECMP width inside a DRG is 8.
   4. REMOTE_PEERING_CONNECTION (RPC): The DRG will choose the route with the lowest network distance.

   If two routes have identical network distances, the DRG selects the route with the highest priority route source (STATIC > VCN > VIRTUAL_CIRCUIT> IPSEC_TUNNEL).

   If two routes have the same route source, then the DRG makes an arbitrary but stable selection.

4. If conflicting routes are imported from attachments of the same type, the conflict is resolved differently depending on the attachment type:
   • VCN attachments: If identical CIDRs are imported from two VCN attachments, only one is selected using an arbitrary but stable decision procedure.
   • VIRTUAL_CIRCUIT and IPSEC_TUNNEL attachments: If multiple routes with the same CIDR and different AS_PATH lengths are imported into a DRG route table, the route with the lowest AS_PATH length is selected. Otherwise, one route is chosen using an arbitrary but stable decision procedure.
   • RPC attachments: If identical CIDRs are imported from two RPC attachments, one of them is chosen using an arbitrary stable decision procedure.

Route Propagation and Import Route Distribution Control on DRG

You can attach network resources, such as VCNs, OCI FastConnect virtual circuits, or IPSec VPN tunnels to a dynamic routing gateway (DRG). The routes associated with these network resources are propagated to the DRG. Use an import route distribution policy to import them into a DRG route table as dynamic routes.

Route Propagation on DRG

The following shows which routes are associated with the network resource of each DRG attachment type and is propagated to the DRG:

• VCN attachment

  The routes in the VCN route table that is associated with the DRG attachment in the VCN, and the VCN CIDRs as well as its subnet CIDRs. Once a VCN is attached to a DRG, the DRG is represented as a DRG attachment in the VCN. A route table of the VCN can be associated with the DRG attachment for the VCN ingress routing through the DRG. It's the routes in this VCN route table that are propagated to the DRG. If a VCN route table is not associated with the DRG attachment, then only the VCN CIDRs and its subnet CIDRs are propagated to the DRG.

  When these routes are imported into a DRG route table, this VCN attachment will be the next-hop attachment in the routes.

• IPSec tunnel attachment

  The routes advertised by the IPSec customer premises equipment (CPE) when Border Gateway Protocol (BGP) dynamic routing is used on the IPSec Connection or the configured static routes if static routing is used on the IPSec Connection.

  When these routes are imported to a DRG route table as dynamic routes, the IPSec tunnel attachment will be the next-hop attachment for the routes.

• VC attachment

  The routes advertised by the OCI FastConnect CPE through BGP.

  When these routes are imported to a DRG route table, the VC attachment is the next-hop attachment in the routes.

• RPC attachment

  All routes in the DRG route table associated with the remote DRG’s RPC attachment will be propagated to the local DRG.

  When these routes are imported to a local DRG route table, the RPC attachment will be the next-hop for the routes.

Import Route Distribution Control on DRG

For a given DRG route table, you can create and apply an import route distribution policy to control which routes get imported to the route table. You can match by attachment type (for example, match all VCN attachments), a specific attachment, or match all.

The auto-generated DRG route table for VCN attachments has a default import route distribution that has a match all statement to import routes from all DRG attachments

The auto-generated DRG route table for IPSec, OCI FastConnect VC, and RPC attachments has a default import route distribution that has a match type VCN statement to only import routes from all VCN attachments.

Note:

This default import distribution policy does not import routes propagated by other attachment types into this DRG route table.

If you use the auto-generated DRG route table for all your VCN attachments, then you will achieve fully meshed routing connectivity among your VCNs, and all your VCNs will have routes to reach all your on-premises networks and VCNs in the remote region.

If you want to establish more restricted routing connectivity or create routing segmentation in your network, then you can use separate DRG route tables with different import route distribution policies for different DRG attachments. For example, we created 3 routing segments on the same DRG by using different DRG route tables and different import route distributions for the VCNs:

• A fully meshed connectivity between VCN-1, VCN-2 and VCN-3
• Connectivity between VCN-4 and VCN-5
• Connectivity between VCN-6 and the on-premises networks

The following image is an example of DRG Import Route Distribution Control.
Description of the illustration drg-import-route-distribution-control.png

drg-import-route-distribution-control-oracle.zip

Although by default, all the attachments use the auto-generated DRG route table for its type, real network designs often require some attachments of the same type to have different route rules and different route import distribution policies. It's a good practice to create separate DRG route tables for these attachments.

DRG Routing Operation

A dynamic routing gateway (DRG) routes traffic between its attachments. For a given traffic flow, there is an ingress attachment and an egress attachment on the DRG.

The DRG uses an ingress routing model when traffic enters the DRG through the ingress attachment, DRG uses the DRG route table associated with the ingress attachment to decide where the traffic goes. If a route for the destination exists in the ingress attachment’s DRG route table, then the next-hop of the route must be another attachment on the DRG. It could be a VCN attachment (if the destination is in a VCN), an IPSec or Virtual Circuit attachment (if the destination is in an on-premises network connected to the DRG through IPSec tunnels or OCI FastConnect), or an RPC attachment (if the destination is in a remote region). If there is no matching route for the destination, then the traffic is dropped.

The following image is an example of a DRG route operation.

Description of the illustration drg-routing-operation.png

drg-routing-operation-oracle.zip

This example shows the DRG routing lookup for traffic that comes from Attachment-1 and goes to a destination network that is on Attachment-2. The routing lookup takes place in the DRG route table of the ingress attachment (Attachment-1). The route table has a route rule for the destination with the egress attachment (Attachment-2) as the next-hop attachment.

Since DRG attachment is a logical point-to-point connection between the DRG and the network resource behind the attachment, the DRG doesn't need to do another routing lookup on the egress attachment, it will just forward the traffic to the next network resource through the attachment. The next network resource could be a VCN, or the routing device on the other side of an IPSec Tunnel or an OCI FastConnect virtual circuit, or a DRG in a remote region. It is up to the next resource to perform its own routing lookup to decide where to forward the traffic. For example, if the next-hop attachment is a VC attachment, then the DRG will route the traffic through the OCI FastConnect virtual circuit. The routing device on the other end of the virtual circuit performs its own routing upon receiving the traffic. If the next-hop is a VCN attachment, then the VCN ingress routing through the DRG takes place.

The following image shows the routing lookup process along a multi-hop network path.

Description of the illustration routing-lookup-multi-hop-network-path.png

routing-lookup-multi-hop-network-path-oracle.zip

In this example, the image shows the routing path between an on-premises network 10.254.0.0/16 and a VCN subnet 10.1.1.0/24. The default local routing in the VCN is used for simplicity. To achieve the end-to-end routing connectivity, there are multiple DRG route tables and VCN route tables used at different points for routing lookup for each direction:

• DRG Route Table for ATT-VC

  The DRG route table for the OCI FastConnect VC attachment: Routes traffic from the on-premises network to VCN-1.

• DRG Route Table for ATT-VCN-1

  The DRG route table for the VCN-1 attachment: Routes traffic from VCN-1 to the on-premises network.

• VCN Route Table for DRG Attachment

  The VCN route table for the DRG attachment in the VCN: VCN ingress routing through the DRG into VCN-1.

  If there is no user specified route table associated with the DRG attachment in the VCN, then the default local route for the VCN CIDRs is used. That is, the DRG will route the traffic directly to the destinations in the VCN. This is the implicit VCN local route and is invisible to the users.

• Subnet Route Table

  The VCN route table for the subnet 10.1.1.0/24: Routes traffic from the VCN-1 subnet to the DRG.

The following image shows traffic routing from VCN-1 to the on-premises network.

Description of the illustration traffic-route-vcn-prem.png

traffic-route-vcn-prem-oracle.zip

In this example, the VCN subnet route table and the DRG route table for the VCN-1 attachment route the traffic from a resource in the VCN-1 subnet to a resource in the on-premises network.

The following image shows the DRG route table for the OCI FastConnect VC attachment:

Description of the illustration traffic-route-prem-vcn.png

traffic-route-prem-vcn-oracle.zip

In this example, the VCN route table associated with the DRG attachment in the VCN for DRG ingress routing routes the traffic from the on-premises resource to a resource in the VCN-1 subnet.