NAT on DRG - Part 5 - Overlapping CIDRs between OCI and On-Premises with BGP

Introduction

In NAT on DRG - Part 3 - Overlapping CIDRs between OCI and On-Premises an Oracle Cloud Infrastructure (OCI) VCN and an on-premises network with the same IP range were connected over a Site-to-Site VPN, with NAT on DRG doing the translation on the IPSec tunnel attachments. The tunnels in that part were policy-based: the encryption domain decided which traffic went into the tunnel, and it was also the place where the Dynamic Routing Gateway (DRG) learned its route to on-premises.

This part builds the same use case with BGP dynamic routing. That is how most production VPNs are built, and it is also how a FastConnect virtual circuit exchanges routes, so the routing model in this tutorial is the one to expect there as well. The topology, the addresses and the NAT rule are identical to Part 3:

Side Real subnet CIDR Shadow CIDR seen by the other side
OCI VCN uc3-vcn-oci 10.0.1.0/24 192.168.10.0/24
On-premises LAN 10.0.1.0/24 192.168.20.0/24

What changes is how each side learns where the other side is. With BGP there is one question that decides everything: does the DRG translate the routes that are exchanged over BGP? The answer, tested for this tutorial, is no:

NAT on DRG translates packets, not routes. With BGP you therefore have to put the shadow CIDRs into the route exchange yourself, on both sides:

Direction What has to be advertised How
On-premises to Oracle 192.168.20.0/24 (the shadow CIDR of on-premises) The on-premises router advertises it with BGP.
Oracle to on-premises 192.168.10.0/24 (the shadow CIDR of OCI) A static route for it in the DRG route table of the tunnel attachments, which the DRG then advertises.

[!NOTE] A NAT rule describes traffic entering the DRG through the attachment that holds the policy. For an IPSec tunnel attachment that is traffic arriving from on-premises. For traffic leaving through the tunnel the DRG applies the reverse translation automatically.

The on-premises network is simulated with a VCN in another region, with one Oracle Linux instance that runs Libreswan for the IPSec tunnels and FRRouting (FRR) for BGP. The lab is the one from Part 3 with the VPN converted to BGP, which is why the resources in the screenshots are named uc3-....

image

Other Use Cases

This tutorial is part of a series. Every part solves a different overlap scenario with the same building blocks.

Diagram

Overlapping IPs between VCN subnet CIDR and on-premises or remote network with stateless 1:1 NAT on DRG, with BGP route exchange over the IPSec tunnels

image

The diagram shows every object you configure in this tutorial:

Prerequisites

[!NOTE] The names, CIDRs and IP addresses in this tutorial come from the lab in the screenshots. You can use your own, as long as the shadow CIDRs do not overlap with anything else in your network and have the same prefix length as the CIDR they replace.

Steps

Step 01: Review the VCN on the OCI side

We start on the OCI side with an ordinary VCN, one public subnet and one test instance.

image

image

image

The internet gateway is only here so that you can reach the test instance over SSH.

image

image

Step 02: Review the simulated on-premises network

The on-premises network is simulated with a VCN in another region. It uses the same address space as the OCI side, which is the whole point of this tutorial. If you have a real on-premises network and router, you can read this step for reference and continue with Step 05.

image

image

The VCN has two subnets. Together they give the Libreswan instance the two sides that a real router has.

image

image

image

[!NOTE] The instance uc3-inst-op forwards traffic that is not addressed to itself. For that to work, Skip source/destination check must be enabled on its VNICs and IP forwarding must be enabled in the operating system (net.ipv4.ip_forward = 1).

Step 03: Configure the security lists

The security lists must allow two kinds of traffic: the IPSec tunnel itself and the ping that we use as a test.

The lab allows ICMP from all three private ranges. That is broader than this tutorial needs. On the OCI side the only range that matters is the on-premises shadow CIDR 192.168.20.0/24. On the simulated on-premises side you need the OCI shadow CIDR 192.168.10.0/24 and also the local 10.0.1.0/24, because traffic from the LAN hosts passes through the LAN VNIC of the Libreswan router, and the security list is checked there as well.

image

image

[!NOTE] On the OCI side the IPSec rules are not needed, because the tunnels end on the DRG and not on an instance in the VCN. They are needed on the simulated on-premises side, where the tunnels end on the Libreswan instance.

Step 04: Configure the routing on the on-premises side

On-premises hosts must know how to reach OCI. They use the shadow CIDR of the OCI side, 192.168.10.0/24, and the route for it points to the on-premises router. The router itself learns that prefix with BGP in Step 08.

image

image

Step 05: Create the DRG and attach the VCN

Back on the OCI side we create the DRG. It gets a VCN attachment now, and two IPSec tunnel attachments are added automatically in the next step.

image

image

image

[!NOTE] The screenshots of the DRG were taken while one of the two tunnels was down. They show a warning that the IPSec connection has only a single active tunnel, and the DRG list shows Not redundant. The warning disappears when both tunnels are up.

Step 06: Create the Site-to-Site VPN with BGP dynamic routing

The Site-to-Site VPN connects the DRG to the on-premises router. It needs a customer-premises equipment (CPE) object that represents the public IP address of the router, and an IPSec connection that creates two tunnels for redundancy. The difference with Part 3 is the routing type of the tunnels. With BGP dynamic routing each tunnel gets:

Tunnel Oracle VPN IP address Inside IP, Oracle Inside IP, on-premises On-premises AS
1 168.138.66.213 10.255.255.1/30 10.255.255.2/30 65000
2 168.138.65.198 10.255.255.5/30 10.255.255.6/30 65000

image

image

image

image

image

If you start from the policy-based VPN of Part 3, you do not have to create a new IPSec connection. You can edit each tunnel and change its routing type. With the OCI CLI that looks like this for the first tunnel:

cat > tunnel-1-bgp.json <<'EOF'
{
  "routing": "BGP",
  "bgpSessionConfig": {
    "oracleInterfaceIp": "10.255.255.1/30",
    "customerInterfaceIp": "10.255.255.2/30",
    "customerBgpAsn": "65000"
  }
}
EOF

oci raw-request --http-method PUT \
  --target-uri "https://iaas.ca-montreal-1.oraclecloud.com/20160918/ipsecConnections/<ipsec_connection_ocid>/tunnels/<tunnel_ocid>" \
  --request-body file://tunnel-1-bgp.json

[!NOTE] Changing the routing type re-provisions the tunnel and removes its encryption domain. The tunnel stays down until the on-premises router is changed to a route-based configuration as well.

Step 07: Configure Libreswan on the on-premises side

The on-premises router needs a route-based IPSec configuration: one tunnel interface per tunnel, and traffic selectors that accept everything. If you use a real router, configure it with IKEv1, a pre-shared key, a tunnel interface per tunnel with the inside addresses from Step 06, and 0.0.0.0/0 as local and remote traffic selector.

With Libreswan a tunnel interface is a VTI interface. Both connections have the same selectors (0.0.0.0/0 on both sides), so each one gets its own mark. The mark is what lets the Linux kernel keep the two tunnels apart, and it is also what keeps the 0.0.0.0/0 selectors from capturing all other traffic of the router: only traffic that is routed into a VTI interface gets the mark and is encrypted.

conn uc3-t2
    auto=start
    authby=secret
    keyexchange=ike
    ikev2=no
    type=tunnel
    left=%defaultroute
    leftid=40.233.76.145
    leftsubnet=0.0.0.0/0
    right=168.138.66.213
    rightid=168.138.66.213
    rightsubnet=0.0.0.0/0
    ike=aes256-sha256;modp1536
    phase2alg=aes256-sha256
    pfs=yes
    encapsulation=yes
    ikelifetime=28800s
    salifetime=3600s
    mark=1/0xffffffff
    vti-interface=vti1
    vti-routing=no
    vti-shared=no
    leftvti=10.255.255.2/30
conn uc3
    auto=start
    authby=secret
    keyexchange=ike
    ikev2=no
    type=tunnel
    left=%defaultroute
    leftid=40.233.76.145
    leftsubnet=0.0.0.0/0
    right=168.138.65.198
    rightid=168.138.65.198
    rightsubnet=0.0.0.0/0
    ike=aes256-sha256;modp1536
    phase2alg=aes256-sha256
    pfs=yes
    encapsulation=yes
    ikelifetime=28800s
    salifetime=3600s
    mark=2/0xffffffff
    vti-interface=vti2
    vti-routing=no
    vti-shared=no
    leftvti=10.255.255.6/30
# /etc/ipsec.d/uc3-t2.secrets
40.233.76.145 168.138.66.213 : PSK "<shared secret of tunnel 1>"

# /etc/ipsec.d/uc3.secrets
40.233.76.145 168.138.65.198 : PSK "<shared secret of tunnel 2>"
sudo systemctl enable --now ipsec
sudo systemctl restart ipsec

There are two settings to point out:

image

image

image

Step 08: Configure BGP on the on-premises side

The tunnels are up, but nothing is routed through them yet. That is the job of BGP. On the simulated on-premises router BGP is provided by FRRouting (FRR).

This is the step where the overlap has to be dealt with on the on-premises side. The router must not advertise its real LAN 10.0.1.0/24: the DRG ignores that prefix, because it overlaps with the attached VCN, and NAT on DRG does not translate it into the shadow CIDR. The router advertises the shadow CIDR 192.168.20.0/24 instead. No interface on the router has an address in that range, so a static route to Null0 puts the prefix in the routing table, which is what BGP needs to be able to advertise it. That route never handles a packet: traffic for on-premises arrives with its destination already translated back to 10.0.1.x.

The two prefix lists make the exchange explicit. The router advertises only its own shadow CIDR, and it accepts only the shadow CIDR of OCI. The second filter matters, because Oracle also advertises the real 10.0.1.0/24 of the VCN (you see why in Step 09), and that prefix is the on-premises LAN itself.

sudo dnf -y install frr
sudo sed -i 's/^bgpd=no/bgpd=yes/' /etc/frr/daemons
frr defaults traditional
hostname uc3-inst-op
log syslog informational
service integrated-vtysh-config
!
! The on-premises shadow CIDR. It only exists so that BGP has something to advertise.
ip route 192.168.20.0/24 Null0
!
ip prefix-list TO-OCI seq 10 permit 192.168.20.0/24
ip prefix-list FROM-OCI seq 10 permit 192.168.10.0/24
!
router bgp 65000
 bgp router-id 10.0.99.146
 no bgp ebgp-requires-policy
 neighbor 10.255.255.1 remote-as 31898
 neighbor 10.255.255.1 description OCI-tunnel-1
 neighbor 10.255.255.1 timers 5 15
 neighbor 10.255.255.5 remote-as 31898
 neighbor 10.255.255.5 description OCI-tunnel-2
 neighbor 10.255.255.5 timers 5 15
 !
 address-family ipv4 unicast
  network 192.168.20.0/24
  neighbor 10.255.255.1 soft-reconfiguration inbound
  neighbor 10.255.255.1 prefix-list FROM-OCI in
  neighbor 10.255.255.1 prefix-list TO-OCI out
  neighbor 10.255.255.5 soft-reconfiguration inbound
  neighbor 10.255.255.5 prefix-list FROM-OCI in
  neighbor 10.255.255.5 prefix-list TO-OCI out
 exit-address-family
!
line vty
sudo chown frr:frr /etc/frr/frr.conf
sudo systemctl enable --now frr

image

[!NOTE] The line timers 5 15 sets the BGP keepalive and hold time to 5 and 15 seconds. With the default hold time of 180 seconds it takes up to three minutes before the router notices that a tunnel is gone. In the lab a failure of one tunnel caused about 14 seconds of packet loss with these timers, and no loss when the tunnel came back.

Step 09: Configure the DRG route tables

The DRG has two kinds of attachments, and each kind gets its own DRG route table. With BGP these tables do two jobs at once: they forward packets, and they decide what is exchanged over BGP.

The route table of the IPSec tunnel attachments gets two static routes, both to the VCN attachment, and each one is there for a different reason:

Destination Why it is there
10.0.1.0/24 Forwarding. A packet from on-premises is translated by the NAT policy first, and the DRG then looks up the translated destination, which is the real CIDR of the VCN.
192.168.10.0/24 Advertising. The DRG advertises the routes of this table to the on-premises router. This route makes it advertise the shadow CIDR of OCI. No packet ever uses it, because a packet for 192.168.10.x has been translated to 10.0.1.x before the route lookup.

This is the Oracle half of the answer to the overlap: the DRG does not translate the routes it advertises, so the shadow CIDR has to be put into the table by hand.

[!NOTE] Only the second route has to be static. The first one, 10.0.1.0/24, is static here because this tutorial uses a custom DRG route table without an import route distribution. The autogenerated DRG route table for IPSec attachments learns that route dynamically from the VCN attachment, and an import route distribution that accepts the VCN attachment does the same for a custom table. The shadow CIDR 192.168.10.0/24 has no such source: no attachment owns it and the NAT policy does not create a route for it, so a static route is the only way to get it into the table.

image

The route table of the VCN attachment handles traffic that comes from the VCN and is addressed to the on-premises shadow CIDR 192.168.20.0/24. This traffic has to go to a tunnel. A static route rule in a DRG route table cannot point to an IPSec tunnel, so the route is imported from the tunnels with an import route distribution. With BGP the imported route is the one that the on-premises router advertises.

image

image

image

[!NOTE] If this route is missing, traffic from the VCN to on-premises is dropped at the DRG even though the tunnels and the BGP sessions are up. Check what the on-premises router advertises (Step 13) first.

Step 10: Configure the VCN route table on the OCI side

The instances in the VCN must send traffic for the on-premises shadow CIDR to the DRG.

image

Step 11: Create the DRG NAT policy and rule

This is the step the whole tutorial is about. One DRG NAT policy with one rule does all the translation, in both directions.

The policy is associated with the IPSec tunnel attachments, so the rule is written for traffic that arrives from on-premises. Such a packet has an on-premises host as source (10.0.1.x) and the shadow address of an OCI instance as destination (192.168.10.x):

  Original Translated Meaning
Source 10.0.1.0/24 192.168.20.0/24 The real addresses of on-premises are presented in OCI as 192.168.20.0/24.
Destination 192.168.10.0/24 10.0.1.0/24 The shadow CIDR of OCI is replaced with the real CIDR of the VCN.

image

image

[!NOTE] It is tempting to write this rule from the point of view of the VCN: source 10.0.1.0/24 to 192.168.10.0/24 and destination 192.168.20.0/24 to 10.0.1.0/24. That rule is accepted without any warning, but on an IPSec tunnel attachment it never matches a packet and no traffic passes. Always write the rule for the traffic that enters the DRG through the attachment that holds the policy.

Step 12: Associate the NAT policy with the IPSec tunnel attachments

The policy is associated with both IPSec tunnel attachments, so that the translation is the same no matter which tunnel carries the traffic.

image

Step 13: Verify the BGP route exchange

Before sending traffic it is worth looking at what is exchanged over BGP, because this is where this tutorial differs from every other part of the series. The check is done on the on-premises router.

image

image

image

image

The OCI Console shows the same exchange from the Oracle side.

image

image

image

image

The received routes are the proof of the statement in the introduction. A NAT policy is associated with these tunnel attachments, and the DRG still advertises 10.0.1.0/24 untranslated. The shadow CIDR 192.168.10.0/24 is only there because of the static route that was added in Step 09.

[!NOTE] What happens when the on-premises router advertises its real CIDR 10.0.1.0/24? In the lab the DRG did not import it into any DRG route table, and it was not even listed among the routes received on the tunnel. Only after the router advertised 192.168.20.0/24 did the route in uc3-drgrt-vcn-oci appear.

Step 14: Test the connectivity

Everything is in place, so it is time to send a packet. The last octet of an address is preserved by the 1:1 translation, which tells you which address to ping:

Instance Real IP address Address to use from the other side
uc3-inst-oci 10.0.1.92 192.168.10.92
uc3-inst-op-lan 10.0.1.130 192.168.20.130

This is what happens to a ping from uc3-inst-oci to the on-premises host:

Where Source Destination What happens
Leaves uc3-inst-oci 10.0.1.92 192.168.20.130 The route table uc3-rt-oci sends 192.168.20.0/24 to the DRG.
Route lookup in uc3-drgrt-vcn-oci 10.0.1.92 192.168.20.130 The route for 192.168.20.0/24, learned with BGP, points to an IPSec tunnel.
Leaves the DRG on the tunnel attachment 192.168.10.92 10.0.1.130 The DRG applies the reverse of the rule in uc3-policy-oci.
On-premises router 192.168.10.92 10.0.1.130 The packet comes out of the tunnel and is forwarded to the LAN.

And this is the reply:

Where Source Destination What happens
Leaves uc3-inst-op-lan 10.0.1.130 192.168.10.92 The route table uc3-rt-op sends 192.168.10.0/24 to the router 10.0.1.164.
On-premises router 10.0.1.130 192.168.10.92 The route for 192.168.10.0/24, learned with BGP, points to a tunnel interface.
Enters the DRG on the tunnel attachment 192.168.20.130 10.0.1.92 The rule in uc3-policy-oci translates source and destination.
Route lookup in uc3-drgrt-ipsec 192.168.20.130 10.0.1.92 10.0.1.0/24 points to uc3-att-vcn-oci.

image

image

A packet capture on the on-premises host shows the translation at work. The host never sees the real address 10.0.1.92 of the OCI instance, only its shadow address:

[opc@uc3-inst-op-lan ~]$ sudo tcpdump -ni any icmp
IP 10.0.1.130 > 192.168.10.92: ICMP echo request, id 18, seq 1, length 64
IP 192.168.10.92 > 10.0.1.130: ICMP echo reply, id 18, seq 1, length 64

[!NOTE] NAT on DRG does not translate DNS. If on-premises systems find the OCI systems by name, or the other way around, the DNS records they use must point to the shadow addresses, not to the real ones.

Conclusion

An OCI VCN and an on-premises network with exactly the same subnet CIDR communicate over a Site-to-Site VPN with BGP dynamic routing. The NAT rule is the same single rule as in Part 3, on the same IPSec tunnel attachments. What is different is the way the routes get where they need to be.

The main finding of this part is that NAT on DRG and BGP are independent of each other. The DRG translates packets, and it does not translate the routes that it advertises or receives. A real CIDR that overlaps with an attached VCN is of no use in the exchange: Oracle advertises it untranslated, and the DRG ignores it when it is received. Both sides therefore advertise their shadow CIDR. On the on-premises side that is a prefix the router originates itself, and on the Oracle side it is a static route in the DRG route table of the tunnel attachments whose only purpose is to be advertised.

Compared to the policy-based VPN of NAT on DRG - Part 3 - Overlapping CIDRs between OCI and On-Premises there is no encryption domain to keep in line with the translation, and failover between the tunnels is handled by BGP. The same routing model, an exchange of shadow prefixes over BGP, is what to expect when the NAT policy is associated with a FastConnect virtual circuit attachment, although that has not been tested for this series.

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.