NAT on DRG - Part 3 - Overlapping CIDRs between OCI and On-Premises

Introduction

The most painful place to find an IP overlap is between Oracle Cloud Infrastructure (OCI) and an on-premises network. Inside OCI you could still re-create a VCN with a different CIDR. An on-premises network with thousands of devices is not going to be re-addressed because a cloud project picked the same 10.0.0.0/16. The same problem appears when you connect a partner or a customer over a VPN and have no say in their address plan.

NAT on DRG lets the Dynamic Routing Gateway (DRG) translate the addresses on the Site-to-Site VPN itself. A DRG NAT policy is associated with the IPSec tunnel attachments, and each side is presented to the other under a unique “shadow” CIDR:

Side Real subnet CIDR Shadow CIDR seen by the other side
OCI VCN uc3-vcn-oci 10.0.1.0/24 192.168.10.0/24
On-premises LAN 10.0.1.0/24 192.168.20.0/24

An instance in OCI reaches an on-premises host on 192.168.20.x, and an on-premises host reaches OCI on 192.168.10.x. The on-premises router needs no NAT configuration at all: all the translation is done by the DRG.

This part has more moving pieces than the earlier ones, because a VPN brings two things of its own:

[!NOTE] A NAT rule describes traffic entering the DRG through the attachment that holds the policy. For an IPSec tunnel attachment that is traffic arriving from on-premises. For traffic leaving through the tunnel the DRG applies the reverse translation automatically.

For this tutorial the on-premises network is simulated with a second VCN in another region, with a Libreswan instance as the customer-premises equipment (CPE). Everything on the OCI side is exactly what you would configure against a real on-premises router.

image

Other Use Cases

This tutorial is part of a series of five. Every part solves a different overlap scenario with the same building blocks.

Diagram

Overlapping IPs between VCN subnet CIDR and on-premises or remote network with stateless 1:1 NAT on DRG

image

The diagram shows every object you configure in this tutorial:

Prerequisites

[!NOTE] The names, CIDRs and IP addresses in this tutorial come from the lab in the screenshots. You can use your own, as long as the shadow CIDRs do not overlap with anything else in your network and have the same prefix length as the CIDR they replace.

Steps

Step 01: Review the VCN on the OCI side

We start on the OCI side with an ordinary VCN, one public subnet and one test instance.

image

image

image

The internet gateway is only here so that you can reach the test instance over SSH.

image

image

Step 02: Review the simulated on-premises network

The on-premises network is simulated with a VCN in another region. It uses the same address space as the OCI side, which is the whole point of this tutorial. If you have a real on-premises network and router, you can read this step for reference and continue with Step 05.

image

image

The VCN has two subnets. Together they give the Libreswan instance the two sides that a real router has.

image

image

image

[!NOTE] The instance uc3-inst-op forwards traffic that is not addressed to itself. For that to work, Skip source/destination check must be enabled on its VNICs and IP forwarding must be enabled in the operating system (net.ipv4.ip_forward = 1).

Step 03: Configure the security lists

The security lists must allow two kinds of traffic: the IPSec tunnel itself and the ping that we use as a test.

The lab allows ICMP from all three private ranges. That is broader than this tutorial needs. On the OCI side the only range that matters is the on-premises shadow CIDR 192.168.20.0/24. On the simulated on-premises side you need the OCI shadow CIDR 192.168.10.0/24 and also the local 10.0.1.0/24, because traffic from the LAN hosts passes through the LAN VNIC of the Libreswan router, and the security list is checked there as well.

image

image

[!NOTE] On the OCI side the IPSec rules are not needed, because the tunnels end on the DRG and not on an instance in the VCN. They are needed on the simulated on-premises side, where the tunnels end on the Libreswan instance.

Step 04: Configure the routing on the on-premises side

On-premises hosts must know how to reach OCI. They use the shadow CIDR of the OCI side, 192.168.10.0/24, and the route for it points to the on-premises router. This is the only thing the on-premises network has to know about the translation.

image

image

Step 05: Create the DRG and attach the VCN

Back on the OCI side we create the DRG. It gets a VCN attachment now, and two IPSec tunnel attachments are added automatically in the next step.

image

image

image

[!NOTE] NOTE The screenshots show a warning that the IPSec connection has only a single active tunnel, and the DRG list shows Not redundant. They were taken while one of the two tunnels was down. The warning disappears when both tunnels are up.

Step 06: Create the Site-to-Site VPN

The Site-to-Site VPN connects the DRG to the on-premises router. It needs a customer-premises equipment (CPE) object that represents the public IP address of the router, and an IPSec connection that creates two tunnels for redundancy.

image

image

image

Step 07: Configure the encryption domain of the tunnels

This step is specific to NAT on DRG over a policy-based VPN, and it is where a configuration most often goes wrong. A policy-based tunnel has an encryption domain: a list of CIDRs on the Oracle side and a list of CIDRs on the on-premises side. The tunnel only carries traffic between those CIDRs.

The DRG applies the NAT rule before a packet goes into the tunnel and after it comes out. The packets inside the tunnel therefore look like this:

Side of the tunnel Address inside the tunnel Why
Oracle 192.168.10.0/24 The OCI side has already been translated to its shadow CIDR.
On-premises 10.0.1.0/24 The on-premises side uses its real addresses. It is not aware of any NAT.

So the Oracle side of the encryption domain is the shadow CIDR of OCI, not the real CIDR of the VCN.

There is a second reason to edit the encryption domain. In policy-based mode the DRG learns its routes towards the tunnel from the on-premises CIDRs of the encryption domain. The real on-premises CIDR 10.0.1.0/24 overlaps with the VCN and does not give the DRG a usable route. The instances in OCI send their traffic to the on-premises shadow CIDR 192.168.20.0/24, so that is the destination the DRG needs a route for. We get that route by adding 192.168.20.0/24 as a second on-premises CIDR.

You can do the same with the OCI CLI. Repeat the command for both tunnels:

cat > tunnel-encryption-domain.json <<'EOF'
{
  "routing": "POLICY",
  "encryptionDomainConfig": {
    "oracleTrafficSelector": ["192.168.10.0/24"],
    "cpeTrafficSelector": ["10.0.1.0/24", "192.168.20.0/24"]
  }
}
EOF

oci raw-request --http-method PUT \
  --target-uri "https://iaas.ca-montreal-1.oraclecloud.com/20160918/ipsecConnections/<ipsec_connection_ocid>/tunnels/<tunnel_ocid>" \
  --request-body file://tunnel-encryption-domain.json

[!NOTE] NOTE The on-premises CIDR 192.168.20.0/24 never carries traffic through the tunnel. Its only purpose is to give the DRG a route for the on-premises shadow CIDR that points to the tunnel. The on-premises router is configured with the single pair 10.0.1.0/24 and 192.168.10.0/24, as you see in the next step.

[!NOTE] NOTE Changing the encryption domain re-provisions the tunnel. Expect it to go down for about half a minute.

Step 08: Configure Libreswan on the on-premises side

Now the on-premises router is configured to bring up both tunnels. If you use a real router, configure it with the values from this step: IKEv1, a pre-shared key, and the traffic selectors 10.0.1.0/24 (local) and 192.168.10.0/24 (remote).

With Libreswan there is one thing to take care of. Both tunnels use exactly the same traffic selectors, and the Linux kernel can hold only one IPSec policy for a given pair of subnets. Two plain connections with identical selectors keep taking that policy from each other, and the tunnels flap. The solution is to give each connection its own mark and its own VTI interface, so that the kernel can tell them apart.

conn uc3-t2
    auto=start
    authby=secret
    keyexchange=ike
    ikev2=no
    type=tunnel
    left=%defaultroute
    leftid=40.233.76.145
    leftsubnet=10.0.1.0/24
    right=168.138.66.213
    rightid=168.138.66.213
    rightsubnet=192.168.10.0/24
    ike=aes256-sha256;modp1536
    phase2alg=aes256-sha256
    pfs=yes
    encapsulation=yes
    ikelifetime=28800s
    salifetime=3600s
    mark=1/0xffffffff
    vti-interface=vti1
    vti-routing=no
    vti-shared=no
conn uc3
    auto=start
    authby=secret
    keyexchange=ike
    ikev2=no
    type=tunnel
    left=%defaultroute
    leftid=40.233.76.145
    leftsubnet=10.0.1.0/24
    right=168.138.65.198
    rightid=168.138.65.198
    rightsubnet=192.168.10.0/24
    ike=aes256-sha256;modp1536
    phase2alg=aes256-sha256
    pfs=yes
    encapsulation=yes
    ikelifetime=28800s
    salifetime=3600s
    mark=2/0xffffffff
    vti-interface=vti2
    vti-routing=no
    vti-shared=no
# /etc/ipsec.d/uc3-t2.secrets
40.233.76.145 168.138.66.213 : PSK "<shared secret of tunnel 1>"

# /etc/ipsec.d/uc3.secrets
40.233.76.145 168.138.65.198 : PSK "<shared secret of tunnel 2>"
sudo systemctl enable --now ipsec
sudo systemctl restart ipsec

Notice that leftsubnet is the real on-premises CIDR and rightsubnet is the shadow CIDR of OCI. This matches the encryption domain from Step 07.

The last piece is the route to the OCI shadow CIDR. With vti-routing=no Libreswan does not add routes itself, so we add one route per tunnel with a different metric. The first tunnel is preferred and the second one takes over when the first one fails.

sudo ip route replace 192.168.10.0/24 dev vti1 metric 100
sudo ip route replace 192.168.10.0/24 dev vti2 metric 200

[!NOTE] NOTE A route that is added with ip route is lost after a reboot, and a route over a tunnel that is down sends traffic into a black hole. In the lab a small systemd service checks every few seconds which tunnels have an active security association (ipsec whack --trafficstatus) and adds or removes the route of each tunnel accordingly. A production router has its own mechanism for this.

image

image

image

[!NOTE] NOTE The Console lists an entry for 192.168.20.0/24 as well, but the on-premises router is only configured with the pair 10.0.1.0/24 and 192.168.10.0/24, and that is the only pair that carries traffic.

Step 09: Configure the DRG route tables

The DRG has two kinds of attachments now, and each kind gets its own DRG route table.

The route table of the IPSec tunnel attachments handles traffic that arrives from on-premises. The NAT policy on the tunnel attachment translates the packet first, and the DRG looks up the translated destination. That destination is the real CIDR of the VCN, so the table has a static route for 10.0.1.0/24 to the VCN attachment.

image

[!NOTE] NOTE This route is static because this tutorial uses a custom DRG route table without an import route distribution. The autogenerated DRG route table for IPSec attachments learns 10.0.1.0/24 dynamically from the VCN attachment, and an import route distribution that accepts the VCN attachment does the same for a custom table.

The route table of the VCN attachment handles traffic that comes from the VCN and is addressed to the on-premises shadow CIDR 192.168.20.0/24. This traffic has to go to the tunnel. You cannot create that route as a static route: a static route rule in a DRG route table accepts a VCN attachment or a remote peering connection as next hop, but not an IPSec tunnel. The route has to be learned dynamically, with an import route distribution.

image

image

image

[!NOTE] NOTE If this route is missing, traffic from the VCN to on-premises is dropped at the DRG even though the tunnels are up and the NAT rule is correct. Check the on-premises CIDR blocks of the encryption domain (Step 07) first.

Step 10: Configure the VCN route table on the OCI side

The instances in the VCN must send traffic for the on-premises shadow CIDR to the DRG.

image

Step 11: Create the DRG NAT policy and rule

This is the step the whole tutorial is about. One DRG NAT policy with one rule does all the translation, in both directions.

The policy is associated with the IPSec tunnel attachments, so the rule is written for traffic that arrives from on-premises. Such a packet has an on-premises host as source (10.0.1.x) and the shadow address of an OCI instance as destination (192.168.10.x):

  Original Translated Meaning
Source 10.0.1.0/24 192.168.20.0/24 The real addresses of on-premises are presented in OCI as 192.168.20.0/24.
Destination 192.168.10.0/24 10.0.1.0/24 The shadow CIDR of OCI is replaced with the real CIDR of the VCN.

image

image

[!NOTE] NOTE It is tempting to write this rule from the point of view of the VCN: source 10.0.1.0/24 to 192.168.10.0/24 and destination 192.168.20.0/24 to 10.0.1.0/24. That rule is accepted without any warning, but on an IPSec tunnel attachment it never matches a packet and no traffic passes. Always write the rule for the traffic that enters the DRG through the attachment that holds the policy.

Step 12: Associate the NAT policy with the IPSec tunnel attachments

The policy is associated with both IPSec tunnel attachments, so that the translation is the same no matter which tunnel carries the traffic.

image

Step 13: Test the connectivity

Everything is in place, so it is time to send a packet. The last octet of an address is preserved by the 1:1 translation, which tells you which address to ping:

Instance Real IP address Address to use from the other side
uc3-inst-oci 10.0.1.92 192.168.10.92
uc3-inst-op-lan 10.0.1.130 192.168.20.130

This is what happens to a ping from uc3-inst-oci to the on-premises host:

Where Source Destination What happens
Leaves uc3-inst-oci 10.0.1.92 192.168.20.130 The route table uc3-rt-oci sends 192.168.20.0/24 to the DRG.
Route lookup in uc3-drgrt-vcn-oci 10.0.1.92 192.168.20.130 The dynamic route for 192.168.20.0/24 points to an IPSec tunnel.
Leaves the DRG on the tunnel attachment 192.168.10.92 10.0.1.130 The DRG applies the reverse of the rule in uc3-policy-oci. The packet now matches the encryption domain.
Arrives at uc3-inst-op-lan 192.168.10.92 10.0.1.130 The host sees a ping from the shadow address of the OCI instance.

And this is the reply:

Where Source Destination What happens
Leaves uc3-inst-op-lan 10.0.1.130 192.168.10.92 The route table uc3-rt-op sends 192.168.10.0/24 to the router 10.0.1.164.
On-premises router 10.0.1.130 192.168.10.92 The packet matches the tunnel selectors and is encrypted.
Enters the DRG on the tunnel attachment 192.168.20.130 10.0.1.92 The rule in uc3-policy-oci translates source and destination.
Route lookup in uc3-drgrt-ipsec 192.168.20.130 10.0.1.92 10.0.1.0/24 points to uc3-att-vcn-oci.

image

image

A packet capture on the on-premises host shows the translation at work. The host never sees the real address 10.0.1.92 of the OCI instance, only its shadow address:

[opc@uc3-inst-op-lan ~]$ sudo tcpdump -ni any icmp
IP 192.168.10.92 > 10.0.1.130: ICMP echo request, id 11, seq 1, length 64
IP 10.0.1.130 > 192.168.10.92: ICMP echo reply, id 11, seq 1, length 64

[!NOTE] NOTE NAT on DRG does not translate DNS. If on-premises systems find the OCI systems by name, or the other way around, the DNS records they use must point to the shadow addresses, not to the real ones.

Conclusion

An OCI VCN and an on-premises network with exactly the same subnet CIDR now communicate over a Site-to-Site VPN. Nothing was re-addressed, and the on-premises router does no NAT at all. It only needs a tunnel with the right selectors and a route to the OCI shadow CIDR. The translation is done by the DRG on the IPSec tunnel attachments, with a single rule.

Compared to the earlier parts, the VPN adds two requirements that are easy to miss. The encryption domain must describe the packets as they are inside the tunnel, which means the shadow CIDR on the Oracle side and the real CIDR on the on-premises side. And because a static DRG route cannot point to an IPSec tunnel, the route to the on-premises shadow CIDR has to be learned from the tunnel, by adding that shadow CIDR to the on-premises side of the encryption domain and importing it into the DRG route table of the VCN attachment.

The rule itself follows the same principle as in the rest of the series: it is written for traffic that enters the DRG through the attachment that holds the policy. In NAT on DRG - Part 4 - Shared services VCN the same technique is used with source NAT only, to give several overlapping VCNs access to one shared services VCN.

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.