Note:

Maximize IPSec Bandwidth using Oracle Interconnect for Microsoft Azure

Introduction

Encryption is important and this tutorial will show you how to enable encryption on our interconnected regions we have with Microsoft Azure around the world.

This redundant interconnect is based on private virtual circuits, which are low predictable network latency on a private connection with ExpressRoute from Microsoft Azure and FastConnect from Oracle Cloud Infrastructure (OCI) side. Those private virtual circuit are not encrypted by default.

This tutorial will use Oracle Interconnect for Microsoft Azure (Oracle Interconnect for Azure), adding encryption from IPSec tunnels and also show how to maximise the bandwidth on the IPSec tunnels since that can be a limiting factor for some use cases.

Note: IPSec on top of OCI FastConnect is not limited to this Oracle Interconnect for Azure, it can be used on any OCI FastConnect. We just used it for this tutorial.

Objectives

Create encrypted tunnels over the Microsoft Azure/OCI interconnect to validate possible network bandwidth on combined IPSec tunnels. To find location where this is possible, see Interconnect for Azure.

Those connections are based upon Microsoft Azure ExpressRoute and OCI FastConnect. When creating these interconnects, they are using a private virtual circuit which are acceptable for most customers in most of the cases, but some customers require encryption on the network links, and this tutorial is for those.

We will create 8 IPSec tunnels on one virtual circuit to utilize the full bandwidth on the encrypted connection since IPSec tunnels might not reach the same bandwidth per tunnel as the underlying virtual circuit.

This option with encrypted traffic over a dedicated private virtual circuit may also be applied to our interconnect to Google Cloud, and FastConnect connections between on-premises and OCI, but setup will vary.

Logical Network Design

There are options to allow only IPSec encrypted traffic on the connection or allow both unencrypted and encrypted traffic on the same virtual circuit. In this tutorial, we are going to allow both on the interconnect, but allow only encrypted traffic to reach VCN1 and unencrypted traffic to reach VCN2 in OCI, both from source in Microsoft Azure.

In real world implementations, the option to allow only IPSec traffic on a virtual circuit might be most appropriate for customers that has strict requirements on allowing only encrypted traffic.

Logical Network Design

Bandwidth with IPSec

IPSec tunnels encrypts and decrypts traffic as fast as they can but are bound to encryptions algorithms protocols that is supported by the two endpoints who builds up the tunnels and it is required CPU cycles to encrypt/decrypt, which in turn will limit the amount of traffic that can pass the tunnel per second. To overcome this limit per tunnel, it is possible to create several tunnels and make use of Equal Cost Multi-Pathing (ECMP), which can route packets over several tunnels if both ends supports ECMP.

Note: Single network stream will only use one tunnel. To make use of ECMP you need to spread load to different endpoints/ports, which normally is the case in real world. Several endpoints on one side connects to several endpoints on other side.

ECMP Enabled on IPSEC traffic

Detailed Network Setup

The following image shows the network setup in both Microsoft Azure and OCI. Microsoft Azure side consist of a normal hub/spoke setup where the network gateways are located in the hub vNet and is peered with spoke vNET (here called azure-vNET) which act as a source when doing these tests. On OCI side, the setup with IPSec over OCI FastConnect requires that we create separate route tables for virtual circuit and IPSec attachment.

Detailed Network setup

Traffic Flow

To allow only encrypted traffic between Microsoft Azure vNET and VCN1 and allow unencrypted traffic to VCN2, we have separate route tables for VCN1 and VCN2 in OCI and:

The traffic flow is shown in the the following image.

Detailed Network setup including network traffic flow

Prerequisites

Task 1: Preparation

Here are some info that is good to know when doing preparation setup.

Microsoft Azure ExpressRoute gateway

We used ultra performance SKU since we wanted to enable FastPath to minimize network latency between Microsoft Azure and OCI and we needed to have a bandwidth of at least 5Gbps. For information on differences between SKU’s of ExpressRoute Gateways, see https://learn.microsoft.com/en-us/azure/expressroute/expressroute-about-virtual-network-gateways.

Microsoft Azure VPN gateway

We chose a VPN gateway SKU that can handle the traffic we planning to push through. VpnGw4 supports 5Gbps throughput and the interconnect virtual circuit that we will deploy is also 5Gbps. For more information, see About VPN gateway SKUs. When it comes to VPN gateway, one important thing is to enable private IP. It was not possible to enable private IP during deployment from the Microsoft Azure portal, so needed to create VPN Gateway with public IP first and then enable to private IPs after deployment.

Enable private IP in VNG for VPN

In OCI, VCN1 and VCN2 attachments need to use different route tables.

Setup should at this stage be like below.

Starting Point

Task 2: Create Route Tables and Import Route Distributions

Create these route tables and import route distributions. Keep them all empty.

  1. Go to the OCI Console, navigate to Networking, Customer Connectivity, Dynamic Routing Gateway, DRG and click your DRG.

  2. Create the following route tables and import route distributions.

    • VCN1 route table.
    • VCN2 route table.
    • Virtual circuit route table.
    • IPSec route table.
    • VCN1 import routes.
    • VCN2 import routes.
    • Virtual Circuit import routes.
    • IPSec import routes.

    Leave all route tables and import route distributions empty.

Task 3: Assign Correct Import Route Distribution to Each Route Table

Click each of the route table names, Edit, select Enable import route distribution and then click drop-down menu to select the correct import route distribution. For VCN1, we enabling ECMP as well.

Mapping

After editing, the assignment should look like this:

Assignment

Task 4: Create Oracle Interconnect for Azure

Follow this documentation: Setting Up a Connection.

One exception is that the documentation does not explain the new capabilities to allow IPSec only traffic or allow all traffic.

Interconnect

Since we plan to allow both encrypted and non-encrypted traffic on the virtual circuit in this tutorial, we need to choose All traffic, rest are the same.

Task 5: Specify Route Tables for OCI FastConnect and VCNs

Since default is to use the autogenerated route tables for each attachment, we need to change them to the ones we have created.

Attachment Name DRG Route table
FastConnect VC Attachment Virtual Circuit route table
VCN1 attachment VCN1 route table
VCN2 attachment VCN2 route table

To do this, we need to edit the attachments. Click each of the attachment, Edit and Show advanced options, then select the DRG route table for each.

DRG route table for VCN attachment

The VCN should look like this:

VCN attachments - route tables

The virtual circuit should look like this:

VCN attachments - route tables

Task 6: Create IPSec VPN Tunnels

OCI Site-to-Site VPN connection use one IP address at remote end (Customer-Premises Equipment (CPE) from OCI perspective) and use one IP address per tunnel at OCI head end. For Border Gateway Protocol (BGP) session, we need to tweak the setup a bit to make use of both tunnels for each of the four VPN connection we will setup.

  1. Get the Microsoft Azure Virtual Network Gateway (VNG) VPN private IP address.

    To see the private IP for the VPN gateway at Microsoft Azure side, you need to click See more. This will be used for terminating the VPN at Microsoft Azure side.

    private IP 1

    Then it will show private IP address.

    private IP 2

  2. Create BGP IP in Microsoft Azure VNG.

    Create 4x custom APIPA BGP IP at Microsoft Azure VPN gateway, one for each VPN connection from OCI side.

    APIPA BGP

  3. Create CPE device in OCI. Now we know the private CPE IP at Microsoft Azure site, which will be used for all VPN tunnels, we will create a virtual representation of the CPE device in Microsoft Azure, in OCI, with the private IP and need to enable Allow IPSec over FastConnect. To enable this, go to Networking, Customer Connectivity, and click Customer-premises equipment.

    Create CPE

  4. Create Site-to-Site VPN in OCI.

    Go to Networking, Customer Connectivity, Site-to-Site VPN and click Create IPSec connection.

    Since we enabled IPSec over OCI FastConnect on the CPE, there will be new options shown which we need to specify.

    • Oracle head end IP (separate IP for each tunnel, see the following table for this setup).
    • Associated virtual circuit (same for all tunnels).
    • Route table for IPSec tunnel (same for all tunnels in this example).

    IPSec over FC settings

    We will use these head end private IP’s at OCI side when creating IPSec tunnels.

    Azure CPE IP VPN Connection name OCI tunnel name OCI headend private IP
      VPN1 Tunnel1 192.168.1.1
      VPN1 Tunnel2 192.168.1.2
      VPN2 Tunnel3 192.168.1.3
    10.30.0.6 VPN2 Tunnel4 192.168.1.4
      VPN3 Tunnel5 192.168.1.5
      VPN3 Tunnel6 192.168.1.6
      VPN4 Tunnel7 192.168.1.7
      VPN4 Tunnel8 192.168.1.8

    From OCI side, this is the configuration used for the BGP setup:

    VPN connection name Tunnel name Azure APIPA BGP IP OCI APIPA BGP IP
    VPN1 Tunnel1 169.254.21.5/31 169.254.21.4/31
    VPN1 Tunnel2 169.254.21.5/30 169.254.21.6/30
    VPN2 Tunnel3 169.254.21.9/31 169.254.21.8/31
    VPN2 Tunnel4 169.254.21.9/30 169.254.21.10/30
    VPN3 Tunnel5 169.254.21.13/31 169.254.21.12/31
    VPN3 Tunnel6 169.254.21.13/30 169.254.21.14/30
    VPN4 Tunnel7 169.254.21.17/31 169.254.21.16/31
    VPN4 Tunnel8 169.254.21.17/30 169.254.21.18/30

    To establish VPN connection to Microsoft Azure, use the standard documentation, starting from Create IPSec Connection and remember the above special setup for IPSec over OCI FastConnect. For more information, see VPN Connection to Azure.

    Need to do this for all 4 VPN connections (8 tunnels).

Task 7: Create Local Network Gateway (LNG) in Microsoft Azure

LNG is the virtual representation of VPN endpoint in OCI. Create 8 LNG with the parameters as shown in the following table.

LNG

Parameters for all LNG is shown in table. When creating LNG, these parameters were used (enable BGP settings).

Name Endpoint IP address ASN number BGP peer IP address
OCI1 192.168.1.1 31898 169.254.21.4
OCI2 192.168.1.2 31898 169.254.21.6
OCI3 192.168.1.3 31898 169.254.21.8
OCI4 192.168.1.4 31898 169.254.21.10
OCI5 192.168.1.5 31898 169.254.21.12
OCI6 192.168.1.6 31898 169.254.21.14
OCI7 192.168.1.7 31898 169.254.21.16
OCI8 192.168.1.8 31898 169.254.21.18

The following image shows a visual representation of each VPN connection.

VPN Connection representation

Task 8: Create Connection between vng-VPN and each LNG in Microsoft Azure

  1. Go to the Site-to-Site VPN page, click Show and Copy to copy the Shared secret (PSK) from OCI console for each tunnel.

    VPN Connection 2

  2. In Microsoft Azure, go to vng-VPN and create connection to each of the LNG.

    VPN Connection 3

  3. Paste the shared secret into Microsoft Azure console for each connection and click drop-down menu to select the correct BGP address, VNG and Local Network Gateway. The BGP IP’s we used in Task 6.

    VPN Connection 4

    After this all tunnels should go up within a couple of minutes. Verify in the Microsoft Azure:

    VPN Connection 5

    Verify in the OCI:

    VPN Connection 6

Task 9: Test the Connection

It is now time to see how this works. We will test bandwidth from one Microsoft Azure VM to VMs in OCI, 4x VMs are placed within a VCN that only allow IPSec traffic and one VM in another VCN that allow unencrypted traffic through Microsoft Azure ExpressRoute/OCI FastConnect.

VMs on both side has enough of cores to handle this network bandwidth. We will use TCP protocol as test scenario since most applications use that protocol.

We understand that network latency will impact bandwidth but since we wanted to capture the difference between with and without IPSec, it will not be noticeable. The aim here is not to show the maximum bandwidth, it is to understand the impact of having IPSec encryption on top on an interconnect compared to just using the interconnect without encryption.

The reason for having 4 VMs on OCI side for IPSec traffic is that one VPN tunnel cannot saturate the 5Gbps virtual circuit we have for interconnect so we will use ECMP to distribute the traffic through different VPN tunnels to it is endpoints, which each has different IP’s and ports it responds to. This is normally the case in real world as well, many source endpoints talk to many destination endpoints.

Test Setup 1

For bandwidth tests, we will use iperf3: https://iperf.fr/

Server on OCI side (listening side):

OCI VM1_a = $ iperf3 -s -p 5201
OCI VM1_b = $ iperf3 -s -p 5202
OCI VM1_c = $ iperf3 -s -p 5203
OCI VM1_d = $ iperf3 -s -p 5204
OCI VM2   = $ iperf3 -s -p 5201

Task 9.1: Test the Bandwidth on Virtual Circuit

Run the following command to test maximum bandwidth on virtual circuit, from Microsoft Azure VM to OCI VM2.

$ iperf3 -c <OCIVM2 IP> -p 5201
Summary output:
[ ID]   Interval         Transfer     Bitrate         Retr
[  6]   0.00-10.00  sec  6.13 GBytes  5.27 Gbits/sec  336296  sender
[  6]   0.00-10.04  sec  6.12 GBytes  5.24 Gbits/sec          receiver

We can see that we utilized the 5Gbps bandwidth for the virtual circuit, achieving 5.24Gbps.

Test 9.2: Test the Combined IPSec Bandwidth using ECMP

Run the following command to test maximum bandwidth on IPSec tunnels, from Microsoft Azure VM to 4x VMs on OCI at the same time.

$ iperf3 -c <OCIVM1_a IP> -p 5201 & iperf3 -c <OCIVM1_b IP> -p 5202 &
iperf3 -c <OCIVM1_c IP> -p 5203 & iperf3 -c <OCIVM1_d IP> -p 5204 &

This is one of the test runs we did.

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.81 GBytes  1.56 Gbits/sec  4018   sender
[  5]   0.00-10.04  sec  1.81 GBytes  1.55 Gbits/sec         receiver

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.24 GBytes  1.07 Gbits/sec  32114  sender
[  5]   0.00-10.04  sec  1.24 GBytes  1.06 Gbits/sec         receiver

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   931 Mbits/sec  1016   sender
[  5]   0.00-10.04  sec  1.08 GBytes   921 Mbits/sec         receiver

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.78 GBytes  1.53 Gbits/sec  63713  sender
[  5]   0.00-10.04  sec  1.78 GBytes  1.52 Gbits/sec         receiver

Summary for this test runs are 5.05Gbps (1.55 + 1.06 + 0.92 + 1.52). If we take an average over the test runs we got 4.51Gbps.

Bandwidth result 1

So we can utilize almost all of the network bandwidth with IPsec encryption over an OCI FastConnect virtual circuit.

Task 9.3: Test the Bandwidth using One IPSec Tunnel

There are cases where ECMP cannot be used (not supported on other end), so we measured which bandwidth we can get out of just one IPSec tunnel. One VM on Microsoft Azure to one VM on OCI.

Here is the summary table show a fairly good network bandwidth for one IPSec encrypted tunnel.

Bandwidth result 1 tunnel

We have now shown that customer that would like to encrypt network traffic on a OCI FastConnect link using IPSec encryption can utilize several VPN tunnels and increase total bandwidth for IPSec traffic beyond just one VPN tunnel, getting almost the same total bandwidth as the underlying virtual circuit.

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.