Archive Your Mirrored Network Traffic to OCI Object Storage

Oracle Cloud Infrastructure Virtual Test Access Point (VTAP) is a network traffic mirroring service that captures a copy of your network traffic from a specified source, applies filters to focus on relevant data, and sends it to a target for analysis. This provides opportunities for you to improve your network troubleshooting, security monitoring, network performance analysis, and compliance auditing.

Architecture

This architecture shows how you can use Oracle Cloud Infrastructure (OCI) to archive your mirrored traffic from VTAP to OCI Object Storage.

Archiving your network traffic can be required for compliance reasons. Also, archiving your network traffic has advantages when troubleshooting elusive or intermittent network issues. You can selectively analyze the network capture of past production traffic as needed.

The following diagram illustrates this reference architecture.

Description of the illustration oci-vtap-archiver.png

oci-vtap-archiver-oracle.zip

For illustrative purposes, a simple HTTP Web Server is in a public subnet with its clients in the first private subnet. The clients use the HTTP GET curl command to download files from the HTTP file server. These clients are set up as VTAP sources in this illustration. We mirror only the HTTP traffic with VTAP. The OCI Network Load Balancer receives the mirrored traffic from the VTAP and load balances it among its backend server nodes. These backend nodes then upload the network capture to OCI Object Storage. You can potentially have web servers or a database instance as the source for your VTAP setup in your environment. The rest of the design will typically remain the same in your implementation.

The vertical dotted lines between the following components indicate that additional VTAP flows are configurable: Client 1 to Client #n, acting as VTAP sources, and VTAP Sink 1 to VTAP Sink #m, nodes doing the archival to OCI Object Storage.

The Terraform configuration will create a VCN with the following three subnets:

• Public subnet: Contains a single host, which acts as both an HTTP file server and a jumpbox to access nodes in the two private subnets. You might need jumpbox or bastion server in a public subnet in your production environment to access nodes in a private subnet for troubleshooting or other maintenance purposes.
• Private subnet: Hosts nodes that download a dummy file from the HTTP file server to create HTTP traffic. These nodes act as sources for the VTAP, and their traffic is mirrored by the VTAP with an appropriate capture filter. We'll refer to these nodes as VTAP Source nodes. Each VTAP Source node has its own separate VTAP.
• Private subnet: Contains a network load balancer (NLB) that acts as the target for the VTAPs. OCI Flexible Network Load Balancer has backend nodes that perform network capture of the VTAP traffic as pcap files and archive them to a bucket. We call these nodes VTAP Sink nodes. The VTAP Sink nodes and NLB reside in the same private subnet.

VTAP is configured with a capture filter to capture only network traffic of HTTP GET requests fired by these VTAP Sources, to the HTTP file server in our public subnet. The VTAP is set on the primary VNIC of the VTAP Source nodes.

You can choose the region and compartment for your deployment. All resources are created in the specified region and compartment. The OCI Object Storage bucket to archive the pcap files is also created.

The architecture has the following components:

• Region

  An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).

• Virtual cloud network (VCN) and subnets

  A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.

• VTAP

  A Virtual Test Access Point (VTAP) provides a way to mirror traffic from a designated source to a selected target to facilitate troubleshooting, security analysis, and data monitoring.

• Network load balancer (NLB)

  OCI Flexible Network Load Balancer provides automated traffic distribution from one entry point to multiple servers in a backend set. Network load balancers ensure that your services remain available by directing traffic only to healthy servers based on Layer 3/Layer 4 (IP protocol) data. Here we use OCI Flexible Network Load Balancer to load balance VXLAN UDP traffic to VTAP Sink nodes.

• Object storage

  Oracle Cloud Infrastructure Object Storage provides quick access to large amounts of structured and unstructured data of any content type, including database backups, analytic data, and rich content such as images and videos. You can safely and securely store and then retrieve data directly from the internet or from within the cloud platform. You can scale storage without experiencing any degradation in performance or service reliability. Use standard storage for "hot" storage that you need to access quickly, immediately, and frequently. Use archive storage for "cold" storage that you retain for long periods of time and seldom or rarely access.

• Service gateway

  The service gateway provides access from a VCN to other services, such as Oracle Cloud Infrastructure Object Storage. The traffic from the VCN to the Oracle service travels over the Oracle network fabric and does not traverse the internet.

• Internet gateway

  The internet gateway allows traffic between the public subnets in a VCN and the public internet.

Recommendations

Use the following recommendations as a starting point. Your requirements might differ from the architecture described here.

• VCN

  When you create a VCN, determine the number of CIDR blocks required and the size of each block based on the number of resources that you plan to attach to subnets in the VCN. Use CIDR blocks that are within the standard private IP address space.

  Select CIDR blocks that don't overlap with any other network (in Oracle Cloud Infrastructure, your on-premises data center, or another cloud provider) to which you intend to set up private connections.

  After you create a VCN, you can change, add, and remove its CIDR blocks.

  When you design the subnets, consider your traffic flow and security requirements. Attach all the resources within a specific tier or role to the same subnet, which can serve as a security boundary.

  Use regional subnets.

• Network Load Balancer connection limits

  OCI L3/L4 Network Load Balancer is a free service and auto-scales dynamically based on the traffic. Network load balancers have a default concurrent connection limit of 330,000 connections per Availability Domain (AD). In three AD regions, by default, network load balancers have a concurrent connection limit of one million.

• Security lists

  Use security lists to define ingress and egress rules that apply to the entire subnet.

• Network security groups (NSGs)

  You can use NSGs to define a set of ingress and egress rules that apply to specific VNICs. We recommend using NSGs rather than security lists, because NSGs enable you to separate the VCN's subnet architecture from the security requirements of your application.

• See the vtap.tf file in GitHub for details on the capture filter.
• See the cloud_init/vtap_sink.yml file for details on how tcpdump is configured and decapsulation of VXLAN encapsulated mirrored traffic works.

Considerations

When implementing this solution, consider the following:

• Internet protocol traffic

  This solution is developed and tested only for IPv4 traffic.

• Permissions

  You must have the required Oracle Cloud Infrastructure Identity and Access Management permissions for the chosen compartment and region, to create all the necessary OCI resources for this deployment.

• Configurable parameters

  See the variables.tf file to view all configurable parameters.

• VTAP sources and rules
  • A VTAP must always have a source, a target, and an associated capture filter.
  • A capture filter must always have at least one associated rule.
  • A VNIC can never be a source for more than one VTAP.

Deploy

Download the code from GitHub, customize the code, and deploy it. Terraform will set up all the resources required within your OCI tenancy.

You can use one-click deployment using OCI Resource Manager or download code to deploy from a local development machine.

The links are available on GitHub.

1. Go to GitHub.
2. The Deploy section of README document is displayed.
3. Follow the instructions in the README document.

Explore More

Learn more about Oracle Cloud Infrastructure and network mirroring:

Review these additional resources:

• Virtual Test Access Points (OCI Documentation)
• VTAP sources and targets (OCI Documentation)
• VCN Flow Logs (OCI Documentation)
• Oracle Cloud Infrastructure Documentation
• Best practices framework for Oracle Cloud Infrastructure
• Oracle Cloud Cost Estimator
• Cloud Adoption Framework

Acknowledgments

• Author: Mayur Raleraskar