Ingest Oracle Cloud Infrastructure WAF Logs By Using Oracle Functions and Events
Oracle Cloud Infrastructure (OCI) Web Application Firewall (WAF) is an Oracle Cloud service that protects your web applications against threats. Logs are available within the WAF Service. This document describes the architecture for compiling WAF logs and forward those logs to OCI Logging for further consumption by a third party, such as Splunk.
Architecture
This architecture describes how to compile WAF logs and forward those logs to OCI Logging for further consumption by a third party, such as Splunk.
The following diagram illustrates this reference architecture.
Description of the illustration waf_logs_function_events.png
The architecture has the following components:
- Region
An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).
- Web Application Firewall
A Web Application Firewall (WAF) filters and monitors HTTP traffic between a web application and the Internet. It is a protocol layer 7 defense that protects web applications from most malicious attacks.
- Availability domains
Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain is unlikely to affect the other availability domains in the region.
- Tenancy
A tenancy is a secure and isolated partition that Oracle sets up within Oracle Cloud when you sign up for Oracle Cloud Infrastructure. You can create, organize, and administer your resources in Oracle Cloud within your tenancy.
- Virtual cloud network (VCN) and subnets
A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.
- Object storage
Object storage provides quick access to large amounts of structured and unstructured data of any content type, including database backups, analytic data, and rich content such as images and videos. Use standard storage for "hot" storage that you need to access quickly, immediately, and frequently. Use archive storage for "cold" storage that you retain for long periods of time and seldom or rarely access.
- LoggingLogging is a highly scalable and fully managed service that provides access to the following types of logs from your resources in the cloud:
- Audit logs: Logs related to events emitted by the Audit service.
- Service logs: Logs emitted by individual services such as API Gateway, Events, Functions, Load Balancing, Object Storage, and VCN flow logs.
- Custom logs: Logs that contain diagnostic information from custom applications, other cloud providers, or an on-premises environment.
Recommendations
- VCN and Subnet
When you attach the OCI function to a subnet and VCN, you must ensure that the subnet can properly reach both Object Storage and the OCI Logging Endpoints.
- Object Storage Service
This service collects the WAF Logs automatically emitted by the WAF Service. You need to ensure this buck has “Emit Events” enabled
- Events Service
This service watches the WAF Log Object Storage Bucket for new Object Creation Events. We recommend filtering for only objects created in this specific bucket.
- Functions Service
This service is triggered on new Object Creation Events.
- Logging
This architecture captures logs from Object storage generated by the WAF Service. We also recommend that you enable Function logging in the event you need to troubleshoot.
- Access controlThe OCI Function used authenticates by resource principal. In order to fulfill this pattern you will need a dynamic group with a policy that allows the following:
Allow dynamic-group WAFLogs to use logging-family in compartment compartmentAllow dynamic-group WAFLogs to read buckets in compartment compartment_nameAllow dynamic-group WAFLogs to read objects in compartment compartment_name
- Service gateway
When you deploy the Function in your Subnets, use a service gateway to communicate with the Object Storage and OCI Logging service endpoints.
Considerations
When implementing this architecture, consider the following factors:
- Performance
The architecture scales based on the number of events generated by WAF service. The OCI Logging service is highly scalable.
Events, Functions, and Streaming are also highly scalable, and is used as a temporary conduit to store event information sent from the Logging service. The Streaming Service also acts as a load balancer. Consider adjusting the number of partitions and streams, based on the amount of log data that you expect.
- Availability
Oracle ensures high availability of the Functions, Events, Streaming, Logging services, which are cloud native and fully managed.
Streaming includes the following high-availability capabilities:- Constant flow of log data
- Multi-threaded and horizontally scalable service
- Near real-time ingestion
- Resilience against short-term outages
- Optimized for efficient data usage
- Visualization and Analysis Once logs are ingested, you can choose from several implementation patterns, such as the following:
- Oracle Logging Analytics
- Ingest Logs into Splunk using the OCI Logging Addon for Splunk
- Visualize Logs by using the OCI App for Splunk