Implement a SIEM System in Splunk Using Logs Streamed from Oracle Cloud
A security information and event management (SIEM) system is a critical operations tool to manage the security of your cloud resources. Oracle Cloud Infrastructure includes native threat detection, prevention, and response capabilities, which you can leverage to implement an efficient SIEM system using Splunk.
Splunk Enterprise administrators can use the Logging and Streaming services with OCI Logging Addon for Splunk to stream logs from resources in the cloud to an existing or new Splunk environment. Administrators can also integrate with other Splunk plugins and data sources, such as threat intelligence feeds, to augment the generation of alerts based on log data.
Architecture
In this architecture, the Logging service captures logs from the load balancer and the virtual cloud network (VCN) flow. There is a separate stream for each log, and each log is connected to its stream with a service connector hub. As a Splunk Enterprise administrator, you can collect the streamed data for further analysis by using the Logging Addon for Splunk.
The following diagram illustrates this reference architecture.
Description of the illustration siem-logging-oci.png
The architecture has the following components:
- Region
An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).
- Availability domain
Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain is unlikely to affect the other availability domains in the region.
- Virtual cloud network (VCN) and subnets
A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.
- Load balancer
The Oracle Cloud Infrastructure Load Balancing service provides automated traffic distribution from a single entry point to multiple servers in the back end.
- Virtual machines (VMs)
The Oracle Cloud Infrastructure Compute service enables you to provision and manage compute hosts in the cloud. You can launch compute instances with shapes that meet your resource requirements for CPU, memory, network bandwidth, and storage. After creating a compute instance, you can access it securely, restart it, attach and detach volumes, and terminate it when you no longer need it.
- LoggingLogging is a highly scalable and fully managed service that provides access to the following types of logs from your resources in the cloud:
- Audit logs: Logs related to events emitted by the Audit service.
- Service logs: Logs emitted by individual services such as API Gateway, Events, Functions, Load Balancing, Object Storage, and VCN flow logs.
- Custom logs: Logs that contain diagnostic information from custom applications, other cloud providers, or an on-premises environment.
- Streaming
Oracle Cloud Infrastructure Streaming provides a fully managed, scalable, and durable storage solution for ingesting continuous, high-volume streams of data that you can consume and process in real time. You can use Streaming for ingesting high-volume data, such as application logs, operational telemetry, web click-stream data; or for other use cases where data is produced and processed continually and sequentially in a publish-subscribe messaging model.
- Service connectors
Oracle Cloud Infrastructure Service Connector Hub is a cloud message bus platform that orchestrates data movement between services in OCI. You can use it to move data between services in Oracle Cloud Infrastructure. Data is moved using service connectors. A service connector specifies the source service that contains the data to be moved, the tasks to perform on the data, and the target service to which the data must be delivered when the specified tasks are completed.
- Logging Addon for Splunk
Logging Addon for Splunk is a plugin that ingests logs and other data directly from the Streaming service. You can use the plugin with Splunk Enterprise (version 8.0 or higher).
Recommendations
Use the following recommendations as a starting point. Your requirements might differ from the architecture described here.
- VCN
When you create a VCN, determine the number of CIDR blocks required and the size of each block based on the number of resources that you plan to attach to subnets in the VCN. Use CIDR blocks that are within the standard private IP address space.
Select CIDR blocks that don't overlap with any other network (in Oracle Cloud Infrastructure, your on-premises data center, or another cloud provider) to which you intend to set up private connections.
After you create a VCN, you can change, add, and remove its CIDR blocks.
When you design the subnets, consider your traffic flow and security requirements. Attach all the resources within a specific tier or role to the same subnet, which can serve as a security boundary.
- Splunk version
The logging addon for Splunk works with Python 3 on Splunk 8.0. For lower versions, Splunk recommends using a heavy forwarder running Splunk 8.0 to ingest the data and forward it to the indexer for the lower version.
- Logging
This architecture captures logs from the Load Balancing service and VCN flow logs. Each compute instance attached to a VCN has one or more virtual network interface cards (VNICs). Use VCN flow logs to troubleshoot security rules and to audit the traffic to and from the VNICs.
- Access control
The logging addon for Splunk supports access both by instance principals and using API signing keys. Oracle recommends using an instance principal, to avoid storing long-lived tokens. If you're not using an instance principal, use an API signing key.
Depending on the access method that you choose, define a least-privilege policy as shown in the following examples:- If you choose the instance-principal access method:
Allow dynamic-group SplunkInstance to use stream-pull in compartment <compartment> - If you choose the API signing key method:
Allow group Splunk to use stream-pull in compartment <compartment>
- If you choose the instance-principal access method:
- Service gateway
If you deploy a Splunk forwarder inside your tenancy, use a service gateway to communicate with the Streaming service endpoints.
Considerations
When implementing this architecture, consider the following factors:
- Performance
The architecture scales based on the number of events generated by the log group. Logging is a highly scalable service.
Streaming is also highly scalable, and is used as a temporary conduit to store event information sent from the Logging service. It also acts as a load balancer. Consider adjusting the number of partitions and streams, based on the amount of log data that you expect.
- Availability
Oracle ensures high availability of the Streaming and Logging services, which are cloud native and fully managed services.
Streaming includes the following high-availability capabilities:- Constant flow of log data
- Multi-threaded and horizontally scalable service
- Near real-time ingestion
- Resilience against short-term outages
- Optimized for efficient data usage
- ExtensibilityApart from VCN flow logs and load balancer logs, you can stream other logs to Splunk by using the logging addon for Splunk. The following are a few examples:
- IAM audit logs
- Function invocation logs
- API Gateway access and execution logs
- Events service logs
Consider deploying Splunk on a compute instance in the cloud. The Explore More section includes a link to the Terraform stack that you can use to provision Splunk on Oracle Cloud Infrastructure.
Deploy
The Terraform code for this reference architecture is available in GitHub. You can pull the code into Oracle Cloud Infrastructure Resource Manager with a single click, create the stack, and deploy it. Alternatively, you can download the code from GitHub to your computer, customize the code, and deploy the architecture by using the Terraform CLI.
- Deploy by using Oracle Cloud Infrastructure Resource
Manager:
- Click
If you aren't already signed in, enter the tenancy and user credentials.
- Review and accept the terms and conditions.
- Select the region where you want to deploy the stack.
- Follow the on-screen prompts and instructions to create the stack.
- After creating the stack, click Terraform Actions, and select Plan.
- Wait for the job to be completed, and review the plan.
To make any changes, return to the Stack Details page, click Edit Stack, and make the required changes. Then, run the Plan action again.
- If no further changes are necessary, return to the Stack Details page, click Terraform Actions, and select Apply.
- Click
- Deploy by using the Terraform CLI:
- Go to GitHub.
- Clone or download the repository to your local computer.
- Follow the instructions in the
READMEdocument.