Scale and optimize Jenkins on Oracle Cloud Infrastructure Container Engine for Kubernetes
Jenkins is a CI/CD system still used by developers to automatically verify, build and deploy their code, so that they can focus only on delivering new features.
With the release of Kubernetes, distributed and shared-nothing architectures have gradually become the norm, so much that all the most recent CI/CD systems are deployed on Kubernetes. As for Jenkins, the controller/agent architecture might not be scalable enough for large organizations and development teams, but thanks to the effort of the community it is possible to containerize Jenkins and deploy it in a Cloud Native way.
You can host Jenkins on Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) to centralize your build automation and scale it as your software projects grow. By implementing a scalable Jenkins on OKE, developers will be able to develop new services with agility and also scale their CI/CD pipelines easily.
Architecture
This high level reference architecture shows an example of multiple Jenkins deployments on OKE. By deploying Jenkins on OKE as a Service it is possible to scale as needed.
Each Jenkins controller pod has a persistent volume attached, to keep pipeline history and the additional plugins installed by teams. The Jenkins agents will be created every time a pipeline job is triggered and destroyed automatically at the end.
Every Jenkins instance can be exposed either individually through a Load Balancer or by using a Kubernetes Ingress.
The following diagram illustrates this reference architecture.
oci-jenkins-oke-arch-oracle.zip
The architecture has the following components:
- Region
An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).
- Availability domains
Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain is unlikely to affect the other availability domains in the region.
- Virtual cloud network (VCN) and subnets
A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.
- Load balancer
The Oracle Cloud Infrastructure Load Balancing service provides automated traffic distribution from a single entry point to multiple servers in the back end.
- Security list
For each subnet, you can create security rules that specify the source, destination, and type of traffic that must be allowed in and out of the subnet.
- Container Engine for Kubernetes
Oracle Cloud Infrastructure Container Engine for Kubernetes is a fully managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud. You specify the compute resources that your applications require, and Container Engine for Kubernetes provisions them on Oracle Cloud Infrastructure in an existing tenancy. Container Engine for Kubernetes uses Kubernetes to automate the deployment, scaling, and management of containerized applications across clusters of hosts.
Recommendations
- VCN
When you create a VCN, determine the number of CIDR blocks required and the size of each block based on the number of resources that you plan to attach to subnets in the VCN. Use CIDR blocks that are within the standard private IP address space.
Select CIDR blocks that don't overlap with any other network (in Oracle Cloud Infrastructure, your on-premises data center, or another cloud provider) to which you intend to set up private connections.
After you create a VCN, you can change, add, and remove its CIDR blocks.
When you design the subnets, consider your traffic flow and security requirements. Attach all the resources within a specific tier or role to the same subnet, which can serve as a security boundary.
- Security
Use Oracle Cloud Guard to proactively monitor and maintain the security of your OCI resources in Oracle Cloud Infrastructure proactively. Cloud Guard uses detector recipes that you can define to examine your resources for security weaknesses and to monitor operators and users for risky activities. When any misconfiguration or insecure activity is detected, Cloud Guard recommends corrective actions and assists with taking those actions, based on responder recipes that you can define.
For resources that require maximum security, Oracle recommends that you use security zones. A security zone is a compartment associated with an Oracle-defined recipe of security policies that are based on best practices. For example, the resources in a security zone must not be accessible from the public internet and they must be encrypted using customer-managed keys. When you create and update resources in a security zone, OCI validates the operations against policies in the security-zone recipe, and denies operations that violate any of the policies.
- Load balancer bandwidth
While creating the load balancer, you can either select a predefined shape that provides a fixed bandwidth, or specify a custom (flexible) shape where you set a bandwidth range and let the service scale the bandwidth automatically based on traffic patterns. With either approach, you can change the shape at any time after creating the load balancer.
- Container Engine for Kubernetes and compute shapes
As a minimum, two node pools are deployed on Container Engine for Kubernetes: one for the Jenkins controllers, and another one for the Jenkins agents.
To save costs and optimize the whole solution, it is recommended to use VM.Standard.A1.Flex instances for Jenkins controllers, as Jenkins is fully compatible with ARM instances. For the agent node pool, the suggestion is to use preemptible instances as generally the pipeline jobs are short-lived and fault tolerant.
- Oracle Cloud Infrastructure
Registry
Oracle Cloud Infrastructure Registry is a private OCI-compliant container repository to host images. In this architecture it is optional and can be used to host your own Jenkins custom images with all the necessary plugins installed. Creating a custom Jenkins controller image is recommended to enforce security and package all the commonly used plugins in one single container image.
Considerations
Consider the following points when deploying this reference architecture.
- Performance and scalability
By using Kubernetes, it is easy to scale both Jenkins controller and agents as the development team grows or shrinks. Furthermore, as the agent pods are created every time there is a pipeline job running, it is possible to customize the resources used by these pods on a per pipeline base.
- Availability
To maximize availability, it is recommended to have multiple OKE nodes provisioned in different availability or fault domains. In addition, to protect Jenkins from voluntary and involuntary disruptions, it is possible to use standard Kubernetes mechanism, such as pod disruption budgets and pod anti-affinity.
- Cost
The aim of this architecture is also to minimize costs when using Jenkins on OKE. It is possible to use ARM nodes for Jenkins controllers, which cost a quarter of the flexible compute instances. As for Jenkins agents, preemptible instance nodes enable customers to save half of the budget.