Note:

This tutorial requires access to Oracle Cloud. To sign up for a free account, see Get started with Oracle Cloud Infrastructure Free Tier.
It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.

Secure Kubernetes Web Services using Nginx Ingress Controller, OAuth2 Proxy, and Oracle Identity Domains

Introduction

Securing Kubernetes hosted web services should be easy. Whether your application lacks built-in support for external Identity Provider (IdP) integration or you need a Single Sign-On (SSO) solution to simplify the management and reduce the need for individual authentication mechanisms per service, OAuth2 Proxy offers the flexibility to connect any application to almost any Identity Provider (IdP).

OAuth2 Proxy acts as a reverse proxy sitting in front of your Kubernetes services. It intercepts traffic, redirects users to an external authorization server (like Okta or GitHub) for login, and validates access tokens before forwarding requests to backend services.

Objective

Deploy a simple web application to Kubernetes, expose it using the Nginx Ingress Controller, and configure OAuth2 Proxy to use Oracle Identity Domains as IdP.

Prerequisites

Access to an Oracle Cloud Infrastructure (OCI) tenancy.
An Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) cluster configured to use a public subnet for load balancers.
(Optional) Access to update records for a Domain Name System (DNS) domain.

Task 1: Create an OKE Cluster

Log in to the OCI Console, navigate to Oracle Container Engine for Kubernetes, and click Create.
In the Create cluster wizard, click Quick Create.
Enter the following information and click Next.
- Name: Enter the cluster name.
- Kubernetes version: Select the version you want to use.
- Kubernetes API endpoints: Select Public endpoint.
- Node type: Select Managed.
- Kubernetes worker nodes: Select Private workers.
- Node count: 1.
Review and click Create Cluster. Wait for the cluster to become available.
Update the security list associated with the service subnet.
1. Click the Service subnet.
2. Click Security List and add the following rules.
  - Ingress:
  - Egress:
Update the security list associated with the worker nodes subnet to allow connections from the load balancers.
1. Click the VCN name.
2. Under Resources, select Security Lists and click the nodes security list.
3. Add the following ingress rule to the nodes security list.
4. Click Add Ingress Rules.

Task 2: Set up the Nginx Ingress Controller

Once the cluster is available, you can access the cluster using OCI Cloud Shell. Click Access Cluster, and copy and run the following command in OCI Cloud Shell.
Run the kubectl get node command to get the list of available nodes.
Add the Nginx Ingress Controller helm repository. For more information, see Overview of Ingress NGINX Controller and helm.
```
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
```

Create the values-override.yaml file.

controller:
  allowSnippetAnnotations: true
  service:
    annotations:
      oci.oraclecloud.com/load-balancer-type: "lb"
      service.beta.kubernetes.io/oci-load-balancer-backend-protocol: "TCP"
      service.beta.kubernetes.io/oci-load-balancer-shape: "flexible"
      service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "10"
      service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "10"

Deploy the Nginx Ingress Controller.

helm install ingress-nginx ingress-nginx/ingress-nginx -f values-override.yaml

Save the following yaml file named application.yaml.

---
# Create ClusterIP service
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - port: 5000
    targetPort: 5000
  selector:
    app: httpbin
---
# Deployment of a sample web application
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 2
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: mendhak/http-https-echo
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 5000
        env:
        - name: HTTP_PORT
          value: "5000"

Create the application resources.
```
kubectl apply -f application.yaml
```
Create a DNS record for the public IP address associated with the load balancer.
- Get the public IP address of the load balancer.
```
$ kubectl get svc ingress-nginx-controller
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller   LoadBalancer   10.96.152.30   158.180.61.74   80:31957/TCP,443:30838/TCP   32s
```
- Create the DNS record. If you do not have a DNS server, we will rely on free wildcard DNS service nip.io.
In this case the FQDN is 158-180-61-74.nip.io.

Save the following yaml file named ingress.yaml.

Note: Make sure to update the host entries with your FQDN.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin-ingress-nginx
spec:
  ingressClassName: nginx
  tls:
  - hosts:
      - "<YOUR-FQDN>"
  rules:
    - host: "<YOUR-FQDN>"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: httpbin
                port:
                  number: 5000

Run the following command to create the ingress resource.
```
kubectl apply -f ingress.yaml
```

Test the connectivity to the service.

$ curl -k https://<YOUR-FQDN> 
{
  "path": "/",
  "headers": {
    "host": "158-180-61-74.nip.io",
    "x-request-id": "b0320217391a922acecbfe10758b3ffe",
    "x-real-ip": "10.0.10.167",
    "x-forwarded-for": "10.0.10.167",
    "x-forwarded-host": "158-180-61-74.nip.io",
    "x-forwarded-port": "443",
    "x-forwarded-proto": "https",
    "x-forwarded-scheme": "https",
    "x-scheme": "https",
    "user-agent": "curl/7.87.0",
    "accept": "*/*"
  },
  "method": "GET",
  "body": "",
  "fresh": false,
  "hostname": "158-180-61-74.nip.io",
  "ip": "10.0.10.167",
  "ips": [
    "10.0.10.167"
  ],
  "protocol": "https",
  "query": {},
  "subdomains": [
    "158-180-61-74"
  ],
  "xhr": false,
  "os": {
    "hostname": "httpbin-644874bcdb-ll4mb"
  },
  "connection": {}
}

Task 3: Set up a Confidential Application in Oracle Identity Domains

Navigate to Oracle Identity Domains in the OCI Console and click the OracleIdentityCloudService domain.
Select Integrated Applications from the left-side menu and click Add application.
Select Confidential Application and click Launch workflow.
Enter the application Name, Description and click Next.
Configure the application oAuth parameters.
1. In the Authorization section, select Authorization code.
2. Enter the following URLs and click Next.
  - Redirect URL: https://<YOUR-FQDN>/oauth2/callback.
  - Post-logout redirect URL: https://<YOUR-FQDN>.
  - Logout URL: https://<YOUR-FQDN>/oauth2/sign_out.
In Web tier policy, select Skip and do later and click Finish.
Click Activate to enable the application.
Navigate to Groups in the left-side menu and associate the groups of users allowed to authenticate with this application.
Note down the Client ID and Client Secret.
Click OracleIdentityCloudService domain and expand the Domain URL.

For example: The Identity Domain URL will be https://idcs-01234567890abcdef.identity.oraclecloud.com (we strip the port number).

Task 4: Deploy the OAuth2 Proxy

We are using the OAuth2 Proxy to handle OAuth2 or OpenID Connect (OIDC) complexity and ensure all the requests forwarded to the applications are authenticated.

Save the following yaml file named oauth2-proxy.yaml and replace the placeholders with the applicable values.

Note:

<Identity Domain URL>: https://idcs-01234567890abcdef.identity.oraclecloud.com/

<Identity Domain FQDN>: idcs-01234567890abcdef.identity.oraclecloud.com

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: oauth2-proxy
  template:
    metadata:
      labels:
        k8s-app: oauth2-proxy
    spec:
      containers:
      - args:
        - --provider=oidc
        - --provider-display-name="Oracle Identity Domains"
        - --oidc-issuer-url=<Identity Domain URL>
        - --redirect-url=https://<YOUR-FQDN>/oauth2/callback
        - --upstream=file:///dev/null
        - --http-address=0.0.0.0:4180
        - --email-domain=*
        - --set-xauthrequest=true
        - --session-cookie-minimal=true
        - --whitelist-domain=<Identity Domain FQDN>
        env:
        - name: OAUTH2_PROXY_CLIENT_ID
          value: "<APPLICATION_CLIENT_ID>"
        - name: OAUTH2_PROXY_CLIENT_SECRET
          value: "<APPLICATION_CLIENT_SECRET>"
        # docker run -ti --rm python:3-alpine python -c 'import secrets,base64; print(base64.b64encode(base64.b64encode(secrets.token_bytes(16))));'     
        - name: OAUTH2_PROXY_COOKIE_SECRET
          value: "<OUTPUT_OF_THE_ABOVE_DOCKER_COMMAND>"
        image: quay.io/oauth2-proxy/oauth2-proxy:latest
        imagePullPolicy: Always
        name: oauth2-proxy
        ports:
        - containerPort: 4180
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
spec:
  ports:
  - name: http
    port: 4180
    protocol: TCP
    targetPort: 4180
  selector:
    k8s-app: oauth2-proxy
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: oauth2-proxy
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      set $xheader "";
      if ( $request_uri = "/oauth2/sign_out" ){
        set $xheader "https://<Identity Domain FQDN>/oauth2/v1/userlogout;
      }
      proxy_set_header X-Auth-Request-Redirect ${xheader};
spec:
  ingressClassName: nginx
  rules:
  - host: <YOUR-FQDN>
    http:
      paths:
      - path: /oauth2
        pathType: Prefix
        backend:
          service:
            name: oauth2-proxy
            port:
              number: 4180
  tls:
  - hosts:
    - <YOUR-FQDN>

Deploy the OAuth2 Proxy to OKE.

kubectl apply -f oauth2-proxy.yaml

Confirm the pod is running.

kubectl get pods -l k8s-app=oauth2-proxy

Configure Nginx to use OAuth2 Proxy for request authentication.

Update the ingress.yaml file with the auth annotations.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin-ingress-nginx
  annotations:
    nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
    nginx.ingress.kubernetes.io/auth-response-headers: "x-auth-request-user, x-auth-request-email"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
      - "<YOUR-FQDN>"
  rules:
    - host: "<YOUR-FQDN>"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: httpbin
                port:
                  number: 5000

Try to connect to your application. You should be redirected to the Oracle authentication page. Once the authentication is successful, you should be redirected to your application.

There are two headers identifying the authenticated user.
```
"x-auth-request-user": "xxxx@oracle.com"
"x-auth-request-email": "xxxx@oracle.com"
```
To sign out from the application, you can go to /oauth2/sign_out on your page.
```
https://<YOUR-FQDN>/oauth2/sign_out
```
Known Issue: Post sign out redirect is not working. This issue is tracked here: Add an id_token_hint parameter to the logout provider URL for OIDC provider.

Task 5: Clean up Resources

Run the following command to delete all the Kubernetes deployed resources.

kubectl delete -f oauth2-proxy.yaml -f ingress.yaml -f application.yaml
helm uninstall ingress-nginx

Delete the application from Oracle Identity Domains.
Delete the OKE cluster.
Delete the VCN created for the OKE cluster.
Note:
- You need to wait for the OKE cluster and node pool deletion to end.
- The VCN name contains the name of the OKE cluster: oke-vcn-quick-<oke-cluster-name>-<random-string>.

Acknowledgments

Author - Andrei Ilas (Principal Cloud Architect)

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.

Title and Copyright Information

Secure Kubernetes Web Services using Nginx Ingress Controller, OAuth2 Proxy, and Oracle Identity Domains

F96412-01

April 2024

Oracle and/or its affiliates.

Secure Kubernetes Web Services using Nginx Ingress Controller, OAuth2 Proxy, and Oracle Identity Domains

Introduction

Objective

Prerequisites

Task 1: Create an OKE Cluster

Task 2: Set up the Nginx Ingress Controller

Task 3: Set up a Confidential Application in Oracle Identity Domains

Task 4: Deploy the OAuth2 Proxy

Task 5: Clean up Resources

Related Links

Acknowledgments

More Learning Resources