Note:

Post Messages to Kafka Stream with Resource Principal Authentication using OCI Functions

Introduction

Discover the seamless synergy between Oracle Cloud Infrastructure (OCI) Functions and OCI Streaming in orchestrating real-time data pipelines. In this tutorial, we will explore a secure and efficient approach to message production, leveraging Oracle resource principal authentication method and the singleton design pattern. Learn how resource principal authentication simplifies security management by seamlessly authenticating serverless functions with OCI Streaming services, while the singleton pattern optimizes resource utilization and enhances message production efficiency. We delve into the convergence of OCI Functions and OCI Streaming, empowering developers to architect robust and scalable event-driven solutions effortlessly.

OCI Streaming: OCI has an array of tools for handling data in the cloud, with OCI Streaming being one such service tailored for high-throughput, real-time data streams. By leveraging OCI Streaming, developers can construct scalable and dependable data pipelines that efficiently ingest, process, and distribute data streams.

OCI Functions: OCI Functions is a fully managed, multi-tenant, highly scalable, on-demand, Functions-as-a-Service platform. It is built on enterprise-grade OCI and powered by the Fn Project open source engine. Use OCI Functions (sometimes abbreviated as Functions, and formerly known as Oracle Functions) when you want to focus on writing code to meet business needs.

Resource Principal Authentication: You can use a resource principal to authenticate and access OCI resources. The resource principal consists of a temporary session token and secure credentials that enables the OCI Functions to authenticate itself to other OCI services, such as OCI Streaming.

Note:

Objectives

Prerequisites

Task 1: Set up Dynamic Groups

Go to your domain, click Dynamic Groups and create the following groups.

ALL {resource.type = 'fnfunc', resource.compartment.id = 'pasteYourCompartmentOCID'}

Task 2: Create Policies

Go to Policies and create the following policies.

Allow dynamic-group MyFunctions to {STREAM_INSPECT, STREAM_READ, STREAM_CONSUME, STREAM_PRODUCE} in compartment YOUR-COMPARTMENT-NAME
Allow dynamic-group MyFunctions to read repos in compartment YOUR-COMPARTMENT-NAME

Task 3: Create OCI Container Registry

  1. Go to Developer Services, click Container registry and create a private repository for the Fn image.

    Name: lab/fn-java-streaming.

    T3_1

  2. Check the repositories and take a note of the Namespace.

    T3_1

  3. Open the terminal shell where you have OCI CLI and Docker installed, and proceed with the log in on the registry. Check what is the correct URL for your region. In this tutorial, we are using Brazil East (Sao Paulo) where the registry URL is gru.ocir.io.

    docker login gru.ocir.io
Username: <your container namespace>/youruser
Password: YOUR_AUTH_TOKEN_CREATED_EARLIER

    T3_3

Task 4: Create the Java OCI Functions to receive the Message and Publish to the OCI Streaming Topic

Note Make sure you select your private subnet, the same subnet of your stream pool.

  1. Go to Developer Services, under Functions, click Applications and then click Create application.

    T9_1

  2. Create a few configurations to store OCI Streaming settings.

    Note This configuration variables are used inside this tutorial Java sample code to be able to produce messages to your stream, make sure you pass the correct ones from your environment.

    Secret Name Value
    BOOTSTRAP_SERVERS paste your Stream Pool Bootstrap servers on port 9092
    STREAM_POOL_OCID paste your Stream Pool OCID
    STREAM_TOPIC_NAME paste your Stream Name

    T4_2

  3. Go to the terminal shell where you have Docker, OCI CLI, Fn Project CLI installed and run the following commands to initialize the function.

    Note: If you followed the tasks, your Docker log in command has already been executed by now, if not, proceed with the Docker log in steps in Task 3.

    fn create context oci-cloud --provider oracle
fn use context oci-cloud
fn update context oracle.compartment-id PASTE_YOUR_COMPARTMENT_OCID
fn update context api-url https://functions.sa-saopaulo-1.oraclecloud.com
fn update context registry gru.ocir.io/PASTE_YOUR_REGISTRY_NAMESPACE/lab

    Note: In this tutorial, we are using Brazil East(Sao Paulo) region, if you are using a different region, you need to change the api-url and registry locations.

  4. Get the Java function sample code from here: fn-java-streaming.zip and unzip it on your bastion console shell machine.

    # check your file is there
ls -lrt
# unzip the file
unzip fn-java-streaming.zip

    T9_1

    This simple Java code project will produce a message to the topic called my-pvt-topic from your stream pool. It will use the Fn configuration values created in step 3.

    • HelloFunction.java.

      T9_1

    • KafkaProducerSingleton.java.

      T9_1

  5. Build the new code and deploy the function.

    cd fn-java-streaming
ls -lrt
fn deploy --app fn-lab

    T9_1

    T9_1

  6. Invoke the function.

    # Invoke the function to check if it is working as expected.
echo -n 'Hello from OSS' | fn invoke fn-lab fn-java-streaming

    T9_1

    Note: The first call may take upto 1 minute to call the function.

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.