Note:

Integrate Oracle Cloud Infrastructure Container Instances with Oracle Cloud Infrastructure Events Service

Introduction

Oracle Cloud Infrastructure (OCI) Events Service creates automation based on the state changes of resources. For example, when a new file is uploaded to an Object Storage bucket, it can automatically create a serverless function as a response to this event. Serverless function is a lightweight resource with a limited execution time. In certain use cases it may not have sufficient compute power and storage, and it is also bounded by a 5 min execution time limit to fully respond to the event. However, it can create a serverless container instance that has unlimited execution time with much higher compute and storage resources.

Objective

Create a serverless OCI container instance with OCI Events.

Architecture

This example shows how to automatically create an OCI container instance when a new media file is uploaded to OCI Object Storage bucket. The emitted event executes a serverless function. The function creates a serverless container instance that starts a transcoding container. The transcoding container uses FFmpeg open source software to transcode the media file to multiple resolutions and different bit rates. It supports both HLS and DASH streaming protocols. On completion, it creates playlists for different resolutions and bit rates, creates a master manifest file and a thumbnail, and uploads all the files to the destination bucket. The complete data flow is illustrated in the picture below:

Image1

Transcoding of large media files can be quite time consuming and requires more disk space than OCI Functions support. Therefore, the role of the function is to create a serverless container instance that can efficiently execute the transcoding job.

Prerequisites

Both OCI Functions and container instance services use AIM resource principal to authenticate and access OCI resources. Create a dynamic group matching OCI Functions and container instance resource types in a given compartment. Use the following matching rule:

All{resource.compartment.id = 'compartment-id', Any{resource.type = 'fnfunc', resource.type ='computecontainerinstance'}}

where, compartment-id is OCID of your compartment. You can get compartment OCID in OCI console from Identity & Security. Under Identity, click Compartments. A compartment hierarchy in your tenancy is displayed. Find your compartment and copy its OCID.

After creating the dynamic group, you should set specific Oracle Cloud Infrastructure Identity and Access Management (OCI IAM policies) for OCI services that can be used by the dynamic group.

At a minimum, the following policies are required:

Allow dynamic-group <dynamic group name> to manage object-family in compartment id <compartment OCID>
Allow dynamic-group <dynamic group name> to manage compute-container-family in compartment id <compartment OCID>
Allow dynamic-group <dynamic group name> to use virtual-network-family in compartment id <compartment OCID>
Allow dynamic-group <dynamic group name> to read repos in tenancy

Task 1: Create a Container Image in OCI registry

  1. Download the spinup-container-instance-on-event.zip file and unzip it to a client machine running Docker.

  2. In a terminal window, go to the spinup-container-instance-on-event/container directory and create a local container image by running:

    docker build -t transcoder . --no-cache

  3. Check that the container image was created on the local machine.

    docker images

    In the output of this command, you should see “transcoder:latest” image listed.

  4. Create an Authentication Token and save it in your records. After that create a repo in OCI Registry where the container image will be stored.

    Note: The OCI registry must be in the tenancy root compartment and the user account associated with the auth token will need relevant privileges for the repo.

  5. Log in to Oracle Cloud Infrastructure Registry.

    docker login <region-key>.ocir.io

    where <region-key> is the key for the Oracle Cloud Infrastructure Registry region you are using. See Availability by Region topic in the OCI Registry documentation.

    When prompted for username, enter your username in the format <tenancy-namespace>/<username>. If your tenancy is federated with Oracle Identity Cloud Service, use the format <tenancy-namespace>/oracleidentitycloudservice/<username>.

    When prompted for password, enter the auth token you copied earlier as the password.

  6. Create a tag to the image that you are going to push to OCI Registry.

    docker tag transcoder:latest <region-key>.ocir.io/<tenancy-namespace>/<repo-name>/transcoder:latest

  7. Push the container image to OCI registry.

    docker push <region-key>.ocir.io/<tenancy-namespace>/<repo-name>/transcoder:latest

    For more details, see Pushing Images Using the Docker CLI

Task 2: Create Network Infrastructure for OCI Functions and Container Instances

Create a VCN with a subnet which will be used by OCI Functions and container instance services. Note that a public subnet requires an internet gateway in the VCN, and a private subnet requires a service gateway in the VCN. For more information, see:

Task 3: Create OCI Functions Application

  1. Create OCI fn Application attached to the subnet that OCI Functions and container instance services will be using. In this tutorial, we have named the application “process-new-file”.

  2. In the terminal window on the client machine, install fn project CLI.

  3. Go to the spinup-container-instance-on-event/function directory and edit the func.yaml file. Set the values of the environment variables:

    AVAILABILITY_DOMAIN: <name of the availability domain>
COMPARTMENT_ID: <compartment OCID>
SHAPE: <container instance shape, currently supported shapes are CI.Standard.E4.Flex and CI.Standard.E3.Flex>
CONFIG_SHAPE_MEMORY: <amount of memory (GB)>
CONFIG_SHAPE_OCPUS: <number of OCPUs>
IMAGE_URL: <URL of the container image in OCI registry>
OUTPUT_BUCKET: <name of the output Object Storage bucket where the transcoded files will be stored>
STREAMING_PROTOCOL: <HLS or DASH>
SUBNET_ID: <subnet OCID>

  4. Save the file and deploy the create-container-instance function.

    fn deploy --app process-new-file

Task 4: Create OCI Event Rule

In the OCI Console, create an Event Rule that fires when a new file is uploaded to the Object Storage bucket. Here is an example of the event rule:

Image2

Input bucket must be an existing Object Storage bucket where the input media files are uploaded.

Task 5: Test the Flow

Upload a new file to the input bucket. Shortly after that you should see that OCI event new-file-upload is emitted. Here is a screenshot from OCI event metrics:

Image3

It triggers execution of the create-container-instance function that you can see in the OCI Functions metrics:

Image4

This function creates a new “transcode” container instance using the configured container instance shape:

Image5

The started container downloads the media file from the Object Storage bucket and transcodes it to 3 different resolutions and bitrates.

It creates a new folder in the output Object Storage bucket with the name of the input file, and uploads the playlist files for each stream (resolution and bitrate) to this folder. Here is an example:

Image6

At the end, it creates a thumbnail of the media content and uploads it to the thumbnails folder of the output. bucket.

Note: This example uses H.264 codec but it can be used with other codecs supported by FFmpeg. You can set the codec library in the spinup-container-instance-on-event/container/transcode.sh file prior to building the container image (see Task 1).

For troubleshooting, you can turn on logging in the OCI fn application. Once the container instance is created, you can view logs of the transcoding container while the container is running.

Acknowledgments

Author - Michael Prestin (Master Principal Cloud Architect)

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.