Note:

Read Messages Published to an OCI Stream from Oracle Access Governance Event Data Publisher

Introduction

Event Data Publishing is a process to export one-time data and continually publish ongoing data events to external systems, such as an Oracle Cloud Infrastructure (OCI) cloud account. With Oracle Access Governance, you have the flexibility to export one-off and continually publish data events, such as identity, identity collections, policy, resource, access to resources, and so on, to your cloud tenancy. You may use this data for deriving insights, storing data for compliance, or for analyzing access management and governance data.

An event refers to any change in data state that occurs when there’s creation, modification, or deletion of Oracle Access Governance components, such as identity, policies, resources, and so on. Using Event Data Publisher, administrators get complete control over access and identity data and may use it to automate event logging and streamline compliance reporting.

Data Event Publishing flow uses OCI buckets for one-off export, and publish subsequent updates either to OCI streams or OCI buckets depending on the file size.

Data Event Publishing Flow

In this tutorial, we will walk you through how user updates in OCI are captured and published to OCI streams by Oracle Access Governance and how to consume and decode these stream messages using a script through the OCI Cloud Shell.

Note: The Event Data Publisher supports exporting identities, identity collections, policies, and resources, however, this tutorial focuses on capturing events for identities.

Overview of the Shell Script

The purpose of this shell script is to retrieve and decode messages published in an OCI stream. These messages are encoded in a way that makes them unreadable in their raw format. By running the script, you can see the messages in a clear, human-readable format, ensuring that critical information is accessible even after it disappears from the OCI Console.

Specifically, the script creates a cursor to mark a starting point for reading messages from the specified OCI stream. The cursor is based on a specified offset and a partition whose values are fetched by producing a test stream message. Using the cursor, the script fetches messages from the OCI stream. Eventually, the script processes and decodes each retrieved message.

Audience

Oracle Access Governance and Oracle Cloud Infrastructure Identity and Access Management (OCI IAM) administrators.

Objectives

Prerequisites

Task 1: Retrieve Key Parameters for your OCI Stream

In this task, we will log in to the OCI Console to retrieve key parameters for the stream.

  1. Log in to the OCI Console, navigate to Analytics & AI, Messaging and click Streaming.

  2. Make sure to select the compartment where your stream is located and click the stream to view details.

  3. Note down the values for stream OCID and Messages Endpoint.

    Grab Stream Parameters

  4. Click Produce Test Message, enter the sample message as Data and click Produce. It should return a Success message.

    Produce Test Stream Message

  5. Click Cancel to close the Test Stream window.

  6. Click Load Messages to check the test message.

  7. From the Recent Messages section, note down the Partition and Offset values.

    Grab Stream Message Parameters

Task 2: Update User Attributes in OCI

In this task, we will make changes to user attributes in OCI.

  1. Go to the OCI Console, navigate to Identity & Security and click Domains.

  2. Select your domain and click Users. Click the user that you would like to update and click Edit User.

  3. Update the values of Country, Title, Department, Cost Center, User Type, Employee Number and so on.

  4. Click Save changes.

Note: Optionally, you can update a few more users if preferred.

Task 3: Perform Data Load in Oracle Access Governance

In this task, we will perform data load in Oracle Access Governance to synchronize the OCI changes.

  1. Log in to Oracle Access Governance, navigate to Service Administration and Orchestrated Systems.

  2. Locate the orchestrated system for OCI, click the three-dots (⋮) icon and select Manage Integration.

  3. Click Load data now and wait for the data load to complete.

    Perform data load

Task 4: Configure OCI Cloud Shell to Run Script to Capture Messages from the Stream

In this task, we will configure OCI Cloud Shell and use the provided script to decode messages published by Oracle Access Governance.

  1. Go to the OCI Console, click Cloud Shell from the upper-right corner and wait for the cloud shell window to initialize.

    Navigate to Cloud Shell

    Start Cloud Shell

  2. Initialize the following parameters.

    export STREAM_OCID=<STREAM_OCID_VALUE>
export ENDPOINT=<MESSAGES_ENDPOINT>
export CURSOR_TYPE="AFTER_OFFSET"
export PARTITION=<PARTITION>
export OFFSET=<OFFSET>

    • Replace the STREAM_OCID, ENDPOINT, PARTITION and OFFSET parameters for the stream with the values captured in Task 1.

    • The CURSOR_TYPE is set to AFTER_OFFSET to include all messages generated after the specified offset value for reading. Do not modify this value.

      Note: A sample value for the offset can be obtained from the test message, but you should choose a value after which you will like to capture the messages. This is what the CURSOR_TYPE as AFTER_OFFSET does.

  3. Create a shell script file and make it executable.

    touch ag-streaming.sh
chmod u+x ag-streaming.sh

  4. Open the script with vi editor.

    vi ag-streaming.sh

  5. Press i to enter the Insert Mode. Paste the following content, then press Esc and :wq! to save the changes.

    #!/bin/bash

# Validate required environment variables

required_vars=("STREAM_OCID" "ENDPOINT" "CURSOR_TYPE" "PARTITION" "OFFSET")

for var in "${required_vars[@]}"; do
   if [ -z "${!var}" ]; then
      echo "Error: Environment variable $var is not set."
      exit 1
   fi
done

# Create a cursor for the OCI Stream
# echo "Creating a cursor for the OCI Stream..."
oci streaming stream cursor create-cursor \
   --partition "$PARTITION" \
   --stream-id "$STREAM_OCID" \
   --type "$CURSOR_TYPE" \
   --offset "$OFFSET" \
   --endpoint "$ENDPOINT" > cursor.json

# Extract cursor value
cursor=$(jq -r '.data.value' cursor.json)
if [ -z "$cursor" ]; then
   echo "Error: Failed to retrieve cursor value."
   exit 1
fi

# Read messages from the OCI Stream
echo "Reading messages from the OCI Stream..."
messages=$(oci streaming stream message get \
   --stream-id "$STREAM_OCID" \
   --cursor "$cursor" \
   --endpoint "$ENDPOINT")

# Check if messages were retrieved
if [ -z "$messages" ] || ! echo "$messages" | jq -e '.data[] | select(.key != null)' > /dev/null 2>&1; then
   echo "No valid messages found."
   exit 1
fi

# Extract and decode the data from messages
echo "Processing messages..."

# Extract and count the number of messages
message_count=$(echo "$messages" | jq -r '.data[] | select(.key != null) | .value' | wc -l)

echo "$messages" | jq -r '.data[] | select(.key != null) | .value' | while read -r value; do
   if [ -n "$value" ]; then
      counter=$((counter + 1))
      echo
      echo "Decoding message $counter of $message_count..."

      # Base64 decode the message
      decoded_message=$(echo "$value" | base64 --decode 2>/dev/null || echo "Error decoding message")
      final_decoded_message=$(echo "$decoded_message" | base64 --decode 2>/dev/null || echo "Error decoding message further")

      # Print the decoded message
      echo "Decoded Message : $final_decoded_message"
   fi
done

  6. Run the script and redirect the output messages to a file (messages_dump.txt is used as an example). Alternatively, you can run the script to display messages on the console.

    ./ag-streaming.sh > messages_dump.txt

  7. Download the messages_dump.txt file from the OCI Cloud Shell to check the decoded messages for the user updates previously made.

In this tutorial, we learned how to read messages from an OCI stream that are published from event data publisher in Oracle Access Governance. We saw how we can leverage OCI Command Line Interface (CLI) using OCI Cloud Shell to read the messages based on a specified offset. These messages contain changes pertaining to OCI users, groups, resources and policy associations and are therefore important from the auditing, compliance and reporting perspective.

Next Steps

After completing these tutorial, you can analyze the contents of the data event messages published from Oracle Access Governance to OCI streams. You can optionally set up further processes to analyze the event messages for auditing and compliance requirements. You can also move data from streams to autonomous data warehouse to perform advanced analytics and visualization. Data can also be moved to OCI Object Storage for long term storage, or to run Hadoop/Spark jobs.

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.