Scenario: Creating Dimensions for a Monitoring Target

Learn how to create dimensions for a custom metric generated by a connector. Send log data from the Logging service to metrics (Monitoring service) using Connector Hub.

This scenario involves creating a connector to generate a custom metric with dimensions referencing log data. Use this connector to move log data from Logging to Monitoring. After the data is moved, you can filter the new custom metrics using the dimensions created by the connector.

Required IAM Policy

If you're a member of the Administrators group, you already have the required access to complete this scenario. Otherwise, you need access to Monitoring .

The workflow for creating the connector includes a default policy when needed to provide permission for writing to the target service. If you're new to policies, see IAM Policies Overview.

Goal

This topic describes the goal of this scenario.

The goal of this scenario is to filter update events for Object Storage buckets. For example, find updates that changed buckets to public access. Finding public buckets can help prevent leakage of secrets. In addition to public access type, this scenario sets up filters for bucket name, compartment name, availability domain, versioning status, and a static value.

Setting Up This Scenario

This topic describes the tasks involved in setting up this scenario.

This scenario creates a metric from a log using the Connector Hub service. Setting up this scenario involves the following tasks:

Create a connector to move logs from Logging to a custom metric with dimensions in Monitoring.

Log data: Bucket - Update event from audit logs (_Audit log group)

Note

While this scenario uses the _Audit log group and the bucket update event, you can use the same approach with any log available in your tenancy.
Metric namespace: bucket_events
Metric name: update
Static value buckets-from-connector (dimension name: mytags)

Extracted value using path dimensions:

Note

Each new dimension value creates a new metric stream. To avoid generating too many unique metric streams, which could potentially result in throttling, we recommend excluding GUIDs or UUIDs (such as compartment OCIDs) from the dimensions.


Path property (dimension name)	Value
`bucketName`	`logContent.data.additionalDetails.bucketName`
`compartmentName`	`logContent.data.compartmentName`
`availabilityDomain`	`logContent.data.availabilityDomain`
`publicAccessType`	`logContent.data.additionalDetails.publicAccessType`
`versioning`	`logContent.data.additionalDetails.versioning`

Filter the moved data with the dimensions (static and extracted values).

You can complete these tasks in the Oracle Cloud Infrastructure Console, CLI, or API.

For help with troubleshooting, see Troubleshooting Connectors and Troubleshooting Notifications.

Note

For complete descriptions of fields on the Create connector page, see Creating a Connector.

Task 1: Create the connector

Create a connector that moves update events for Object Storage buckets to a custom metric with custom dimensions.

On the Connectors list page, select Create connector. If you need help finding the list page, see Listing Connectors.

1. Basic Connector Information

Enter identifying information.

Connector name: Enter a user-friendly name for the new connector and an optional description. Avoid entering confidential information. Example: Bucket - Update
Select a compartment: Select the compartment to store the new connector in.

Select Next.

2. Configure Connector Source

Specify the source log for the connector to transfer data from.

Select source: Select Logging.
Configure source: Select audit logs.
- Compartment name: The compartment containing the log data.
- Log group: _Audit
- Log filter task: Select the update event for Object Storage buckets.
  - Filter type: Event type
  - Service name: Object Storage
  - Event type: Object - Update

Select Next.

3. Configure Connector Task

Select Next (skip this section).

4. Configure Connector Target

Specify the metric to send the data to.

Target: Select Monitoring.

Configure target: Enter the custom metric namespace and custom metric name that you want to use for the filtered log data.

Compartment: Select the compartment where you want to store the metric data points corresponding to the bucket update event.
Metric namespace: Enter a new metric namespace: bucket_events
Note

When typing a new metric namespace, press ENTER to submit it.
Metric: Enter a new metric: update
Next, create custom dimensions to tag the log data with the static value "buckets-from-connector" and to extract bucket name, compartment name, compartment OCID, availability domain, public access type, and versioning status.
Add dimensions: Select to open the Add dimensions panel and extract the bucket name from the log data (dimension name bucketName).
- View available dimensions: Optionally view available dimensions.
  The six latest rows of log data are retrieved from the log specified under Configure source.
- Add path (under Edit dimensions): Select to list paths for selection (Dimension path) and then optionally edit the automatically populated value in Dimension name for the selected path.
- Add static path (under Static path): Select to manually enter a dimension name and value.
  Note
  If no log data is available, then you can manually enter a path value with a custom dimension name under Static path. The path must start with logContent, using either dot (.) or index ([]) notation. Dot and index are the only supported JMESPath selectors. For example:
  - logContent.data (dot notation)
  - logContent.data[0].content (index notation)
  Example path for bucket update event, using dot notation:
```
logContent.data.additionalDetails.bucketName
```
  Example entries:
  
  Dimension name Dimension path
  
  bucketName logContent.data.additionalDetails.bucketName
  
  For more information about valid path notation, see JmesPathDimensionValue.

Repeat extraction for each additional value you want to use as a dimension (filter).

The Dimension name is automatically populated from your selected path. You can optionally edit the default name.


Dimension name	Dimension path
`compartmentName`	`logContent.data.compartmentName`
`availabilityDomain`	`logContent.data.availabilityDomain`
`publicAccessType`	`logContent.data.additionalDetails.publicAccessType`
`versioning`	`logContent.data.additionalDetails.versioning`

Tag the log data with the static value "buckets-from-connector" (dimension name "mytags"): Under Static path, enter the following:
- Dimension name: mytags
- Dimension value: buckets-from-connector
Save changes: Select to save the dimensions and close the Add dimensions panel.

Create policy: Select to accept the default policy provided for the entered target configuration.

Select Next.

Connector Preview

Review the connector configuration and then select Create.

The creation process begins, and its progress is displayed. On completion, the connector's details page opens.

The new connector immediately begins moving log data to your custom metric update with your specified dimensions.

Next, filter the moved data using the dimensions.

Task 2: Filter the moved data

This topic describes how to use the Console to filter data moved from a log to a custom metric with dimensions.

Open the navigation menu and select Observability & Management. Under Monitoring, select Metrics Explorer.

On the Metrics Explorer page, fill in the fields for a new query.

Compartment: The compartment containing the metric.
Metric namespace: bucket_events
Metric name: update
Interval: 1m
Statistic: Count

Metric dimensions: Add the following dimensions.

Dimension name	Dimension value
`mytags`	`buckets-from-connector`
`bucketName`	`bucket-20210722-1148`
`compartmentName`	`development`
`availabilityDomain`	`PHX-AD-3`
`publicAccessType`	`ObjectRead`
`versioning`	`Enabled`

Select Update Chart.

The chart shows metric data filtered by dimension.

Congratulations! You are now filtering log data that the connector moved to a metric.

For information about using the API and signing requests, see REST API documentation and Security Credentials. For information about SDKs, see SDKs and the CLI.

Define the log source for the connector: Save a file named source.json.

Example source.json file

{
  "kind": "logging",
  "logSources": [
    {
      "compartmentId": "<compartment_OCID>",
        "logGroupId": "_Audit",
        "logId": null
    }
  ]
}

Define the target (metric and dimensions) for the connector: Save a file named target.json.

Example target.json file

{
  "kind": "monitoring",
  "compartmentId": "<compartment_OCID>",
  "metricNamespace": "bucket_events",
  "metric": "update",
  "dimensions": [
    {"name": "mytags", "dimensionValue": {"kind": "static", "value": "buckets-from-connector"}},
    {"name": "bucketName", "dimensionValue": {"kind": "jmesPath", "path": "logContent.data.additionalDetails.bucketName"}},
    {"name": "compartmentName", "dimensionValue": {"kind": "jmesPath", "path": "logContent.data.compartmentName"}},
    {"name": "availabilityDomain", "dimensionValue": {"kind": "jmesPath", "path": "logContent.data.availabilityDomain"}},
    {"name": "publicAccessType", "dimensionValue": {"kind": "jmesPath", "path": "logContent.data.additionalDetails.publicAccessType"}},
    {"name": "versioning", "dimensionValue": {"kind": "jmesPath", "path": "logContent.data.additionalDetails.versioning"}}
  ]
}

Create a connector: Open a command prompt and run the oci sch service-connector create command:

Example command for creating a connector

oci sch service-connector create
 --compartment-id <compartment_OCID>
 --display-name "Bucket - Update"
 --source file://source.json
 --target file://target.json

Query the metric data using the custom dimensions: Open a command prompt and run the oci monitoring metric-data summarize-metrics-data command:

Example command for querying metric data using custom dimensions

oci monitoring metric-data summarize-metrics-data
 --compartment-id <compartment_OCID>
 --namespace bucket_events
 --query-text update[1m]{mytags = "buckets-from-connector", bucketName = "bucket-20210722-1148", coompartmentName = "development", availabilityDomain = "PHX-AD-3", publicAccessType = "ObjectRead", versioning = "Enabled"}.count()

For help with troubleshooting, see Troubleshooting Connectors and Troubleshooting Notifications.

For information about using the API and signing requests, see REST API documentation and Security Credentials. For information about SDKs, see SDKs and the CLI.

Use the following operations:

CreateServiceConnector: Create a connector.

Example CreateServiceConnector request

POST /20200909/serviceConnectors
Host: service-connector-hub.us-phoenix-1.oraclecloud.com
<authorization and other headers>
{
  "compartmentId": "<compartment_OCID>",
  "displayName": "Bucket - Update",
  "source": {
    "kind": "logging",
    "logSources": [
      {
        "compartmentId": "<compartment_OCID>",
        "logGroupId": "_Audit",
        "logId": null
      }
    ]
  },
  "target": {
    "compartmentId": "<compartment_OCID>",
    "dimensions": [
      {"name": "mytags", "dimensionValue": {"kind": "static", "value": "buckets-from-connector"}},
      {"name": "bucketName", "dimensionValue": {"kind": "jmesPath", "path": "logContent.data.additionalDetails.bucketName"}},
      {"name": "compartmentName", "dimensionValue": {"kind": "jmesPath", "path": "logContent.data.compartmentName"}},
      {"name": "availabilityDomain", "dimensionValue": {"kind": "jmesPath", "path": "logContent.data.availabilityDomain"}},
      {"name": "publicAccessType", "dimensionValue": {"kind": "jmesPath", "path": "logContent.data.additionalDetails.publicAccessType"}},
      {"name": "versioning", "dimensionValue": {"kind": "jmesPath", "path": "logContent.data.additionalDetails.versioning"}}
    ]
    "kind": "monitoring",
    "metric": "update",
    "metricNamespace": "bucket_events"
  }
}

SummarizeMetricsData: Query the metric data by dimensions.

Example SummarizeMetricsData request

POST /20180401/metrics/actions/summarizeMetricsData?compartmentId=<compartment_OCID>
Host: telemetry.us-phoenix-1.oraclecloud.com
<authorization and other headers>
{
  "namespace": "bucket_events",
  "query": "update[1m]{mytags = "buckets-from-connector", bucketName = "bucket-20210722-1148", coompartmentName = "development", availabilityDomain = "PHX-AD-3", publicAccessType = "ObjectRead", versioning = "Enabled"}.count()"
}

For help with troubleshooting, see Troubleshooting Connectors and Troubleshooting Notifications.

Oracle Cloud Infrastructure Documentation

Scenario: Creating Dimensions for a Monitoring Target

Required IAM Policy

Goal

Setting Up This Scenario

Task 1: Create the connector

1. Basic Connector Information

2. Configure Connector Source

3. Configure Connector Task

4. Configure Connector Target

Connector Preview

Task 2: Filter the moved data