Span Enrichment

About Span Enrichment

Provide users with the ability to enrich spans with custom attributes or change, combine or remove data from existing attributes.
Enrichment operations are defined through span rules which contain one or more actions to be applied in a particular order on a given span.
There are two types of span rules: Custom and Oracle-provided.
Span enrichment is configured at APM Domain level.

Span Enrichment Examples:

Removal of Personal Identifiable Information (PII) from spans.
Combine data from multiple attributes into a single one.
Extract data from an attribute. For example, a partial path from a longer URL.
Apply mathematical functions on data. For example, convert a unit from seconds to milliseconds.
Map an attribute value to a new value. For example, map the value '1' to the color 'blue', value '2' to the color 'red'.
Distribute numeric values into buckets. For example, a temperature value range that is mapped into 'cold', 'warm' and 'hot' buckets.

Topics:

Span Enrichment Usage

Span Enrichment is configured through Span Rules which operate on a span and are used to modify the span's attributes. The Span Rules toolbox provides a variety of actions that can be performed as part of a rule.

Span Rules are managed within groups. A group is a collection of rules that are applied in a specified order. Each rule can apply one or more actions on a span and the output of each rule can be used as input to follow-up rules. Rule groups require a Span Filter that is used to select the spans on which to apply the rules.

Multiple groups can be defined. This is useful when you need to maintain different types of rules for different types of Spans as it creates logical entities with functionality designed for a specific purpose. This reduces the overall complexity of your Rule Set.

If multiple groups have Span Filters set up that match the same span, then all of them will apply their rules to that span. The order in which this is performed is determined by the alphabetical order of the group names. Keep this in mind when you name or rename your groups. Oracle recommends prefixing the names with a number to avoid the group names getting sorted alphabetically. For example, "1. Remove Location Data", "2. Convert Length Units".

Keep in mind that when multiple groups are operating on the same span, this means that each group will be operating on a version of the span that may have been modified by a previous group in the chain. This can be useful, as you can write rules in follow-up groups that rely on modifications created by previous groups. On the other hand, it can also cause follow-up rules to fail if an attribute that a rule relies on, is removed or modified by a previous group.

Note

The above is only important when a span matches more than one group.

If your groups have clearly disjoint filters, then a span will always be modified by a single group and the above considerations become irrelevant.

Attributes in Span Events

Spans may contain a list of events which are also referred to as log messages. These are typically used to store information about important moments during the execution of the span. For example, if a span describes a database query, then events could be generated to mark the moment in time when the connection was open, then when the query request was actually submitted and when the response was received.

An event is identified by a name, timestamp and a set of attributes (key-value pairs) that provide further context for the recorded information.

All Span Rule actions described below, which require an attribute as input or output, can also be applied on the attributes of individual events. Since the event attributes are independent of regular span attributes, you will need to use the following event reference syntax to identify the proper event attribute:

event['NameOfEvent']:'NameOfAttribute'.

Note

Event attributes cannot be used as Span Filters.

Create and Manage Span Rules

You can create and manage Span Rules depending on your reporting needs.

To create Span Rules for a given APM domain, do the following:

Go to the APM Domains page and click on the desired APM domain.

The Domain Details page is displayed.
Go to Resources, located at the bottom left, and click Span Enrichment.

The Span Enrichment pane is displayed.
Click on Create group to create a new span rule group.
The Create span rule group pane is displayed.
- Enter the Name of the new group.
- Select a Span Filter to use with this group.
- Optionally, provide a Description for the new group.
- Click Next to proceed to Rule actions pane to start adding span rules to the new group depending on the span rule type: Custom span rule or Oracle provided span rule.
- Enter the Rule name.
- Optionally, select a Rule span filter to further narrow down the scope of the rule's operation.
  
  Note
  
  Span Filters are created separately. For details, see Span Filters.
- Optionally, provide a Rule description.
- Select a span rule type.
  - If you select Oracle Provided span rule, you are offered a list of rules to choose from.
    
    The rules have already been created by Oracle in order to assist the users. The list includes the following:
    Adds the client IP address to a span.
    It adds the client IP address to a span as an attribute named ClientIP.
    
    EBS suite.
    It provides the E-Business Suite template.
    
    OpenTelemetry to APM naming conversion.
    It converts span attribute names from the OpenTelemetry naming convention to the APM naming convention. For information about using OpenTelemetry with APM, see Configure OpenTelemetry Data Sources .
    Click on a rule from the list to add it to the group.
    
    Note
    
    You cannot configure any actions since they are managed internally by the rule itself. Also, you cannot see what the actions are for an Oracle provided rule.
  - If you select to create a Custom span rule, you need to define one or more actions for it. (A rule always contains at least one action).
    Before you can choose an Action, you have to select an Action target and Action type.
    
    The Action target specifies the part of the span on which the action operates. There are two targets available:
    
    Entity: Target the entire span.
    
    Attribute: Target a single attribute.
    
    The Action type specifies the type. It is mainly used to group actions logically together. For example, string or numeric.
    
    Now select the appropriate action from the Available Actions drop-down.
    Since actions have various contexts, the window will change accordingly with each action type. For more details about actions, see Working with Actions.
    For more information about the available actions for Action target and Action type, see Available Actions.
    Use the Add Action button to add more actions to the rule.
    
    Use the action item menu at the top right of each Action to move it up or down in the rule ordering, or delete an action.

Working with Actions

Actions that have a target of Attribute always require you to provide the name of the attribute whose value will be used as input to the action. The configuration window for each action refers to it as Input Attribute.

Some actions also require you to provide the name of an attribute in which to store the result of the operation. This is referred to as the Output Attribute. If the attribute exists on the span, any value it has at that time will be overwritten by the result of the action. If it does not yet exist, it will be added to the span with the new value. You can use this mechanism to replace the value of any attribute on the span (even the input attribute) with the result of the action's operation.

Using this, you can feed the results of any action to a follow-up action (or even a rule within the same rule group).

You can also use it to store values into temporary attributes that can be used by follow up rules or actions, but without the need to keep them as part of the final result. Just make sure to remove the temporary attributes as a last step using the Remove Attribute action.

Note

A Block action rejects the span and you cannot use it later on, even from another Rule Group.

Available Actions

Action Target	Action Type	Available Actions
Entity	Passthrough Actions that control whether the span in its entirety is passed through or blocked. Blocked spans are rejected from the processing pipeline. Rejected spans will no longer be processed by APM, will not be stored anywhere, and will not be available for viewing later on in other APM tools or dashboards.	Block: Blocks this span unconditionally. Any follow up actions and rules will not be applied to it and the span will no longer be available in APM OCI tools and products. Sample: Samples (accepts) spans based on a sample percentage. For example, a sample percentage of 25% will (randomly) accept 25% of all spans, and will block the rest. Spans blocked here are treated the same as spans that were blocked by the Block action.
Attribute	String Actions that allow various operations on the value of attribute which is a string. For example, changing the case, replacing a value or extracting a portion of the string.	Append: Appends a static string to the value of an attribute using an optional separator string. Concatenate: Concatenates two attribute values. An optional separator string can be provided on which the values will be joined. The result is returned as a new attribute. Extract: Extracts a portion of an attribute's value based on a regular expression. The result is returned as a new attribute. The regular expression must contain a capturing group operator which identifies the portion of the string to capture. If the regular expression doesn't match, or there is no capture group, the action will produce no output. You can optionally provide a fallback value, which will be used as the output of the action in case the extract itself produced no result. Lowercase: Converts the value of an attribute to lower case. Search & Replace: Replaces one or more portions of an attribute's value using a regular expression. The amended string is returned as a new attribute. The regular expression may contain one capture group. Without a capture group, the full content of what the expression matches will be replaced. When a capture group is present, only the content that is captured by the group will be replaced. The optional match occurrence can be used if only a specific occurrence of a match must be replaced. If this is not set, or set to 0, all matched occurrences will be replaced. This is the default behavior. Trim: Trims any leading and trailing spaces from an attribute's value. Uppercase: Converts the value of an attribute to upper case.
Attribute	Numeric Actions that allow various mathematical operations on the attribute values (provided they are numbers). Numeric actions support both actual numbers and numbers represented as strings in the attribute value, that is the following two are equivalent: "amount" : 2.5 "amount" : "2.5" Numeric actions can take as input either a string representing the name of an attribute whose value to use in the operation, or a number which will be used directly in the computation. All numeric actions have an option where you can apply rounding to the output value. The rounding can be one of the following: Round: Rounds the number up or down to the nearest whole number. Floor: Rounds the number down to the nearest whole number. Ceil: Rounds the number up to the nearest whole number. No rounding: Default option.	Absolute: Returns the absolute value of the input value. Add: Sums two values. Bucketize: Maps a numeric attribute value into bucket names. If the attribute value falls within the range of a bucket, the bucket name will be returned. If no bucket matches, a fallback value will be returned. Divide: Divides the first by the second value. If the second value is 0, the result will be 0. Maximum: Returns the maximum of the two values. Minimum: Returns the minimum of the two values. Multiply: Multiplies the two values. Negate: Negates (changes the sign) of the value. Subtract: Subtracts the second from the first attribute.
Attribute	Generic Generic actions provide attribute level operations, such as removing an attribute from the span.	Map attribute value: Takes the value of an attribute and maps it to a new value. Remove attribute: Removes the attribute from the span. If the attribute is an event name reference, `event['EventName']`, then the entire span event will be removed. Rename attribute: Renames the attribute. Set attribute: Sets an attribute to a static value. If the attribute doesn't exist, it is added, otherwise it is assigned the new value, overwriting the existing one.

Examples of Action Configuration

This section provides examples of the configuration steps required for some individual actions.

Bucketize Example:
This action uses a mini-editor to set up a number of buckets and their threshold values. You can use the control on the right to create, update and remove buckets.

Buckets are automatically ordered based on their threshold values. A fallback value is used when a value is too large for the last bucket. The following image shows an example with five buckets and a fallback value:
Map Example:
This action uses a mini-editor to set up map of input / output value pairs. Both values can be numbers or strings. Use the control on the right to create, update and/or remove a mapping.
The Export button allows you to download any mappings to a file (in CSV format) to your computer.
The Import button allows you to populate the mappings from a CSV file on your computer. This is particularly useful if you have a large number of mappings.

Span Enrichment Rules Validation

Span enrichment configurations can be complex to write and it can sometimes be difficult to know beforehand if a rule or rule group will work as expected. To help with this, the Span Enrichment Configuration Editor provides an option to test a rule or rule group on a sample span before saving it.

To use the test option, do the following:

Select a group, and then click on Edit to open the Edit span rule group panel. Another way is to create a new group and the Edit panel get displayed.
Click through to the Configure rule actions step.
Click Test to test the entire rule group. Alternatively, to test an individual rule, select the rule and click Test rule.
Note

Test rule is not available when the rule group contains only one rule.
The Test span rule group panel is displayed.
The rules to test will be shown with a checkbox next to them. You can toggle those as desired.
Enter a sample span in the Span sample text box.
Click Test span to perform a test of the selected rules on the given span.

The test run will report whether the span is matching the configured span filters or not.

If the filters match, then the rules will be applied to the span and the resulting differences (if any) will be displayed in the table below. Any attributes that were added as a result of the rule will be highlighted in green. Attributes that were removed will be highlighted in red. Modified attributes will be highlighted in yellow.

Trace Explorer: To get accurate samples of spans from your traffic, click Trace Explorer located at the top right to open Trace Explorer and see the actual spans that have been processed. You can select a span, open the context menu on the right (the three vertical dots) and click Copy root span data.

A popup is displayed with the contents of the span. You can copy the span data and use it during testing.

Another option is to paste your own spans or enter them manually, as long as they conform to the span format as reported by Trace Explorer.

Span Property Aliases

Note

Few of the actual span properties are referred to by a different name when used in Span Enrichment or Span Filters features. Other than the visual difference, this has no impact on how you setup your rules and filters. For example, you should always use the alias when referring to the given property.

Here are the properties that currently have an alias:

Span Property	Span Enrichment Alias	Remarks
`td-micros`	`SpanDuration`	`td-micros` is in microseconds, while `SpanDuration` is in milliseconds. Time unit conversion is applied automatically by Span Enrichment and Span Filters.
`name`	`OperationName`	The name of the span represents the operation performed
`ts-micros`	`StartTime`	`ts-micros` is in microseconds, while `StartTime` is in milliseconds. Time unit conversion is applied automatically by Span Enrichment and Span Filters.
`id`	`SpanId`
`parent-id`	`ParentId`
`trace-id`	`TraceId`

Oracle Cloud Infrastructure Documentation