ssid-metadata - Defining and mapping statistics store identifiers (ssids)
All entities in a statistics store identifier (ssid) must have metadata that describes the class, resource, statistic and so on. While some metadata comes directly from the underlying provider (such as kstats), you must explicitly define the remaining metadata either in JavaScript Object Notation (JSON) files or dynamically, using functions such as sstore_resource_add(3SSTORE).
Additionally, you can transform (map) an ssid from one of the provider classes into a more stable and administrator-friendly name. This also allows ssids to have names that are independent of the current provider implementation. See the sstore(7) man page for a description of the provider classes. You can also define these mappings in JSON files.
You can also provide general metadata and mapping data to the statistics store using JSON files delivered to the /usr/lib/sstore/metadata/json directory hierarchy or by using the application-level interfaces offered for providing statistics data, such as sstore_resource_add(3SSTORE).
You can find JSON files delivered as part of Oracle Solaris in the path: /usr/lib/sstore/metadata/json/solaris. There is also a site and vendor directory alongside Oracle Solaris. Vendor-delivered files should appear in a eponymous subdirectory.
This man page describes the format and content of the files in the /usr/lib/sstore/metadata/json hierarchy.
Some best practices for choosing ssid names are as follows:
Keep all identifiers administrator-focused. Choose names and abstractions already used in administrative commands and documentation, especially for resource names. This also simplifies mapping.
Use all lowercase names wherever possible. Do not use mixed case. Choose - over _ as a separator.
Whenever possible, follow the existing partition name conventions instead of inventing similar names even if they are only slightly different.
If conflicts in filenames are problematic, use url-encoding package names as part of the filename.
Whenever possible, keep statistics as counters rather than absolute values. This lets you sample incrementing counters at arbitrary intervals without worrying about artifacts introduced by high frequencies in the signal being sampled.
All JSON files loaded by the statistics store contain either a JSON object or an array of JSON objects. Each loaded JSON object is verified using a schema defined in another JSON file loaded from /usr/lib/sstore/metadata/json-schema. For information on how to check syntax and compliance with the appropriate schema before loading into the statistics store, see the soljsonfmt(1) man page.
Each JSON object specifies its controlling schema using the $schema key-value pair; the value of the key is the ID specified in the schema file. Thus, the file names are not important save that they must not collide. The function of each JSON object is determined solely by its controlling schema.
There are nine different types of JSON schemas defined and thus, there are nine different types of JSON metadata files defined. The following list provides information on how each JSON metadata file uses its specified schema:
Used by JSON file to define a new class in statistics store
Used by JSON file to define stats to collect persistently
Used by JSON file to define mappings for events
Used by JSON file to define mappings for events
Used by JSON file to define mappings for resources
Used by JSON file to define mappings for statistics
Used by JSON file to define data needed for a statistic
Used by JSON file to define mappings for topology links
Used by JSON file to define mappings for enabling threshold checks
It is a useful convention to name files according to what they define. For more information, see the files in the /usr/lib/sstore/metadata/json/solaris directory.
The ssid-schema describes JSON schema objects and how to interpret them for authoring new JSON objects.
The following fields are common for all metadata objects:
Type: string
Names the metadata object.
The value of id must be unique. The format of id is schema-specific and ids for new objects should follow the format of ids for existing ones
Type: enum
Has a value of either stable or unstable. A stable object will not change incompatibly and is a committed interface. Unstable objects are not browsable.
Type: string
Describes the containing object. Might be displayed as part of the user interface.
Type: string
Defines the controlling schema and purpose of this object.
Type: boolean
Determines whether the object can be discovered without either explicit specification or by asking for all objects. For more information, see the sstore(1) man page.
Unlike resources (instances), statistics, and events, classes cannot be mapped based on other provider classes. You must always uniquely define classes in JSON metadata objects using the //:class schema.
Authorizations can be used to permit specified (daemon) users to create resources under classes. This is useful to allow new resources to be created for objects they manage by otherwise unprivileged daemons. For more information, see the sstore-security(7) man page.
//:stat (Stat Metadata Definition)These objects are controlled by the schema //:stat. The following values are required:: id, description, stability, type, and units.
The id member names the class and stat like a regular ssid, but without a resource. For example:
id: //:class.io//:stat.writes
The type member describes the type of data in the statistic, so that statistical operations can work correctly on these fields. The supported types are as follows:
accum-boot, addr, counter, epoch, event, partition, percentage, rate, relative-boot, scalar, string
The units member describes the units of the statistic. These currently include the following units:
RPM, amperes, bytes, calls, cpus, degrees-C, errors, events, instructions, interrupts, microseconds, milliseconds, name-value pair, operations, pages, packets, processes, seconds, threads, volts, watts
If required, you can specify authorizations in a statistic. This permits applications running under separate UIDs to update statistics they generate but not other statistics. For more information, see the sstore-security(7) man page.
Threshold MappingYou can set thresholds for some stats defined by ssid's. Statistics store queries the stat value periodically and compares it with the thresholds for the stat. It will generate FMA alerts if the stat crosses a given threshold value. Threshold mappings are specified by JSON data. The JSON metadata objects are controlled by the schema //:threshold. The following values are required: description and ssid-threshold-map-list.
The ssid-threshold-map-list member is a list of ssid-maps consisting of ssid and ssid-threshold-list.
The ssid-threshold-list member is a list of strings representing floating-point numbers. A negative threshold value represents a lower threshold which means the stat is checked for lower than the absolute value of the threshold. A positive threshold value represents a higher threshold which means the stat is checked for higher than the given threshold value.
If required, you can specify ssid-query-interval. This defines the time interval between threshold checks for the ssid. The default value for ssid-query-interval is 10 seconds.
If required, you can specify query-interval which defines the minimum time interval for threshold checks applicable for all the ssid's in ssid-threshold-map-list. The default value for query-interval is 10 seconds.
In the following example, threshold mappings are enabled for the stats that provide the CPU and memory utilization by all the sstored daemons on the system. An FMA alert is generated for a sstored daemon if its CPU or memory utilization crosses 10% which is the threshold mapping set for these ssid's. There is also a threshold mapping set for the repository size of the sstored daemon. An FMA alert is generated if the repository grows to 90% of its allocated capacity or falls to within 5% of its allocated capacity. The defined thresholds are checked every 5 minutes (300 seconds).
{
"$schema": "//:threshold",
"copyright": "Copyright (c) 2020, Oracle and/or its affiliates.",
"description": "List of SSID and threshold mappings for sstore daemon",
"id": "threshold-sstore",
"query-interval": 10,
"ssid-threshold-map-list": [
{
"ssid": "//:class.proc//:res.sstore*//:stat.cpu-percentage",
"ssid-query-interval": 300,
"ssid-threshold-list": [
"10.0"
]
},
{
"ssid": "//:class.proc//:res.sstore*//:stat.memory-percentage",
"ssid-query-interval": 300,
"ssid-threshold-list": [
"10.0"
]
},
{
"ssid": "//:class.fs//:res.mountpoint//var/share/sstore/repo//:stat.capacity",
"ssid-query-interval": 300,
"ssid-threshold-list": [
"90.0", "-5.0"
]
}
]
}
A thresholds JSON file to monitor zpool capacity on the system is delivered at /usr/lib/sstore/metadata/json/thresholds/zpool-usage.json. Note that threshold mapping does not work for ssid's which includes an operator //:op.
To provide stable and administration-friendly names, ssids in the provider namespaces are mapped into an independent namespace. These mappings are defined by JSON data that describes transformations from the provider namespaces into the independent namespace. These JSON objects define ssids that contain Perl-compatible regular expressions (PCREs) to match portions of the provider namespaces, and then replace them with the mapped ssids.
There are a number of lookup functions available for use during namespace mapping. These are implemented as an extension to regexp backreferences. The general syntax of a lookup function is as follows:
[\\(FUNCTION_USE)A{}...]\\(FUNCTION_USE)(FUNCTION_NAME){[ARGUMENT]}
The FUNCTION_USE determines whether the argument to the function is a backreference or a literal string and whether the result of the lookup function is added to the replacement string or pushed onto a stack of arguments for a later function call. You can specify one of the following values:
Executes FUNCTION_NAME by using a backreference as an argument, specified by number, replacing the function call text in the replacement string with the result.
Executes FUNCTION_NAME by using the argument specified in the replacement string, replacing the function call text in the replacement string with the result.
Executes FUNCTION_NAME by using a backreference as an argument, specified by number, removing the function call text from the replacement string, and adding the result to the argument stack.
Executes FUNCTION_NAME by using the argument specified in the replacement string, removing the function call text from the replacement string, and adding the result to the argument stack.
The FUNCTION_NAME determines which lookup function is executed. These are implemented as an extension to regexp backreferences:
Adds argument to the current argument stack for the next function that is called with a literal argument of {}
Name of the zone specified by the fsid/zoneid tuple in the argument
Disk controller for the disk device in the argument
Physical device path under /devices for the logical device name in the argument
Mountpoint of the NFS device number in the argument
File system type of the fsid/zoneid tuple in the argument
Media type of the network device in the argument
NFS device name corresponding to a mountpoint path in the argument
Device ID of the disk device in the argument
Type of network link contained in the argument
Mountpoint corresponding to the fsid/zoneid tuple in the argument
Name of the upper link for the network link in the argument
Name of the lower link for the network link in the argument
Processor set that contains the CPU and the ID specified in the argument
Name of the ZFS pool a device in the argument belongs to
Replaces all occurrences of the second argument in the first argument with the third argument. If the third argument is omitted, then all occurrences of the second argument are removed from the first argument
Value of the statistic named in the specified argument
Disk name corresponding to the mountpoint in the argument
Device name of the physical link in the argument
Network link name for the physical device in the argument
Name of the ZFS pool corresponding to the fsid/zoneid tuple in the argument
Name of the ZFS pool the given mountpoint belongs to
Name of the zone specified by the integer zoneid in the argument
You can provide the ARGUMENTs to a given function in one of the following ways:
As a backreference:
\\VS{1}
As a literal string:
\\vS{//:class.cpu//:res.id/0//:stat.xcalls}
Using an argument stack created through one or more calls to the argument lookup function, followed by the function itself. The function must be called using the literal argument form of {}.
Whenever you use this syntax, all arguments on the stack at the time of the function call are removed from the stack. For more information, see the following examples:
The following example shows how you can call the R() replacement function to replace all _ characters with - characters, and add the result to the replacement string:
\\AA{0}\\aA{_}\\aA{-}\\vR{}
The following example shows how you can invoke the S lookup function with the result obtained by first applying the R replacement lookup function:
\\AA{0}\\aA{_}\\aA{-}\\aR{}\\vS{}
If a lookup function fails due to some reason such as the non-existence of a zone with the backreferenced zoneid, then the replacement operation does not succeed. See the following sections for examples of usage and application.
Resource MappingUsing //:resource-mapping, you can map resources into a new part of the namespace under a given class. The mapping is based on a series of PCRE transformations defined in JSON files. These transformations are applied to an existing portion of the namespace, usually a provider-specific class such as kstat or dtrace.
These entries provide a way to represent both physical resources known to the system, such as individual CPUs, and logical resources, such as file systems, which are derived from information obtained from system configuration. When the underlying resource or stat from which this entry is derived is removed from the system, the corresponding resource entry is inactivated or removed from the namespace along with any of its descendants such as stats or events.
To map a given resource, perform the following steps:
Define a namespace (unique identifier for the type of resource identifier, such as id and core) in the metadata for the parent class.
Provide a series of PCRE transformation entries conforming to the //:resource-mapping JSON schema either by JSON files delivered to the /usr/lib/sstore/metadata/json directory or by using the application-level interfaces offered for providing statistics data.
If required, define metadata that only applies to the resources defined under a given class by using instance-metadata in the metadata for the parent class.
In the following example, the cpu class defines an id namespace for resources:
"namespaces": [
{
"name-type": "number", "resource-name": "id"
}
],
The above example declares that when the cpu class is added to the namespace, only new resources with a name that use an id/ prefix are expected. In addition, every entry in that namespace must be numeric, such as id/0.
After you declare the resource namespace, a fragment of a JSON file or metadata otherwise provided to the system matches resources in the source namespace and maps them into the independent namespace. This mapping is shown in the following example:
"transforms": [
{
"match":
"//:class.kstat//:res.misc/cpu_info/cpu_info([0-9]+)/\\1",
"replace: //:class.cpu//:res.id/\\1"
}
]
Note that the pattern backreference \\1 is used in both the match and the replace strings, and refers to the text that matched the first parenthesized term in the match PCRE. \\g{1} has the same meaning as \\1, but is not ambiguous when used with numbers.
Provider namespaces do not provide all the information you need. However, we need to read the value of a kstat statistic to perform curation.
In the following example, the match expression matches the kstat statistic that provides the core id of each CPU. The replace expression uses backreference 0 (the entire string that matched) and looks up the value of that kstat statistic using the \\VS{0} extension to PCREs, substituting the value of that statistic into the string, and creating the core.
Note that on all CPUs with multiple strands per core, the process described above results in multiple definitions for the core.
"transforms": [
{
"match":
"//:class.kstat//:res.misc/cpu_info/cpu_info([0-9]+)
/\\1//:stat.core_id",
"replace": "//:class.core//:res.id/\\VS{0}"
}
]
Statistic Mapping
Statistics can be mapped into a new part of the namespace under a given class or under a given class and resource. The mapping is based on a series of PCRE transformations defined in JSON files. These transformations are applied to an existing portion of the namespace, usually a provider-specific class such as kstat or dtrace.
These entries provide a way to represent observability metrics related to a particular class or class and resource obtained from information provided by an existing provider. When the underlying class, resource, or stat from which this entry is derived is removed from the system or otherwise inactivated, the corresponding stat entry is inactivated or removed from the namespace along with any of its descendants (such as partitions). Similarly, when the related class or resource is added to the namespace, the related statistic mappings are automatically applied.
You can perform the following steps to map a given stat:
Declare the stat in the stat-names metadata for the parent class.
If you need to map the stat under a resource, ensure that the resource already exists in a provider-specific namespace or is mapped as mentioned in the previous step.
Provide a series of PCRE transformation entries conforming to the //:stat-mapping JSON schema.
Define the metadata conforming to the //:stat JSON schema describing the statistic, so that the system understands how to process and display the related statistic value.
In the following example, the cpu class declares the following statistics:
"stat-names": [
"//:stat.usage",
"//:stat.fpu-usage",
"//:stat.integer-pipe-usage",
"//:stat.interrupt-count",
"//:stat.interrupt-time",
"//:stat.xcalls"
]
In the above example, when the cpu class is added to the namespace, only statistics with one of the names listed are allowed under the class itself or under a resource under the class.
After declaring the statistic in the related class metadata, a fragment of a JSON file or metadata otherwise provided to the system matches existing statistics in a source namespace and maps them into the independent namespace. One such example is shown below:
"transforms": [
{
"match": "//:class.kstat//:res.processor_group/pg_hw_perf_cpu
/Floating_Point_Unit/([0-9]+)//:stat.hw_util$",
"replace": "//:class.cpu//:res.id/\\1//:stat.fpu-usage"
}
]
The above example shows how you can create a statistic node in the independent cpu namespace for each CPU in the kstat provider-specific namespace only when a hw_util stat exists. The newly created statistic node points directly to the original stat under the original resource. Any statistic operations that read the value of this stat also read the value of the original stat. The following entries result from applying the above transformation:
//:class.cpu//:res.id/0//:stat.fpu-usage //:class.cpu//:res.id/1//:stat.fpu-usage
In some cases, you might need to represent a set of related statistics in aggregate form using statistic partitions. To define a partitioned stat, perform the following steps:
Declare the partitions for the stat in the partitions metadata for the related stat
Map each source statistic to a specific entry in a partition by defining a series of PCRE transformations conforming to the //:stat-mapping JSON schema.
For example, the following mapping currently exists for creating an aggregate statistic that represents the total number of interrupts received by the system for each interrupt level in the independent cpu namespace:
"transforms": [
{
"match": "//:class.kstat//:res.misc/cpu/intrstat/([0-9]+)//:stat.level-
([0-9]+)-count$",
"replace": "//:class.cpu//:res.id/\\1//:stat.interrupt-
count//:part.level(\\2)"
}
]
In the example above, for each CPU identifier found in the provider-specific kstat namespace, an interrupt-count statistic is created in the independent cpu namespace. For each new independent statistic created, a level partition is added. For each level partition, a series of keys is added that maps to the source statistic for the matching resource and class. The independent statistic, when read, returns a set of key-value pairs representing the source statistic values instead of a single value, as follows:
1 3097281 2 1781870 3 41 4 157 5 5167174
Entries in the partition are interpreted as strings, so numbers or any other alphanumeric character can be used for partition entry names. Statistics that are aggregated in this fashion must be related as the system assumes that each partition entry represents a portion of an aggregate value for the statistic.
Topology MappingTopology mappings help define the structure of the system in a browsable manner. These mappings are used to define the logical or physical relationships between different resources in the namespace. These mappings are bi-directional, for example, the links between a CPU and the processor set that contains it or between a CPU and the cores contained within it.
Resource and statistics mappings are fairly simple, as shown in the previous sections. Topology mappings are a little more complex, since both the ssids to be included in the mapping must be identified. Therefore, match and replace expressions are required for each node, with the requirement that the first match expression be the same for both nodes. Often, multiple match replace steps are required to obtain the desired ssids. For example, the following JSON fragment identifies the topology links to be created between a core object and a chip object:
"node-1": [
{
"match": "//:class.kstat//:res.misc/cpu_info/cpu_info([0-9]+)/\\1",
"replace": "\\g{0}//:stat.chip_id"
}, {
"match": "//:class.kstat//:res.misc/cpu_info/cpu_info([0-9]+)/\\1//:stat.chip_id",
"replace": "//:class.chip//:res.id/\\VS{0}"
}
], "node-2": [
{
"match": "//:class.kstat//:res.misc/cpu_info/cpu_info([0-9]+)/\\1",
"replace": "\\g{0}//:stat.core_id"
}, {
"match": "//:class.kstat//:res.misc/cpu_info/cpu_info([0-9]+)/\\1//:stat.core_id",
"replace": "//:class.core//:res.id/\\VS{0}"
}
]
Start with a CPU in the kstat namespace, then add the chip_id statistic, and then look it up to get the correct chip object for node 1. For node 2, start with the same CPU, and then get the core id statistics and look it up to get the correct core instance in the independent namespace.
Provider mappingProvider specific nodes can be mapped into the namespace for providers, such as the event provider, which does not have a namespace. A provider node can be added by applying a series of PCRE transformations defined in JSON files to the resources in the user space and the independent namespace.
These mappings provide a way to gather provider-specific data for a provider node. When the resource that contains the provider node is removed from the system or inactivated, the corresponding provider nodes are inactivated. Similarly, when the resource is added, the corresponding provider nodes are activated.
To map a provider node, the node must be declared in the metadata of the parent class, and a series of PCRE transformation entries conforming to the //:provider-mapping JSON schema must be provided.
For example, to map event nodes under the resources of the cpu class, you need to declare the event nodes in the class metadata. The following example declares three event nodes that need to be added to each resource under the cpu class.
"events": [
{
"description": "Administrative events for a cpu",
"event-source": "//:audit",
"name": "//:event.adm-action",
"stability": "stable"
}, {
"description": "FMA faults/defects for a cpu",
"event-source": "//:fma",
"name": "//:event.fault",
"stability": "stable"
}, {
"description": "FMA alerts for a cpu",
"event-source": "//:fma",
"name": "//:event.alert",
"stability": "stable"
}
]
After you declare the provider node in the class metadata, a JSON file conforming to the //:provider-mapping JSON schema is provided. The file contains a series of PCRE transformations. When these transformations are applied to the resources in the user space or independent namespace, they generate a string of the following format:
<provider name>:<map string used for mapping provider node>
For example, if the provider-mapping for the event provider for the cpu class is as follows:
{
"$schema": "//:provider-mapping",
"copyright": "Copyright (c) 2020, Oracle and/or its affiliates.",
"description": "construct cpu fmri from a cpu resource ssid for the event provider",
"id": "//:class.cpu//:provider-mapping.event",
"transforms": [
{
"match": "//:class.cpu//:res.id/(\\d+)$",
"replace": "event:cpu:///cpuid=\\1"
}
]
}
Then the above transformation, when applied to a resource of the cpu class, generates event:<FMRI for the cpu resource>. The part before the first : is the provider name event and the remaining string is used by the event provider for mapping event nodes.
Map string for event providerThe map string for mapping event nodes is the FMRI for the resource of which the events need to be aggregated in the event nodes being mapped. This map string is used for associating the relevant incoming events to the event nodes being mapped.
sstore(1), soljsonvalidate(1), ssid(7), sstore(7), sstoreadm(1)