Go to main content

man pages section 7: Standards, Environments, Macros, Character Sets, and Miscellany

Exit Print View

Updated: Wednesday, August 8, 2018
 
 

ssid-metadata (7)

Name

ssid-metadata - Defining and mapping statistics store identifiers (ssids)

Description

All entities in a statistics store identifier (ssid) must have metadata that describes the class, resource, statistic and so on. While some metadata comes directly from the underlying provider (such as kstats), you must explicitly define the remaining metadata either in JavaScript Object Notation (JSON) files or dynamically, using functions such as sstore_resource_add(3SSTORE).

Additionally, you can transform (map) an ssid from one of the provider classes into a more stable and administrator-friendly name. This also allows ssids to have names that are independent of the current provider implementation. See the sstore(7) man page for a description of the provider classes. You can also define these mappings in JSON files.

You can also provide general metadata and mapping data to the statistics store using JSON files delivered to the /usr/lib/sstore/metadata/json directory hierarchy or by using the application-level interfaces offered for providing statistics data, such as sstore_resource_add(3SSTORE).

You can find JSON files delivered as part of Oracle Solaris in the path: /usr/lib/sstore/metadata/json/solaris. There is also a site and vendor directory alongside Oracle Solaris. Vendor-delivered files should appear in a eponymous subdirectory.

This man page describes the format and content of the files in the /usr/lib/sstore/metadata/json hierarchy.

Best Practices

Some best practices for choosing ssid names are as follows:

  • Keep all identifiers administrator-focused. Choose names and abstractions already used in administrative commands and documentation, especially for resource names. This also simplifies mapping.

  • Use all lowercase names wherever possible. Do not use mixed case. Choose - over _ as a separator.

  • Whenever possible, follow the existing partition name conventions instead of inventing similar names even if they are only slightly different.

  • If conflicts in filenames are problematic, use url-encoding package names as part of the filename.

  • Whenever possible, keep statistics as counters rather than absolute values. This lets you sample incrementing counters at arbitrary intervals without worrying about artifacts introduced by high frequencies in the signal being sampled.

Metadata File Names and Format

All JSON files loaded by the statistics store contain either a JSON object or an array of JSON objects. Each loaded JSON object is verified using a schema defined in another JSON file loaded from /usr/lib/sstore/metadata/json-schema. For information on how to check syntax and compliance with the appropriate schema before loading into the statistics store, see the soljsonfmt(1) man page.

Each JSON object specifies its controlling schema using the $schema key-value pair; the value of the key is the ID specified in the schema file. Thus, the file names are not important save that they must not collide. The function of each JSON object is determined solely by its controlling schema.

There are eight different types of JSON schemas defined and thus, there are eight different types of JSON metadata files defined. The following list provides information on how each JSON metadata file uses its specified schema:

//:class

Used by JSON file to define a new class in statistics store

//:collection

Used by JSON file to define stats to collect persistently

//:event-filter-generator

Used by JSON file to define mappings for events

//:provider-mapping

Used by JSON file to define mappings for events

//:resource-mapping

Used by JSON file to define mappings for resources

//:stat-mapping

Used by JSON file to define mappings for statistics

//:stat

Used by JSON file to define data needed for a statistic

//:topology-mapping

Used by JSON file to define mappings for topology links

It is a useful convention to name files according to what they define. For more information, see the files in the /usr/lib/sstore/metadata/json/solaris directory.

The ssid-schema describes JSON schema objects and how to interpret them for authoring new JSON objects.

The following fields are common for all metadata objects:

id

Type: string

Names the metadata object.

The value of id must be unique. The format of id is schema-specific and ids for new objects should follow the format of ids for existing ones

stability

Type: enum

Has a value of either stable or unstable. A stable object will not change incompatibly and is a committed interface. Unstable objects are not browsable.

description

Type: string

Describes the containing object. Might be displayed as part of the user interface.

$schema

Type: string

Defines the controlling schema and purpose of this object.

browsable

Type: boolean

Determines whether the object can be discovered without either explicit specification or by asking for all objects. For more information, see the sstore(1) man page.

//:class: (Class Metadata Definition)

Unlike resources (instances), statistics, and events, classes cannot be mapped based on other provider classes. You must always uniquely define classes in JSON metadata objects using the //:class schema.

Authorizations can be used to permit specified (daemon) users to create resources under classes. This is useful to allow new resources to be created for objects they manage by otherwise unprivileged daemons. For more information, see the sstore-security(7) man page.

//:stat (Stat Metadata Definition)

These objects are controlled by the schema //:stat. The following values are required:: id, description, stability, type, and units.

id

The id member names the class and stat like a regular ssid, but without a resource. For example:

id: //:class.io//:stat.writes
type

The type member describes the type of data in the statistic, so that statistical operations can work correctly on these fields. The supported types are as follows:

accum-boot, addr, counter, epoch, event, partition, percentage, rate, relative-boot, scalar, string

units

The units member describes the units of the statistic. These currently include the following units:

RPM, amperes, bytes, calls, cpus, degrees-C, errors, events, instructions, interrupts, microseconds, milliseconds, name-value pair, operations, pages, packets, processes, seconds, threads, volts, watts

If required, you can specify authorizations in a statistic. This permits applications running under separate UIDs to update statistics they generate but not other statistics. For more information, see the sstore-security(7) man page.

Namespace Mapping

To provide stable and administration-friendly names, ssids in the provider namespaces are mapped into an independent namespace. These mappings are defined by JSON data that describes transformations from the provider namespaces into the independent namespace. These JSON objects define ssids that contain Perl-compatible regular expressions (PCREs) to match portions of the provider namespaces, and then replace them with the mapped ssids.

There are a number of lookup functions available for use during namespace mapping. These are implemented as an extension to regexp backreferences. The general syntax of a lookup function is as follows:

[\\(FUNCTION_USE)A{}...]\\(FUNCTION_USE)(FUNCTION_NAME){[ARGUMENT]}

The FUNCTION_USE determines whether the argument to the function is a backreference or a literal string and whether the result of the lookup function is added to the replacement string or pushed onto a stack of arguments for a later function call. You can specify one of the following values:

V

Executes FUNCTION_NAME by using a backreference as an argument, specified by number, replacing the function call text in the replacement string with the result.

v

Executes FUNCTION_NAME by using the argument specified in the replacement string, replacing the function call text in the replacement string with the result.

A

Executes FUNCTION_NAME by using a backreference as an argument, specified by number, removing the function call text from the replacement string, and adding the result to the argument stack.

a

Executes FUNCTION_NAME by using the argument specified in the replacement string, removing the function call text from the replacement string, and adding the result to the argument stack.

The FUNCTION_NAME determines which lookup function is executed. These are implemented as an extension to regexp backreferences:

A()

Adds argument to the current argument stack for the next function that is called with a literal argument of {}

C()

Disk controller for the disk device in the argument

D()

Physical device path under /devices for the logical device name in the argument

E()

Mountpoint of the NFS device number in the argument

F()

File system type of the fsid in the argument

G()

Media type of the network device in the argument

H()

NFS device name corresponding to a mountpoint path in the argument

I()

Device ID of the disk device in the argument

L()

Type of network link contained in the argument

M()

Mountpoint corresponding to the fsid in the argument

N()

Name of the upper link for the network link in the argument

O()

Name of the lower link for the network link in the argument

P()

Processor set that contains the CPU and the ID specified in the argument

Q()

Name of the ZFS pool a device in the argument belongs to

R()

Replaces all occurrences of the second argument in the first argument with the third argument. If the third argument is omitted, then all occurrences of the second argument are removed from the first argument

S()

Value of the statistic named in the specified argument

U()

Disk name corresponding to the mountpoint in the argument

V()

Device name of the physical link in the argument

W()

Network link name for the physical device in the argument

X()

Name of the ZFS pool corresponding to the fsid in the argument

Y()

Name of the ZFS pool the given mountpoint belongs to

Z()

Name of the zone specified by the integer zoneid in the argument

You can provide the ARGUMENTs to a given function in one of the following ways:

  • As a backreference:

    \\VS{1}
  • As a literal string:

    \\vS{//:class.cpu//:res.id/0//:stat.xcalls}
  • Using an argument stack created through one or more calls to the argument lookup function, followed by the function itself. The function must be called using the literal argument form of {}.

    Whenever you use this syntax, all arguments on the stack at the time of the function call are removed from the stack. For more information, see the following examples:

    • The following example shows how you can call the R() replacement function to replace all _ characters with - characters, and add the result to the replacement string:

      \\AA{0}\\aA{_}\\aA{-}\\vR{}
    • The following example shows how you can invoke the S lookup function with the result obtained by first applying the R replacement lookup function:

      \\AA{0}\\aA{_}\\aA{-}\\aR{}\\vS{}

If a lookup function fails due to some reason such as the non-existence of a zone with the backreferenced zoneid, then the replacement operation does not succeed. See the following sections for examples of usage and application.

Resource Mapping

Using //:resource-mapping, you can map resources into a new part of the namespace under a given class. The mapping is based on a series of PCRE transformations defined in JSON files. These transformations are applied to an existing portion of the namespace, usually a provider-specific class such as kstat or dtrace.

These entries provide a way to represent both physical resources known to the system, such as individual CPUs, and logical resources, such as file systems, which are derived from information obtained from system configuration. When the underlying resource or stat from which this entry is derived is removed from the system, the corresponding resource entry is inactivated or removed from the namespace along with any of its descendants such as stats or events.

To map a given resource, perform the following steps:

  1. Define a namespace (unique identifier for the type of resource identifier, such as id and core) in the metadata for the parent class.

  2. Provide a series of PCRE transformation entries conforming to the //:resource-mapping JSON schema either by JSON files delivered to the /usr/lib/sstore/metadata/json directory or by using the application-level interfaces offered for providing statistics data.

  3. If required, define metadata that only applies to the resources defined under a given class by using instance-metadata in the metadata for the parent class.

In the following example, the cpu class defines an id namespace for resources:

"namespaces": [
   {
	"name-type": "number", "resource-name": "id"
   }
],

The above example declares that when the cpu class is added to the namespace, only new resources with a name that use an id/ prefix are expected. In addition, every entry in that namespace must be numeric, such as id/0.

After you declare the resource namespace, a fragment of a JSON file or metadata otherwise provided to the system matches resources in the source namespace and maps them into the independent namespace. This mapping is shown in the following example:


Note -  Familiarity with PCREs will help you understand the mapping process better.
"transforms": [
   {
      "match":
      "//:class.kstat//:res.misc/cpu_info/cpu_info([0-9]+)/\\1",
      "replace: //:class.cpu//:res.id/\\1"
   }
]

Note that the pattern backreference \\1 is used in both the match and the replace strings, and refers to the text that matched the first parenthesized term in the match PCRE. \\g{1} has the same meaning as \\1, but is not ambiguous when used with numbers.

Provider namespaces do not provide all the information you need. However, we need to read the value of a kstat statistic to perform curation.

In the following example, the match expression matches the kstat statistic that provides the core id of each CPU. The replace expression uses backreference 0 (the entire string that matched) and looks up the value of that kstat statistic using the \\VS{0} extension to PCREs, substituting the value of that statistic into the string, and creating the core.

Note that on all CPUs with multiple strands per core, the process described above results in multiple definitions for the core.

"transforms":  [
    {
	  "match":
	  "//:class.kstat//:res.misc/cpu_info/cpu_info([0-9]+)
/\\1//:stat.core_id",
	  "replace": "//:class.core//:res.id/\\VS{0}"
    }
]
Statistic Mapping

Statistics can be mapped into a new part of the namespace under a given class or under a given class and resource. The mapping is based on a series of PCRE transformations defined in JSON files. These transformations are applied to an existing portion of the namespace, usually a provider-specific class such as kstat or dtrace.

These entries provide a way to represent observability metrics related to a particular class or class and resource obtained from information provided by an existing provider. When the underlying class, resource, or stat from which this entry is derived is removed from the system or otherwise inactivated, the corresponding stat entry is inactivated or removed from the namespace along with any of its descendants (such as partitions). Similarly, when the related class or resource is added to the namespace, the related statistic mappings are automatically applied.

You can perform the following steps to map a given stat:

  1. Declare the stat in the stat-names metadata for the parent class.

  2. If you need to map the stat under a resource, ensure that the resource already exists in a provider-specific namespace or is mapped as mentioned in the previous step.

  3. Provide a series of PCRE transformation entries conforming to the //:stat-mapping JSON schema.

  4. Define the metadata conforming to the //:stat JSON schema describing the statistic, so that the system understands how to process and display the related statistic value.

In the following example, the cpu class declares the following statistics:

"stat-names": [
      "//:stat.usage",
      "//:stat.fpu-usage",
      "//:stat.integer-pipe-usage",
      "//:stat.interrupt-count",
      "//:stat.interrupt-time",
      "//:stat.xcalls"
]

In the above example, when the cpu class is added to the namespace, only statistics with one of the names listed are allowed under the class itself or under a resource under the class.

After declaring the statistic in the related class metadata, a fragment of a JSON file or metadata otherwise provided to the system matches existing statistics in a source namespace and maps them into the independent namespace. One such example is shown below:


Note -  Familiarity with PCREs will help you understand the mapping process better.
"transforms": [
	{
	    "match": "//:class.kstat//:res.processor_group/pg_hw_perf_cpu
/Floating_Point_Unit/([0-9]+)//:stat.hw_util$",
	    "replace": "//:class.cpu//:res.id/\\1//:stat.fpu-usage"
	}
]

The above example shows how you can create a statistic node in the independent cpu namespace for each CPU in the kstat provider-specific namespace only when a hw_util stat exists. The newly created statistic node points directly to the original stat under the original resource. Any statistic operations that read the value of this stat also read the value of the original stat. The following entries result from applying the above transformation:

//:class.cpu//:res.id/0//:stat.fpu-usage
//:class.cpu//:res.id/1//:stat.fpu-usage

In some cases, you might need to represent a set of related statistics in aggregate form using statistic partitions. To define a partitioned stat, perform the following steps:

  1. Declare the partitions for the stat in the partitions metadata for the related stat

  2. Map each source statistic to a specific entry in a partition by defining a series of PCRE transformations conforming to the //:stat-mapping JSON schema.

For example, the following mapping currently exists for creating an aggregate statistic that represents the total number of interrupts received by the system for each interrupt level in the independent cpu namespace:

"transforms": [
    {
	"match": "//:class.kstat//:res.misc/cpu/intrstat/([0-9]+)//:stat.level-
([0-9]+)-count$",
	"replace": "//:class.cpu//:res.id/\\1//:stat.interrupt-
count//:part.level(\\2)"
    }
]

In the example above, for each CPU identifier found in the provider-specific kstat namespace, an interrupt-count statistic is created in the independent cpu namespace. For each new independent statistic created, a level partition is added. For each level partition, a series of keys is added that maps to the source statistic for the matching resource and class. The independent statistic, when read, returns a set of key-value pairs representing the source statistic values instead of a single value, as follows:

1 3097281
2 1781870
3 41
4 157
5 5167174

Entries in the partition are interpreted as strings, so numbers or any other alphanumeric character can be used for partition entry names. Statistics that are aggregated in this fashion must be related as the system assumes that each partition entry represents a portion of an aggregate value for the statistic.

Topology Mapping

Topology mappings help define the structure of the system in a browsable manner. These mappings are used to define the logical or physical relationships between different resources in the namespace. These mappings are bi-directional, for example, the links between a CPU and the processor set that contains it or between a CPU and the cores contained within it.

Resource and statistics mappings are fairly simple, as shown in the previous sections. Topology mappings are a little more complex, since both the ssids to be included in the mapping must be identified. Therefore, match and replace expressions are required for each node, with the requirement that the first match expression be the same for both nodes. Often, multiple match replace steps are required to obtain the desired ssids. For example, the following JSON fragment identifies the topology links to be created between a core object and a chip object:

"node-1": [
	{
	    "match": "//:class.kstat//:res.misc/cpu_info/cpu_info([0-9]+)/\\1",
	    "replace": "\\g{0}//:stat.chip_id"
	}, {
	    "match": "//:class.kstat//:res.misc/cpu_info/cpu_info([0-9]+)/\\1//:stat.chip_id",
	    "replace": "//:class.chip//:res.id/\\VS{0}"
	}
], "node-2": [
	{
	    "match": "//:class.kstat//:res.misc/cpu_info/cpu_info([0-9]+)/\\1",
	    "replace": "\\g{0}//:stat.core_id"
	}, {
	    "match": "//:class.kstat//:res.misc/cpu_info/cpu_info([0-9]+)/\\1//:stat.core_id",
	    "replace": "//:class.core//:res.id/\\VS{0}"
	}
]

Start with a CPU in the kstat namespace, then add the chip_id statistic, and then look it up to get the correct chip object for node 1. For node 2, start with the same CPU, and then get the core id statistics and look it up to get the correct core instance in the independent namespace.

Provider mapping

Provider specific nodes can be mapped into the namespace for providers, such as the event provider, which does not have a namespace. A provider node can be added by applying a series of PCRE transformations defined in JSON files to the resources in the user space and the independent namespace.

These mappings provide a way to gather provider-specific data for a provider node. When the resource that contains the provider node is removed from the system or inactivated, the corresponding provider nodes are inactivated. Similarly, when the resource is added, the corresponding provider nodes are activated.

To map a provider node, the node must be declared in the metadata of the parent class, and a series of PCRE transformation entries conforming to the //:provider-mapping JSON schema must be provided.

For example, to map event nodes under the resources of the cpu class, you need to declare the event nodes in the class metadata. The following example declares three event nodes that need to be added to each resource under the cpu class.

"events": [
	  {
	      "description": "Administrative events for a cpu",
	      "event-source": "//:audit",
	      "name": "//:event.adm-action",
	      "stability": "stable"
	  }, {
	      "description": "FMA faults/defects for a cpu",
	      "event-source": "//:fma",
	      "name": "//:event.fault",
	      "stability": "stable"
	  }, {
	      "description": "FMA alerts for a cpu",
	      "event-source": "//:fma",
	      "name": "//:event.alert",
	      "stability": "stable"
	  }
]

After you declare the provider node in the class metadata, a JSON file conforming to the //:provider-mapping JSON schema is provided. The file contains a series of PCRE transformations. When these transformations are applied to the resources in the user space or independent namespace, they generate a string of the following format:

<provider name>:<map string used for mapping provider node>

For example, if the provider-mapping for the event provider for the cpu class is as follows:

{
	  "$schema": "//:provider-mapping",
	  "copyright": "Copyright (c) 2015, Oracle and/or its affiliates. All rights
	  reserved.",
	  "description": "construct cpu fmri from a cpu resource ssid for the event provider",
	  "id": "//:class.cpu//:provider-mapping.event",
	  "transforms": [
	      {
		  "match": "//:class.cpu//:res.id/(\\d+)$",
		  "replace": "event:cpu:///cpuid=\\1"
	      }
	  ]
}

Then the above transformation, when applied to a resource of the cpu class, generates event:<FMRI for the cpu resource>. The part before the first : is the provider name event and the remaining string is used by the event provider for mapping event nodes.

Map string for event provider

The map string for mapping event nodes is the FMRI for the resource of which the events need to be aggregated in the event nodes being mapped. This map string is used for associating the relevant incoming events to the event nodes being mapped.

See Also

sstore(1), soljsonvalidate(1), ssid(7), sstore(7), sstoreadm(1)