SStore, SStoreData, SStoreHistogram, SStoreWarning, libsstore-python - interfaces to update user-application statistics
enum SStoreHistogramType:
SSTORE_HISTOGRAM_TYPE_LINEAR
SSTORE_HISTOGRAM_TYPE_LOG10
SSTORE_HISTOGRAM_TYPE_LOG2
SSTORE_HISTOGRAM_TYPE_LOG10_LINEAR
enum SStoreWarningType:
SS_WARN_OK
SS_WARN_INVALID
SS_WARN_INTERNAL
SS_WARN_CONN_BROKEN
SS_WARN_UNAUTHORIZED
SS_WARN_NOENT
class SStoreData:
Properties:
ss_hdl
ssid
Methods:
add_point(timestamp, val)
reset()
class SStoreHistogram:
Properties:
ss_hdl
type
range_min
range_max
step
Methods:
quantize(val)
Static Methods:
bucket(type, range_min, range_max, step, val)
class SStoreWarning
Properties:
type
id
description
class SStore:
Methods:
data_attach(ids)
data_attach_histogram(id, histogram)
data_update(data)
data_bulk_update(data_arr)
warnings()
The sstore python API exposes the libsstore interfaces to update user-application statistics. It has the following classes:
It provides the following methods to update user-application statistics:
ids – Array of strings representing the list of ssids for statistics. For more information, see the ssid(7) man page.
An array of ctypes.C_uint
It is an init routine for counter-type numeric statistics that sets up a shared memory region between sstored and the client process. Only one call to this method is necessary during the runtime of a client process, after which update to the statistics can be made directly within the shared memory region returned on successful execution of the method.
An ssid at index i in the ids array maps to the i'th element in the array returned from this method.
For example, if addrs is the array returned from the data_attach() function, then to update the value of ids[i] with val:
addrs[i] = val
It performs the following tasks:
Sets up an array of ulong shared between the calling process and sstored
Enables any resources in ids that are defined statically and are not currently enabled. For information about statically defined resources, see sstore and ssid-metadata man pages.
Marks the statistics in ids as actively provided.
Note that simply updating the values does not have any impact on sstored, and these values are not actually sampled until asked for either data reading interfaces in sstore or when persistently enabled.
Once the data is being captured, sstored attempts to read from the shared memory region every second regardless of how quickly the provider updates the value.
Statistics that are updated using the shared-memory method remain mapped by sstored until the client dies, or releases the SStore instance.
id – String representing the ssid of the statistic
histogram – instance of SStoreHistogram
It is similar to the data_attach() function except for the following details:
It allows the user-application to map a histogram, which is an array of counters corresponding to its buckets, in the shared-memory region between sstored and the client process.
It takes an ssid and an instance of histogram as input and replaces the buckets in the histogram with the array of ulong mapped in the shared-memory region.
After the shared-memory region is setup, the user-application can update the histogram using the quantize() method, which will update the relevant bucket in the shared-memory region.
data – Dictionary with ssid as the key and the value of the corresponding statistic being the value
It allows user-applications to update their statistics for all types of data. This method updates non-numeric or non-counter type statistics that cannot be updated using the shared-memory safely without some addition synchronization mechanism.
It should be called every time the user-application wants to update the value of one or more statistics. It performs the following tasks:
Updates the value of the given statistics with the given values. There is a one-to-one relationship between the ssids in ids and values in vals.
Enables any resources in ssids that are defined statically and are not currently enabled. For information about statically defined resources, see sstore and ssid-metadata man pages.
Marks the statistics in ids as actively provided.
Sampling behavior is identical to the data_attach() function. Most importantly, values are only recorded when asked for by data reading interfaces in sstore or when persistently enabled.
data_arr – Array of SStoreData instances
It is similar to the data_update() function except for the following details:
It allows a user-application to provide multiple data points, that is a set of (timestamp,value) pairs for a statistic at once. This removes the requirement of updating statistics in real-time.
A statistic provided using this method must define min-update-interval, which is the minimum number of seconds between two bulk updates in the metadata of the statistic. This is just a pass-through interface to allow the consumer of the statistic to set some expectations on the update frequency of the statistic.
The data points provided using this method must be in chronological order. This means that the user-application can only provide data points that are more recent than the data points provided earlier. Any data point that does not meet this requirement will be ignored.
An array of SStoreWarning instances set during the most recent call to the data_attach(), data_attach_histogram(), data_update(), or data_bulk_update() functions.
Warnings are set when the operation succeeds partially. For example, in the data_update() function, if one of the statistics specified in ids is invalid, it will update the values of all the valid statistics and return a warning for the invalid statistic.
It provides a way to add one or more data points for a statistic, which can then be passed through the data_bulk_update() function. It provides the following methods:
timestamp – Number representing the time in microseconds since epoch
val – A python object representing the value for a statistic
Adds a new data point, that is the (timestamp, value) pair
Removes all the data points previously added using the add_point() function
It defines different types of histograms and provides methods to manage it. It has the following properties:
One of the SStoreHistogramType values
Number representing the upper bound of the first bucket in a linear histogram. In log-linear histograms, this is the lower magnitude of the logarithmic range.
Number representing the lower bound of the last bucket in a linear histogram. In log-linear histograms, this is the upper magnitude of the logarithmic range.
Bucket step for linear histograms. Number of steps in each linear range in log-linear histograms. Not used for plain log histograms.
It supports the following histogram types:
A histogram with fixed, linear bucket ranges.
The number of buckets depend on range_min, range_max, and step, and the bucket ranges are:
[0, range_min) [range_min, range_min + step) [range_min + step, range_min + step + step) ... [range_max, 2^64)
A histogram with bucket ranges increasing in powers of 2.
The bucket ranges are:
[0, 2^0) # Value range: 0-0 [2^0, 2^1) # Value range: 1-1 [2^1, 2^2) # Value range: 2-3 [2^2, 2^3) # Value range: 4-7 ... [2^63, 2^64)
A histogram with bucket ranges increasing in powers of 10.
The bucket ranges are:
[0, 10^0) # Value range: 0-0 [10^0, 10^1) # Value range: 1-9 [10^1, 10^2) # Value range: 10-99 ... [10^19, 2^64)
A histogram with fixed, linear bucket ranges which increase in log10 steps.
The number of buckets depend on range_min, range_max, and step. The first bucket will contain a count of values in the range:
[0, 10^range_min)
and the last bucket will contain count of values in the range:
[10^(range_max+1), 2^64)
Between the first and last bucket, each of the following intermediate ranges will have step number of buckets of size (10^intermediate_range_max / step):
[10^range_min, 10^(range_min+1)) [10^(range_min+1), 10^(range_min+2)) ... [10^(range_max-1), 10^range_max) [10^range_max, 10^(range_max+1))
The value of step must at least be 10, and must be able to evenly divide 10^(range_min+1).
For example, if range_min=1, range_max=2 and step=10, the bucket ranges would be:
[0, 10) [10, 20) [20, 30) ... [90, 100) [100, 200) [200, 300) ... [800, 900) [900, 1000) [1000, 2^64)
Provides the following method to update the histogram:
val – Numeric value to be added to the histogram
It expands the number of buckets as needed to increment the contents of the bucket which will hold the given value.
Parameters:
One of the SStoreHistogramType values
Number representing the upper bound of the first bucket in a linear histogram. In log-linear histograms, this is the lower magnitude of the logarithmic range.
Number representing the lower bound of the last bucket in a linear histogram. In log-linear histograms, this is the upper magnitude of the logarithmic range.
Bucket step for linear histograms. Number of steps in each linear range in log-linear histograms. Not used for plain log histograms.
Numeric value to be added to the histogram
Returns a string representing start of the bucket in a histogram, with the given parameters, that the given value falls in. For any invalid inputs, 0 will be returned.
It represents a warning set during a call to a method in SStore. It has the following properties:
One of the SStoreWarningType values
ssid of the statistic
Reason for the failure
The following metadata files must be installed in /usr/lib/sstore/metadata/json/solaris for this example:
class.app.solaris.exapp.json
stat.app.solaris.exapp.stat1.json
stat.app.solaris.exapp.stat2.json
After installing the files, svc:/system/sstore:default must be restarted.
class.app.solaris.exapp.json:
{
"$schema": "//:class",
"description": "Example class",
"id": "app/solaris/exapp",
"namespaces": [
{
"name-type": "string",
"resource-name": "name"
}
],
"stat-names": [
"//:stat.stat1",
"//:stat.stat2"
],
"stability": "stable",
"static-instances": [
{
"name": "exapp/ins",
"namespace": "name"
}
]
}
stat.app.solaris.exapp.stat1.json:
{
"$schema": "//:stat",
"description": "Example count statistic one",
"id": "//:class.app/solaris/exapp//:stat.stat1",
"stability": "stable",
"type": "counter",
"units": "calls"
}
stat.app.solaris.exapp.stat2.json:
{
"$schema": "//:stat",
"description": "Example count statistic two",
"id": "//:class.app/solaris/exapp//:stat.stat2",
"stability": "stable",
"type": "counter",
"units": "calls"
}
da_example.py: Example Script to provide stats using data_attach()
import time
from libsstore import SStore
ssids = [
"//:class.app/solaris/exapp//:res.name/exapp/ins//:stat.stat1",
"//:class.app/solaris/exapp//:res.name/exapp/ins//:stat.stat2"
]
# Get an instance of SStore class
ss = SStore()
# try to set up the shared memory region
try:
stats = ss.data_attach(ssids)
except:
print("data_attach() failed. Reason {0}".format(
ss.err_description))
exit(1)
# shared memory region is setup; update stats every second
while 1:
stats[0] += 1
stats[1] += 3
time.sleep(1)
Once the sstore has been restarted, run the following command in one terminal to observe the effect of running the da_example.py in another terminal:
$ sstore capture //:class.app/solaris/exapp//:*//:*
The attach_example program must be run as a user with appropriate stats store privileges. For a description of the authorizations required to provide and update statistics, see the sstore-security man page.
Example 2 Update Counter Type Stats Using the data_update() FunctionThe following metadata files must be installed in /usr/lib/sstore/metadata/json/solaris for this example:
class.example.json
stat.example.json
After installing the files, svc:/system/sstore:default must be restarted.
class.example.json:
{
"$schema": "//:class",
"description": "example class",
"id": "example",
"stability": "stable",
"stat-names": [
"//:stat.one",
"//:stat.two"
]
}
stat.example.json
[
{
"$schema": "//:stat",
"description": "example stat one",
"id": "//:class.example//:stat.one",
"stability": "stable",
"type": "counter",
"units": "calls"
},
{
"$schema": "//:stat",
"description": "example stat two",
"id": "//:class.example//:stat.two",
"stability": "stable",
"type": "counter",
"units": "calls"
}
]
du_example.py: Script to provide stats via data_update()
import time
from libsstore import SStore
ssids = [
"//:class.example//:stat.one",
"//:class.example//:stat.two"
]
# Get an instance of SStore class
ss = SStore()
# Update stats every second
data = {id: 0 for id in ssids}
while 1:
for i, id in enumerate(ssids):
data[id] += i + 1
try:
ss.data_update(data)
except:
print("Failed to update stats because {0}".format(
ss.err_description))
exit(1)
# Check for warnings
for w in ss.warnings():
print("Failed to update stat {0} because {1}".format(
w.id, w.description))
time.sleep(1)
Once the sstore has restarted after installing the metadata, run du_example.py as a user with appropriate stats store authorizations. For a description of the authorizations required to provide and update statistics, see the sstore-security man page.
In another terminal, start recording the stats updated through the data_update() function as follows:
sstore capture //:class.example//:stat.*
Sample output:
$ sstore capture //:class.example//:stat.* TIME VALUE IDENTIFIER 2016-03-31T13:36:12 6 //:class.example//:stat.one 2016-03-31T13:36:12 12 //:class.example//:stat.two 2016-03-31T13:36:13 7 //:class.example//:stat.one 2016-03-31T13:36:13 14 //:class.example//:stat.two 2016-03-31T13:36:14 8 //:class.example//:stat.one 2016-03-31T13:36:14 16 //:class.example//:stat.twoExample 3 Update Partitioned Stat Using the data_update() Function
The following metadata files are required for this example, and are installed in /usr/lib/sstore/metadata/json/solaris:
class.example_partition.json
stat.example_partition.json
After installing the files, svc:/system/sstore:default must be restarted.
class.example_partition.json:
{
"$schema": "//:class",
"description": "example class",
"id": "example/partition",
"stability": "stable",
"stat-names": [
"//:stat.one"
]
}
stat.example_partition.json
{
"$schema": "//:stat",
"description": "example stat one",
"id": "//:class.example/partition//:stat.one",
"partitions": [
"name",
"place"
],
"stability": "stable",
"type": "counter",
"units": "calls"
}
du_partition_example.py: Example script to update partitioned stats
import time
import random
from libsstore import SStore
part_keys = [
["john", "tom", "bill"],
["home", "work", "party"]
]
ssid = "//:class.example/partition//:stat.one"
# Get an instance of SStore class
ss = SStore()
# Update stats every second
while 1:
# Add 5 random partition records
val = {}
for i in range(5):
k = []
for keys in part_keys:
k.append(keys[random.randint(0, len(keys) - 1)])
k = tuple(k)
if k in val:
val[k] += i
else:
val[k] = i
# try to update the stat
try:
ss.data_update({ssid: val})
except:
print("Failed to update stats because {0}".format(
ss.err_description))
exit(1)
# Check for warnings
for w in ss.warnings():
print("Failed to update stat {0} because {1}".format(
w.id, w.description))
time.sleep(1)
Once the sstore has restarted after installing the metadata, run data_update_partition() function as a user with appropriate stats store authorizations. For a description of the authorizations required to provide and update statistics, see the sstore-security man page.
In another terminal, start recording the stats updated through the data_update() function as follows:
sstore capture //:class.example/partition//:stat.one sstore capture //:class.example/partition//:stat.one//:part.name sstore capture //:class.example/partition//:stat.one//:part.place sstore capture "//:class.example/partition//:stat.one//:part.place(party)//:part.name"
Sample Output:
$ sstore capture //:class.example/partition//:stat.one
TIME VALUE IDENTIFIER
2016-04-12T16:36:31 140.0 //:class.example/partition//:stat.one
2016-04-12T16:36:32 150.0 //:class.example/partition//:stat.one
2016-04-12T16:36:33 160.0 //:class.example/partition//:stat.one
2016-04-12T16:36:34 170.0 //:class.example/partition//:stat.one
$ sstore capture //:class.example/partition//:stat.one//:part.name
TIME VALUE IDENTIFIER
2016-04-12T16:36:45 //:class.example/partition//:stat.one//:part.name
john: 86.0
tom: 92.0
bill: 102.0
2016-04-12T16:36:46 //:class.example/partition//:stat.one//:part.name
john: 86.0
tom: 93.0
bill: 111.0
$ sstore capture //:class.example/partition//:stat.one//:part.place
TIME VALUE IDENTIFIER
2016-04-12T16:36:50 //:class.example/partition//:stat.one//:part.place
work: 151.0
party: 95.0
home: 84.0
2016-04-12T16:36:51 //:class.example/partition//:stat.one//:part.place
work: 153.0
party: 102.0
home: 85.0
$ sstore capture "//:class.example/partition//:stat.one//:part.place(party)//:part.name"
TIME VALUE IDENTIFIER
2016-04-12T16:37:07 //:class.example/partition//:stat.one//:part.place(party)//:part.name
tom: 44.0
bill: 57.0
john: 53.0
2016-04-12T16:37:08 //:class.example/partition//:stat.one//:part.place(party)//:part.name
tom: 44.0
bill: 58.0
john: 53.0
Example 4 Update Counter Type Stats Using the data_bulk_update() Function
The following metadata files are required for this example and must be installed in /usr/lib/sstore/metadata/json:
class.example_bulk.json
stat.example_bulk.json
After installing the files, svc:/system/sstore:default must be restarted.
class.example_bulk.json:
{
"$schema": "//:class",
"description": "example class",
"id": "example/bulk",
"stability": "stable",
"stat-names": [
"//:stat.one",
"//:stat.two"
]
}
stat.example_bulk.json
[
{
"$schema": "//:stat",
"description": "example stat one",
"id": "//:class.example/bulk//:stat.one",
"min-update-interval": 10,
"stability": "stable",
"type": "counter",
"units": "calls"
},
{
"$schema": "//:stat",
"description": "example stat two",
"id": "//:class.example/bulk//:stat.two",
"min-update-interval": 10,
"stability": "stable",
"type": "counter",
"units": "calls"
}
]
dbu_update.py: Example script to update counter stats using
data_bulk_update()
import time
from libsstore import SStore, SStoreData
MICRO = 1000000
SLEEP_INTERVAL = 10
part_keys = [
["john", "tom", "bill"],
["home", "work", "party"]
]
ssids = [
"//:class.example/bulk//:stat.one",
"//:class.example/bulk//:stat.two"
]
# Get an instance of SStore class
ss = SStore()
# init an instance of SStoreData for each ssid
data = [SStoreData(ss, id) for id in ssids]
# start providing stats
now_ts = int(time.mktime(time.localtime())) * MICRO
ts = now_ts - (SLEEP_INTERVAL * MICRO)
vals = [0 for id in ssids]
while 1:
# Add data points for the last 10 seconds for each ssid
for i in range(SLEEP_INTERVAL):
for j in range(len(ssids)):
vals[j] += (j + 1)
data[j].add_point(ts, vals[j])
ts += MICRO
# try to bulk update
try:
ss.data_bulk_update(data)
except:
print("Failed to update stats because {0}".format(
ss.err_description))
exit(1)
# Check for warnings
for w in ss.warnings():
print("Failed to update stat {0} because {1}".format(
w.id, w.description))
# Reset the data to avoid resending the same data points
# again
for d in data:
d.reset()
time.sleep(SLEEP_INTERVAL)
Once the sstore has restarted after installing the metadata, run dbu_example.py as user with appropriate stats store authorizations. For a description of the authorizations required to provide and update statistics, see the sstore-security man page.
In another terminal, start recording the stats updated through the data_bulk_update() function as follows:
sstore capture //:class.example/bulk//:stat.* sstore export //:class.example/bulk//:stat.*
Sample output:
$ sstore capture //:class.example/bulk//:* 10 TIME VALUE IDENTIFIER 2016-04-12T14:44:50 10 //:class.example/bulk//:stat.one 2016-04-12T14:44:50 20 //:class.example/bulk//:stat.two 2016-04-12T14:45:00 20 //:class.example/bulk//:stat.one 2016-04-12T14:45:00 40 //:class.example/bulk//:stat.two 2016-04-12T14:45:10 30 //:class.example/bulk//:stat.one 2016-04-12T14:45:10 60 //:class.example/bulk//:stat.two $ sstore export //:class.example/bulk//:* TIME VALUE IDENTIFIER 2016-04-12T14:44:41 1 //:class.example/bulk//:stat.one 2016-04-12T14:44:42 2 //:class.example/bulk//:stat.one 2016-04-12T14:44:43 3 //:class.example/bulk//:stat.one 2016-04-12T14:44:44 4 //:class.example/bulk//:stat.one 2016-04-12T14:44:45 5 //:class.example/bulk//:stat.one 2016-04-12T14:44:46 6 //:class.example/bulk//:stat.one 2016-04-12T14:44:47 7 //:class.example/bulk//:stat.one 2016-04-12T14:44:48 8 //:class.example/bulk//:stat.one 2016-04-12T14:44:49 9 //:class.example/bulk//:stat.one 2016-04-12T14:44:50 10 //:class.example/bulk//:stat.one 2016-04-12T14:44:41 2 //:class.example/bulk//:stat.two 2016-04-12T14:44:42 4 //:class.example/bulk//:stat.two 2016-04-12T14:44:43 6 //:class.example/bulk//:stat.two 2016-04-12T14:44:44 8 //:class.example/bulk//:stat.two 2016-04-12T14:44:45 10 //:class.example/bulk//:stat.two 2016-04-12T14:44:46 12 //:class.example/bulk//:stat.two 2016-04-12T14:44:47 14 //:class.example/bulk//:stat.two 2016-04-12T14:44:48 16 //:class.example/bulk//:stat.two 2016-04-12T14:44:49 18 //:class.example/bulk//:stat.two 2016-04-12T14:44:50 20 //:class.example/bulk//:stat.twoExample 5 Update Partitioned Stat Using the data_bulk_update() Function
The following metadata files are required for this example and must be installed in /usr/lib/sstore/metadata/json:
class.example_bulk_partition.json
stat.example_bulk_partition.json
After installing the files, svc:/system/sstore:default must be restarted.
class.example_bulk_partition.json:
{
"$schema": "//:class",
"description": "example class",
"id": "example/bulk/partition",
"stability": "stable",
"stat-names": [
"//:stat.one"
]
}
stat.example_bulk_partition.json
{
"$schema": "//:stat",
"description": "example stat one",
"id": "//:class.example/bulk/partition//:stat.one",
"min-update-interval": 10,
"partitions": [
"name",
"place"
],
"stability": "stable",
"type": "counter",
"units": "calls"
}
dbu_partition_example.py: Example script to update partitioned
stats via data_bulk_update()
import time
import random
from libsstore import SStore, SStoreData
MICRO = 1000000
SLEEP_INTERVAL = 10
part_keys = [
["john", "tom", "bill"],
["home", "work", "party"]
]
ssid = "//:class.example/bulk/partition//:stat.one"
# Get an instance of SStore class
ss = SStore()
# Get an instance of SStoreData for our stat
data = SStoreData(ss, ssid)
# start providing stats
now_ts = int(time.mktime(time.localtime())) * MICRO
ts = now_ts - (SLEEP_INTERVAL * MICRO)
while 1:
# Add data points for the last 10 seconds for each ssid
for i in range(SLEEP_INTERVAL):
# Add 5 random partition records
val = {}
for i in range(5):
k = []
for keys in part_keys:
k.append(keys[random.randint(0, len(keys) - 1)])
k = tuple(k)
if k in val:
val[k] += i
else:
val[k] = i
# Add the data point to ss_data
data.add_point(ts, val)
ts += MICRO
# try to bulk update
try:
ss.data_bulk_update([data])
except:
print("Failed to update stats because {0}".format(
ss.err_description))
exit(1)
# Check for warnings
for w in ss.warnings():
print("Failed to update stat {0} because {1}".format(
w.id, w.description))
# Reset the data to avoid resending the same data points
# again
data.reset()
time.sleep(SLEEP_INTERVAL)
Once the sstore has restarted after installing the metadata, run dbu_partition.py as a user with appropriate stats store authorizations. For a description of the authorizations required to provide and update statistics, see the sstore-security man page.
In another terminal, start recording the stats updated through data_bulk_update() function as follows:
sstore capture //:class.example/bulk/partition//:stat.one 10 sstore capture //:class.example/bulk/partition//:stat.one//:part.name 10 sstore capture //:class.example/bulk/partition//:stat.one//:part.place 10 sstore capture //:class.example/bulk/partition//:stat.one//:part.place(party)//:part.name 10
Sample Output:
$ sstore capture //:class.example/bulk/partition//:stat.one 10
TIME VALUE IDENTIFIER
2016-04-12T15:05:08 200.0 //:class.example/bulk/partition//:stat.one
2016-04-12T15:05:18 300.0 //:class.example/bulk/partition//:stat.one
2016-04-12T15:05:28 400.0 //:class.example/bulk/partition//:stat.one
$ sstore capture //:class.example/bulk/partition//:stat.one//:part.name 10
TIME VALUE IDENTIFIER
2016-04-12T15:05:48 //:class.example/bulk/partition//:stat.one//:part.name
john: 193.0
tom: 224.0
bill: 183.0
2016-04-12T15:05:58 //:class.example/bulk/partition//:stat.one//:part.name
john: 241.0
tom: 248.0
bill: 211.0
$ sstore capture //:class.example/bulk/partition//:stat.one//:part.place 10
TIME VALUE IDENTIFIER
2016-04-12T15:06:18 //:class.example/bulk/partition//:stat.one//:part.place
work: 290.0
party: 305.0
home: 305.0
2016-04-12T15:06:28 //:class.example/bulk/partition//:stat.one//:part.place
work: 321.0
party: 351.0
home: 328.0
$ sstore capture "//:class.example/bulk/partition//:stat.one//:part.place(party)//:part.name" 10
TIME VALUE IDENTIFIER
2016-04-12T15:06:58 //:class.example/bulk/partition//:stat.one//:part.place(party)//:part.name
john: 170.0
tom: 121.0
bill: 194.0
2016-04-12T15:07:08 //:class.example/bulk/partition//:stat.one//:part.place(party)//:part.name
john: 178.0
tom: 122.0
bill: 200.0
See attributes(7) for descriptions of the following attributes:
|
sstore(7), sstored(8), sstore(1), ssid-metadata(7), sstore-security(7)