Go to main content

Using Oracle® Solaris 11.4 StatsStore and System Web Interface

Exit Print View

Updated: February 2019
 
 

Troubleshooting Retrieving Statistics

The following sections describe how to diagnose and fix certain issues.

Ensure Services Are Online

Verify that the system services associated with the statistics store and System Web Interface are online and not in maintenance.

If a service is not online, use the svcs -Lx command to investigate. See Troubleshooting Services Problems in Managing System Services in Oracle Solaris 11.4.

StatsStore

The Oracle Solaris 11.4 OS includes StatsStore; you do not need to install it. The statistics store, CLI, and APIs are delivered by the sstore package. The sstored daemon is invoked automatically during system startup and restarted if any failures occur.

Verify that both of the following services are online:

  • sstore – The sstored daemon captures statistics and events for the OS and applications.

  • sysstat – The sysstatd daemon provides common OS statistics to the statistics store.

$ svcs sstore sysstat
STATE          STIME    FMRI
online         13:55:28 svc:/system/sstore:default
online         13:55:28 svc:/system/sysstat:default

System Web Interface

The System Web Interface is delivered by the webui-server package. You might need to install the webui-server package:

$ pkg install webui-server

Make sure the svc:/system/webui/server:default service is online.

System Web Interface Problems

Check the following log files:

  • System Web Interface log files in svcs -L webui/server and /var/webui/logs

  • Apache web server log files in svcs -L http:apache24 and /var/apache2/2.4/logs

Ensure the Certificate and Key are Installed

The /var/webui/conf/webui.conf file and the conf property group of the webui/server service show where your certificate file and key file must be installed.

$ svccfg -s webui/server:default listprop -o propname,value conf/cert*file
cert_file          /var/webui/conf/authdb/server/server.crt
cert_key_file      /var/webui/conf/authdb/server/server.key
$ ls -l /var/webui/conf/authdb/server

Degraded Performance

If your System Web Interface performance is degraded, try one of the following steps:

  • Close some instances of the System Web Interface if you have multiple tabs or browsers open.

  • Check the performance impact of data you are showing. Select the Properties option from the Visualization Actions menu, and note the value of “Performance impact” at the bottom of the panel. If you can, reduce the number of expensive statistics you are monitoring.

    You can also check for expensive statistics at the command line:

    $ sstore info //:class.net/ip//:stat.out-bytes
     Identifier: //:class.net/ip//:stat.out-bytes
       agg-name: bytes
    description: bytes sent
     partitions: link
     partitions: protocol
     partitions: local-address
     partitions: remote-address
     partitions: application
           ssid: //:class.net/ip//:stat.out-bytes
          units: bytes
      stability: unstable
      expensive: True
           type: counter
  • Flush the browser cache.

  • Restart the browser.

No Matches Found for Statistic

A “no matches found” error message is shown in the following cases:

  • The class, resource, or statistic does not exist.

  • The SSID is specified incorrectly.

  • You used a wildcard in an SSID that is classified as not browsable or not stable.

SSID Does Not Exist

In the following example, resource identifiers 0-3 exist, but the specified identifier, 4, does not exist:

$ sstore list //:class.cpu//:res.id/4
Warning (//:class.cpu//:res.id/4) - lookup error: no matches found
$ sstore list //:class.cpu//:res.id/*
IDENTIFIER
//:class.cpu//:res.id/0
//:class.cpu//:res.id/1
//:class.cpu//:res.id/2
//:class.cpu//:res.id/3

SSID is Incorrectly Specified

In the following example, a colon character was omitted from the delimiter in front of the operation:

$ sstore capture //:class.proc//:res.*//:stat.cpu-percentage//op.top
TIME                VALUE IDENTIFIER
Warning (//:class.proc//:res.*//:stat.cpu-percentage//op.top) - lookup error: no matches found
$ sstore capture //:class.proc//:res.*//:stat.cpu-percentage//:op.top
TIME                VALUE IDENTIFIER
2016-05-16T17:33:22 0.20751953125 //:class.proc//:res.sstored/1696/root//:stat.cpu-percentage//:op.top
...

SSID is Unbrowsable or Unstable

If an SSID is classified as unbrowsable or unstable, use one of the following methods to get information about that SSID:

  • Specify the full SSID explicitly

  • If you want to use a wildcard in the SSID to match multiple SSIDs, use the -a option with the sstore subcommand.

The following example shows that the kstat class is not browsable, though you can get information when you specify the SSID without the wildcard.

$ sstore list //:class.kstat*
Warning (//:class.kstat*) - lookup error: no matches found
$ sstore info //:class.kstat
 Identifier: //:class.kstat
description: kstat free-for-all namespace
  stability: stable
  browsable: False

All resources in the kstat class are classified as unstable.

$ sstore list //:class.kstat//:res.*
Warning (//:class.kstat//:res.*) - lookup error: no matches found

You can get information or record data if you fully specify the SSID.

$ sstore info //:class.kstat//:res.system/cpu/0/intr
 Identifier: //:class.kstat//:res.system/cpu/0/intr
  stability: unstable
       type: Untyped
description: CPU Interrupt statistics

If you need to use a wildcard, use the -a option to get information or record data.

$ sstore list -a //:class.kstat//:res.system/cpu/0/intr//:*
IDENTIFIER
//:class.kstat//:res.system/cpu/0/intr//:stat.crtime
//:class.kstat//:res.system/cpu/0/intr//:stat.level-count
//:class.kstat//:res.system/cpu/0/intr//:stat.level-time
//:class.kstat//:res.system/cpu/0/intr//:stat.snaptime
$ sstore info //:class.kstat//:res.system/cpu/0/intr//:stat.snaptime
           Identifier: //:class.kstat//:res.system/cpu/0/intr//:stat.snaptime
            stability: unstable
                 type: counter
kstat name/value type: Relative time
          description: kstat snapshot time
          kstat scale: 1000000000

Some, but not all, //:class.net/ip statistics are classified as unstable.

The following command records values for eight statistics:

$ sstore capture //:class.net/ip//:stat.*

The following command records values for 26 statistics:

$ sstore capture -a //:class.net/ip//:stat.*

If you name the statistic explicitly, you can get information or record data without using the -a option.

$ sstore info //:class.net/ip//:stat.out-bytes
 Identifier: //:class.net/ip//:stat.out-bytes
   agg-name: bytes
description: bytes sent
 partitions: link
 partitions: protocol
 partitions: local-address
 partitions: remote-address
 partitions: application
       ssid: //:class.net/ip//:stat.out-bytes
      units: bytes
  stability: unstable
  expensive: True
       type: counter
$ sstore capture //:class.net/ip//:stat.out-bytes
TIME                VALUE IDENTIFIER
2016-05-18T15:15:06 1201563.0 //:class.net/ip//:stat.out-bytes

The sstore export command does not require the -a option because that data is already available.

$ sstore export //:class.net/ip//:stat.*
TIME                VALUE IDENTIFIER
2016-05-18T14:10:41 34731 //:class.net/ip//:stat.in-delivers

Size of the Statistics Store Repository

An important feature of the statistics store repository is that the data you request (whether through CLI, API, or the System Web Interface) is saved, enabling you to later compare current and historical values. To prevent the statistics store repository from growing too large, a size limit is set, and older data is pruned from the repository to stay within the defined limit.

The sstored daemon prunes data by saving fewer and fewer samples of older data. The same set of data might be pruned multiple times as necessary to keep the repository within the size limit as new data is added. For example, if you had recorded some data every second, sstored might prune older data to one value every five seconds, then one value every minute, then one value every hour and so forth as needed.

You can do some data pruning yourself by using the sstoreadm purge command. See the sstoreadm(1) man page for details.

The repository size limit is a property of the sstore service:

$ svcprop -p config sstore
config/max-process-size integer 536870912
config/max-repo-size integer 2048
config/repo-path astring /var/share/sstore/repo

The max-repo-size is a global property of the service, not an instance property. The max-process-size is a property of the sstore:default service instance.

$ svccfg -s sstore listprop -o propname,value config/*
max-repo-size        2048
repo-path            /var/share/sstore/repo
$ svccfg -s sstore:default listprop -o propname,value config/*
max-process-size        536870912

The value of max-repo-size is in megabytes. You can use the zfs list command to check the current usage:

$ zfs list rpool/VARSHARE/sstore
NAME                    USED  AVAIL  REFER  MOUNTPOINT
rpool/VARSHARE/sstore  26.8M   245G  26.8M  /var/share/sstore/repo

The following command shows how to change the maximum statistics store repository size to 4 GB:

$ svccfg -s sstore
svc:/system/sstore> listprop config/max-repo-size
config/max-repo-size integer     2048
svc:/system/sstore> setprop config/max-repo-size=4096
svc:/system/sstore> listprop config/max-repo-size
config/max-repo-size integer     4096
svc:/system/sstore> refresh
svc:/system/sstore> exit
$ svcprop -p config/max-repo-size sstore
4096

Repository Version Not Supported

If a system update includes a statistics store repository format change, the system/sstore:default service might fail to start in the updated image with the following messages in the service log file:

Method "start" exited with status 0.
sstore: Repo version unsupported

Run in Debug Mode

Another way to get more information about the statistics store is to run the statistics store daemon with the --debug and --log options:

# /usr/lib/sstored --debug --log --log-path=/var/tmp/log --no-cache --events --no-auth