Monitoring the System Using the CLI

C H A P T E R 8

This chapter describes the CLI commands you can use to monitor the 5800 system. For information on using the GUI to monitor the system, see Monitoring the 5800 System Using the GUI.

This chapter contains the following sections:

Obtaining System Status

Displaying Performance Statistics

Viewing the System Software Version

Obtaining FRU Listings

Obtaining Disk Status

Obtaining Voltage, Temperature, and Fan Speed Data

Note - For instructions on accessing the CLI commands described in this chapter, see Using the Administrative Interfaces.

Obtaining System Status

Obtain basic system state information with the command sysstat.

This command provides an estimate of free space in the system that is available for data storage. For a detailed breakdown of space usage per disk, refer to the df command described in Obtaining Disk Status.

For example:

ST5800 $ sysstat
Cell 0: Online. Estimated Free Space: 7.49 TB
8 nodes online, 32 disks online.
Data VIP 10.8.60.104, Admin VIP 10.8.60.103
Data services Online, Query Engine Status: HAFaultTolerant
Data Integrity check not completed since boot
Data Reliability check last completed at Wed Sep 05 07:12:43 UTC 2007
Query Integrity established as of Wed Sep 05 01:31:20 UTC 2007
NDMP status: Backup ready.
ST5800 $

The output that the sysstat command produces is explained below. Data reported is for all online disks in the entire system.

Data services Online indicates that the system is available to read and write to via the API, while Data services Offline means that the system is not available to read and write to via the API.

Query Engine Status reports the states of the query engine, as follows:

HAFaultTolerant - Query services are available and highly fault tolerant.

FaultTolerant - Query services are available, but not as fault tolerant as in the HAFaultTolerant state.

Operational - Query services are available, but not fault tolerant.

Starting - The query engine is starting up. This process may include creating the query database or recreating the connection to the database. Query services are not available during this process.

Unknown - The query engine is in an undetermined state. This may be because it is too early in the starting up process to establish a connection to the query engine, or the query engine is in the process of restarting.

Stopped - The query engine is stopped; query services are not available.

Unavailable - The query engine is not returning any status at this time, probably because it is in a transitional state; query services may not be available.

Nonoperational - The query engine is corrupted; no query services will be available until the system has completed recreating the engine.

Data Integrity check indicates when the system last completed checking each fragment on the system for integrity against bit rot. Each cycle of this testing might take up to one week to complete, so the check will be listed as not complete for the first week after a system reboot.

Data Reliability check indicates when the system last completed a full cycle of testing to detect and recover any missing fragments, indicating that the system has full reliability. Each cycle of this testing takes approximately 12 hours to complete, so the check will be listed as not complete for the first 12 hours after a system reboot.

Query Integrity established provides assurance that a query of data stored on the 5800 system will accurately reflect the contents of the object archive. Exceptions would be data that was stored or deleted from the 5800 system while the query was in progress, as well as objects that were stored after the query integrity time and for which the store operation returned the special error status isIndexed=false to the storing application.

NDMP Status check indicates the status of the Network Data Management Protocol (NDMP), which allows you to back-up the data stored on the system to tape and restore that data in the event of catastrophic system loss. This check indicates whether the data has been backed up and is available for restoration and also whether the backup or restore is in progress.

Obtain extended system state with the command sysstat --verbose.

Verbose output includes the online/offline status of each node and disk in the system. The online/offline status reported by the command refers to the logical system status. To see the state of hardware components, refer to the hwstat command described in Obtaining FRU Listings.

For example:

ST5800 $ sysstat --verbose
NODE-101     [ONLINE]
DISK-101:0   [ONLINE]
DISK-101:1   [OFFLINE]
DISK-101:2   [ONLINE]
DISK-101:3   [ONLINE]
NODE-102     [ONLINE]
DISK-102:0   [ONLINE]
DISK-102:1   [ONLINE]
DISK-102:2   [ONLINE]
DISK-102:3   [ONLINE]
NODE-103     [ONLINE]
DISK-103:0   [ONLINE]
.
.
.

Note - If a disk is listed as off line, the disk should be replaced.

Displaying Performance Statistics

Display real-time performance metrics about throughput and operations using the command perfstats.

The metrics displayed reflect activity during a specified time internal (default is 15 seconds). There is a delay between the time an action occurred on the system and the time it is displayed by the perfstats command.

For example:

ST5800 $ perfstats
 
Cell Performance Statistics:
 
                                 Avg         Avg
                     # Ops      Op/sec     KB/sec
                   -------- ----------  ------------
  Add MD:                 0     0.00          0.00
  Store:                  0     0.00          0.00
  Retrieve:               1     0.20          0.15
  Retrieve MD:            0     0.00          0.00
  Delete:                 0     0.00             -
  Query:                687    22.90             -
  WebDAV Put:             0     0.00          0.00
  WebDAV Get:             0     0.00          0.00
 
Hive Performance Statistics:
 
Load 1m: 4.12 Load 5m: 4.21 Load 15m: 4.43
Disk Used: 241.28 GB  Disk Total: 13.38 TB  Usage: 1.8%

Display performance statistics for repeated intervals of time, for a specified period of time, using the commands perfstats --howlong minutes and perfstats --interval seconds.

Note - To specify that the perfstats command should run indefinitely, use --howlong 0.

Display performance statistics for a single node in the system using the command perfstats --node node_id.

For example:

ST5800 $ perfstats --node NODE-101
 
NODE-101 Performance Statistics:
 
 
 
                                 Avg         Avg
                     # Ops      Op/sec     KB/sec
                   -------- ----------  ------------
  Add MD:                 0     0.00          0.00
  Store:                  0     0.00          0.00
  Retrieve:               1     0.20          0.15
  Retrieve MD:            0     0.00          0.00
  Delete:                 0     0.00             -
  Query:                687    22.90             -
  WebDAV Put:             0     0.00          0.00
  WebDAV Get:             0     0.00          0.00
 
Hive Performance Statistics:
 
Load 1m: 4.12 Load 5m: 4.21 Load 15m: 4.43
Disk Used: 241.28 GB  Disk Total: 13.38 TB  Usage: 1.8%

Viewing the System Software Version

Display the version of the system software using the command version.

For example:

ST5800 $ version
ST5800 1.1 release [1.1-11076]

Display version information for each node, the service node, and the switches using the command version --verbose.

For example:

Note - In normal operation, all nodes should be running the same version of Service Management Daughter Card (SMDC) and the same version of Basic Input Output System (BIOS).


ST5800 $ version --verbose
ST5800 1.1 release [1.1-11076]
Service Node:
        BIOS Version: 1.1.3
        SMDC Version: 4.13
Switch:
        Overlay Version (sw#1): 11068
        Overlay Version (sw#2): 11068
 
NODE-101:
        BIOS version: 0.1.8
        SMDC version: 4.18
NODE-102:
        BIOS version: 0.1.8
        SMDC version: 4.18
NODE-103:
        BIOS version: 0.1.8
        SMDC version: 4.18
NODE-104:
        BIOS version: 0.1.8
        SMDC version: 4.18
 
.
.
.
 
 
ST5800 $

Obtaining FRU Listings

Obtain a list of field-replaceable units (FRUs) with the command hwstat --cellid cellid.

For example:

ST5800 $ hwstat --cellid 8
Component     Type    FRU ID                                          Status
------------  ------  ----------------------------------------------  --------
NODE-101      NODE    91230b2c-6ac5-d311-0180-c96e5981e000            ONLINE
DISK-101:0    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGHYS3D    ENABLED
DISK-101:1    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGHLS7D    ENABLED
DISK-101:2    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGHJY1D    ENABLED
DISK-101:3    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGHW6ED    ENABLED
NODE-102      NODE    96230b2c-6ac5-d311-0180-c1645981e000            ONLINE
DISK-102:0    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGHW8VD    ENABLED
DISK-102:1    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGE8G9D    ENABLED
DISK-102:2    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGHVX2D    ENABLED
DISK-102:3    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGHW0ND    ENABLED
NODE-103      NODE    92230b2c-6ac5-d311-0180-7e6d5981e000            ONLINE
DISK-103:0    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGHVZLD    ENABLED
DISK-103:1    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGHN3RD    ENABLED
DISK-103:2    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGHVR4D    ENABLED
DISK-103:3    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGHVXSD    ENABLED
.
.
.
ST5800 $

Obtain information about a specific FRU using the command hwstat --FRUID fruid.

For example:

ST5800 $ hwstat --FRUID NODE-107
Component     Type    FRU ID 																	Status
------------  ------  -------------------------------------------- --------
NODE-107		 NODE    72cda8b6-aec3-d311-0080-2a835981e000 															 ONLINE
DISK-107:0    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGLX7GD 		ENABLED
DISK-107:1    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGLY5PD 	ENABLED
DISK-107:2    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGGY8VD 	ENABLED
DISK-107:3    DISK    ATA_____HITACHI_HDS7250S______KRVN63ZAGLXA7D 	ENABLED
 
ST5800 $

Obtaining Disk Status

Use the df command to display a summary of disk usage. Note the following information about the utilization numbers displayed:

The used value in the display is not equivalent to the total number of object bytes stored in the system. The used value includes space consumed by data parity, object headers and footers, and query indexes.

Storage utilization statistics displayed by df are refreshed every three minutes.

When using df to view storage utilization, the system reserves 15% of raw storage space to allow for data recovery on a full system.

Obtain a summary of disk usage in an easily readable format with the command df --human-readable.

The displayed numbers refer to the logical space used by, or available for, the user’s data storage. Reserved space is reserved by the system for data recovery operations, and is not available to the user. Total space is a sum of Available + Used + Reserved. The use percentage is calculated as Used / (Used + Available).

For example:

ST5800 $ df --human-readable
Total:  13.39 TB; Avail:  13.17 TB; Used: 219.52 GB; Usage: 1.6%

Obtain information on the physical space available on the disk with the command df --physical.

Note - The system can no longer accept objects for storage when any disk in the system reaches 80% capacity.

For example:.

ST5800 $ df --physical
All sizes expressed in 1K blocks
DISK-101:0: Total: 449128448; Avail: 434057216; Used:  15071232; Usage: 3.4%
DISK-101:1: Total: 449128448; Avail: 444561408; Used:   4567040; Usage: 1.0%
DISK-101:2: Total: 449128448; Avail: 444561408; Used:   4567040; Usage: 1.0%
DISK-101:3: Total: 449128448; Avail: 444561408; Used:   4567040; Usage: 1.0%
DISK-102:0: Total: 449128448; Avail: 444561408; Used:   4567040; Usage: 1.0%.
.
.
.

Obtaining Voltage, Temperature, and Fan Speed Data

Use the command sensors to display voltage, temperature, and fan speed data, as collected by system sensors.

For example:

ST5800 $ sensors
 
NODE-101:
        DDR Voltage           2.60 Volts
        CPU Voltage           1.42 Volts
        VCC 3.3V              3.32 Volts
        VCC 5V                5.12 Volts
        VCC 12V               12.03 Volts
        Battery Voltage       2.98 Volts
        CPU Temperature       49 degrees C
        System Temperature    32 degrees C
        System Fan 1 speed    11340 RPM
        System Fan 2 speed    11340 RPM
        System Fan 3 speed    11070 RPM
        System Fan 4 speed    10980 RPM
        System Fan 5 speed    11070 RPM
 
NODE-102:
        DDR Voltage           2.60 Volts
        CPU Voltage           1.43 Volts
        VCC 3.3V              3.32 Volts
        VCC 5V                5.10 Volts
        VCC 12V               12.10 Volts
        Battery Voltage       2.98 Volts
        CPU Temperature       49 degrees C
        System Temperature    33 degrees C
        System Fan 1 speed    11700 RPM
        System Fan 2 speed    11430 RPM
.
.
.
 
ST5800 $