Solstice DiskSuite 4.2.1 User's Guide

Chapter 6 Managing the System

This chapter describes how to perform system management tasks related to DiskSuite.

Use the following to proceed directly to the section that provides step-by-step instructions for using DiskSuite Tool.

Use the following to proceed directly to the section that provides step-by-step instructions for using the command line interface.

Overview of Managing the System

The tasks associated with managing systems running DiskSuite include:

For general information on DiskSuite, see Solstice DiskSuite 4.2.1 Reference Guide.

Prerequisites for Managing the System

Here are the prerequisites for the steps in this chapter:

Working With the Graphical View of the SPARCstorage Array

This section explains how to:

Preliminary Information for Working Graphically with the SPARCstorage Array

DiskSuite Tool's Disk View window presents a graphical view of the SPARCstorage Array. When you display the Disk View window, the icon for a SPARCstorage Array controller resembles a fibre channel icon, both on the Disk View canvas and in the Controllers window.

Each tray within the SPARCstorage Array is represented by an "etched-in" frame, with a title showing the tray number. Differences between the SPARCstorage Array 100 and 200 are:

Figure 6-1 SPARCstorage Array 100

Graphic

If one or both of the busses on the tray has no disks, the bus is "stubbed-out" by showing an empty frame. If one or more of the trays have no disks, DiskSuite Tool does not stub-out the tray(s).

Figure 6-2 SPARCstorage Array 200

Graphic

You can also use the Disk View window to display the SPARCstorage Array's unique 12-digit identification number, known as the World Wide Name. The last four digits appear just below the controller icon on the canvas (and on the LCD panel on the SPARCstorage Array itself). You can see the entire number by displaying a controller's Information window.

The Disk View window enables you to check at a glance the status of SPARCstorage Array components, including the battery and fan, and to see if a disk is spun up or down.

How to Select Objects in the Disk View Window (DiskSuite Tool)

This task explains how to select objects in the Disk View window. The selection model is hierarchical: selecting an object selects not only the object, but all of its child objects as well.

  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information for Working Graphically with the SPARCstorage Array").

  2. Click Disk View to display the Disk View window.

  3. To select an object, click it.

    For SPARCstorage Arrays:

    • Selecting all slices in a disk selects the disk. Selecting a disk selects all slices in the disk.

    • Selecting all disks in a tray selects the tray. Selecting a tray selects all disks in the tray.

    • Selecting all trays on a controller selects the controller. Selecting a controller selects all trays on the controller.

    For non-SPARCstorage Array configurations:

    • Selecting all slices in a disk selects the disk. Selecting a disk selects all slices in the disk.

    • Selecting all disks on a controller selects the controller. Selecting a controller selects all disks on the controller.

How to Check the Status of SPARCstorage Array Disks (DiskSuite Tool)

Use this task to view the status of SPARCstorage Array disks (spun up or down).

  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information for Working Graphically with the SPARCstorage Array").

  2. Click Disk View to display the Disk View window.

  3. Look at the canvas. A disk that is spun down appears with a down arrow icon beneath the disk.

    If necessary, display the desired controllers that contain the disks by selecting them from the Controllers window.


    Note -

    You can use DiskSuite Tool to spin down non-SPARCstorage Array disks. However, the Disk View window shows a spun up or spun down status only for SPARCstorage Array disks. Also, any I/O performed on a non-SPARCstorage Array disk that is spun down automatically spins up the disk.


Where to Go From Here

To spin up SPARCstorage Array disks that have been spun down, refer to "How to Start a Disk (DiskSuite Tool)".

How to Check the Status of a SPARCstorage Array Controller's Fan and Battery (DiskSuite Tool)

  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information for Working Graphically with the SPARCstorage Array").

  2. Click Disk View to display the Disk View window.

  3. Look at the icons that appear beside each controller. (If necessary, display the desired controllers by selecting them in the Controllers window.)

    DiskSuite Tool indicates the status for a SPARCstorage Array controller's fans and battery with the same icons that appear on the SPARCstorage Array LCD screen:

    • Wrench icon - the object needs maintenance.

    • Wrench and Fan icons - a fan has failed, or excessive internal temperatures have occurred. In most cases, the SSA will be taken off-line automatically.

    • Wrench and battery icons - the NVRAM battery is low, or not installed.


    Note -

    The Disk View window also shows problems on non-SPARCstorage Array disks. If a wrench icon appears beneath one of these disks, there is a disk problem.


    Figure 6-3 shows a controller that has the wrench and battery icons displayed.

    Figure 6-3 SPARCstorage Array Wrench and Battery Icons

    Graphic


    Note -

    One or more failures can occur simultaneously. In this case, multiple icons can appear. The Controller Information window will also contain information on the failures, as will the Problem Log.


Where to Go From Here

For information on resolving problems with SPARCstorage Arrays, refer to Chapter 7, Troubleshooting the System.

How to Display a SPARCstorage Array Controller's World Wide Name (DiskSuite Tool)

  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information for Working Graphically with the SPARCstorage Array").

  2. Click Disk View to display the Disk View window.

  3. Look at the controllers on the canvas. (If necessary, display the desired controllers by selecting them in the Controllers window.)

    The last four digits of the World Wide Name appear below each SPARCstorage Array controller.

  4. To view the entire 12-digit number, select the controller, display its pop-up menu and select Info.

    The entire World Wide Name appears in the Info window as the Serial Number.

Administering the SPARCstorage Array

This section explains how to use DiskSuite Tool to:

DiskSuite Tool provides an integrated solution for working with SPARCstorage Arrays, enabling you to view the physical layout of the SPARCstorage Array hardware, and to perform various administrative tasks. You must use the DiskSuite Tool Disk View window to perform these tasks; DiskSuite does not provide equivalent command line utilities. To administer a SPARCstorage Array from the command line, use the ssaadm(1M) command.


Caution - Caution -

Use this functionality with care. It provides a powerful way to manage the SPARCstorage Array. As a minimum precaution, you should have a current backup of your data before using DiskSuite Tool in this way.


Preliminary Information For Enabling and Disabling NVRAM

How to Enable NVRAM on a Controller, Tray, or Disk (DiskSuite Tool)

  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information For Enabling and Disabling NVRAM").

  2. Refer to "How to Stop a Disk (DiskSuite Tool)" to quiesce all I/O.

    Be sure that no I/O is active on metadevices defined that exist on disks within a SPARCstorage Array. In particular, ensure that ownership of any diskset metadevices has been released since an implicit I/O stream exists while ownership of a diskset is maintained.

  3. Click Disk View.

    The Disk View window appears.

  4. In the Disk View window, select a controller, tray, or individual disk or disks.

  5. Choose Fast Write from the Object menu, then choose Enable.

  6. A Fast Write dialog box appears, reminding you that all I/O to the object should be stopped. Click Continue.

  7. A confirmation appears, indicating that fast write has been enabled. Click OK.

  8. Refer to "How to Start a Disk (DiskSuite Tool)" to restart the disks.

How to Enable NVRAM For Synchronous Writes on a Controller, Tray, or Disk (DiskSuite Tool)

This procedure enables fast writes for synchronous writes only. To enable fast writes for "all" writes, use the previous procedure.

  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information For Enabling and Disabling NVRAM").

  2. Refer to "How to Stop a Disk (DiskSuite Tool)" to quiesce all I/O.

    Be sure that no I/O is active on metadevices that exist on disks within a SPARCstorage Array. In particular, ensure that ownership of any diskset metadevices has been released since an implicit I/O stream exists while ownership of a diskset is maintained.

  3. Click Disk View.

    The Disk View window appears.

  4. In the Disk View window, select a controller, tray, or individual disk or disks.

  5. Choose Fast Write from the Object menu, then choose Synchronous.

  6. A Fast Write dialog box appears, reminding you that all I/O to the object should be stopped. Click Continue.

  7. A confirmation appears, indicating that fast write for synchronous writes has been enabled. Click OK.

  8. Refer to "How to Start a Disk (DiskSuite Tool)" to restart the disks.

How to Disable NVRAM on a Controller, Tray, or Disk (DiskSuite Tool)

  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information For Enabling and Disabling NVRAM").

  2. Click Disk View.

    The Disk View window appears.

  3. In the Disk View window, select a controller, tray, or individual disk or disks.

  4. Choose Fast Write from the Object menu, then choose Disable.

  5. A confirmation appears, indicating that fast write has been disabled. Click OK.

Preliminary Information for Purging and Flushing NVRAM Data

How to Flush Outstanding Writes From NVRAM (DiskSuite Tool)

This task flushes out all outstanding writes for the selected controller (and all disks), tray (and all disks in that tray), or individual disk(s), from the NVRAM to disk.

  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information for Purging and Flushing NVRAM Data").

  2. Click Disk View.

    The Disk View window appears.

  3. In the Disk View window, select a controller, tray, or individual disk or disks.

  4. Choose Sync NVRAM from the Object menu.

  5. A confirmation dialog box appears, indicating the NVRAM will be synced. Click OK.

How to Purge Fast Write Data NVRAM (DiskSuite Tool)

Use this task only if you can no longer access the SPARCstorage Array, such as when a drive has failed. If you can access drives in the SPARCstorage Array, refer to "How to Flush Outstanding Writes From NVRAM (DiskSuite Tool)". Purging fast write data gets rid of any outstanding writes.

  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information for Purging and Flushing NVRAM Data").

  2. Click Disk View.

    The Disk View window appears.

  3. In the Disk View window, select a controller, tray, or disk and display the object's pop-up menu.

  4. Select Purge NVRAM. A confirmation dialog box appears, indicating that the NVRAM will be purged. Click Purge.

Preliminary Information for Reserving and Releasing Disks


Note -

This functionality is currently available only for SPARC systems using SPARCstorage Array disks.


How to Reserve a Disk for Host Exclusive Use (DiskSuite Tool)

  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information for Reserving and Releasing Disks").

  2. Click Disk View.

    The Disk View window appears.

  3. Select a controller, tray, or single disk and choose Reserve Disks from the Object menu.

  4. A confirmation dialog box appears, indicating the disk reservation. Click OK.

    There is no visual indication that the disk has been reserved. However, if DiskSuite Tool is run on a host other than the host issuing the reserve command, a status of "Reserved" is shown for those disks in the Information window, and a "lock" icon appears in the Disk View canvas.

How to Release a Disk Reserved by Host (DiskSuite Tool)

  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information for Reserving and Releasing Disks").

  2. Click Disk View.

    The Disk View window appears.

  3. Select a controller, tray, or single disk and choose Release Disks from the Object menu.

  4. A confirmation dialog box appears, indicating the disk release. Click OK.

Preliminary Information for Stopping and Starting Disks


Caution - Caution -

The ssaadm(1M) command does not generate an event notification for DiskSuite when used to start and stop a disk. For this reason, do not use DiskSuite Tool and ssaadm(1M) together when stopping and starting disks. Doing so could cause DiskSuite Tool to display a disk`s status incorrectly. Always use one or the other to both stop and start a disk.


How to Stop a Disk (DiskSuite Tool)

Prerequisites for Stopping a Disk (SPARCstorage Array)

Because SPARCstorage Array disks are in trays, to stop one disk you should stop all disks in the tray to ensure proper shutdown. This involves:


Note -

Stopping the disk also causes a sync_cache to flush outstanding writes in NVRAM.


  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information for Stopping and Starting Disks").

  2. Check for other DiskSuite objects on the affected controller, disk, or tray.

    In the Disk View window, display the controller. Display the controller's Info window and select Device Mappings to view which objects (metadevices, hot spares, state database replicas) might be using the physical device(s).

  3. Stop all access to the disk.

    For example, unmount any file systems associated with the disk, or stop all I/O to a database using the disk.

  4. In the Disk View window, select a controller, tray, or single disk and choose Stop Disks from the Object menu.

  5. A Stop Disk dialog box appears, reminding you that all I/O to the disk should be stopped. Click Continue.

  6. The system notifies you when the disk is stopped. Click OK.

    For a SPARCstorage Array, a down arrow appears beneath each disk that is stopped, indicating that the disk is spun down. If you stop a SCSI disk, no down arrow appears.

How to Start a Disk (DiskSuite Tool)

  1. Make sure you have met the prerequisites ("Prerequisites for Managing the System") and have read the preliminary information ("Preliminary Information for Stopping and Starting Disks").

  2. Click Disk View.

    The Disk View window appears. SPARCstorage Array disks that are spun down appear on the canvas with a down arrow beneath the disk.

  3. Select a controller, to start all disks connected to it, or select individual disks using Control-click.

    You can also select multiple controllers using Control-click.

  4. Choose Start Disks from the Object menu.

  5. The system notifies you when the disk is started. Click OK.

    For a SPARCstorage Array disk, the down arrow disappears from beneath the disk, indicating it has spun up. There is no indication for a SCSI disk.

Monitoring and Graphing Performance

This section explains how to:

Performance Monitoring vs. Performance Analysis

DiskSuite Tool gives you the capability for simple performance monitoring of metadevices and physical disks. The goal of performance monitoring is to maximize system performance by making small changes while maintaining data availability. Performance monitoring assumes that you are using a given configuration that you cannot easily or often reconfigure.

Performance analysis, on the other hand, aims to maximize system throughput or minimize latency by trying out various configurations until the "best performance" is achieved. Performance analysis assumes that you can set up multiple configurations and collect a different set of data from each configuration. This requires a different methodology and set of tools to cover such tasks as device configuration, file system management, data collection, test load generation, and the like.

The performance monitoring capability within DiskSuite Tool is best suited for the day-to-day operation of DiskSuite in spotting performance problems associated with hot-spots and bottlenecks. This enables you to monitor general performance trends, look for abrupt changes, and compare data collected for different parts of a configuration. For example, while monitoring a RAID5 metadevice, you might notice one column (slice) that shows much more I/O than the other columns. By moving that column to another disk, you are able to better balance the I/O for the entire metadevice.

Preliminary Information for Performance Monitoring and Graphing

How to View Device Statistics (DiskSuite Tool)

Use this task to display a device statistic sheet that shows a "snap-shot" for the current values of the statistical variables.

  1. Select an object from either the Metadevice Editor window or the Disk View window.

    The MetaDB object, hot spare pools, and slices are not able to show statistics.

  2. Select Statistics from the Object menu in either window to display the Device Statistics window for the object.

    For reference information on the Device Statistics window, see the online help.

  3. To view a new snap-shot of statistical information, click Update.


    Note -

    DiskSuite Tool displays a separate Device Statistics window for each disk selected in the Disk View window. Thus, if you select a controller or tray with many disks, you'll see as many separate Device Statistics windows as disks selected.


How to Graph Device Statistics (DiskSuite Tool)

Use this task to display instantaneous statistical information in the form of a graph for a device.

  1. Select Statistics Graphs from the Browse menu in the Metadevice Editor window to display the Statistics Graphs window.

  2. Drag an object from either the Metadevice Editor window or the Disk View window to the Statistics Graphs window.

    The Statistics Graphs window will display statistics and a blank area for the graph of the object. The MetaDB object, hot spare pools, and slices are not able to graph statistics.

  3. Choose the statistical information to be graphed.

    The Statistics Graphs window contains two "y-axis" (two vertical scales) of statistics. Thus, you can graph any two of the types of statistics available:

    Ops/sec - The number of disk I/O operations per second, measured in terms of reads, writes, or as a total.

    Kbytes/sec - The number of kilobytes of disk I/O, measured in terms of reads, writes, or as a total.

    Load - The disk load measured either as the percentage of time busy, or as a wait queue, which represents the average number of transactions waiting for service.

  4. When done using the Statistics Graphs window, select Put Away from the All Graphs menu.

    If you do not put away a device, the Statistics Graphs window automatically includes it the next time it is opened.

How to Add Devices to the Statistics Graph Window (DiskSuite Tool)

  1. Select Statistics Graphs from the Browse menu in the Metadevice Editor window to display the Statistics Graphs window.

  2. Drag an object from either the Metadevice Editor window or the Disk View window to the Statistics Graphs window.

    The Statistics Graphs window will display statistics and a blank area for the graph of the object. The MetaDB object, hot spare pools, and slices are not able to graph statistics.

  3. Refer to Step 3 in the "How to Graph Device Statistics (DiskSuite Tool)" task ("How to Graph Device Statistics (DiskSuite Tool)").

How to Remove Devices From the Statistics Graph Window (DiskSuite Tool)

  1. Select Statistics Graphs from the Browse menu in the Metadevice Editor window to display the Statistics Graphs window.

  2. To remove a single device, select it and click the Put Away button.

    The Statistics Graphs window clears the device from the list.

  3. To remove all devices, select Put Away from the All Graphs menu in the Statistics Graphs window.

Integrating SunNet Manager With DiskSuite

DiskSuite and SunNet Manager can be configured for cooperative ease of use. DiskSuite has the capability to forward its driver messages using asynchronous SNMP traps via a daemon that monitors the system console log. Messages, such as failures and errors, can then be detected by a SunNet Manager console. When you receive DiskSuite messages within SunNet Manager, you can launch DiskSuite Tool.

For more information on SunNet Manager, see the Site/SunNet/Domain Manager Administration Guide.

How to Enable SunNet Manager to Launch DiskSuite Tool (SunNet Manager)

SunNet Manager automatically receives DiskSuite console messages. To enable SunNet Manager to launch DiskSuite Tool, follow these steps:

  1. With SunNet Manager running, select Save To File from the File menu, then Exit from the File menu.

  2. Edit the /opt/SUNWconn/snm/struct/elements.schema file to change the elementCommand structure for your server. The entry should look like this:


    instance elementCommand (
     
    ...
     
    (component.sun-server "DiskSuite Tool..."
    "/usr/sbin/metatool")
     
    ...

    Add this entry to the "component" category.

  3. Restart SunNet Manager with the -i option.

    The -i option reads the modified elements.schema file and reinitializes the SunNet Manager database.

How to Launch DiskSuite Tool From SunNet Manager (SunNet Manager)

Use this task to launch DiskSuite Tool from SunNet Manger. This task assumes you have followed the steps in the above task, "How to Enable SunNet Manager to Launch DiskSuite Tool (SunNet Manager)".

  1. Select a host object (that has DiskSuite installed) within the SunNet Manger canvas.

  2. Display the host's pop-up menu and choose Tools.

  3. From the Tools menu, choose DiskSuite Tool.

    DiskSuite Tool is displayed for the selected host.

Integrating SNMP Alerts With DiskSuite

DiskSuite can send SNMP trap data (alerts) to any network management console capable of receiving SNMP messages. This is accomplished through a separate DiskSuite daemon, mdlogd, that you choose to install. Through a configuration file, you specify the trap variables as well as the kinds of messages to look for.

To use the DiskSuite SNMP daemon, you must install the optional SUNWmdnr and SUNWmdnupackages. See Solstice DiskSuite 4.2.1 Installation and Product Notes.

How to Configure DiskSuite SNMP Support (Command Line)

This task assumes you have installed the DiskSuite SNMP packages, SUNWmdnr and SUNWmdnu.

The steps to configure DiskSuite for SNMP support are:

For more information, refer to the mdlogd(4) man page.

When you reboot the system after installing the SUNWmdnr and SUNWmdnu packages, the following message appears:


Starting mdlogd ...
 
        /etc/lvm/mdlogd.cf: no configuration information

DiskSuite displays this message because you have not yet configured the mdlogd.cf file.

  1. Make sure that the mdlogd daemon is configured to load automatically at boot.

    (A postinstall script should have taken care of this when you added the package.)

  2. Edit and save the SNMP trap configuration file, mdlogd.cf.

    1. Change the line ENTERPRISE = to the SNMP identifier for the enterprise to which the host running the daemon belongs.

    2. Change the line OBJECTID = to the SNMP identifier of the host running the daemon.

    3. Configure the SubStrings the daemon will look for. These are messages generated by DiskSuite.

    4. Change the trap destination tuple to the name of the host to receive the SNMP trap; the port; and the protocol.

      Most likely you will always use port 162 and protocol udp, as these are the defaults for SNMP traps.

    5. Leave the Generic SNMP Trap # set to 6.

    6. You can use your own coding scheme for Specific Trap #.

      For example, use number 1 for a low priority message, number 2 for medium priority, and 3 for high priority.

Example -- mdlogd.cf File


##ident "@(#)mdlogd.cf 1.1 96/02/15"
# DiskSuite SNMP Trap configuration file.
...
ENTERPRISE = 1.3.6.1.4.1.42
OBJECTID = 1.3.6.1.4.1.860
#
# SubString	Trap Destination      SNMP Trap #   Specific Trap #
#              (host:port:protocol)   0 < n <= 6       0 < n
 
"NOTICE: md:"  "spin:162:udp"            6             1
"WARNING: md:" "spin:162:udp"            6             2

This example dispatches SNMP traps for DiskSuite errors written to /dev/console to a host named spin.

Here's what the trap generated by the error message "WARNING: md: d6: /dev/dsk/c3t3d0s7 needs maintenance" would look like when received by a SunNet Manager console:


Wed Feb 21 15:40:41 1996 [ spin ] : Trap:
 
    sequence=2
    receive-time=Wed Feb 21 15:40:41 1996
    version=0
    community=public
    enterprise=Sun Microsystems
    source-time=00:00:00.00
    trap-type=enterprise specific trap: 2
 
    1.3.6.1.4.1.860 = Feb 21 15:40:41 1996 spin WARNING:
            md: d6: /dev/dsk/c3t3d0s7 needs maintenance

Integrating Storage Manager With DiskSuite

DiskSuite and Storage Manager can be configured for cooperative ease of use. The Storage Manager application contains two tools, Disk Manager and File System Manager, that enable you to manage a server's disk configurations and file systems. For example, you could create a metadevice in DiskSuite Tool, then drag it to File System Manager to create a UFS file system on the metadevice.

You can configure DiskSuite so that it can launch Storage Manager from the DiskSuite Tool Tools menu.

For more information on Storage Manager, refer to Appendix A, Using Storage Manager.

How to Enable DiskSuite to Launch Storage Manager (Command Line)

This task assumes you have installed the Storage Manager packages. Refer to Solstice DiskSuite 4.2.1 Installation and Product Notes for more information.

Enabling DiskSuite to launch Storage Manager's tools, involves configuring the /usr/lib/lvm/metatool-toolsmenu file. For more information, refer to the metatool-toolsmenu(4) man page.

The supplied metatool-toolsmenu file has two lines already configured to use File System Manager and Disk Manager. Uncomment (remove the leading # sign) the two lines and save the file.

Example -- metatool-toolsmenu File

Here is a sample metatool-toolsmenu file, with two entries that enable File System Manager and Disk Manager to be launched from DiskSuite Tool.


#
# metatool `Tools' menu registry file...
#
# Entries are of the form:
#
# :0:<name>:<command>:
#
# 0 is a format specifier.  `0' is the only valid specifier for metatool.
#
# <name> becomes a menu item1.5 in the Metadevice Editor and DiskView Tools menu
# <command> is passed to system() when item is chosen from the menu.
#
# ':' is a field delimiter, one of: `+' `|' `:' `^'
#
# sample entries:
	+0+File System Manager...+/opt/SUNWadm/2.2/bin/stomgr -F+
	:0:Disk Manager...:/opt/SUNWadm/2.2/bin/stomgr -D:
#
#

How to Launch File System Manager and Disk Manager (DiskSuite Tool)

This task assumes you have configured the metatool-toolsmenu file, as explained in the previous task.

  1. Start DiskSuite Tool.


    # metatool &
    

    DiskSuite Tool's menu options now includes a "Tools" item, with File System Manager and Disk Manager selections.

  2. Select the appropriate item from the Tools menu to start either File System Manager or Disk Manager.