JavaScript is required to for searching.
Skip Navigation Links
Exit Print View
Sun ZFS Storage 7000 System Administration Guide
search filter icon
search icon

Document Information

Preface

1.  Introduction

2.  Status

3.  Configuration

Configuration

Introduction

Initial

Initial Configuration

Prerequisites

Summary

BUI

Configuring Management Port

CLI

Performing Initial Configuration with the CLI

Network

Network Configuration

Devices

Datalinks

Interfaces

IP MultiPathing (IPMP)

Performance and Availability

Routing

Routing Entries

Routing Properties

BUI

Configuration

Addresses

Routing

CLI

Tasks

BUI

CLI

Infiniband Upgrade Procedures for Q3.2010

Storage

Introduction

Configure

Configuration Rules and Guidelines

Verification

Allocation on SAS-1 Systems

Allocation on SAS-2 Systems

Profile Configuration

Import

Add

Unconfig

Scrub

Tasks

BUI

SAN

SAN

Terminology

Targets and Initiators

Target and Initiator Groups

BUI

CLI

Terms

SAN Terminology

FC

Fibre Channel

Target Configuration

Clustering Considerations

Initiator Configuration

Switch Considerations

Clustering Considerations

Performance Considerations

Troubleshooting

Queue Overruns

Link-level Issues

BUI

Changing modes of FC ports

Viewing discovered FC ports

Creating FC Initiator Groups

Associating a LUN with an FC initiator group

CLI

Changing modes of FC ports

Viewing discovered FC ports

Creating FC Initiator Groups

Associating a LUN with an FC initiator group

Scripting Aliases for Initiators and Initiator Groups

Protocol Support

IP based Protocol Support

Initiator Support

Introduction

Supported InfiniBand Client-Side Data Protocols

Supported Fiber Channel Initiators

FCMPxIO

Configuring FC Client Multipathing

Configuring Solaris Initiators

Configuring Windows Initiators

Windows Tunables - Microsoft DSM Details

Configuring Linux Initiators

Configuring VMware ESX Initiators

Troubleshooting

See Also

iSCSI

Introduction

Target Configuration

Clustering Considerations

Initiator Configuration

Planning Client Configuration

Solaris iSCSI/iSER and MPxIO Considerations

Troubleshooting

Observing Performance

BUI

Creating an Analytics Worksheet

CLI

Adding an iSCSI target with an auto-generated IQN

Adding an iSCSI target with a specific IQN and RADIUS authentication

Adding an iSCSI initiator which uses CHAP authentication

Adding an iSCSI target group

Adding an iSCSI initiator group

SRP

Introduction

Target configuration

Clustering Considerations

Initiator configuration

Observing Performance

Multipathing Considerations

Linux with OFED SRP Initiator

OFED 1.5 Issues

VMWare 4.0

Path Selection Plugin (psp)

Storage Array Type Plugin (satp)

VMWare ESX 4.0 Issues

BUI

iSER Target Configuration

SRP Target Configuration

CLI

Users

Introduction

Roles

Authorizations

Properties

Users

Roles

BUI

CLI

Tasks

BUI

CLI

Generic

Preferences

Introduction

BUI

CLI

SSH Public Keys

Alerts

Introduction

Actions

Send Email

Send SNMP trap

Send Syslog Message

Resume/Suspend Dataset

Resume/Suspend Worksheet

Execute Workflow

Threshold Alerts

BUI

CLI

Tasks

BUI

Cluster

Clustering

Features and Benefits

Drawbacks

Terminology

Subsystem Design

Cluster Interconnect I/O

Resource Management Concepts

Takeover and Failback

Configuration Changes in a Clustered Environment

Clustering Considerations for Storage

Clustering Considerations for Networking

Clustering Considerations for Infiniband

Redundant Path Scenarios

Preventing 'Split-Brain' Conditions

Estimating and Reducing Takeover Impact

Setup Procedure

Node Cabling

JBOD Cabling

BUI

Unconfiguring Clustering

4.  Services

5.  Shares

6.  Analytics

7.  Application Integration

Glossary

Index

FC

Fibre Channel

Fibre Channel (FC) is a gigabit-speed networking technology used nearly exclusively as a transport for SCSI. FC is one of several block protocols supported by the appliance; to share LUNs via FC, the appliance must be equipped with one or more optional FC cards.

Target Configuration

By default, all FC ports are used by the appliance to connect to a tape SAN for purposes of backup; to enable sharing via FC, one or more FC ports must be configured to be in target mode. Configuring a port to be in target mode requires the appliance to be reset, but multiple ports may be configured to be in target mode simultaneously. The mode of a port will be preserved across any subsequent reboots or upgrades. You must have root permissions to change the mode of a port.

Each FC port is assigned a World Wide Name (WWN), and -- as with other block protocols -- FC targets may be grouped into target groups, allowing port bandwidth to be dedicated to specific LUNs or groups of LUNs. Once an FC port is configured as a target, the remotely discovered ports can be examined and verified.

Refer to the Implementing Fibre Channel SAN Boot with Oracle's Sun ZFS Storage Appliance whitepaper at http://www.oracle.com/technetwork/articles/servers-storage-admin/fbsanboot-365291.html for details on FC SAN boot solutions using the Sun ZFS Storage Appliance.

Clustering Considerations

In a cluster, initiators will have two paths (or sets of paths) to each LUN: one path (or set of paths) will be to the head that has imported the storage associated with the LUN; the other path (or set of paths) will be to that head's clustered peer. The first path (or set of paths) are active; the second path (or set of paths) are standby; in the event of a takeover, the active paths will become unavailable, and the standby paths will (after a short time) be transitioned to be active, after which I/O will continue. This approach to multipathing is known as asymmetric logical unit access (ALUA) and -- when coupled with an ALUA-aware initiator -- allows cluster takeover to be transparent to higher-level applications.

Initiator Configuration

Initiators are identified by their WWN, and as with other block protocols, aliases can be created for initiators. To aid in creating aliases for FC initiators, a WWN can be selected from the WWNs of discovered ports. Further, and as with other block protocols, initiators can be collected into groups. When a LUN is associated with a specific initiator group, the LUN will only be visible to initiators in the group. In most FC SANs, LUNs will always be associated with the initiator group that corresponds to the system(s) for which the LUN has been created.

Switch Considerations

In general, the FC target works with off-the-shelf FC switches. In particular, the following switches have been tested and are known to work:

Switch vendor
Model
Brocade
All models supporting speeds of 4 Gb/sec and/or 8 Gb/sec
Cisco
All models supporting speeds of 4 Gb/sec and/or 8 Gb/sec
QLogic
All models supporting speeds of 4 Gb/sec and/or 8 Gb/sec.
Clustering Considerations

As discussed in target clustering considerations, the appliance is an ALUA-compliant array. Properly configuring an FC initiator in an ALUA environment requires an ALUA-aware driver, and may require initiator-specific tuning. The Supported Fiber Channel Initiator Table lists the initiators that are currently qualified with ALUA.

Performance Considerations

FC performance can be observed via analytics, whereby one can breakdown operations or throughput by initiator, target, or LUN:

image:Image

For operations, one can also breakdown by offset, latency, size and SCSI command, allowing one to understand not just the what but the how and why of FC operations.

Troubleshooting

Queue Overruns

The appliance has been designed to utilize a global set of resources to service LUNs on each head. It is therefore not generally necessary to restrict queue depths on clients as the FC ports in the appliance can handle a large number of concurrent requests. Even so, there exists the remote possibility that these queues can be overrun, resulting in SCSI transport errors. Such queue overruns are often associated with one or more of the following:

While the possibility of queue overruns is remote, it can be eliminated entirely if one is willing to limit queue depth on a per-client basis. To determine a suitable queue depth limit, one should take the number of target ports multiplied by the maximum concurrent commands per port (2048) and divide the product by the number of LUNs provisioned. To accommodate degraded operating modes, one should sum the number of LUNs across cluster peers to determine the number of LUNs, but take as the number of target ports the minimum of the two cluster peers. For example, in an active-active 7410 dual headed cluster with one head having 2 FC ports and 100 LUNs and the other head having 4 FC ports and 28 LUNs, one should take the pessimal maximum queue depth to be two ports times 2048 commands divided by 100 LUNs plus 28 LUNs -- or 32 commands per LUN.

Tuning the maximum queue depth is initiator specific, but on Solaris, this is achieved by adjusting the global variable ssd_max_throttle.

Link-level Issues

To troubleshoot link-level issues such as broken or flakey optics or a poorly seated cable, look at the error statistics for each FC port: if any number is either significantly non-zero or increasing, that may be an indicator that link-level issues have been encountered, and that link-level diagnostics should be performed.

BUI

Changing modes of FC ports

To make use of FC ports, set them to Target mode on the Configuration > SAN screen of the BUI, using the drop-down menu shown in the screenshot below. You must have root permissions to perform this action. Note that in a cluster configuration, you will set ports to Target mode on each head node separately, see the clustering considerations section.

image:Image

After setting desired ports to Target, click the Apply button. A confirmation message will appear notifying you that the appliance will reboot immediately. Confirm that you want to reboot.

When the appliance boots, the active FC targets appear with the image:Image icon and, on mouse-over, the move image:Move icon appears.

Viewing discovered FC ports

Click the info image:Get info icon to view the Discovered Ports dialog where you can troubleshoot link errors. In the Discovered Ports dialog, click a WWN in the list to view associated link errors.

image:Image

Creating FC Initiator Groups

Create and manage initiator groups on the Initiators screen. Click the add image:Add item icon to view unaliased ports. Click a WWN in the list to add a meaningful alias in the Alias field.

image:Image

On the Initiators screen, drag initiators to the FC Initiator Groups list to create new groups or add to existing groups.

image:Image

Click the Apply button to commit the new Initiator Group. Now you can create a LUN that has exclusive access to the client initiator group.

Associating a LUN with an FC initiator group

To create the LUN, roll-over the initiator group and click the add LUN image:Image icon. The Create LUN dialog appears with the associated initiator group selected. Set the name and size and click Apply to add the LUN to the storage pool.

image:Image

CLI

Changing modes of FC ports

dory:configuration san targets fc> set targets="wwn.2101001B32A11639"
                      targets = wwn.2101001B32A11639 (uncommitted)
dory:configuration san targets fc> commit

Viewing discovered FC ports

dory:configuration san targets fc> show
Properties:
                      targets = wwn.2100001B32811639,wwn.2101001B32A12239
Targets:
NAME       MODE       WWN                     PORT                 SPEED
target-000 target     wwn.2100001B32811639    PCIe 5: Port 1       4 Gbit/s
target-001 initiator  wwn.2101001B32A11639    PCIe 5: Port 2       0 Gbit/s
target-002 initiator  wwn.2100001B32812239    PCIe 2: Port 1       0 Gbit/s
target-003 target     wwn.2101001B32A12239    PCIe 2: Port 2       0 Gbit/s
dory:configuration san targets fc> select target-000
dory:configuration san targets fc target-000> show
Properties:
                          wwn = wwn.2100001B32811639
                         port = PCIe 5: Port 1
                         mode = target
                        speed = 4 Gbit/s
             discovered_ports = 6
           link_failure_count = 0
           loss_of_sync_count = 0
         loss_of_signal_count = 0
         protocol_error_count = 0
        invalid_tx_word_count = 0
            invalid_crc_count = 0
Ports:
PORT      WWN                   ALIAS              MANUFACTURER
port-000  wwn.2100001B3281A339  longjaw-1          QLogic Corporation
port-001  wwn.2101001B32A1A339  longjaw-2          QLogic Corporation
port-002  wwn.2100001B3281AC39  thicktail-1        QLogic Corporation
port-003  wwn.2101001B32A1AC39  thicktail-2        QLogic Corporation
port-004  wwn.2100001B3281E339  <none>             QLogic Corporation
port-005  wwn.2101001B32A1E339  <none>             QLogic Corporation

Creating FC Initiator Groups

dory:configuration san initiators fc groups> create
dory:configuration san initiators fc group (uncommitted)> set name=lefteye
dory:configuration san initiators fc group (uncommitted)> 
    set initiators=wwn.2101001B32A1AC39,wwn.2100001B3281AC39
dory:configuration san initiators fc group (uncommitted)> commit
dory:configuration san initiators fc groups> list
GROUP     NAME
group-001 lefteye
         |
         +-> INITIATORS
             wwn.2101001B32A1AC39
             wwn.2100001B3281AC39

Associating a LUN with an FC initiator group

The following example demonstrates creating a LUN called lefty and associating it with the fera initiator group.

dory:shares default> lun lefty
dory:shares default/lefty (uncommitted)> set volsize=10
                      volsize = 10 (uncommitted)
dory:shares default/lefty (uncommitted)> set initiatorgroup=fera
               initiatorgroup = default (uncommitted)
dory:shares default/lefty (uncommitted)> commit

Scripting Aliases for Initiators and Initiator Groups

Refer to the CLI Usage and Simple CLI Scripting and Batching Commands sections for information about how to modify and use the following example script.

script
     /*
      * This script creates both aliases for initiators and initiator
      * groups, as specified by the below data structure.  In this
      * particular example, there are five initiator groups, each of
      * which is associated with a single host (thicktail, longjaw, etc.),
      * and each initiator group consists of two initiators, each of which
      * is associated with one of the two ports on the FC HBA.  (Note that
      * there is nothing in the code that uses this data structure that
      * assumes the number of initiators per group.)
      */
      groups = {
               thicktail: {
                      'thicktail-1': 'wwn.2100001b3281ac39',
                      'thicktail-2': 'wwn.2101001b32a1ac39'
               },
               longjaw: {
                       'longjaw-1': 'wwn.2100001b3281a339',
                       'longjaw-2': 'wwn.2101001b32a1a339'
               },
               tecopa: {
                       'tecopa-1': 'wwn.2100001b3281e339',
                       'tecopa-2': 'wwn.2101001b32a1e339'
               },
               spinedace: {
                       'spinedace-1': 'wwn.2100001b3281df39',
                       'spinedace-2': 'wwn.2101001b32a1df39'
               },
               fera: {
                       'fera-1': 'wwn.2100001b32817939',
                       'fera-2': 'wwn.2101001b32a17939'
               }
       };
       for (group in groups) {
               initiators = [];
               for (initiator in groups[group]) {
                       printf('Adding %s for %s ... ',
                           groups[group][initiator], initiator);
                           try {
                               run('select alias=' + initiator);
                               printf('(already exists)\n');
                               run('cd ..');
                           } catch (err) {
                               if (err.code != EAKSH_ENTITY_BADSELECT)
                                       throw err;
                               run('create');
                               set('alias', initiator);
                               set('initiator', groups[group][initiator]);
                               run('commit');
                               printf('done\n');
                       }
                       run('select alias=' + initiator);
                       initiators.push(get('initiator'));
                       run('cd ..');
               }
               printf('Creating group for %s ... ', group);
               run('groups');
               try {
                       run('select name=' + group);
                       printf('(already exists)\n');
                       run('cd ..');
               } catch (err) {
                       if (err.code != EAKSH_ENTITY_BADSELECT)
                               throw err;
                       run('create');
                       set('name', group);
                       run('set initiators=' + initiators);
                       run('commit');
                       printf('done\n');
               }
               run('cd ..');
       }