Go to main content

SPARC M8 and SPARC M7 Servers Service Manual

Exit Print View

Updated: September 2017
 
 

Prepare a PCIe Card For Removal

If the PCIe card is assigned to an I/O domain on a logical domain, use these steps to avoid a configuration that is unsupported by the SR-IOV feature.

If the PCIe card is not assigned to an I/O domain on a logical domain, go to Remove a PCIe Card Carrier and Card From the Server.

  1. Determine which PCIe card requires service.

    In this example, pci_45 represents the I/O domain name.


    Note -  The following example shows only an excerpt of the output.
    # ldm ls-io -l pci_45
    NAME                                       TYPE   BUS      DOMAIN    STATUS
    ----                                       ----   ---      ------    ------
    pci_45                                     BUS    pci_45   primary   IOV
    [pci@32d]
    /SYS/CMIOU9/PCIE2                          PCIE   pci_45   primary   OCC
    [pci@32d/pci@1]
        SUNW,qlc@0/fp/disk
        SUNW,qlc@0/fp@0,0
        SUNW,qlc@0,1/fp/disk
        SUNW,qlc@0,1/fp@0,0/ssd@w20350080e518b846,1f
        fibre-channel@0,2
        fibre-channel@0,3
        fibre-channel@0,4
        fibre-channel@0,5
    ...
    
    /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0            VF     pci_45   ldg4
    [pci@32d/pci@1/SUNW,qlc@0,2]
        Class properties [FIBRECHANNEL]
            port-wwn = 10:00:00:14:4f:f8:74:85
            node-wwn = 20:00:00:14:4f:f8:74:85
            bw-percent = 0
    ...
    
    /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3            VF     pci_45   ldg4
    [pci@32d/pci@1/SUNW,qlc@0,5]
        Class properties [FIBRECHANNEL]
            port-wwn = 10:00:00:14:4f:fb:27:29
            node-wwn = 20:00:00:14:4f:fb:27:29
            bw-percent = 0
    #
  2. Save the current I/O configuration.

    Before removing any virtual functions, save the original configuration in a file you can use to restore the configuration after you replace the card.

    # ldm ls-io slot-name>file-name

    where:

    • slot-name is a format similar to /SYS/CMIOU9/PCIE2

    • file-name is a format similar to /var/tmp/before_removal.txt

  3. Identify the names of the virtual functions you need to remove from the I/O domains.

    Display these details from the backup file you created in the previous step, for example:

    # cat /var/tmp/before_removal.txt
    NAME                                       TYPE   BUS      DOMAIN    STATUS
    ----                                       ----   ---      ------    ------
    /SYS/CMIOU9/PCIE2                          PCIE   pci_45   primary   OCC
    /SYS/CMIOU9/PCIE2/IOVFC.PF0                PF     pci_45   primary
    /SYS/CMIOU9/PCIE2/IOVFC.PF1                PF     pci_45   primary
    /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0            VF     pci_45   ldg4
    /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF1            VF     pci_45   ldg4
    /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF2            VF     pci_45   ldg4
    /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3            VF     pci_45   ldg4
  4. Remove all virtual functions of the card from all I/O domains listed by the cat command.

    Typically, a card has two or more physical functions, so you must ensure that all virtual functions from all physical functions are removed from the I/O domains. As shown in the preceding example, these virtual functions are indicated under the TYPE column as VF.


    Note -  You can remove virtual functions dynamically without shutting down the domains.
    # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0 ldg4
    # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF1 ldg4
    # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF2 ldg4
    # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3 ldg4
  5. Verify that the virtual functions have been removed from the I/O domains.
    # ldm ls-io /SYS/CMIOU9/PCIE2
    NAME                                       TYPE   BUS      DOMAIN    STATUS
    ----                                       ----   ---      ------    ------
    /SYS/CMIOU9/PCIE2                          PCIE   pci_45   primary   OCC
    /SYS/CMIOU9/PCIE2/IOVFC.PF0                PF     pci_45   primary
    /SYS/CMIOU9/PCIE2/IOVFC.PF1                PF     pci_45   primary
    /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0            VF     pci_45
    /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF1            VF     pci_45
    /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF2            VF     pci_45
    /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3            VF     pci_45

    In this example, the virtual functions listed under the TYPE column now display no domain name.

  6. If the previous command fails, the I/O domain software is keeping the virtual functions busy. In this case, stop the I/O domain and remove each virtual function.
    # ldm stop ldg4
    LDom ldg4 stopped
    
    # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0 ldg4
    # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF1 ldg4
    # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF2 ldg4
    # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3 ldg4 
  7. Back up the virtual function properties.

    If the replacement PCIe card is the same type and model as the original card, and if virtual functions were created from physical functions on that card, save the properties of any virtual function instance for the PCIe card. You can use the file you create to reproduce the original configuration on the new card.

    # ldm ls-io -l slot-name>file-name

    where:

    • slot-name is a format similar to /SYS/CMIOU9/PCIE2

    • file-name is a format similar to /var/tmp/vf_config.txt

  8. Verify that the backup file stored the correct configuration information.

    Display the details from the backup file you created in the previous step, for example:

    # cat /var/tmp/vf_config.txt
    NAME                                       TYPE   BUS      DOMAIN    STATUS
    ----                                       ----   ---      ------    ------
    pci_45                                     BUS    pci_45   primary   IOV
    [pci@32d]
    /SYS/CMIOU9/PCIE2                          PCIE   pci_45   primary   OCC
    [pci@32d/pci@1]
        SUNW,qlc@0/fp/disk
        SUNW,qlc@0/fp@0,0
        SUNW,qlc@0,1/fp/disk
        SUNW,qlc@0,1/fp@0,0/ssd@w20350080e518b846,1f
        fibre-channel@0,2
        fibre-channel@0,3
        fibre-channel@0,4
        fibre-channel@0,5
    ...
    
    /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0            VF     pci_45   ldg4
    [pci@32d/pci@1/SUNW,qlc@0,2]
        Class properties [FIBRECHANNEL]
            port-wwn = 10:00:00:14:4f:f8:74:85
            node-wwn = 20:00:00:14:4f:f8:74:85
            bw-percent = 0
    ...
    
    /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3            VF     pci_45   ldg4
    [pci@32d/pci@1/SUNW,qlc@0,5]
        Class properties [FIBRECHANNEL]
            port-wwn = 10:00:00:14:4f:fb:27:29
            node-wwn = 20:00:00:14:4f:fb:27:29
            bw-percent = 0
    #
  9. Destroy the virtual functions in all physical functions of the card.

    You can do this dynamically while the root domain is running.

    # ldm destroy-vf -n max physical-function-name

    where physical-function-name is a format similar to /SYS/CMIOU9/PCIE2/IOVFC.PF0.

  10. Verify that the virtual functions were removed from all physical functions of the card.

    Display the details for the slot, for example:

    # ldm ls-io -l /SYS/CMIOU9/PCIE2
    NAME                                       TYPE   BUS      DOMAIN    STATUS
    ----                                       ----   ---      ------    ------
    /SYS/CMIOU9/PCIE2                          PCIE   pci_45   primary   OCC
    [pci@32d/pci@1]
        SUNW,qlc@0/fp/disk
        SUNW,qlc@0/fp@0,0
        SUNW,qlc@0,1/fp/disk
        SUNW,qlc@0,1/fp@0,0/ssd@w20350080e518b846,1f
    /SYS/CMIOU9/PCIE2/IOVFC.PF0                PF     pci_45   primary
    [pci@32d/pci@1/SUNW,qlc@0]
        maxvfs = 16
    /SYS/CMIOU9/PCIE2/IOVFC.PF1                PF     pci_45   primary
    [pci@32d/pci@1/SUNW,qlc@0,1]
        maxvfs = 16
    #
  11. If the previous command fails, run the destroy command under delayed-reconfiguration, for example:
    # ldm start-reconf primary
    Initiating a delayed reconfiguration operation on the primary domain.
    All configuration changes for other domains are disabled until the primary
    domain reboots, at which time the new configuration for the primary domain
    will also take effect.
    # 
    # ldm destroy-vf -n max /SYS/CMIOU9/PCIE2/IOVFC.PF0
    ------------------------------------------------------------------------------
    Notice: The primary domain is in the process of a delayed reconfiguration.
    Any changes made to the primary domain will only take effect after it reboots.
    ------------------------------------------------------------------------------
    # shutdown -i6 -g0 -y

    Use the ldm stop -r command, instead of the shutdown command, to stop the other root domain.

  12. Physically remove the PCIe card from the server.

    See Remove a PCIe Card Carrier and Card From the Server.

    For more information about making PCIe hardware changes, refer to the Oracle VM Server for SPARC documentation at http://www.oracle.com/goto/vm-sparc/docs.

Related Information