Use a supported Oracle SPARC T-series server, M-series, or S-series server. See Supported Platforms in the Oracle VM Server for SPARC Installation Guide. The SPARC server must have at least two PCIe buses, so that you can configure a root domain in addition to the primary domain. For more information, see I/O Domain Overview in the Oracle VM Server for SPARC Administration Guide.

Both domains must be configured with at least one PCIe bus. The PCIe buses that you assign to each domain must be unique. You cannot assign the same PCIe bus to two different domains.

By default, after a fresh installation, all PCIe buses are assigned to the primary domain. When adding a new service domain, some of these PCIe buses must be released from the primary domain and then assigned to the secondary domain.

For example, a SPARC T5-2 server with two SPARC T5 processors has 4 PCIe buses. This server can be configured with a primary domain and a secondary domain. You can assign two PCIe buses to the primary domain, and two PCIe buses to the secondary domain.

The network ports used by the primary domain must all be connected to the PCIe buses that are assigned to the primary domain.

Similarly the network ports used by the secondary domain must all be connected to the PCIe buses that are assigned to the secondary domain.

In addition, the primary and secondary domains must have the same number of network ports. Each network port in the primary domain must have a corresponding network port in the secondary domain, and they must be connected to the same physical network.

For example, a SPARC T5-2 server with two SPARC T5 processors has 4 PCIe buses (pci_0, pci_1, pci_2, and pci_3). The server also has 4 onboard network ports. Two network ports are connected to pci_0, and the other two are connected to pci_3. You can assign 2 PCIe buses (pci_0 and pci_1) to the primary domain, and 2 PCIe buses (pci_2 and pci_3) to the secondary domain. That way, both domains have two ports configured. You must ensure that each port is connected to the same physical network as the port in the corresponding domain.

Physical disks or LUNs used by the primary domain must all be accessible through one or several host bus adapters (HBAs) connected to the PCIe buses that are assigned to the primary domain. The primary domain requires at least one disk for booting and hosting the operating system. The primary domain usually has access to all, or a subset of, local SAS disks present on the server through an onboard SAS HBA connected to one of the PCIe buses of the server.

Similarly, physical disks or LUNs used by the secondary domain must all be accessible through one or several HBAs connected to the PCIe buses assigned to the secondary domain. The secondary domain needs at least one disk for booting and hosting the operating system. Depending on the server used, the secondary domain might not have access to any local SAS disks present on the server, or it might have access to a subset of the local SAS disks. If the secondary domain does not have access to any of the local SAS disks then it must have an HBA card on one of its PCIe buses and access to an external storage array LUN that it can use for booting.

In addition, if a Fibre Channel (FC) storage area network (SAN) is used, then the primary and the secondary domains must have access to the same FC disks. So one or more FC HBAs must be connected to the FC SAN and to the PCIe buses that are assigned to the primary domain. And, one or more FC HBAs must be connected to the FC SAN and to the PCIe buses that are assigned to the secondary domain.

For example, a SPARC T5-2 server with two SPARC T5 processors has 4 PCIe buses (pci_0, pci_1, pci_2, pci_3). The server also has 2 onboard SAS HBAs to access the 6 internal SAS disks. One SAS HBA is connected to PCIe bus pci_0 and accesses 4 internal disks. The other SAS HBA is connected to PCIe bus pci_4 and accesses the 2 other internal SAS disks. You can assign 2 PCIe buses (pci_0 and pci_1) to the primary domain, and 2 PCIe buses (pci_2 and pci_3) to the secondary domain. That way, both domains have access to internal SAS disks that can be used for booting. The primary domain has access to four SAS disks, and the secondary domain has access to two SAS disks.

If you want to connect the server to an FC SAN, then you can add an FC HBA to the primary domain (for example on PCIe bus pci_1) and an FC HBA to the secondary domain (for example, on PCIe bus pci_2). Then you should connect both FC HBAs to the same SAN.

The list of all PCIe buses present on the server is displayed, with information indicating whether or not they are available for creating a secondary service domain. Example output from the ovs-agent-secondary command, for this step, is displayed below:

Gathering information about the server...
The server has 2 PCIe buses.
----------------------------------------------------------------------
This is the list of PCIe buses present on the server, and whether
or not they are available for creating a secondary service domain
  Bus         Available   Reason
  ---         ---------   ------
  pci_0              no   Bus is assigned and used by the primary domain
  pci_1             yes   Bus is assigned to the primary domain but it is not used
Enter + or - to show or hide details about PCIe buses.
  +) Show devices in use
Or select one of the following options.
  0) Exit and do not create a secondary service domain
  1) Continue and select PCIe buses to create a secondary service domain
 
Choice (0-1): 1

Use this information to figure out which PCIe buses are available, and which buses you want to use for the secondary service domain. You can display more or less information about the PCIe buses by entering "+" or "-".

A PCIe bus is not available for creating a secondary service in the following cases:

The next step provided by the ovs-agent-secondary command allows you to actually select the PCIe buses that are to be used for the secondary service domain. Typically, this step may appear as follows:

The following PCIe buses can be selected for creating a secondary
service domain.
  Bus        Selected   Slot                 Devices Count
  ---        --------   ----                 -------------
  pci_1            no                        
                        /SYS/MB/PCIE5        
                        /SYS/MB/PCIE6        
                        /SYS/MB/PCIE7        ETH(2)
                        /SYS/MB/PCIE8        FC(2)
                        /SYS/MB/SASHBA1      DSK(2)
                        /SYS/MB/NET2         ETH(2)
Enter + or - to show or hide details about PCIe buses.
  +) Show devices
  -) Hide PCIe slots
Or enter the name of one or more buses that you want to add to the
selection of PCIe buses to create a secondary service domain.
Or select one of the following option.
  0) Exit and do not create a secondary service domain
  1) Add all PCIe buses to the selection
  2) Remove all PCIe buses from the selection
Choice (0-2): pci_1
adding bus pci_1 to selection

Note that in addition to the menu options, which allow you to add all available PCIe buses to the secondary service domain, you can also manually specify a space separated list of PCIe buses by bus name to individually add particular buses to the secondary service domain.

As soon as at least one PCIe bus is marked as selected, the menu options change to allow you to create the secondary service domain with the selected PCIe buses:

The following PCIe buses can be selected for creating a secondary
service domain.
  Bus        Selected   Slot                 Devices Count
  ---        --------   ----                 -------------
  pci_1           yes                        
                        /SYS/MB/PCIE5        
                        /SYS/MB/PCIE6        
                        /SYS/MB/PCIE7        ETH(2)
                        /SYS/MB/PCIE8        FC(2)
                        /SYS/MB/SASHBA1      DSK(2)
                        /SYS/MB/NET2         ETH(2)
Enter + or - to show or hide details about PCIe buses.
  +) Show devices
  -) Hide PCIe slots
Or enter the name of one or more buses that you want to add to the
selection of PCIe buses to create a secondary service domain.
Or select one of the following option.
  0) Exit and do not create a secondary service domain
  1) Add all PCIe buses to the selection
  2) Remove all PCIe buses from the selection
  3) Create a secondary services domain with the selected buses
 
Choice (0-3): 3

A final confirmation screen displays the buses selected for the secondary service domain, before you can proceed to create the secondary service domain. This confirmation screen looks as follows:

You have selected the following buses and devices for the secondary
domain.
  Bus        Current Domain       Slot                 Devices Count
  ---        --------------       ----                 -------------
  pci_1      primary                                   
                                  /SYS/MB/PCIE5        
                                  /SYS/MB/PCIE6        
                                  /SYS/MB/PCIE7        ETH(2)
                                  /SYS/MB/PCIE8        FC(2)
                                  /SYS/MB/SASHBA1      DSK(2)
                                  /SYS/MB/NET2         ETH(2)
Verify that the selection is correct.
  0) Exit and do not create a secondary service domain
  1) The selection is correct, create a secondary domain with pci_1
  2) Go back to selection menu and change the selection
Choice (0-2): 1

After the selection of PCIe buses for the secondary service domain has been confirmed, the secondary domain is created and instructions for configuring the secondary service domain are displayed. The output from the tool looks similar to the following:

ldm add-domain secondary
ldm set-core 1 secondary
ldm set-memory 8G secondary
ldm add-vds secondary-vds0 secondary
ldm add-io pci_1 secondary
ldm start-reconf primary
ldm remove-io pci_1 primary
 
----------------------------------------------------------------------
 
The secondary service domain has been created. Next, you need to
install Solaris on that domain. Then you can configure the Oracle
VM Agent to run with the secondary domain.

Once the secondary service domain is up and running with Solaris,
run the following command to configure the Oracle VM Agent to run
with the secondary domain:

  # ovs-agent-secondary configure

If a reboot is required to complete the creation of the secondary service domain then a corresponding menu is displayed, otherwise the tool terminates and the creation of secondary service domain is already finished. The following menu is displayed if a reboot is required:

To complete the configuration of the Oracle VM Agent, the
system has to be rebooted.

Do you want to reboot the system now?
  1) Yes, reboot the system now
  2) No, I will reboot the system later
 
Choice (1-2): 1

Server Reboot

!!! WARNING !!!
You are not connected to the system console. Rebooting
the server will close this connection with the server.

!!! WARNING !!!
Are you sure that you want to continue?
  1) Yes, continue and reboot the system now
  2) No, cancel the reboot, I will reboot the system later

Choice (1-2): 1

Rebooting the system...

When you have finished creating the new service domain, you need to install it. Complete the instructions in Section 1.7.2.4, “Installing the Secondary Service Domain”.

Once the secondary service domain is correctly installed, you must configure the Oracle VM Agent to use it by running the ovs-agent-secondary command on the control domain, as follows:

# ovs-agent-secondary configure

The first step in the configuration process requires you to confirm that the secondary domain is installed and running. This step is displayed as follows:

The secondary service domain exists and is active. It should be up
and running Solaris 11.3.
 
Confirm that the secondary service domain is up and running Solaris 11.3
 
  1) Yes, the secondary service domain is up and running Solaris 11.3.
  2) No, the secondary service domain is not running Solaris 11.3
 
Choice (1-2): 1

The configuration process notifies you if virtual switches are defined.

The secondary domain can only be configured when no virtual
switches are defined. Remove any virtual switch, and restart
the configuration.

The following virtual switches are defined: 0a010000

You must remove any virtual switches defined in the secondary service domain before you can configure it, as in the following example:

# ldm list-services
VCC
    NAME             LDOM             PORT-RANGE
    primary-vcc0     primary          5000-5127
VSW
    NAME             LDOM             MAC               NET-DEV   ID   DEVICE
    0a010000         primary          00:14:4f:fb:53:0e net0      0    switch@0

    LINKPROP   DEFAULT-VLAN-ID   PVID VID   MTU   MODE   INTER-VNET-LINK
               1                 1          1500         on
VDS
    NAME             LDOM         VOLUME     OPTIONS      MPGROUP    DEVICE
    primary-vds0     primary        
VDS
    NAME             LDOM         VOLUME     OPTIONS      MPGROUP    DEVICE
    secondary-vds0   secondary

# ldm remove-vsw 0a010000

Restart the configuration of the secondary service domain after you remove the virtual switch.

# ovs-agent-secondary configure
The secondary service domain exists and is active. It should be up
and running Solaris 11.3.
Confirm that the secondary service domain is up and running Solaris 11.3
  1) Yes, the secondary service domain is up and running Solaris 11.3.
  2) No, the secondary service domain is not running Solaris 11.3
Choice (1-2): 1

Each network link in the primary domain should have a corresponding network link in the secondary domain connected to the same physical network. By default, a network link in the primary domain is associated with the network link with the same name in the secondary domain. If a network link in the primary domain should be associated with a network link in the secondary domain with a different name, then you need to define a network link mapping. This is achieved in the next step of the configuration process, which is displayed as follows:

Each network link in the primary domain should have a corresponding
network link in the secondary domain connected to the same physical
network. By default, a network link in the primary domain will be
associated with the network link with the same name in the secondary
domain.
 
Network links in the primary domain and corresponding link in the
secondary domain:
 
    Primary    Secondary
    -------    ---------
    net0       net0
    net1       net1
    net4       net4
    net5       net5
    net6       net6
    net7       net7
 
If a network link in the primary domain should be associated with
a network link in the secondary domain with a different name, then
you need to define a network link mapping.
 
Do you need to define a network link mapping?
 
  1) Yes, I need to map a network link in the primary domain to
     a network link in the secondary domain with a different name.
  2) No, each network link in the primary domain has a corresponding
     network link in the secondary domain with the same name.
 
Choice (1-2): 1

Ideally, you should be able to select option 2 here to continue. However, it is possible that network link names may not correspond correctly. In this case, you should select option 1 and redefine the mapping as follows:

Enter the mapping for net0 [net0]:
Enter the mapping for net1 [net1]:
Enter the mapping for net4 [net4]: net2
Enter the mapping for net5 [net5]: net3
Enter the mapping for net6 [net6]:
Enter the mapping for net7 [net7]:
 
Network links in the primary domain and corresponding link in the
secondary domain:
 
    Primary    Secondary
    -------    ---------
    net0       net0
    net1       net1
    net4       net2
    net5       net3
    net6       net6
    net7       net7
 
Is the mapping correct?
 
  1) Yes, the mapping is correct.
  2) No, the mapping is not correct, redo the mapping.
 
Choice (1-2): 1

Note that you are prompted for the mapping for each network link in the primary domain. If you enter a blank line, the existing default mapping is used. If you need to change a mapping, you must specify the network link name in the secondary domain that is connected to the same physical network as the network link listed in the primary domain.

When you have finished redefining the mappings, you should select option 1 to continue to the next step in the configuration process.

Each Fibre Channel (FC) disk accessible from the primary domain should also be accessible from the secondary domain. By default, a FC disk is accessed using the same device path in the primary domain and in the secondary domain. In particular, each disk is accessed using the same disk controller name. If a disk controller in the primary domain should be associated with a disk controller in the secondary domain with a different name, then you must define a disk controller mapping.

It is recommended that Solaris I/O multipathing is enabled in the primary and in the secondary domain on all multipath-capable controller ports, in particular on all FC ports. In this case, all FC disks appear under a single disk controller (usually c0), and disk controller mapping is usually not needed.

The following screen is displayed for this step in the configuration process:

Each Fibre Channel (FC) disk accessible from the primary domain
domain should also be accessible from the secondary domain. By
default, a FC disk will be access using the same device path in
the primary domain and in the secondary domain. In particular,
each disk will be accessed using the same disk controller name.
 
FC disk controllers in the primary domain and corresponding
controller in the secondary domain:
 
    Primary    Secondary
    -------    ---------
    c0         c0
 
If a disk controller in the primary domain should be associated with
a disk controller in the secondary domain with a different name, then
you need to define a disk controller mapping.
 
Do you need to define a disk controller mapping?
 
  1) Yes, I need to map a disk controller in the primary domain to
     a disk controller in the secondary domain with a different name.
  2) No, each disk controller in the primary domain has a corresponding
     disk controller in the secondary domain with the same name.
 
Choice (1-2): 1

Ideally, you should be able to select option 2 to continue. However, it is possible that disk controller names might not correspond correctly. In this case, you should select option 1 and redefine the mapping as follows:

Enter the mapping for c0 [c0]: c1
 
FC disk controllers in the primary domain and corresponding
controller in the secondary domain:
 
    Primary    Secondary
    -------    ---------
    c0         c1
 
Is the mapping correct?
 
  1) Yes, the mapping is correct.
  2) No, the mapping is not correct, redo the mapping.
 
Choice (1-2): 1

Note that you are prompted for the mapping for each FC disk controller in the primary domain. If you enter a blank line, the existing default mapping is used. If you need to change a mapping, you must specify the FC disk controller name in the secondary domain that is connected to the same FC disk listed in the primary domain.

When you have finished redefining the mappings, you should select option 1 to continue to the next step in the configuration process.

The Oracle VM Agent uses a configuration file to access and configure itself for resources in the secondary service domain. In this step of the configuration process, the configuration file is created and saved to disk within the primary control domain:

Creating configuration file
Saving configuration ovm-shadow on the service processor
 
The secondary service domain is configured. Continuing with
the configuration of the Oracle VM Agent.

This command can not be run while the ovs-agent is online.

Do you want to disable the ovs-agent service?
  1) Yes, disable the ovs-agent service
  2) No, exit the ovs-agent-setup tool

Choice (1-2): 1

Finally, the Oracle VM Agent is automatically reconfigured to use the secondary service domain and the Oracle VM Agent is enabled:

Network Configuration
Network Configuration OK
Storage Configuration
Storage Configuration OK
OVS Agent Configuration
OVS Agent Configuration OK
Cluster Configuration
Cluster Configuration OK
LDoms Manager Configuration
LDoms Manager Configuration OK
Virtual I/O Services Configuration
Virtual I/O Services Configuration OK
LDoms Configuration
LDoms Configuration OK
 
Enabling Oracle VM Agent Services

The configuration values that are used for this process map onto the values that you entered for the configuration steps when you first configured Oracle VM Agent for your primary control domain, as described in Configuring Oracle VM Agent for SPARC in the Oracle VM Installation and Upgrade Guide.

When the process is complete, the Oracle VM Agent is enabled and your environment is configured to use both a primary and a secondary service domain.