42 Dynamic Resource Provisioning with Oracle VM Server for SPARC
Integration for a PaaS Solution
In Oracle Enterprise Manager Ops Center, the IaaS cloud platform service is a virtual datacenter (vDC), a virtual environment supported by pooled servers, storage, and network resources. A cloud administrator sets up the vDC to run Oracle VM Servers for SPARC (logical domains). Enterprise Manager Cloud Control users are creating vServers in the logical domains automatically, as needed. Each vServer draws its requirements from the resources of the vDC.
This section explains how to create the vDC and to connect it to Enterprise Manager Cloud Control.
Prerequisites
-
Enterprise Manager 12c Cloud Control Release 5 with Virtual Service Adaptor configured to connect to Ops Center's Enterprise Controller. See Getting Started with Dynamic Resource Provisioning.
-
Enterprise Manager Ops Center Release 3 with managed storage and network assets. See Oracle Enterprise Manager Ops Center Installation for Oracle Solaris Operating System and Configure Reference guides for information about installing the product and discovering assets.
-
Oracle Enterprise Manager Ops Center Client Kit, which includes modifications to enable communication between the two software products. This kit is available from My Oracle Support to
https://support.oracle.com/
.
The minimum requirement for Ops Center is the co-located configuration: one system hosts the Enterprise Controller, one Proxy Controller, and an embedded Oracle database.
-
Requirements for Enterprise Controller:
-
A minimum of T4-1 Sun Server
-
Same or higher version of Oracle Solaris 11 as the version to be provisioned on vServers running Oracle VM Server for SPARC. Supported versions are Oracle Solaris 11, 11.1, and 11.2 but, for optimal performance use the latest SRU of Oracle Solaris 11.2.
-
-
Requirements for Proxy Controller:
-
Co-located with Enterprise Controller or running in a system with the same minimum requirements for server, network, and operating system as for the Enterprise Controller.
-
-
Requirements for Oracle VM Server for SPARC control domain:
-
Oracle VM Server for SPARC 3.1 or later
-
Oracle Solaris 11, 11.1, and 11.2. The version must be the same or later than the version to run in the vServer. To provide different versions of Oracle Solaris 11, create a control domain for each operating system.
-
For optimal performance between storage elements and control domains, equip each server that hosts a control domain with a Fibre Channel HBA of at least 4 Gb and a 10 Gb Ethernet card. A 1 Gb Ethernet card is the minimum.
-
The right size of networks and virtual assets is not uniform. This document makes recommendations but your purposes and site requirements affect them. For a discussion of sizing, see the Oracle blog for some good practices in tuning resources for virtual assets at
https://blogs.oracle.com/jsavit/entry/best_practices_top_ten_tuning
-
-
Requirements for Oracle VMS Server for SPARC logical domain:
-
Oracle VM Server for SPARC 3.1 or later
-
The version of Oracle Solaris that is available in the Ops Center OS image repository.
-
-
Requirements for storage resources:
-
Dedicated LUNs exposed only to appropriate WWNs. The LUNs must be fully-provisioned, not thin-provisioned. You can use Fibre Channel LUNs or iSCSI LUNs or both.
-
For Fibre Channel LUNs, create at least one SAN static storage library
-
For iSCSI LUNs, a SAN dynamic storage library is created when you discover the storage server; however, you cannot use this library for the vDC. Create at least one static SAN storage library. Use a 10 Gb Ethernet network connection and tune the number of iSCSI connections. For 8 threads/connection:
iscsiadm modify initiator-node -c 8
-
For optimal performance between storage elements and control domains, equip each server that hosts a control domain with a Fibre Channel HBA of at least 4 Gb and a 10 Gb Ethernet card. A 1 Gb Ethernet card is the minimum.
-
-
Requirements for networks:
-
A fabric backed by an isolated switch to provide separate interfaces to each control domain. an isolated switch is an isolated Ethernet switch or a set of non-routable VLANs spanning multiple network switches. The switch provides the VLAN IDs that provision virtual networks.
-
For host-managed or fully-managed fabric, you must reserve a range of VLAN IDs to be mapped to the control domains. When a private network is needed, one is created on demand, using one of the VLAN IDs, and deleted when no longer needed.
-
For an unmanaged fabric, you must create the private networks manually, using the Create Private Networks action and add them to the network domain. When a private network is needed, an available network is assigned.
-
-
A dedicated network domain. Do not use the Default network domain.
-
At least one public network to provision OS and to access vServers. Public networks supply a block of IP addresses for provisioning the vServers. The number of IP addresses equals the number of vServers that can be supported. Consider the size of the network that can accommodate all your current and future requirements. Class C (for example, 10.0.0.1/24) networks enable you to create 256 vServers, Class 16C networks (for example, 10.0.0.2/20) enable you to create 4096 vServers.
-
Private networks for Clusterware and internal host connectivity.
-
-
A set of dedicated VLANs IDs, configured on one or more network interfaces.
-
-
Requirement for software libraries:
-
Place the libraries on a storage server with high transfer rates.
-
To provision an OS version, the Oracle Solaris 11 Software Update Library must contain the packages for that version. This repository is created during the product installation. In addition, you must download the Oracle Solaris 11 SRU library.
-
Limit the Oracle Solaris 11 Software Library to versions of releases and SRUs that are in active use.
-
Add packages from client kit to the Oracle Solaris 11 Software Library.
-
Overview of the Configuration of PaaS Solution
To set up the PaaS solution, you perform tasks on each component of the solution:
Configure Enterprise Manager Cloud Control
The Enterprise Manager's Virtual Service Adapter is the interface between Enterprise Manager and Enterprise Manager Ops Center. To configure the integration, the main component of Ops Center, the Enterprise Controller, is added as a staging server, as described in Adding the Stage Server.
When the configuration of Oracle Enterprise Manager Ops Center is complete, use the following information to configure Enterprise Manager Cloud Control:
-
IP address of the Enterprise Controller
-
username and password of the cloud user account
Obtain the Client Kit
Go to https://support.oracle.com/
to obtain the Ops Center Client Kit, p21190506_121050_Generic.zip
. The kit is also available in the OC DVD bundle.
The kit contains packages that provision logical domains, modified to enable PaaS operations, and a script:
-
ORCLsysman-iaas-cli.pkg
-
ORCLsysman-paas-cli.pkg
-
post-script-paas-12.3.0.<buildnumber>.p5p
Extract the files in the compressed file to a location of your choice. These packages and script will be uploaded to Ops Center's repository for Oracle Solaris 11 and used in an OS Update Profile.
Configure Enterprise Manager Ops Center
The tasks in this section take 2-3 hours to complete. Some of them can be done simultaneously, that is, you do not have to wait for one job to complete before starting another one.
-
Add Client Kit Contents to Software Library for Oracle Solaris 11
-
Preparing Network Resources for the vDC: fabric, network domain, and public network
-
Preparing Storage Resources: a NAS storage library and a SAN storage library
Configure Resources for vDC
A vDC relies on pools of networks, storage, and control domains. It provisions its servers from images in a current OS repository according to profiles in the repository.
Add Client Kit Contents to Software Library for Oracle Solaris 11
The package that enables the PaaS solution is post-script-paas-12.3.0.1407.p5p
. This file is in the compressed file you downloaded from My Oracle Support and resides in the directory where you extracted the contents of the compressed file. This procedure imports the file into Op Center's Oracle Solaris 11 repository so that it can enable communication between the Enterprise Manager Cloud Control staging server and the Ops Center resources.
Use the Add Content action to add ISO image files to the Oracle Solaris 11 Software Update Library manually.
When the job is completed, the file is listed in the Contents tab of the library.
Preparing Network Resources for the vDC
Fabrics are the physical infrastructure, consisting of network interface cards and network switches. Network domains are the logical infrastructure, combining network resources from at least one fabric. The networks used by virtual assets are created from the VLAN IDs managed by a network domain. The physical fabrics support these virtual networks, through the network domain, in a manner that depends on their type. To supply network resources to a vDC, you create a network domain from an appropriate fabric.
You must have the role of Network Admin to create network domains.
Identify a Fabric
To see the available fabrics, expand Networks in the Assets pane. Select Fabrics. For each fabric, the Dashboard in the center pane includes a Management Capability field. You have these options for providing physical network resources to a new network domain:
-
Use an existing host-managed fabric.
-
Convert an unmanaged fabric to host-managed and then use it.
-
Create a new host-managed fabric.
-
Use an unmanaged fabric.
To convert an unmanaged fabric to a host-managed fabric, use the Assign VLAN ID Range action. Specify a range that matches VLANs on the isolated switch that supports the fabric. If the fabric is not supported by an isolated switch, you can use an arbitrary VLAN ID range representing configured, non-routed VLANS, for example, 1001-1200.
To create a new host-managed fabric:
- In the Navigation pane, navigate to Networks, then select Fabrics from the drop-down list.
- In the Actions pane, click Define Ethernet Fabric.
- In the wizard, enter a name in the Fabric Name field. Enter a description. Click Next
- Enter the range of VLAN IDs. If the fabric is supported by an isolated switch, specify a range that matches the switch's VLANs. If the fabric is not supported by an isolated switch, use an arbitrary VLAN ID range, for example, 1001-1200.
- Select the networks to be associated with the fabric, then click Next.
- Review the Summary, then click Finish. The new fabric is host-managed
To use an unmanaged fabric, assign a set of its public networks as private networks or convert the fabric to an host-managed fabric.
Create a Network Domain
After Ops Center manages the physical fabric, its network resources are in the system-defined Default Network Domain, but you cannot use the Default network domain for a virtual datacenter. Instead, create a network domain specifically to support a server pool that contains a vDC.
A network domain is a container for networks and handles the relationship between the physical fabrics and the server pool. The fabrics provide network resources such as links and IP subnets. Within the network domain, networks that have been discovered or specified are available for assignment. These are called public networks. Their network resources have been defined. Another type of network is dynamic, that is, the network is created when it is needed, using an IP address space available to the network domain. Because they exist only for a specific purpose and only within a specific network domain, they are called private networks.
To use virtual datacenters, the network domain must provide private networks. To use existing public networks for a virtual datacenter, add each network to the network domain, making it a static private network, because existing public networks already have an IP subnet addresses and VLAN IDs.
Note:
Do not attempt to use the Default Network Domain to support a virtual datacenter. Although you are not prevented from specifying the Default Network Domain, its networks will not be available for selection.
Although the network domain is providing the networks to the server pools, this is only a logical association using VLAN IDs. To use the network resources, each physical server that supports the control domains in the server pool must have a physical connection to a fabric in the network domain.
Preparing Storage Resources
When Ops Center discovers a storage device, it creates a dynamic storage library, but these libraries cannot be used to support a vDC. Instead, create static storage libraries:
-
Create a NAS storage library to store the metadata of the control domains. This storage is exclusive for the vDC. Do not use this library for any other purpose.
-
Create at least one SAN library to store the operational data.
The storage libraries are block storage, which use Logical Unit Numbers (LUNs). The LUNs are backed by Fibre Channel disks or iSCSI disks in the form of LUNs.
SAN LUNs are associated with Fibre Channel or iSCSI target groups. Fibre Channel targets use a dedicated optical network and iSCSI targets use the IP network, but in both protocols, the targets in a target group expose one or more LUNs as a storage resource for the target's initiators.
For both protocols, acts as the initiator. To allow the target group and to identify each other as eligible initiators and targets, each one's Fibre Channel World Wide Number (WWN) or iSCSI IQN must be registered with the other one. recognizes the targets because the WWNs or IQNs of the storage server are recorded when the storage server is discovered. Any LUNs that have been assigned to that WWN or IQN are eligible to be used in a storage library.
Create a NAS Storage Library
These are the tasks for setting up a NAS storage library:
Identify the NAS Share
The NFS share is a file system on a NAS storage device. You can view the shares on managed storage servers from the user interface.
Create a NAS Storage Library
-
Expand Libraries in the Navigation pane. The new library will be created in the Filesystem Storage section.
-
Click New NAS Software Library in the Actions pane.
-
Enter a name for the library and a description. For example, identify how the new library will be used.
-
Do not associate a server pool with the new storage library. The server pool will be created later. Click Next.
-
By default, the wizard displays the option for using an exported share of a storage device. Because this procedure has set up an NFS share, click the Other option.
-
Enter the URL or IP address for the NFS server.
-
(Optional) You can specify the version of NFS that this storage uses. By default, uses the operating system's default NFS version. To specify a different version, allow the version to be changed.
-
Click the Override NFS Version option.
-
Click the drop-down list of NFS versions. Select a version.
-
-
Click the Next button to review a summary of the storage library.
-
Click the Finish button to submit the job.
When the job is completed, you can see the new storage library in the Libraries section of the Navigation pane.
Create SAN Storage Libraries using Fibre Channel LUNs
. This section describes how to set up a static block storage library, backed by Fibre Channel LUNs. The number of LUNs in the storage library determines the number of virtual hosts that the library can support.
What You Will Need
You will need the following to set up and use a block storage library:
-
LUNs backed by a managed SAN server using the Fibre Channel protocol
-
The role of Storage Admin
To create LUNs on the storage server, you must have the user account and password to log into the storage server and create the LUNs as targets.
Configure the Storage Server
If you need to create LUNs and make them available to , see the storage server's documentation for instructions in how to perform the following:
-
Configure the initiator and the targets. The initiator () must be able to recognize the targets (LUNs) and the targets must be able to recognize the initiator. recognizes the targets because the WWNs of the storage server are recorded when the storage server is discovered. Any LUNs that have been assigned to that WWN are eligible to be used in a storage library. On the storage server, you must specify 's WWN as an initiator and assign LUNs to that initiator.
-
Enable multipathing on the Fibre Channel ports. Multipathing is enabled by default on Oracle Solaris x86-based systems, but is disabled by default on Oracle Solaris SPARC-based systems. Use the
stmsboot -e
command to enable multipathing. -
Create new LUNs. It can take several hours for a new LUN to be displayed in 's user interface.
To verify iSCSI LUNs are available to the virtualization host, use the following command:
# iscsiadm list target -S
To verify Fibre Channel LUNs are available to the virtualization host, use the following command for Oracle Solaris 11:
# fcadm lu
Create a Static Block Storage SAN Library
To specify the LUNs in a storage library, you specify the server pool that will use the storage library and then select LUNs from among the LUNs available to the server pool.
When the job is completed, the new storage library is displayed in the Libraries section of the Navigation pane in the Static Block Storage section. The center pane shows the library's Summary tab.
Create an OS Update Profile
The Oracle Solaris 11 OS Update profile installs packages as part of an OS Provisioning profile. This example shows how to create an OS Update profile that installs the integration package. You then add this profile to an OS Provisioning profile.
Deploy Control Domains
Create an Provisioning Profile for Oracle VM Server for SPARC
This profile provisions the Oracle Solaris 11 OS and includes the OS Update profile you created in Create an OS Update Profile.
The profile appears in the center pane and in the Profiles and Policies section of Plan Management.
Create an OS Configuration Profile for Oracle VM Server for SPARC
The OS Configuration profile lets you define the network interface that you want to use on the operating system. The first interface you define is the boot interface and also the primary interface. You can add more than one network interface. When you apply the plan to a target, you can select which network interface is the primary interface and define the IP address.
-
Control domain configuration such as CPU Threads or Whole-cores, memory, Crypto Units, Virtual Console Port Range and Virtual Disk Server name.
-
Options such as enabling MPxIO and SR-IOV, detaching the unused buses, and saving NVRAMRC values.
-
OS management option to install the Agent Controller.
-
Networking options to use IPMP or Link Aggregation.
The recommended minimum configurations for the control domain are described in the following sections.
CPU Resource Allocation
The number of system CPUs determines the number of control domain CPU threads:
-
For less than 16 system CPUs, set the control domain CPU Threads to 2.
-
For between 16 and 64 system CPUs, set the control domain CPU Threads to 4.
-
For more than 64 system CPUs, set the control domain CPU Threads to 8.
You can select to allocate CPU resources either as CPU Threads or Whole-core. Whole-core is the default value in Oracle Enterprise Manager Ops Center. When you allocate as Whole-core, all the CPU Threads in the core are allocated to the control domain. For example, when you allocate two cores in UltraSPARC T2 servers, the control domain is allocated with all the 16 CPU Threads in the core. You can also set the maximum cores constraint when you select Whole-core allocation type. The maximum number of cores constraint specifies the number of cores that must be assigned to the domain.
Crypto Units
Crypto units are the resources on the supported platforms that provide high-performance, dedicated cryptographic engines. These can be used for tasks such as encrypting and decrypting network traffic between a Secure Socket Layer (SSL) web server and an application server.
Each CPU core has one Crypto unit and four or eight CPU threads. Because the Crypto unit is part of a core, the Crypto unit is bound only to domains that contain at least one thread from the parent core. Crypto units cannot be split as CPU threads are split. For example, you have assigned the Crypto unit for the first CPU core to the control domain. When a new logical domain is assigned a thread from the first CPU core and the Crypto unit for that core is already assigned, the control domain cannot assign that Crypto unit to the new logical domain. Allocation of Crypto units might not succeed, especially when a core is split between domains. An Oracle VM Server might allocate fewer Crypto units or none at all.
You must assign at least one Crypto unit to the control domain because the Crypto unit enables domain migration.
The use of Crypto Units is not mandatory although it might speed the logical domain migration. Allocation of Crypto Units might not be available in all the hardware.
RAM
The amount of RAM for the control domain depends on the size of the system RAM and the load of the system.
-
For system RAM less than 8 GB, set the control domain's RAM to 1 GB.
-
For system RAM between 8 GB to 16 GB, set the control domain's RAM to 2 GB.
-
For system RAM greater than 64 GB, set the control domain's RAM to 8 GB.
In Oracle Enterprise Manager Ops Center the default value is 4 GB as a recommended starting point for logical domains, and the minimum value is 1GB.
NVRAMRC Value
Automatic booting on a SPARC system uses the default boot device that is defined in the non-volatile RAM (NVRAM). User-defined commands that are executed during start-up are stored in the NVRAMRC file in the NVRAM. When you run an OS provisioning job on a SPARC machine, Oracle Enterprise Manager Ops Center resets the configuration to the factory default configuration and removes the information that is stored in the NVRAMRC file. The control domain OS configuration profile gives you the option to preserve the information in the NVRAMRC file before resetting the server to the factory defaults, and then restore the information after the reset.
To create an OS Configuration profile:
The job is initiated and the new OS configuration profile appears under Profiles and Policies in the Plan Management section. You can select the OS Configuration profile to view its details in the center pane.
Apply the Deployment Plan
Apply a deployment plan to provision Oracle VM Server for SPARC.
When you apply a deployment plan to provision Oracle VM Server for SPARC, you must have the following information to complete the installation:
-
In the server hardware, obtain the network interface that is physically connected to the network that is managed by Oracle Enterprise Manager Ops Center.
-
Both tagged and untagged networks are listed for network configuration. When OpenBoot PROM (OBP) is used, only untagged networks can be used for OS provisioning as you cannot boot from a tagged network.
-
The IP address for the boot interface.
-
If you want to identify the network interface using the MAC address, you can select to enter the MAC address instead of the boot interface.
-
The details of network connection that you want to use to configure the OS after booting. For multiple network configuration, the first interface is always overwritten by the boot interface that is defined during the OS provisioning step. For single network configuration, it is same as the boot interface network.
-
Do not select the option to place the Oracle VM Server for SPARC in a server pool.
Create the Server Pool
A server pool is a group of control domains that has access to the same virtual and physical networks and storage resources. Server pools provide load balancing, high availability capabilities, and sharing of the resources for all the members of the pool.
This procedure creates a server pool containing the control domains you created, both storage libraries you created, and the network domain you created.
Before you begin, you must know the network interface that is the physical connection to the system that supports the control domain.
The server pool is created and listed in the Server Pools view of the Assets section.
Set Up and Configure a Virtual Data Center
Create the vDC
The vDC inherits network resources and storage resources from the server pool. The vDC enables cloud users to create virtual machines and run applications. The cloud user provides networking to its virtual machines from the public networks that are allocated to each account or by creating private networks. The cloud user can use the public networks, or create a private vNet for use in the account. When the cloud users create a private vNet, either a dynamic private network is created or the static private network is available for use in that account.
Preparing the vServers
A vServer is an entity that provides the outward interface of a stand-alone operating system that consumes CPU, storage, and memory resources. A vServer has its own identity, local storage, interfaces, and configuration that exist for the full lifetime of the vServer. You determine the creation of new vServers according to the account quota limits and applications requirements.
When creating a vServer, the following account resources are required:
-
A vServer type: vServer Types are profiles that define the computing resources such as virtual CPU, memory, and storage size. You select the best suitable vServer type from those available for the account. vServer types are visible to cloud users during the vServer creation process.
-
A server template: Server templates designate the OS provisioning and OS configuration for the vServer. You can either select a server template from those that exist for the account or create a new server template.
-
One or more virtual networks: For the vServer network connectivity you must choose one or more vNets from the available vNets or create new ones before creating a vServer. vServers are only assigned to virtual networks at vServer creation time.
Create a new vServer Type
When you create a vServer type, the VM hosting details display the following information in the wizard based on the resources defined:
-
The number of virtualization servers in the vDC that have sufficient physical resources to host a vServer with the selected resources.
-
An estimation of number of vServers that can be hosted with the total number of physical resources of the vDC.
-
A warning when the current value of the memory size exceeds the selected storage size.
Create a vServer Template
Create a vServer template from the new OS profiles you created: provisioning, which includes the update profile, and configuration.
The new template is created and available to create vServers.
Validate the Integration and PaaS
Test the connections from Enterprise Manager Cloud Control and Enterprise Manager Ops Center to the cloud user. In this test, create two vServers with the same storage and network resources. If successful, one vServer can send information to and receive information from the other vServer.
Create Volumes for vServer
Use the following procedure to create volumes from shared disks. In normal operations, Enterprise Manager creates the volumes.
Verify Shared Storage and Connectivity
To verify connectivity, create two vServers. From the console, log into one vServer and use the ping
command with the IP address of the other vServer. This succeeds on public network or private network.
To verify shared storage:
-
For a Fibre Channel SAN storage:
fcadm lu
-
For iSCSI SAN storage:
iscsiadm list target -S
The Client Kit also contains the new_vDC.ksh
script that can be useful for diagnosing problems. Enterprise Manager uses the file when it starts the integration, using the file path /tmp/INPUT/bootstrap.xml
but you can also perform this manually.
Note:
After you finish testing, delete the bootstrap.xml
file and the oc-pass.txt
files because they contain security information such as passwords and URLs.
Diagnosing Problems
If a virtual datacenter is unable to create vnets, edit the network domain to increase the limit of private networks it can provide simultaneously.
If control domains are already running and you change the fabric that supports the server pool, you will need to configure the Fibre Channel controller:
fcadm hba-port
- Identify active ports.
fcadm remote-port -p <each_portID_from_output>
# cfgadm -c configure <controller_to_be_configured>