3 Networks for Virtualization

This chapter includes the following sections:

3.1 Introduction to Networks for Virtualization

Oracle Enterprise Manager Ops Center provides extensive management support for your datacenter network infrastructure. It supports both Ethernet and InfiniBand network protocols.

Network management in Oracle Enterprise Manager Ops Center provides a full lifecycle management of network domains, networks, and fabrics. It provides the following services:

  • Discover and manage switches

  • Define Ethernet fabrics

  • Create and define networks

  • Create and manage network domains

  • Automatic network discovery during asset discovery

  • Provision to create private networks on demand

  • IP address allocation that includes reserve and release a subnet member

  • DHCP management for host interface configuration

  • UI support for complex network configurations of virtualization deployments

This chapter provides a brief description about the network management in Oracle Enterprise Manager Ops Center, the different types of network infrastructure that can be setup and managed for virtualization technologies like Oracle Solaris Zones, Oracle VM Server for SPARC and Oracle VM Server for x86.

The prerequisites and how networks are connected to or assigned to the virtualization hosts, virtual host and the OS are described in this chapter.

See the Related Resources for Networks section for more information about networking.

3.2 Roles for Networks for Virtualization

This table lists the tasks and the role required to complete the task. Contact your administrator if you do not have the necessary role or privilege to complete a task.

See the Related Resources for Networks section for information about the different roles and the permissions they grant.

Table 3-1 Network Tasks and Roles

Task Role

Create Networks

Network Admin

Create Network Domains

Network Admin

Create Private Networks

Network Admin

Define Networks

Network Admin

Assign Networks

Network Admin

Define Ethernet Fabric

Network Admin

Add Fabric

Network Admin

Assign Fabric

Network Admin

Assign Network

Network Admin

Delete Network Domain

Network Admin

Remove Network

Network Admin

Delete Network

Network Admin

Remove Fabric

Network Admin

Delete Managed Fabric

Network Admin

Assign VLAN ID Range

Network Admin

Edit Attributes

Network Admin

Attach Network

Virtualization Admin

Connect Guests

Virtualization Admin

Associate Network Domain

Virtualization Admin

Detach Networks

Virtualization Admin

3.3 Actions for Networks for Virtualization

You can perform the following actions from the virtualization hosts, server pools, and guests:

  • Attach Network

  • Connect Guests

  • Associate Network Domain

  • Detach Networks

  • Disconnect Guests from Network

  • Modify Physical Connectivity

3.4 Location of Network Information in the User Interface

This table shows where to find network information.

Table 3-2 Location of Network Information in the BUI

Object Location

To see fabrics

Expand Networks in the Navigation pane, then select Fabrics in the filter.

To see all network domains

Expand Networks in the Navigation pane, then select Networks in the filter. Network Domains are listed.

To see all network

Expand Networks in the Navigation pane, then select Networks in the filter. Network Domains are listed. Expand a Network Domain to view all networks.

To attach networks to virtualization host

Expand Assets in the Navigation pane and select the Virtualization host. Click Attach Network in the Actions pane.

To connect guests to network

Expand Assets in the Navigation pane and select the guest. Click Connect Network in the Actions pane.

To associate network domains to server pool

Expand Assets in the Navigation pane, then Server Pools in the Resource Management Views. Select the server pool and click Associate Network Domain in the Actions pane.

To attach networks to server pool

Expand Assets in the Navigation pane, then Server Pools in the Resource Management Views. Select the server pool and click Attach Networks in the Actions pane.

3.5 Manage Networks

Networks are managed in the following way in Oracle Enterprise Manager Ops Center:

  • Discovering an asset automatically discovers the network.

  • Discovering and managing a switch automatically discovers all the fabrics in the switch.

  • Defining the Ethernet fabrics and assigning VLAN IDs to the fabrics.

  • Defining networks by providing network address, gateway, VLAN ID, fabrics, and network services.

  • Creating networks by providing network address, gateway, fabric, and network services. Oracle Enterprise Manager Ops Center automatically allocates the VLAN IDs or P-Keys.

  • Creating network domains that are administrative containers for networks. The network domains handle the relationship between the physical fabrics and networks constructed on the fabrics.

3.6 Physical Fabrics Management

Networks are built on the physical fabrics that provide network resources such as links and IP subnets. The physical fabrics can be fully managed, host managed, or unmanaged in Oracle Enterprise Manager Ops Center.

Depending on the network infrastructure in your datacenter, you can use Oracle Enterprise Manager Ops Center to manage the physical fabrics in the following way:

  • Fully managed fabrics

    When you discover a physical switch in Oracle Enterprise Manager Ops Center, all the fabrics that the switch supports are discovered and managed. A physical fabric can be partitioned to support many logical fabrics. Each port on an Ethernet switch can support 128 fabrics through its VLAN ID. Each partition on an InfiniBand switch can support 32000 partition keys. You can create VLANs or partitions as required. The fully managed fabrics facilitates to create dynamic private network for each VLAN ID or partition key.

  • Host managed fabrics

    The host of the switch is a managed asset in Oracle Enterprise Manager Ops Center. You must manually enable the VLAN IDs or the partition keys on the switch ports connected to the host.

    The host managed fabrics facilitates to create dynamic private network for each VLAN ID or partition key.

  • Unmanaged fabrics

    The switches are not managed in Oracle Enterprise Manager Ops Center. The VLAN IDs or partition keys are not available to manage the fabrics through a host. The networks are defined or discovered while discovering an asset in Oracle Enterprise Manager Ops Center. You cannot create dynamic private networks on unmanaged fabrics. Instead, you can assign some of the managed networks on these fabrics as private, called as static private networks.

3.7 Networks and Network Domains

You can create, define, or discover the networks in Oracle Enterprise Manager Ops Center as follows:

3.7.1 Create and Define Public Networks

Use the Create Network or Define Network option in the UI to create public networks on the fully managed and host managed fabrics. When you use Create Network option, Oracle Enterprise Manager Ops Center automatically assigns the VLAN IDs or P-Keys.

When you use Define Network option, you must specify the VLAN IDs for host managed Ethernet fabrics. For InfiniBand networks, the P-Keys are automatically assigned. For unmanaged fabric, use Define Network option to create public networks on the fabrics. You do not require to specify any VLAN IDs for Ethernet fabrics.

3.7.2 Create Network Domains

A network domain is a container for managed networks that handles the relationship between the physical fabrics that support the networks and the virtualization hosts or server pools that use the networks.

The Oracle Enterprise Manager Ops Center software always has a Default Network Domain and all public networks are members of that domain. You can create a user-defined network domain. When you create a network domain, you assign the fabrics and associate the networks that are already known to the system.

3.7.3 Create Private Networks

Use the Create Private Network option to create private networks on fully managed and host managed fabrics.

The Create Private Network option is useful for environment where you want to isolate applications any services from the public network. Private networks are created within a specific user-defined network domain for a specific purpose.

3.7.4 Dynamic Private Network Creation

To facilitate dynamic private network creation from virtual datacenter, associate user-defined network domains with the server pools of the virtual datacenter.

Select fully managed and host managed fabrics in the network domain to dynamically create private networks on demand. When you use unmanaged fabrics, then you must assign some of the existing managed networks assigned to the fabric. You must choose the managed networks that are not in use and do not route to other networks.

3.8 Properties of a Network

Figure 3-1 is an example of the network characteristics that appear in the Details tab.

Figure 3-1 Network Details Tab

Description of Figure 3-1 follows
Description of "Figure 3-1 Network Details Tab"

You cannot change the network IP address or the network type. Use the Edit Network Attributes action to change the network name and description, default gateway, MTU size, and to change the static IP routes. To change the MTU size, see the instructions for the Maximum Transmission Unit (MTU).

For 15811770:

Use the Edit Managed IP Ranges action to change the range of IP addresses that are available from the selected network. You specify the range with the starting IP address and the ending IP address. You have the option to exclude a specific IP address from the range. When you attach the network to a virtualization host, server pool, or virtual datacenter, the IP address is not available. You cannot exclude an IP address that is in use, which can be difficult to determine. For example, in a virtual data center, an account is assigned a range of IP addresses for its exclusive use. While the account exists, the IP addresses are in use, regardless of whether there is network activity.

3.9 VLAN and VLAN Tags

For fabrics based on Ethernet protocol, the ability to use VLAN tags is an attribute of each network. Use the Edit Network Attributes action to add or change the VLAN capability only for networks on fully managed fabrics.

For Bug 16745166, 15811770

When one CIDR supports both tagged and untagged networks, you can distinguish them by the default User Friendly Name (UFN), as shown in Figure 3-2. appends the VLAN ID or tag to the UFN. For an untagged network with no VLAN ID, the [UNTAG] string is appended.

Figure 3-2 VLAN Tags

Description of Figure 3-2 follows
Description of "Figure 3-2 VLAN Tags"

3.10 IP Multipathing Groups

Using IP Multipathing (IPMP), two or more physical network interface cards (NIC) form a group that use one IP address. If one NIC fails, the other NIC in the group maintains network access.

A network interface can be a physical network interface card (NIC) or, for an Oracle Solaris OS asset, it can be an IPMP group or link aggregation. You can implement both methods on the same network because they work at different layers of the network stack.

For information about how IPMP groups work in Oracle Solaris 11, see theRelated Resources for Networks section.

Note:

IPMP groups are supported only for IPv4 protocol.

IPMP provides increased reliability, availability, and network performance for systems with multiple physical interfaces because IPMP detects a physical interface failure and migrates network access to another member transparently.

Using IPMP, you can configure two or more physical interfaces into an IPMP group. If an interface in the group fails or is removed for maintenance, IPMP migrates the failed interface's IP addresses to another member of the group. The failover feature of IPMP preserves connectivity and prevents disruption of any existing connections.

The association between an IPMP group and a network must be unique. You can associate an IPMP group with only one network and you can associate a network with only one IPMP group or individual NICs.

In an IPMP group, you define whether each interface is a failover or a standby interface. The actions of each type differ if the current network interface fails, as follows:

  • Network access changes from the failed interface to the failover interface in the IPMP group and uses the failover interface data address. You must provide the data address for an interface that is defined as failover.

  • Network access changes from the failed interface to the standby interface in the IPMP group but does not change its data address. The data address of the failed interface migrates to the standby interface.

Link-based failure detection in an IPMP group is always enabled if your interface supports this type of failure detection. You can set up probe-based failure detection by providing a test address for each interface in the group.

You can create a single IPMP group while provisioning an operating system. If you create IPMP groups manually, Oracle Enterprise Manager Ops Center identifies and displays the groups on the UI. See Creating IPMP Groups for information and procedures for creating IPMP groups.

3.11 Link Aggregation

A network interface can be a physical network interface card (NIC) or, for an Oracle Solaris OS asset, it can be an IPMP group or link aggregation. You can implement both methods on the same network because they work at different layers of the network stack.

In an aggregated link, two or more NICs form a group and all members of the link aggregation provide network access at the same time. In addition to the high availability and load balancing that an IPMP group provides, an aggregated link can provide increased throughput when the network ports are also aggregated.

When interfaces have been aggregated, they are treated as a single network interface. Oracle Enterprise Manager Ops Center includes any link aggregations in the list of available NICs as if the link aggregation were an individual interface. To assign a network with a link aggregation to an Oracle VM Server or global zone, select the link aggregation from the NIC list. You can view the link aggregation details on the Oracle VM Server's or global zone's Network tab as described in Link Aggregation.

Link aggregation is a standard defined in IEEE802.3ad. An aggregated link consists of several interfaces on a system configured as a single, logical unit. Link aggregation increases the speed and high availability of a connection between a server and a switch. The most common protocol used to manage link aggregation is LACP (Linked Aggregation Control Protocol).

For information about how link aggregation works in Oracle Solaris 11, see the Related Resources for Networks section.

In Oracle Solaris 10 and by default in Oracle Solaris 11, the type of link aggregation you create is a trunk aggregation, which has these requirements:

  • All the members of the aggregated link are connected to the same switch.

  • The members of the aggregated link are of the same type. For example, NICs with the e1000g interface cannot be mixed with NICs that use the bge interface.

  • The required driver is GLDv3.

The following is for Bug 16398076

Oracle Solaris 11 supports an alternative to trunk aggregation called Datalink Multipathing Aggregations (DLMP). This type of aggregation overcomes the limitations of trunk aggregation for network virtualization because DLMP aggregation works with more than one switch and provides the benefits of the link layer of the network stack to the aggregation.

In trunk aggregation, every port is associated with every datalink in the link aggregation. In a DLMP aggregation, every port is associated with every datalink in the link aggregation and every port is associated with the primary network interface and any of its VNICs that are configured to use the link aggregation.

For a link aggregation created in Oracle Solaris 11 OS, the MTU size for all of the members of the aggregation must be at least 9216 bytes to allow Oracle VM Servers and logical domains to use VLAN tagged networks. To change the MTU size, see the Maximum Transmission Unit (MTU) instructions.

3.12 Networking for Virtualization and Virtual Datacenter

Oracle Enterprise Manager Ops Center provides systems and users with efficient, controlled and secure sharing of the networking resources. The virtualization properties available in different Oracle Solaris OS version and the virtualization technology are implemented and available through the software UI.

The UI provides options to assign the network connection to the managed assets. You can also select the network interfaces through which the network connection is made to the virtualization host, virtual host and the OS. The attachment and connection varies, depending on the virtualization technology.

3.12.1 Networking for Server Pools

A server pool must have at least one network. When a server pool has more than one network, all virtualization hosts in the server pool are associated with the same set of networks.

When you add a virtualization host to a server pool, the virtualization host is provided access to all the networks defined for the pool. This ensures that all virtual hosts have network access, even when you migrate a virtual host from one virtualization host to another one within the pool.

For zones and Oracle VM Server for SPARC server pool, it is recommended to create server pool that has homogenous network connection. Refer to Server Pools for more detailed information.

Figure 3-3 is an example of network connections to two virtualization hosts in a server pool. This server pool has two virtualization hosts and two network associations.

Figure 3-3 Network Connections for a Server Pool

Description of Figure 3-3 follows
Description of "Figure 3-3 Network Connections for a Server Pool"

3.12.2 Networking for Zones

When you attach networks to zones, VNICs are created. Virtual Network Interface Cards (VNICs) are pseudo interfaces created on top of datalinks. It has an automatically generated MAC address.

You can define the mode of the network to be attached as Shared IP or Exclusive IP. In Shared IP mode, the global zone shares its network interface with one or more zone. You must define the network interface when you assign the network to the global zone. In Exclusive IP mode, a dedicated network interface is allocated to the zone. You can choose the network interface when you assign the network to a zone.

When a network is assigned as shared on a global zone, you can assign the network as exclusive on another global zone. For a global zone, a network can be attached in either shared or exclusive mode only. For non-global zones, a network that is used in a shared mode for one zone cannot be used in exclusive mode for another zone.

While you attach networks to a global zone, you can deploy IP Multipathing (IPMP) to obtain better network performance or link aggregation to provide increased reliability, availability, and network performance for systems with multiple physical interfaces.

Table 3-3 identifies the differences in attaching the network for Oracle Solaris 10 OS and Oracle Solaris 11 OS global zone in Oracle Enterprise Manager Ops Center.

Table 3-3 Differences in Network Connection for Global Zone

Oracle Solaris 10 OS Oracle Solaris 11 OS

You can attach network in Shared IP or Exclusive IP mode.

Networks are always attached in Exclusive IP mode.

You cannot make multiple connections to a network.

You can make multiple connections to a network.

You can deploy IPMP or Link Aggregation for better network performance.

You can deploy only Link Aggregation in Oracle Solaris 11 OS.

In Oracle Solaris 11 OS, the network is always attached in exclusive IP mode, this is because a VNIC is created when the zone boots, and deleted when the zone is halted.

When you connect networks to the global zones, you can also select the tagging mode for the networks configured with VLAN ID. You can select Tagged or Untagged mode for the network connection.

3.12.2.1 Zones Server Pool

Server pool for zones reflect the networking properties of the Oracle Solaris OS version of the global zones in the pool. You can connect to a network only once for Oracle Solaris 10 OS. Whereas, you can make multiple network connections for Oracle Solaris 11 OS.

For zones server pool that contains a mixture of Oracle Solaris 10 and Oracle Solaris 11 OS, you cannot make multiple connections to a network.

Also, create zones server pool that are homogenous in network tagging mode. It can prevent any network outages for the zones created on the members of the server pool.

Before you attach a network to a server pool, verify that each virtualization host in the server pool has a physical network interface to the network so that all members of the pool can continue to share the network resources of the server pool.

3.12.3 Networking for Oracle VM Server for SPARC

You can attach networks to the Oracle VM Server using the physical interfaces or etherstub device that can belong to the control domain, I/O domain, or root domain. When you attach networks to the Oracle VM Server, you can select the service domain and the physical network interfaces available from that domain.

You can also specify the tagging mode for attaching networks configured with VLAN ID. You can select Tagged or Untagged mode for the network connection.

Attaching networks to Oracle VM Server result in the creation of a virtual switch for each network connection. This is not applicable for SR-IOV enabled networks. See SR-IOV Enabled Networks for more information about attaching SR-IOV enabled networks.

You can make multiple connections to a network in the control domain. For each network connection, a virtual switch is required. If there is an already existing virtual switch for the physical interface or etherstub device, you can re-use the virtual switch. If there is no virtual switch for the physical interface or etherstub device, you can either provide a user-friendly name for the virtual switch or a virtual switch is automatically created with a default naming pattern. For an example network 1.1.1.0/24, the virtual switches take the name as 1.1.1.0_24, 1.1.1.0_24_1, 1.1.1.0_24_2 and 1.1.1.0_24_3. This ensures that the switches have unique names.

When a network connection is made to the server, the virtual switch created is incremented. When you create and start a logical domain, you define the virtual switch that connects to the logical domain. Each virtual switch must be connected to a NIC.

When you connect to the physical interfaces from I/O domains and root domains, the virtual switch is created in the control domain. You cannot define the IP address allocation for the network connection. Instead, you can define the IP address in the OS of the guest domain as required. You can define the IP address only when you use the network interfaces from the control domain or primary.

You can create IPMP groups and aggregate links in the control domain.

3.12.3.1 SR-IOV Enabled Networks

An SR-IOV enabled network interface means that there are virtual functions created on the physical functions of the PCIe Endpoint device and you can assign the virtual functions to the logical domains. Oracle Enterprise Manager Ops Center does not create any virtual switch when you connect to a network using SR-IOV enabled network interface. When you select SR-IOV option while attaching networks to the Oracle VM Server, only the interfaces on which the virtual functions are created are available for network configuration.

Guest domains that are assigned with SR-IOV enabled networks cannot be migrated. SR-IOV enabled networks are available only from control domain and root domain.

SR-IOV enabled networks on root domain are available only in the following conditions:

  • Oracle Solaris 11 Update 1 OS (SRU 4.5) is necessary for dynamic attach of networks.

  • Available only from Oracle VM Server for SPARC 3.1 version.

  • Refer to Oracle VM Server for SPARC Release Notes at http://docs.oracle.com/cd/E38405_01/html/E38409/index.html for hardware and firmware requirements for SR-IOV feature.

3.12.3.2 Attach Networks to Oracle VM Server for SPARC Server Pool

Before you attach a network to a server pool, verify that each virtualization host in the server pool has a physical network interface to the network so that all members of the pool can continue to share the network resources of the server pool.

The following options are available when you attach networks to the server pool:

  • You can select the service domain which provides the network interface for the network connection.

  • If the Oracle VM Server is already connected to the network, you can keep the existing connection or make a new connection.

  • If there are any virtual switches available for the network interface, you can re-use the virtual switches.

  • If there are no virtual switches, you can either provide a name for the virtual switch or a virtual switch is automatically created with a default naming pattern.

  • You can select the tagging mode for the networks configured with VLAN ID.

It is recommended to maintain server pool networks attached either in tagged or untagged mode to the server pool members. You can maintain server pool with mixed configuration. There is likely occurrence of network outage in the logical domain OS when you try to migrate the logical domain between servers that have different tagging modes. To avoid such outage, maintain the server pool members with homogenous network condition. Refer to Network Tagging Mode Conditions for more information about selecting tagging modes for the network connection.

3.12.3.3 Connect Networks to Logical Domains

You can connect networks to logical domains in running state. You can make multiple connections to a network. For each connection, you require a virtual switch or virtual function of SR-IOV enabled networks to connect the logical domain to the network.

You can re-use a virtual switch to make multiple connections to a network from the logical domain or use the same virtual switch to connect to a network for different logical domains. A virtual network device or vnet is defined when you connect the logical domain to a network through a virtual switch. For each network connection, a vnet is created. Oracle Enterprise Manager Ops Center tries to re-use the vnets. This reduces the number of vnets created for the network connections.

You can connect network root domains and I/O domains only when their operating systems are managed in Oracle Enterprise Manager Ops Center.

When you connect networks to logical domains, you can define the following parameters for the connection:

  • Select the service domain that will provide the network services.

  • Select the network mode as Tagged or Untagged for an VLAN ID network.

  • Select SR-IOV enabled network connection.

  • Select the virtual switch or the virtual function for SR-IOV enabled network function through which the logical domain is connected to the network.

You need an untagged network connection for provisioning OS on the logical domain. If the network is already configured with an VLAN ID, then select Untagged option while connecting the logical domain to the network.

When the network is connected to logical domain using Oracle Enterprise Manager Ops Center, by default 10 alternate MAC addresses are created.

3.12.4 Networking for Oracle VM Server for x86

During the installation of Oracle VM Server, the network interface used for the management is configured as a bonded interface. The bond is created with one interface and named as bond0. You can create additional bonds to add redundancy and load balancing of your network environment.

Attach networks to the Oracle VM Server or to the server pool that consists of a group of Oracle VM Servers on an Oracle VM Manager. Configure the network interfaces or the bonds to the network to be attached. You can assign different roles or functions to the networks attached to the Oracle VM Server.

The following are the network roles available for an Oracle VM Server:

  • Server Management: Manages the Oracle VM Servers in a server pool. The Oracle VM Manager has one Server Management network.

  • Live Migrate: Migrates the virtual machines from one Oracle VM Server to another in the server pool, without changing the state of the virtual machine.

  • Cluster Heartbeat: Verifies that the Oracle VM Servers in the server pool are running.

  • Virtual Machine: Monitors the network traffic between the virtual machines in a server pool.

  • Storage: Transfers between virtual machines and virtual disks.

The management network created during the installation of Oracle VM Server has the following roles:

  • Server Management

  • Cluster Heartbeat

  • Live Migrate

You can add and remove the roles of this management network, except for the Server Management role.

Depending on the available network interfaces on the Oracle VM Server, you can attach networks to Oracle VM Server and assign different roles to the networks. For example, you can attach the network in which your storage servers are placed and assign the Storage role to that network. You can assign a network with Live Migrate to be used only for migration.

3.12.5 Networking for Virtual Datacenters

Each virtual datacenter uses server, storage, and network resources in a dynamic way, allocating and releasing resources when necessary.

The virtual datacenter inherits its network resources from the network domain that supports the server pool. These networks form the public external networks for the virtual datacenter. These networks can then be assigned to the accounts in the virtual datacenter. When the user of an account creates a private vNet, either a dynamic private network is created or the static private network is made available for use in that account.

For a complete description of networks for virtual datacenters, see Creating vNets and Setting Up Network Resources in Virtual Datacenters.

3.13 Related Resources for Networks

For instructions in performing actions or to learn more about the role of this feature, go to one of the following resources.