The following figure shows the virtualization components that form the Oracle VM Server for SPARC “execution environment.” These components are not strictly separated. The most simple configuration is to combine all of these functions in a single domain. The control domain might also act as an I/O domain and a service domain for other domains.
Figure 3 Components of the Execution Environment
Suppose an attacker attempts to break system isolation and then manipulate the hypervisor or another component of the execution environment to reach a guest domain. You must protect each guest domain as you would any stand-alone server.
The rest of this chapter presents threat possibilities and the various measures that you can take to counter them. Each of these attacks attempt to overcome or eliminate the isolation of the different domains that run on a single platform. The following sections describe the threats to each part of an Oracle VM Server for SPARC system:
The operational environment includes physical systems and their components, datacenter architects, administrators, and members of the IT organization. A security breach can occur at any point in the operational environment.
Virtualization places a layer of software between the actual hardware and the guest domains that run the production services, which increases complexity. As a result, you must carefully plan and configure the virtual system and beware of human error. Also, beware of attempts by attackers to gain access to the operational environment by using “social engineering.”
The following sections describe the distinctive threats that you can counter at the operational environment level.
The primary security concern for a virtualized environment is to sustain server isolation by separating network segments, segregating administrative access, and deploying servers to security classes, which are groups of domains that have the same security requirements and privileges.
Carefully configure virtual resources to avoid some of the following errors:
Creating unnecessary communication channels between production guest domains and the execution environment
Creating unnecessary access to network segments
Creating unintentional connections between discrete security classes
Unintentionally migrating a guest domain to the wrong security class
Allocating insufficient hardware, which might lead to unexpected resource overloading
Assigning disks or I/O devices to the wrong domain
Before you begin, carefully define the operational guidelines for your Oracle VM Server for SPARC environment. These guidelines describe the following tasks to perform and how to perform them:
Managing patches for all components of the environment
Enabling the well-defined, traceable, and secure implementation of changes
Checking log files at a regular interval
Monitoring the integrity and availability of the environment
Regularly perform checks to ensure that these guidelines remain up to date and adequate, and to verify that these guidelines are being followed in everyday operations.
In addition to these guidelines, you can take several more technical measures to reduce the risk of unintentional actions. See Logical Domains Manager.
When moving a physical system to a virtualized environment, you can usually keep the storage configuration as-is by reusing the original LUNs. However, the network configuration must be adapted to the virtualized environment and the resulting architecture might differ considerably from the architecture used on the physical system.
You must consider how to maintain the isolation of the discrete security classes and their needs. Also, consider the shared hardware of the platform and shared components such as network switches and SAN switches.
To maximize security for your environment, ensure that you maintain the isolation of guest domains and security classes. When you design the architecture, anticipate possible errors and attacks and implement lines of defense. A good design helps to confine potential security issues while managing complexity and cost.
Use security classes, which are groups of domains that have the same security requirements and privileges, to isolate individual domains from each other. By assigning guest domains that are in the same security class to a certain hardware platform, even a breach of isolation prevents the attack from crossing into another security class.
The live domain migration feature has the potential to break isolation if a guest domain is inadvertently migrated to a platform that is assigned to a different security class as shown in the following figure. So, carefully plan guest domain migration to ensure that a migration across security class boundaries is not permitted.
Figure 4 Domain Migration Across Security Boundaries
To minimize or eliminate the security vulnerability that the migration operation poses, you must manually exchange and install ldmd-generated host certificates out-of-band between each source machine and target machine pair. For information about how to set up the SSL certificates, see Configuring SSL Certificates for Migration in Oracle VM Server for SPARC 3.6 Administration Guide.
Losing track of all of the virtual network connections might lead to a domain gaining erroneous access to a network segment. For example, such access might circumvent the firewall or a security class.
To reduce the risk of implementation errors, carefully plan and document all of the virtual and physical connections in your environment. Optimize the domain connection plan for simplicity and manageability. Clearly document your plan and verify the correctness of your implementation against your plan before going into production. Even after your virtual environment is in production, verify the implementation against the plan at regular intervals.
You can use VLAN tagging to consolidate several Ethernet segments onto a single physical network. This feature is also available for virtual switches. To mitigate the risks involved with software errors in the implementation of virtual switches, configure one virtual switch per physical NIC and VLAN. To further protect against errors in the Ethernet driver, refrain from using tagged VLANs. However, the probability for such errors is low as this tagged VLAN vulnerability is well known. Intrusion tests on Oracle's Sun SPARC T-series server with the Oracle VM Server for SPARC software have not shown this vulnerability.
Security appliances such as packet filters and firewalls are instruments of isolation and protect the isolation of security classes. These appliances are subject to the same threats as any other guest domain, so using them does not guarantee complete protection against an isolation breach. Therefore, carefully consider all aspects of risk and security before you decide to virtualize such a service.
Resource sharing in a virtualized environment might lead to denial-of-service (DoS) attacks, which overload a resource until it negatively affects another component such as another domain.
In a Oracle VM Server for SPARC environment, only some resources might be affected by a DoS attack. CPU and memory resources are assigned exclusively to each guest domain, which prevents most DoS attacks. Even the exclusive assignment of these resources might slow down a guest domain in the following ways:
Thrashing the cache areas that are shared between strands and are assigned to two guest domains
Overloading memory bandwidth
Unlike CPU and memory resources, disk and network services are usually shared between guest domains. These services are provided to the guest domains by one or more service domains. Carefully consider how to assign and distribute these resources to guest domains. Note that any configuration that permits maximum performance and resource utilization simultaneously minimizes the risk of side effects.
A network link can be saturated or a disk can be overloaded whether exclusively assigned to a domain or shared among domains. Such attacks affect the availability of a service for the duration of the attack. The target of the attack is not compromised and no data is lost. You can easily minimize the effects of this threat, but you should keep it in mind even though it is limited to network and disk resources on Oracle VM Server for SPARC.
Ensure that you assign only required hardware resources to guest domains. Be sure to unassign an unused resource after the resource is no longer required, for example, a network port or DVD drive required only during an installation. By following this practice, you minimize the number of possible entry points for an attacker.
Shared hardware resources such as physical network ports, present a possible target for DoS attacks. To limit the impact of DoS attacks to a single group of guest domains, carefully determine which guest domains share which hardware resources.
For example, guest domains that share hardware resources could be grouped by the same availability or security requirements. Beyond grouping, you can apply different kinds of resource controls.
You must consider how to share disk and network resources. You can mitigate issues by separating disk access through dedicated physical access paths or through dedicated virtual disk services.
All the countermeasures described in this section require that you understand the technical details of your deployment and their security implications. Plan carefully, document well, and keep your architecture as simple as possible. Ensure that you understand the implications of virtualized hardware so that you can prepare to securely deploy the Oracle VM Server for SPARC software.
Logical domains are robust against the effects of sharing CPUs and memory, as little sharing actually occurs. Nevertheless, it is best to apply resource controls such as Solaris resource management within the guest domains. Using these controls protect against bad application behavior for either a virtual or a non-virtualized environment.
Components of the Execution Environment shows the components of the execution environment. Each component provides certain services that together form the overall platform on which to run the production guest domains. Properly configuring the components is vitally important for the integrity of the system.
All of the execution environment components are potential targets for an attacker. This section describes the threats that might affect each component in the execution environment. Some threats and countermeasures might apply to more than one component.
By manipulating the execution environment, you can gain control in numerous ways. For example, you might install manipulated firmware in the Oracle ILOM to snoop on all guest domain I/O from within an I/O domain. Such an attack can access and change the system's configuration. An attacker that takes control of the Oracle VM Server for SPARC control domain can reconfigure the system in any way, and an attacker that takes control of an I/O domain can make changes to attached storage, such as to boot disks.
An attacker who successfully breaks in to either the Oracle ILOM or to any domain in the execution environment can read and manipulate all data that is available to that domain. This access might be gained over the network or by means of an error in the virtualization stack. Such an attack is difficult to perform as usually the Oracle ILOM and the domains cannot be attacked directly.
The countermeasures to protect against the manipulation of the execution environment are standard security practice and should be implemented on any system. Standard security practices present an additional layer of protection around the execution environment that further reduces the risk of intrusion and manipulation.
Ensure that you only create accounts that are required for the applications that run on the system.
Ensure that accounts that are required for administration are secured by using either key-based authentication or strong passwords. These keys or passwords should not be shared among different domains. Also, consider implementing two-factor authentication or a “two-person rule” for taking certain actions.
Do not use anonymous logins for accounts such as root to ensure that you have complete traceability and accountability of the commands run on the system. Instead, use rights to grant individual administrators access only to those functions that they are permitted to perform. Ensure that administrative network access always uses encryption such as SSH and that an administrator's workstation is treated as a high-security system.
Any software that is installed on a system can be compromised, so ensure that you install only the required software to minimize the breach window.
In addition to installing a minimized Oracle Solaris OS, configure software packages to “harden” the software against attack. First, run limited network services to effectively disable all network services except for SSH. This policy is the default behavior on Oracle Solaris 11 systems. For information about how to secure the Oracle Solaris OS, see Oracle Solaris 10 Security Guidelines and Oracle Solaris 11 Security Guidelines.
By necessity, production applications are connected to other systems and as a result are more exposed to external attacks. Do not deploy production applications to a domain that is part of the execution environment. Instead, ensure that you deploy them only to guest domains that have no further privileges.
The execution environment should provide only the necessary infrastructure for these guest domains. Separating the execution environment from the production applications enables you to implement granularity in administration privileges. A production guest domain administrator does not require access to the execution environment and an execution environment administrator does not require access to the production guest domains. If possible, assign the different roles of the execution environment, such as the control domain and I/O domain, to different domains. This type of configuration reduces the amount of damage that can be done if any one of these domains is compromised.
You can also extend role separation to the network environment that is used to connect your different servers.
Connect all servers that are equipped with service processors (SPs) to a dedicated management network. This configuration is also recommended for the domains of the execution environment. If networked at all, host these domains on their own dedicated network. Do not connect the execution environment domains directly to those networks that are assigned to the production domains. While you can perform all administrative work through the single console connection that is made available by the Oracle ILOM SP, this configuration makes administration sufficiently cumbersome to be impracticable. By separating the production and administration networks, you protect against both eavesdropping and manipulation. This type of separation also eliminates the possibility of an attack on the execution environment from the guest domains over the shared network.
Figure 5 Dedicated Management Network
All current Oracle SPARC systems include a built-in system controller (Oracle ILOM), which has the following capabilities:
Manages basic environmental controls such as fan speed and chassis power
Enables firmware upgrades
Provides the system console for the control domain
You can access the Oracle ILOM through a serial connection or use SSH, HTTP, HTTPS, SNMP, or IPMI to access it through a network port. The Fujitsu M10 servers use XSCF instead of Oracle ILOM to perform similar functions.
An attacker that gains control of the Oracle ILOM can compromise the system in many ways, including the following:
Removing power from all running guests
Installing manipulated firmware to gain access to at least one guest domain
These scenarios apply to any system that has such a controller device. In a virtualized environment, the damage can be that much greater than in a physical environment because many domains that are housed in the same system enclosure are at risk.
Likewise, an attacker that gains control over the control domain or an I/O domain can easily disable all dependent guest domains by shutting down the corresponding I/O services.
The Oracle ILOM is usually connected to an administrative network that must be well protected and isolated from normal production networks.
Likewise, an attacker can breach a service domain from the network or through an error in the virtualization stack, and then block guest I/O or perform a system shutdown. While the damage is limited as data is neither lost nor compromised, the damage can affect a large number of guest domains. So, ensure that you protect against the possibility of this threat to limit the potential damage.
As the system service processor, the Oracle ILOM controls critical features such as chassis power, Oracle VM Server for SPARC startup configurations, and console access to the control domain. The following measures enable you to secure the Oracle ILOM:
Placing the network port of the Oracle ILOM in a network segment that is separate from the administrative network, which is used for the domains in the execution environment.
Disabling all services that are not required for operation, such as HTTP, IPMI, SNMP, HTTPS, and SSH.
Configuring dedicated and personal administrator accounts that grant only the required rights. To maximize accountability of the actions taken by administrators, ensure that you create personal administrator accounts. This type of access is especially important for console access, firmware upgrades, and managing startup configurations.
The hypervisor is the firmware layer that implements and controls the virtualization of real hardware. The hypervisor includes the following components:
Actual hypervisor, which is implemented in firmware and supported by the systems' CPUs.
Kernel modules that run in the control domain to configure the hypervisor.
Kernel modules and daemons that run in I/O domains and service domains to provide virtualized I/O, as well as the kernel modules that communicate by means of Logical Domain Channels (LDCs).
Kernel modules and device drivers that run in the guest domains to access virtualized I/O devices as well as the kernel modules that communicate by means of LDCs.
An attacker can hijack guest domains or the entire system by breaking out of the isolated runtime environment provided by the hypervisor. Potentially, this threat can cause the most severe damage to a system.
A modular system design can improve isolation by granting different levels of privileges to guest domains, the hypervisor, and the control domain. Each functional module is implemented in a separate and configurable kernel module, device driver, or daemon. This modularity requires clean APIs and simple communication protocols, reducing the overall risk for error.
Even if exploitation of an error seems unlikely, the potential damage can lead to the attacker controlling the entire system.
Even though you can download system firmware and OS patches directly from an Oracle web site, these patches can be manipulated. Before you install the software, ensure that you verify the MD5 checksums of the software packages. The checksums of all downloadable software is published by Oracle.
Oracle VM Server for SPARC uses several drivers and kernel modules to implement the overall virtualization system. All kernel modules and most binaries that are distributed with the Oracle Solaris OS carry a digital signature. Use the elfsign utility to check the digital signature for each kernel module and driver. You can use the Oracle Solaris 11 pkg verify command to check the integrity of Oracle Solaris binary.
First, you must establish the integrity of the elfsign utility. Use the basic audit and reporting tool (BART) to automate digital signature verification process. Integrating BART and the Solaris Fingerprint Database in the Solaris 10 Operating System (http://www.oracle.com/technetwork/articles/systems-hardware-architecture/o11-005-bart-solaris-fp-db-276999.pdf) describes how to combine BART and the Solaris Fingerprint Database to automatically perform similar integrity checks. Although the fingerprint database has been discontinued, the concepts described in this document can be carried over to use elfsign and BART in a similar manner.
You can use the verified boot feature as a countermeasure for validating kernel modules. To configure the automated validation of kernel modules at boot time, set the verified boot policies in Oracle ILOM. See the documents for your specific platform at http://docs.oracle.com/en/hardware/. To validate kernel modules in the control domain, set the verified boot policies in Oracle ILOM. To validate kernel modules in guest domains, use the Logical Domains Manager to set the verified boot policies.
The control domain, which often has the roles of an I/O domain and a service domain, must be kept safe as it can modify the configuration of the hypervisor, which controls all attached hardware resources.
The shutdown of the control domain can result in a denial of service of the configuration tools. Because the control domain is required only for configuration changes, the guest domains are unaffected if they access their network and disk resources through other service domains.
Attacking the control domain over the network is equivalent to attacking any other properly protected Oracle Solaris OS instance. The damage of a shutdown or similar denial of service of the control domain is relatively low. However, guest domains are affected if the control domain also acts as a service domain for these guest domains.
Avoid configuring administrative network access to the execution environment's domains. This scenario requires that you use the Oracle ILOM console service to the control domain to perform all administration tasks. Console access to all other domains is still possible by using the vntsd service running on the control domain.
Consider this option carefully. Although this option reduces the risk of being attacked over the administrative network, only one administrator can access the console at a time.
For information about securely configuring vntsd, see How to Enable the Virtual Network Terminal Server Daemon in Oracle VM Server for SPARC 3.6 Administration Guide.
The Logical Domains Manager runs in the control domain and is used to configure the hypervisor, and create and configure all domains and their hardware resources. Ensure that Logical Domains Manager use is logged and monitored.
An attacker might take control of an administrator's user ID or an administrator from a different group might gain unauthorized access to another system.
Ensure that an administrator does not have unnecessary access to a system by implementing well-maintained identity management. Also, implement strict, fine-grained access control and other measures such as the two-person rule.
Consider implementing a two-person rule for Logical Domains Manager and other administrative tools by using rights. This rule protects against social engineering attacks, compromised administrative accounts, and human error.
By using rights for the ldm command, you can implement fine-grained access control and maintain complete retraceability. For information about configuring rights, see Oracle VM Server for SPARC 3.6 Administration Guide. Using rights helps safeguard against human errors because not all features of the ldm command are available to all administrators.
Disable unnecessary domain manager services. The Logical Domains Manager provides network services for domain access, monitoring, and migration. Disabling network services reduces the attack surface of Logical Domains Manager to the minimum required to operate it normally. This scenario counters denial of service attacks and other attempts to misuse these network services.
Disable any of the following network services when they are not being used:
Migration service on TCP port 8101
To disable this service, see the description of the ldmd/incoming_migration_enabled and ldmd/outgoing_migration_enabled properties in the ldmd(8) man page.
Extensible Messaging and Presence Protocol (XMPP) support on TCP port 6482
For information about how to disable this service, see XML Transport in Oracle VM Server for SPARC 3.6 Developer’s Guide.
Disabling XMPP prevents some management tools and key Oracle VM Server for SPARC features from working. See Oracle VM Server for SPARC XML Interface.
Simple Network Management Protocol (SNMP) on UDP port 161
Determine whether you want to use the Oracle VM Server for SPARC Management Information Base (MIB) to observe domains. This feature requires that the SNMP service is enabled. Based on your choice, do one of the following:
Enable the SNMP service to use the Oracle VM Server for SPARC MIB. Securely install the Oracle VM Server for SPARC MIB. See How to Install the Oracle VM Server for SPARC MIB Software Package in Oracle VM Server for SPARC 3.6 Management Information Base User’s Guide.
Disable the SNMP service. For information about how to disable this service, see How to Remove the Oracle VM Server for SPARC MIB Software Package in Oracle VM Server for SPARC 3.6 Management Information Base User’s Guide.
Discovery service on multicast address 239.129.9.27 and port 64535
You cannot disable this service while the Logical Domains Manager daemon, ldmd, is running. Instead, use the IP Filter feature of Oracle Solaris to block access to this service, which minimizes the attack surface of the Logical Domains Manager. Blocking access prevents unauthorized use of the utility, which effectively counters denial-of-service attacks and other attempts to misuse these network services. See Chapter 20, IP Filter in Oracle Solaris (Overview) in Oracle Solaris Administration: IP Services and Using IP Filter Rule Sets in Oracle Solaris Administration: IP Services.
Also see Countermeasure: Securing the Oracle ILOM.
A service domain provides some virtual services to guest domains on the system. Services might include a virtual switch, virtual disk, or virtual console service.
Service Domain Example shows an example service domain that offers console services. Often the control domain hosts the console services, and thus is also a service domain. The execution environment domains often combine the functions of a control domain, I/O domain, and service domain in one or two domains.
An attacker who gains control of a service domain can manipulate data or listen to any communication that occurs through the offered services. This control might include console access to guest domains, access to network services, or access to disk services.
While the attack strategies are the same as for an attack on the control domain, the possible damage is less because the attacker cannot modify the system configuration. The resulting damage might include the theft or manipulation of data that is being offered by the service domain but not manipulation of any data sources. Depending on the service, an attacker might be required to exchange kernel modules.
Figure 6 Service Domain Example
If possible, have each service domain offer only one service to its clients. This configuration guarantees that only one service can be compromised if a service domain is breached. However, be sure to weigh the importance of this type of configuration against the additional complexity. Note that having redundant I/O domains is highly recommended.
You can isolate both Oracle Solaris 10 and Oracle Solaris 11 service domains from guest domains. The following solutions are shown in the preferred order of implementation:
Ensure that the service domain and the guest domain do not share the same network port. Also, do not plumb any virtual switch interface on the service domain. For Oracle Solaris 11 service domains, do not plumb any VNICs on the physical ports that are used for virtual switches.
If you must use the same network port for both the Oracle Solaris 10 OS and Oracle Solaris 11 OS, place the I/O domain traffic in a VLAN that is not used by guest domains.
If you cannot implement either of the previous solutions, do not plumb the virtual switch in the Oracle Solaris 10 OS and apply IP filters in the Oracle Solaris 11 OS.
Ensure that access to individual virtual consoles is limited to only those users that must access them. This configuration ensures that no single administrator has access to all consoles, which prevents access to consoles other than those assigned to a compromised account. See How to Create Default Services in Oracle VM Server for SPARC 3.6 Administration Guide.
Any domain that has direct access to physical I/O devices such as network ports or disks is an I/O domain. For information about configuring I/O domains, see Chapter 6, Configuring I/O Domains in Oracle VM Server for SPARC 3.6 Administration Guide.
An I/O domain also might be a service domain if it provides I/O services to guest domains, which gives the domains access to the hardware.
An attacker who blocks the I/O services of an I/O domain ensures that all dependent guest domains are equally blocked. A successful DoS attack might be achieved by overloading the back-end network or disk infrastructure or by injecting a fault into the domain. Either attack might force the domain to hang or panic. Likewise, an attacker who suspends a service domain's services causes any guest domain that depends on these services to immediately hang. If the guest domain hangs, it will resume operation when the I/O service resumes.
DoS attacks are commonly made over the network. Such an attack can be successful because network ports are open for communication and can be overwhelmed by network traffic. A resulting loss of service blocks dependent guest domains. A similar attack on disk resources might be made by means of the SAN infrastructure or by attacking the I/O domain. The only damage is a temporary halt of all dependent guest domains. While the impact of DoS tasks might be substantial, data is neither compromised nor lost, and the system configuration remains intact.
Configuring multiple I/O domains reduces the impact of one domain failing or being compromised. You can assign individual PCIe slots to a guest domain to give it I/O domain capabilities. If the root domain that owns the PCIe bus crashes, that bus is reset, which leads to a subsequent crash of the domain that was assigned the individual slot. This feature does not fully eliminate the need for two root domains that each own a separate PCIe bus.
High availability also contributes to enhanced security because it ensures that services can withstand denial-of-service attacks. The Oracle VM Server for SPARC implements high availability methodologies such as using redundant disk and network resources in redundant I/O domains. This configuration option enables rolling upgrades of the I/O domains and protects against the impact of a failed I/O domain due to a successful DoS attack. With the advent of SR-IOV, guest domains can have direct access to individual I/O devices. However, when SR-IOV is not an option, consider creating redundant I/O domains. See Countermeasure: Granularly Segregating Service Domains.
An I/O domain has direct access to back-end devices, usually disks, which it virtualizes and then offers to guest domains. A successful attacker has full access to these devices and can read sensitive data or manipulate software on the boot disks of the guest domains.
An I/O domain attack is as likely as a successful attack on a service domain or the control domain. The I/O domain is an attractive target given the potential access to a large number of disk devices. Therefore, consider this threat when dealing with sensitive data in a guest domain that runs on virtualized disks.
When an I/O domain is compromised, the attacker has full access to the guest domain's virtual disks.
Protect the contents of the virtual disks by doing the following:
Encrypting the contents of the virtual disks. On Oracle Solaris 10 systems, you might use an application that can encrypt its own data, such as pgp/gpg or Oracle 11g encrypted tablespaces. On Oracle Solaris 11 systems, you might use ZFS encrypted datasets to provide transparent encryption of all data stored in the file system.
Distributing the data over several virtual disks across different I/O domains. A guest domain might create a striped (RAID 1/RAID 5) volume that stripes over several virtual disks that are obtained from two I/O domains. If one of these I/O domains is compromised, the attacker would have difficulty making use of the portion of the data that is available.
While guest domains are not part of the execution environment, they are the most likely target for an attack because they are connected to the network. An attacker who breaches a virtualized system can launch attacks on the execution environment.
The operating system on the guest domain is often the first line of defense against any attack. With the exception of attacks that originate within the datacenter, an attacker must break into a guest domain that has external connections before attempting to break guest domain isolation and capture the complete environment. Therefore, you must harden the guest domain's OS.
To further harden the OS, you can deploy your application in Solaris Zones, which place an additional layer of isolation between the application's network service and the operating system of the guest domain. A successful attack on the service compromises only the zone and not the underlying operating system, which prevents the attacker from expanding control beyond the resources that are allocated to the zone. As a result, eventually breaking guest isolation is more difficult. For more information about securing the guest OS, see Oracle Solaris 10 Security Guidelines and Oracle Solaris 11 Security Guidelines.