This chapter describes how to create Exalogic Virtual Servers (vServers).
It contains the following sections:
Before starting an Exalogic Deployment ensure that the following tasks have been performed:
Exalogic rack has been commissioned and one-command run.
Accounts have been created in Exalogic Control.
Private IPoIB network has been created for the account, enabling secure communications between the Virtual Servers assigned to the Account as described in Section 6.4.2, "Creating a Private IPoIB Network,"
You have created and loaded a Server Template for the operating system you wish to deploy.
You have created a vServer Type which matches the specification of the Virtual Servers you want to create.
A Client Access Network has been created, using a bonded Network Interface for communication between the vServers and an External Load Balancer.
A Distribution group prevents virtual servers assigned to it from running on the same physical nodes. By preventing different vServers of the same type running on the same physical server, you prevent the failure of the underlying physical server from taking out the complete system.
In an IAM Exalogic implementation, three Distribution Groups are required:
IAM_OTD: Prevents two Oracle Traffic Director Servers from running on the same Physical Server
IAM_IAD: Prevents two IAMAccessDomain Servers from running on the same Physical Server
IAM_IAG: Prevents two IAMGovernanceDomain Servers from running on the same physical server
To Create a distribution group perform the following steps:
Log in to Exalogic Control at the URL listed in Section 20.2, "About Identity and Access Management Console URLs."
Expand vDC Management.
Navigate to vDCs - Accounts - Cloud Admin Account.
In the actions window click Create Distribution Group.
Enter a Name, for example: IAM_OTD.
Click Next.
Enter Number of Elements.
This is a number that defines the number of Oracle VM Servers on which the vServers can be placed. For example, where OTD can run on two vServers, then the number of elements is 2.
Click Next.
Click Finish.
Repeat for each Distribution Group to be created. Table 8-1 lists Distribution Groups and the number of elements for each.
The vServerTypes are based on the supplied Exalogic vServerTypes. Depending on your load you may need to increase the size of the default template requirements.
| Name | vServerType | Virtual Networks | Host Name | Distribution Group |
|---|---|---|---|---|
|
webhost1 |
LARGE |
IPoIB-IAMFoot 1 EoIB-clientFoot 2 IPoIB-vserver-shared-storageFoot 3 |
webhost1 webhost1-ext webhost1-stor |
IAM_OTD |
|
webhost2 |
LARGE |
IPoIB-IAM EoIB-client IPoIB-vserver-shared-storage |
webhost2 webhost2-ext webhost2-stor |
IAM_OTD |
|
oamhost1 |
EXTRA_LARGE |
IPoIB-IAM EoIB-client IPoIB-vserver-shared-storage |
oamhost1 oamhost1-ext oamhost1-stor |
IAM_IAD |
|
oamhost2 |
EXTRA_LARGE |
IPoIB-IAM EoIB-client IPoIB-vserver-shared-storage |
oamhost2 oamhost2-ext oamhost2-stor |
IAM_IAD |
|
oimhost1 |
EXTRA_LARGE |
IPoIB-IAM EoIB-client IPoIB-vserver-shared-storage |
oimhost1 oimhost1-ext oimhost1-stor |
IAM_IAG |
|
oimhost2 |
EXTRA_LARGE |
IPoIB-IAM EoIB-client IPoIB-vserver-shared-storage |
oimhost2 oimhost2-ext oimhost2-stor |
IAM_IAG |
Footnote 1 IPoIB-IAM is the internal IPoIB network used for inter vServer communication
Footnote 2 EoIB-client is the Client Access Network which connects to the corporate ethernet
Footnote 3 IPoIB-vserver-shared-storage is the internal network that vServers use to communicate with the ZFS storage appliance.
If your database is on an Exadata machine, you must also include the virtual Network IPoIB-default.
Note:
If are planning to use OAAM in your topology, add extra memory to the virtual servers hosting OAM. The extra memory should be at least an additional 4GB.When you create a vServer, by default, it creates one default volume and allocates the space to swap and the root file system. For a more efficient controlled way to do this, create separate volumes for each vServer to mount for the swap and temp space.
To create separate volumes for each vServer:
Log in to Exalogic Control at the URL listed in Section 20.2, "About Identity and Access Management Console URLs."
Expand vDC Management.
Navigate to vDCs, Accounts, and then Cloud Admin Account.
Select Create Volume from the Actions menu.
Give the volume a name, for example vServer1_tmp, and a description.
Click Next.
On the Volume Configuration screen, enter a size for the volume.
Do not select shared.
Click Next.
On the Volume Summary screen, click Finish to create the volume.
Repeat for each volume to be created.
Table 8-3 lists the vServer types used in this document. These vServer types can be used as a guide. Refer to the Oracle® Fusion Middleware System Requirements and Specifications for Oracle Identity and Access Management for the latest hardware requirements.
To Create a vServer perform the following steps. Refer to Table 8-2, "vServer Information" for data values.
Log in to Exalogic Control at the URL listed in Section 20.2, "About Identity and Access Management Console URLs."
Expand vDC Management.
Navigate to vDCs - Accounts - Cloud Admin Account.
In the Actions window, click Create vServer.
Enter:
Name: For example: otdhost1
Number of vServers:1
Select: Support High Availability
Click Next.
Choose the Server Template you want to deploy.
Click Next.
Choose the vServer Type you wish to create, for example: LARGE
If you have created volumes for swap and tmp, select them here.
Click Next.
Enter all of the virtual networks you want to assign
For each chosen network enter the following:
IP Address Type - Static or Automatic
IP Address - Enter the IP address if you have a predetermined IP address to use.
Hostname - Select the fully qualified host name you wish to assign to the IP address.
Click Next.
Enter the Distribution Group to use.
Click Next.
Click Next on vServerAccessControl screen.
Click Finish.
Repeat for Each vServer to be created.
Note:
Make sure that each vServer has the swap space detailed in Section 9.4, "Meeting Operating System Requirements."Now that the vServers have been created you need to perform the steps in the following sections to make them available for use.
When the vServer is created, it has a default password which is generally ovsroot. Change this to a value appropriate to your organization.
After configuration, your hosts file will look something like:
IP Address Host_Name
For example:
192.168.32.3 oamhost1-stor
Change the hosts file so that it contains both fully qualified and short names for each network, for example:
192.168.10.3 oamhost1.mycompany.com oamhost1 192.168.32.3 oamhost1-stor.mycompany.com oamhost1-stor 192.168.10.3 oamhost1-data.mycompany.com oamhost1-data
Notes:
For clarity the host name of the default network has been changed to -data to show it is the network that is used for Exadata communication.
External Network interface names are assumed to be in DNS.
Now that your vServer has been created you must configure it as appropriate to your organization. This typically includes the following steps:
When you created your vServer, you added the network IPoIB-vserver-shared-storage. This is the network the vServers use to communicate with the ZFS storage appliance. In order for them to communicate properly, you must determine the appropriate IP address of the storage appliance to use.
To determine the IP address perform the following steps:
Log in to Exalogic Control as a Cloud User.
From the navigation pane on the left, select vDC Management.
Under vDC Accounts, expand the name of your account, and select the vServer for which you want to configure access to the storage appliance.
The vServer dashboard is displayed.
Select the Network tab, and note the IP address of the vServer for the IPoIB-vserver-shared-storage network. This corresponds with the -stor entry in the /etc/hosts file
For example: 172.17.0.100
Log in to the storage appliance as root. For example, type:
ssh root@exalogicsn01.mycompany.com
Show the network interfaces using the command:
configuration net interfaces show
The output is similar to the following:
configuration net interfaces show
Interfaces:
INTERFACE STATE CLASS LINKS ADDRS LABEL
igb0 up ip igb0 10.244.64.60/21 igb0
igb1 offline ip igb1 10.244.64.61/21 igb1
ipmp1 up ipmp pffff_ibp1 192.168.10.15/24 ipmp1
pffff_ibp0
ipmp2 up ipmp p8001_ibp0 192.168.20.9/24 IB_IF_8001
p8001_ibp1
ipmp3 up ipmp p8002_ibp0 192.168.21.9/24 IB_IF_8002
p8002_ibp1
ipmp4 up ipmp p8005_ibp0 172.17.0.9/16 IB_IF_8005
p8005_ibp1
p8001_ibp0 up ip p8001_ibp0 0.0.0.0/8 ibp0.8001
p8001_ibp1 up ip p8001_ibp1 0.0.0.0/8 ibp1.8001
p8002_ibp0 up ip p8002_ibp0 0.0.0.0/8 ibp0.8002
p8002_ibp1 up ip p8002_ibp1 0.0.0.0/8 ibp1.8002
p8005_ibp0 up ip p8005_ibp0 0.0.0.0/8 ibp0.8005
p8005_ibp1 up ip p8005_ibp1 0.0.0.0/8 ibp1.8005
pffff_ibp0 up ip pffff_ibp0 0.0.0.0/8 ibp0
pffff_ibp1 up ip pffff_ibp1 0.0.0.0/8 ibp1
Determine the corresponding IP address by looking for the IP address in the same range as 172.17.0.100. In this example it is the one associated with interface ipmp4, for example: 172.17.0.9.
Create an entry in the /etc/hosts file to reflect this, for example:
172.17.0.9 zfsinternal.mycompany.com zfsinternal
In order to maintain optimum performance, you must update the MTU size of each of the InfiniBand interfaces on the vServer to 65520. To do this perform the following steps:
Log in to the vServer as the root user.
Verify the current MTU for bond2 by running the ifconfig command.
Note:
The steps in this procedure usebond2 as an example. This procedure should be repeated for all the InfiniBand interfaces.ifconfig bond2 bond2 Link encap:InfiniBand HWaddr 80:58:08:CA:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:192.168.1.12 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:9 errors:0 dropped:0 overruns:0 frame:0 TX packets:7 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:504 (504.0 b) TX bytes:420 (420.0 b)
Append the line MTU=64000 to the ifcfg file corresponding to the bond2 interface:
echo MTU=64000 >> /etc/sysconfig/network-scripts/ifcfg-bond2
Verify whether the MTU=64000 line was added to the ifcfg-bond2 file:
cat /etc/sysconfig/network-scripts/ifcfg-bond2 | grep MTU
MTU=64000
Find the slave interfaces for bond2:
cd /etc/sysconfig/network-scripts
grep "MASTER=bond2" ifcfg-* | awk -F":" '{print $1}'
ifcfg-ib0.8009
ifcfg-ib1.8009
Set the mode to connected for both the slave interfaces of the bond2 interface:
echo connected > /sys/class/net/ib0.8009/mode echo connected > /sys/class/net/ib1.8009/mode
Perform Steps 2 through 6 for the other InfiniBand interfaces.
Stop and start the vServer as described in Section 20.1.4, "Stopping and Starting vServers."
After the vServer starts, log in again to the vServer using SSH as the root user.
Run the ifconfig command for each InfiniBand interface, and verify that the output of the command displays MTU:64000, as shown in the following example for bond2 and its slave interfaces:
ifconfig bond2 | grep MTU UP BROADCAST RUNNING MASTER MULTICAST MTU:64000 Metric:1 ifconfig ib0.8009 | grep MTU UP BROADCAST RUNNING SLAVE MULTICAST MTU:64000 Metric:1 ifconfig ib1.8009 | grep MTU UP BROADCAST RUNNING SLAVE MULTICAST MTU:64000 Metric:1
If you create separate disk volumes for swap and tmp, update your vServer to use these new volumes.
The disk volumes are added to your virtual server as virtual volumes. They appear in the /dev directory as xvdb/c.
To determine the exact names, run the following command:
fdisk -l
This command shows output similar to:
255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000c520c Device Boot Start End Blocks Id System /dev/xvda1 * 1 32 256000 83 Linux Partition 1 does not end on cylinder boundary. /dev/xvda2 32 1305 10223616 8e Linux LVM Disk /dev/xvdb: 18.3 GB, 18253611008 bytes 255 heads, 63 sectors/track, 2219 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/xvdc: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000
As you can see from the output, /dev/xvda has a partition created on it so it is in use. This is the default disk.
Disks /dev/xvdb and /dev/xvdc do not have a partition and are therefore the attached disk volumes. You can determine which is which by the size of the volumes.
To use these volumes in the vServer, create a partition of type LVM. This enables the use of Linux LVM, and make resizing easier if required later. The procedure is the same if you are using the disk for swap or tmp.
To create an LVM partition:
Choose a disk to work on using the following command:
fdisk disk_name
For example:
fdisk /dev/xvdb
When prompted for a command type n
You are asked if you wish to create an extended or primary partition. Select p for primary.
When promoted for a partition number, enter 1.
You are then asked where on the disk to create the partition. Accept the Default from value of 1. Accept the default end value to use the entire disk.
Now that the partition has been created, give it a type. To do this, when prompted for a command, enter t.
You can see the list of types available by entering the command L.
When prompted for the Hex code, enter the code (from the previous list) for the Linux LVM. This is typically 8e.
Save your changes using the command w.
Validate that the changes are correct using the command fdisk -l
Repeat the procedure for each disk volume
Now that you have disk partitions, create logical volumes to use those disks:
Create a physical volume on the disk partition by using the command:
pvcreate disk_partition
For example
pvcreate /dev/xvdb1
Note:
The number1 at the end of the disk, which denotes the partition number, is the same as the values you saw in the fdisk -l command.Repeat for each disk partition you created above.
Verify that the physical volumes have been created correctly using the following command
pvdisplay
Create a volume group, one for each virtual disk. You can create a single volume group for all disks, but this example uses one per disk.
To create a volume group, use the following command:
vgcreate volume_group_name disk partition
For example:
vgcreate volGroupSwap /dev/xvdb1
Repeat for each volume group. For example: volGroupTemp volGroupSwap.
Validate that the volume groups have been created properly using the following:
vgdisplay
Once you've created the volume groups, create a logical volume inside the volume group using the following command:
lvcreate --name lvname --size 40G volume_group
size is the size of space you wish to assign to the volume group. This equates to the size of the file system.
For example
lvcreate ---name Swap1 ---size 16G volGroupSwap
Repeat for each logical volume to be created.
Validate that the logical volumes were created successful using the following command:
lvdisplay
Creating a Swap File on the New Logical Volume
To use a logical volume for swap:
Create a swapfile using the following command
mkswap volume_group
For example:
mkswap /dev/volGroupSwap/Swap1
Create an entry in the /etc/fstab directory for the new swap file. The entry will look similar to:
/dev/volGroupSwap/Swap1 swap swap defaults 0 0
Comment out the original swap entry.
Validate that the new swap space is being used by issuing the command swapon -s
You can disable the original swap using the following command:
swapoff
Note:
This is not necessary as only your new swap space will be available after a reboot.Moving /tmp to the New Logical Volume
If you have created a logical volume for /tmp, you can enable this by first creating a file system on it and then mounting it as a disk. You do this by performing the following commands:
Create a file system using the command:
mkfs.ext3 volume_name
For example:
mkfs.ext3 /dev/volGroupTemp/Temp1
Add the new file system to /etc/fstab so that it is automatically mounted.
Create an entry similar to:
/dev/volGroupTemp/Temp1 /tmp ext3 defaults 1 1
Mount the file system using the following command
mount -a
Verify that the file system is created correctly using the following command:
df -k