This section is applicable only for the following customers who have Flash storage:
Customers running Exalytics X3-4, X4-4, X5-4, or X6-4 machines
Customers running Exalytics X2-4 machines with Upgrade kits (Flash Upgrade Kit or Memory and Flash Upgrade Kit
Note:
The following customers can ignore this section:
Oracle Exalytics X2-4 customers who do not have Upgrade kits (Flash Upgrade Kit or Memory and Flash Upgrade Kit).
Oracle Exalytics X3-4 or X2-4 customers with Upgrade kits who have already configured Flash.
To verify that you have already configured Flash, navigate to Oracle VM Manager and confirm that the names of the Flash drives in Oracle VM Manager are the same as the names specified in the FlashDriveMappings file.
This section consists of the following topics:
The Flash Card information for X2-4, X3-4, and X4-4 Exalytics Machine is described in the following section:
Each X2-4 and X3-4 Exalytics Machine contains six 400 GB Flash cards. Each Flash card contains four 100 GB drives, for a total of 24 individual drives with a capacity of 2400 GB.
Each X4-4 Exalytics Machine contains three 800 GB Flash card. Each Flash card contains four 200 GB drives, for a total of 12 individual drives with a capacity of 2400 GB.
You next configure Flash to enable names of the Flash drives on Oracle VM Server to be read by Oracle VM Manager.
Note:
Before Oracle Exalytics X2-4 customers with Flash Upgrade kits can configure Flash, they must have Oracle Field Services engineers install six Flash cards on the Exalytics Machine being used for virtualization.
To configure Flash on Oracle VM Server:
Log on to the Exalytics Machine as the root user.
To create a FlashDriveMappings file, enter the following command:
# /opt/exalytics/bin/exalytics_flashovsrepos.sh -script > ~/flashDriveMappings
The FlashDriveMappings file extracts the names of the Flash drives on Oracle VM Server into a text file.
Navigate to the /root directory to view the text file.
On Oracle VM Server, copy the FlashDriveMappings file to the /opt/exalytics/bin/exalytics_ovm_scripts/ folder and then tar the FlashDriveMappings file by entering the following commands:
# cd /opt/exalytics/bin/
# tar -cvf exalytics_ovm_scripts.tar exalytics_ovm_scripts/
Copy the tar file to the machine where Oracle VM Manager is installed.
Connect to machine where Oracle VM Manager is installed.
Navigate to the directory where you copied the scripts and the FlashDriveMappings file.
To untar the file, enter the following command:
# tar -xvf exalytics_ovm_scripts.tar
To confirm that the "expect" package is available, enter the following command:
# rpm -qa | grep expect
If the "expect" package is not available, install it before proceeding to Step 10.
To rename the Flash drives in Oracle VM Manager, navigate to the exalytics_ovm_scripts folder and enter the following command:
# ./exalytics_ovm_rename_flash.sh <path_to _flashmappingfile> <Oracle VM Manager Username> < Oracle VM Manager Password> <Exalytics_Machine_Configuration>
where Exalytics_Machine_Configuration is x2-4, x3-4, x4-4, x5-4, or x6-4.
The names of the Flash drives in Oracle VM Manager now appears the same as the names specified in the FlashDriveMappings file.
Attach the Flash drives to Oracle VM Manager by performing the following tasks:
In Oracle VM Manager, select the Server and VMs tab.
In the left pane, select the server pool on which the virtual machine resides.
From the Perspective list, select Virtual Machines.
In the Management pane, select a virtual machine.
In the toolbar, click Edit.
The Edit Virtual Machine dialog is displayed.
Click the Disks tab.
Under Disk Type, select Physical Disk.
Under Actions, select Select a Virtual Machine Disk.
Select a Physical Disk page is displayed.
Select the Flash card you want to attach to Oracle VM Guest Machine, and click OK.
Repeat the previous step to attach other Flash cards to Oracle VM Guest Machine.
In Oracle VM Manager, select the Servers and VMs tab.
In the left pane, expand Server Pools, and then select exalytics_server_pool.
From the Perspective list, select Virtual Machines.
In the Management pane, select a virtual machine.
Click the expand arrow to the left of the selected virtual machine.
Click the Disks tab.
Confirm that the Flash drives attached to Oracle VM Guest Machine are displayed.
RAID Level 1 is usually referred to as mirroring. RAID Level 1 duplicates data from one drive on a second drive so that if either drive fails, no data is lost. This is useful when reliability is more important than data storage capacity.
The following procedure assumes you have attached the Flash drives to Oracle VM Manager.
Note:
In this example, RAID1 is configured for two (xvdb and xvdc) Flash drives that are on two different Flash cards. Depending on your configuration, you can configure RAID1 for a different number of Flash drives.
Prerequisite: Download and install the patch 21876196 from My Oracle Support on the OVM Server (Dom0):
Connect as a root user to the Oracle Virtual Machine to which you attached the Flash drives.
To Install the mdadm package onto the OVM Server, complete the following steps:
Download p21876196_20000_Linux-x86-64.zip.
My Oracle Support Patch number: 19405370.
Copy p21876196_20000_Linux-x86-64.zip to the OVM Server temporary (/tmp) directory.
Unzip the p21876196_20000_Linux-x86-64.zip file to /tmp/p21876196.
Enter the following command:
# cd /tmp/p21876196
Enter the following command to install the rpms:
# rpm -Uvh --nosignature /tmp/patch/rpms/*rpm
Restart the OVM server.
To backup and remove RAID1 on the OVM guest:
Backup data on the existing RAID:
Start the OVM Guest.
Connect as a root user.
Stop any running services.
Stop auto start of services, if enabled. The device will not be available when the OVM Guest is started.
Enter the following command:
cd /u02
To backup the data to a location other than the RAID, which has enough storage to hold all the data, enter the following command:
tar -cvf /tmp/u02_data.tar *
To unmount the RAID, enter the following command:
umount /u02
Remove auto mount from /etc/fstab:
To backup the existing /etc/fstab, enter the following command:
cp /etc/fstab /etc/fstab.ori
In /etc/fstab, search for the line containing "/u02".
Note the first column of the line, for example "/dev/md2".
Delete or comment out the line containing "/u02".
Save the /etc/fstab file.
Remove the RAID:
To backup the existing /etc/mdadm.conf, enter the following command:
cp /etc/mdadm.conf /etc/mdadm.conf.ori
In the /etc/mdadm.conf file, search for the line containing "/dev/md2".
Delete or comment out the following lines:
mdadm --stop /dev/md2 mdadm --zero-superblock /dev/xvdb mdamd --zero-superblock /dev/xvdc
Shutdown the OVM guest.
Remove the allocated flash drives from the OVM Guest, if allocated as physical drives:
Login to the OVM manager.
Edit the virtual machine.
Go to the Disk tab.
Find the physical disks that were used for the RAID and change Disk Type to Empty.
Click OK. The changes are saved.
Remove the allocated flash drives from the OVM Guest, if allocated as virtual drives:
Login to the OVM manager.
Edit the virtual machine.
Go to the Disk tab.
Find the virtual disks that were used for the RAID and change Disk Type to Empty.
Click OK. The changes are saved.
Go to the Repositories tab.
Highlight the repository used for the virtual drive and then delete the repository.
Reboot the OVM Server.
To create a RAID1 on the OVM Server and mount the RAID1 on an OVM Guest:
Connect as a root user to the Oracle Virtual Machine to which you attached the Flash drives.
Identify the Flash drives.
For example: xvdb and xvdc
To configure RAID1 for the two Flash drives (xvdb and xvdc), enter the following command:
# /sbin/mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/xvdb /dev/xvdc
To add the RAID1 configuration to the RAID configuration file, enter the following command:
# mdadm --detail --scan --verbose >> /etc/mdadm.conf
This enables the RAID configuration to be recognized when Oracle VM Manager restarts.
To create a file system on the new RAID1 device, enter the following command:
# parted /dev/md2
Find the vm.cfg file which corresponds to the VM guest where the RAID is to be mounted:
Login to the OVM manager.
Select the Repositories tab and select the main repository for your system.
Note the repository ID.
Select the Servers and VMs tab.
Edit the OVM Guest, select the configuration tab, and then note the VM ID.
The vm.cfg file is located inside the directory with the repository ID number and VM ID number.
Add /dev/md2p1 to the vm.cfg file.
Start the OVM guest and mount device xvdb.
Add the following line to /etc/fstab on the OVM guest to remount the device when guest is restarted.
# /dev/xvdb1 /u02 ext3 defaults 0 0
To restore the backup files, if a backup was created for the OVM guest, enter the following commands:
# cd /u02 # tar -xvf /tmp/u02_data.tar
This procedure is valid only for Oracle VM Server 3.3.3 running on X5-4 and X6-4 machines.
Note:
In this example, RAID1 is configured for two Flash drives that are on two different Flash cards. Depending on your configuration, you can configure RAID1 for a different number of Flash drives.
For a guest user, the OVM Manager web interface must not be used for flash configurations, like adding a virtual disk or physical disk.
Prerequisite: To create partitions on the Flash cards, if there are no existing partitions:
Enter the following command:
# parted /dev/nvme0n1 mklabel gpt
Warning: The existing disk label on /dev/nvme0n1 will be destroyed and all data on this disk will be lost. Do you want to continue? Yes/No? Yes Information: You may need to update /etc/fstabm.
Enter the following command:
# parted -a optimal /dev/nvme0n1 mkpart primary 0% 25% mkpart primary 25% 50% mkpart primary 50% 75% mkpart primary 75% 100%
Information: You may need to update /etc/fstab.
To verify the created partitions, enter the following command:
# parted /dev/nvme0n1 print
Model: Unknown (unknown) Disk /dev/nvme0n1: 1600GB Sector size (logical/physical): 512B/512B Partition Table: msdos Number Start End Size Type File system Flags 1 1049kB 400GB 400GB primary 2 400GB 800GB 400GB primary 3 800GB 1200GB 400GB primary 4 1200GB 1600GB 400GB primary
Repeat steps 1 through 3 for all the flash drives. That is, for /dev/nvme1n1 and /dev/nvme2n1 to create four partitions on every flash drive.
To configure RAID1 for Flash in Oracle VM Server:
Connect as a root user to the Oracle VM Server to which you attached the Flash drives.
Identify the Flash drives.
Select flash drives from two different flash controllers. For example, the flash cards are displayed as nvme1n1p1, nvme2n1p1. The values nvme1 and nvme2 indicate the controller. Select nvme1n1p1 and nvme2n1p1 for the RAID configuration.
To configure RAID1 for the two Flash drives (nvme1n1p1 and nvme2n1p1), enter the following command:
# /sbin/mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/nvme1n1p1 /dev/nvme2n1p1
mdadm: /dev/nvme1n1p1 appears to be part of a raid array: level=raid1 devices=3 ctime=Tue Nov 10 11:15:30 2015 mdadm: Note: this array has metadata at the start and may not be suitable as a boot device. If you plan to store '/boot' on this device please ensure that your boot-loader understands md/v1.x metadata, or use --metadata=0.90 mdadm: /dev/nvme2n1p1 appears to be part of a raid array: level=raid1 devices=3 ctime=Tue Nov 10 11:15:30 2015 Continue creating array? yes mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md1 started.
To add the RAID1 configuration to the RAID configuration file, enter the following command:
# mdadm --detail --scan >> /etc/mdadm.conf
This enables the RAID configuration to be recognized when Oracle VM Server restarts.
Edit /etc/mdadm.conf file and apply following changes:
Remove the line with the entry: # DEVICE /no/device
Add the following entries for nvme devices:
DEVICE /dev/nvme0n1p* /dev/nvme1n1p* /dev/nvme2n1p*
Create a partition on the new RAID1 device by entering the following command:
# parted /dev/md1
(parted) mklabel msdos (parted) mkpart primary 1MiB -1MiB (parted) unit s print (to validate partition was created correctly) (parted) quit
The partition can range from 1 MiB to any GB value based on user needs.
Note:
The command creates the /dev/md1p1 directory.
Find the vm.cfg file which corresponds to the VM Guest on which you want the RAID device to be mounted:
Go to Oracle VM Manager.
Select the Repositories tab and select the main repository for your system.
Note the repository ID.
Select the Servers and VMs tab.
Edit the OVM Guest, select the Configuration tab, and note the VM ID.
The vm.cfg file will be located under the directory with the repository ID number and VM ID number. It appears as /OVS/Repositories/<repository ID number>/VirtualMachines/VM ID number/vm.cfg.
Add the newly created partition md1p1 to vm.cfg. This will create /dev/xvdb on the VM guest when the guest is started.
Note:
Steps a through c below show a sample repository ID number and VM ID number. Your repository ID and VM ID number will be different and you will need to substitute the ID for your specific VM in the sample commands given as follows.
Save the original /OVS/Repositories/0004fb0000030000b25b4b2848c959f4/VirtualMachines/0004fb000006000043ee01db14c067f2/vm.cfg to vm.ori.
Edit the vm.cfg file using the following command:
vi /OVS/Repositories/0004fb0000030000b25b4b2848c959f4/VirtualMachines/0004fb000006000043ee01db14c067f2/vm.cfg
If a virtual disk is previously allocated, add or modify the following line (with highlighted text):
disk = ['file:/OVS/Repositories/0004fb0000030000b25b4b2848c959f4/VirtualDisks/0004fb0000120000afd68c2d37f266a2.img,xvda,w', 'phy:/dev/md1p1,xvdb,w']
If no other disk is previously allocated, add the following line:
disk = [ 'phy:/dev/md1p1,xvdb,w']
Save a copy of the modified vm.cfg by entering the following command:
cp vm.cfg vm.cfg_PostFlash
Start the Oracle VM Guest and mount the device xvdb:
Login to Oracle VM Guest as root user.
Create a partition using the following command:
parted /dev/xvdb (parted) mklabel msdos (parted) mkpart primary 0 -1MiB (parted) quit
Format the partition using the following command:
mkfs -t ext3 /dev/xvdb1
Create a new directory (/u02) by entering the following command, if it does not exist:
# mkdir -p /u02
Mount the RAID1 on this directory by entering the following command:
# mount /dev/md2 /u02
Update the /etc/fstab on Oracle VM Guest to remount the device when guest is rebooted:
Edit the /etc/fstab on Oracle VM Guest:
vi /etc/fstab
Add the following line:
/dev/xvdb1/u02 ext3 defaults 0 0
Note:
To attach another RAID device or increase the disk space for a guest user, repeat steps 1 through 10 for the different RAID devices.
For information about replacing a defective Flash Card on an X5-4 and X6-4 Exalytics Machine, see Replacing a Defective Flash Card on an X4-4 Exalytics Machine..
This procedure is valid only for Oracle VM Server 3.4.3 running on X5-4 and X6-4 machines.
Note:
In this example, RAID1 is configured for two Flash drives that are on two different Flash cards. Depending on your configuration, you can configure RAID1 for a different number of Flash drives.
For a guest user, the OVM Manager web interface must not be used for flash configurations, like adding a virtual disk or physical disk.
Prerequisite: To create partitions on the Flash cards, if there are no existing partitions:
Enter the following command:
# parted /dev/nvme0n1 mklabel gpt
Warning: The existing disk label on /dev/nvme0n1 will be destroyed and all data on this disk will be lost. Do you want to continue? Yes/No? Yes Information: You may need to update /etc/fstab.
Enter the following command:
# parted /dev/nvme0n1
Information: You may need to update /etc/fstab.
To verify the created partitions, enter the following command:
# parted /dev/nvme0n1 print
Model: Unknown (unknown) Disk /dev/nvme0n1: 3201GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size Type File system Flags 1 1049kB 800GB 800GB primary 2 800GB 1600GB 800GB primary 3 1600GB 2400GB 800GB primary 4 2400GB 3201GB 800GB primary
Repeat steps 1 through 3 for all the flash drives. That is, for /dev/nvme1n1 and /dev/nvme2n1 to create four partitions on every flash drive.
To configure RAID1 for Flash in Oracle VM Server:
Connect as a root user to the Oracle VM Server to which you attached the Flash drives.
Identify the Flash drives.
Select flash drives from two different flash controllers. For example, the flash cards are displayed as /dev/nvme1n1p1, /dev/nvme2n1p1. The values nvme1 and nvme2 indicate the controller. Select nvme1n1p1 and nvme2n1p1 for the RAID configuration.
To configure RAID1 for the two Flash drives (nvme1n1p1 and nvme2n1p1), enter the following command:
# /sbin/mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/nvme1n1p1 /dev/nvme2n1p1
mdadm: /dev/nvme1n1p1 appears to be part of a raid array: level=raid1 devices=3 ctime=Tue Nov 10 11:15:30 2015 mdadm: Note: this array has metadata at the start and may not be suitable as a boot device. If you plan to store '/boot' on this device please ensure that your boot-loader understands md/v1.x metadata, or use --metadata=0.90 mdadm: /dev/nvme2n1p1 appears to be part of a raid array: level=raid1 devices=3 ctime=Tue Nov 10 11:15:30 2015 Continue creating array? yes mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md1 started.
To add the RAID1 configuration to the RAID configuration file, enter the following command:
# mdadm --detail --scan >> /etc/mdadm.conf
This enables the RAID configuration to be recognized when Oracle VM Server restarts.
Edit /etc/mdadm.conf file and apply following changes:
Remove the line with the entry: # DEVICE /no/device
Add the following entries for nvme devices:
DEVICE /dev/nvme0n1p* /dev/nvme1n1p* /dev/nvme2n1p*
Create a partition on the new RAID1 device by entering the following command:
# parted /dev/md1
(parted) mklabel msdos (parted) mkpart primary 1MiB -1MiB (parted) unit s print (to validate partition was created correctly) (parted) quit
The partition can range from 1 MiB to any GB value based on user needs.
Note:
The command creates the /dev/md1p1 directory.
Find the vm.cfg file which corresponds to the VM Guest on which you want the RAID device to be mounted:
Go to Oracle VM Manager.
Select the Repositories tab and select the main repository for your system.
Note the repository ID.
Select the Servers and VMs tab.
Edit the OVM Guest, select the Configuration tab, and note the VM ID.
The vm.cfg file will be located under the directory with the repository ID number and VM ID number. It appears as /OVS/Repositories/<repository ID number>/VirtualMachines/<VM ID number>/vm.cfg.
Add the newly created partition md1p1 to vm.cfg. This will create /dev/xvdb on the VM guest when the guest is started.
Note:
Steps a through c below show a sample repository ID number and VM ID number. Your repository ID and VM ID number will be different and you will need to substitute the ID for your specific VM in the sample commands given as follows.
Save the original /OVS/Repositories/0004fb0000030000b25b4b2848c959f4/VirtualMachines/0004fb000006000043ee01db14c067f2/vm.cfg to vm.ori.
Edit the vm.cfg file using the following command:
vi /OVS/Repositories/0004fb0000030000b25b4b2848c959f4/VirtualMachines/0004fb000006000043ee01db14c067f2/vm.cfg
If a virtual disk is previously allocated, add or modify the following line (with highlighted text):
disk = ['file:/OVS/Repositories/0004fb0000030000b25b4b2848c959f4/VirtualDisks/0004fb0000120000afd68c2d37f266a2.img,xvda,w', 'phy:/dev/md1p1,xvdb,w']
If no other disk is previously allocated, add the following line:
disk = [ 'phy:/dev/md1p1,xvdb,w']
Save a copy of the modified vm.cfg by entering the following command:
cp vm.cfg vm.cfg_PostFlash
Start the Oracle VM Guest and mount the device xvdb:
Login to Oracle VM Guest as root user.
Create a partition using the following command:
parted /dev/xvdb (parted) mklabel msdos (parted) mkpart primary 0 -1MiB (parted) quit
Format the partition using the following command:
mkfs -t ext3 /dev/xvdb1
Create a new directory (/u02) by entering the following command, if it does not exist:
# mkdir -p /u02
Mount the RAID1 on this directory by entering the following command:
# mount /dev/xvdb1 /u02
Update the /etc/fstab on Oracle VM Guest to remount the device when guest is rebooted:
Edit the /etc/fstab on Oracle VM Guest:
vi /etc/fstab
Add the following line:
/dev/xvdb1/u02 ext3 defaults 0 0
Note:
To attach another RAID device or increase the disk space for a guest user, repeat steps 1 through 10 for the different RAID devices.
For information about replacing a defective Flash Card on an X5-4 and X6-4 Exalytics Machine, see Replacing a Defective Flash Card on an X5-4 or X6-4 Exalytics Machine..