Managing Storage

10 Managing Storage

Understand the storage options and how to manage storage for your Oracle Database Appliance deployment.

About Managing Storage
You can add storage at any time without shutting down your databases or applications.
Storage on Single Node Platforms
Review storage options on Oracle Database Appliance single node platforms.
Storage on Multi Node Platforms
Review storage options on Oracle Database Appliance multi node platforms.

About Managing Storage

You can add storage at any time without shutting down your databases or applications.

Oracle Database Appliance uses raw storage to protect data in the following ways:

Fast Recovery Area (FRA) backup. FRA is a storage area (directory on disk or Oracle ASM diskgroup) that contains redo logs, control file, archived logs, backup pieces and copies, and flashback logs.
Mirroring. Double or triple mirroring provides protection against mechanical issues.

The amount of available storage is determined by the location of the FRA backup (external or internal) and if double or triple mirroring is used. External NFS storage is supported for online backups, data staging, or additional database files.

Oracle Database Appliance X7-2M and X7-2-HA models provide storage expansion options from the base configuration. In addition, Oracle Database Appliance X7-2-HA multi-node platforms have an optional storage expansion shelf.

When you add storage, Oracle Automatic Storage Management (Oracle ASM) automatically rebalances the data across all of the storage including the new drives. Rebalancing a disk group moves data between disks to ensure that every file is evenly spread across all of the disks in a disk group and all of the disks are evenly filled to the same percentage. Oracle ASM automatically initiates a rebalance after storage configuration changes, such as when you add disks.

The redundancy level for FLASH is based on the DATA and RECO selection. If you choose High redundancy (triple mirroring), then FLASH is also High redudancy.

WARNING:

Pulling a drive before powering it off will crash the kernel, which can lead to data corruption. Do not pull the drive when the LED is an amber or green color. When you need to replace an NVMe drive, use the software to power off the drive before pulling the drive from the slot. If you have more than one disk to replace, complete the replacement of one disk before starting replacement of the next disk.

See “Adding Optional Oracle Database Appliance X7-2-HA Storage Shelf Drives (CRU)” in the Oracle Database Appliance Service Manual for disk placement.

Related Topics

Adding Optional Oracle Database Appliance X7-2-HA Storage Shelf Drives (CRU)

Parent topic: Managing Storage

Storage on Single Node Platforms

Review storage options on Oracle Database Appliance single node platforms.

Storage Options for Single Node Systems
Oracle Database Appliance X7-2S and X7-2M have NVMe storage configurations with storage expansion options.
Adding NVMe Storage Disks
Depending on the available drives, you can expand Oracle Database Appliance X7-2M storage to add NVMe disks or replace existing NVMe disks.

Parent topic: Managing Storage

Storage Options for Single Node Systems

Oracle Database Appliance X7-2S and X7-2M have NVMe storage configurations with storage expansion options.

Table 10-1 Storage Options for Oracle Database Appliance X7-2S and X7-2M

Configuration	Oracle Database Appliance X7-2S	Oracle Database Appliance X7-2M
Base Configuration	Two (2) 6.4TB NVMe drives populated in slots 0 and 1.	Two (2) 6.4TB NVMe drives populated in slots 0 and 1.
Expansion Options	None	Options: Three (3) 6.4TB NVMe drives populated in slots 2 to 4. Order Qty 1 - 7117431 (3-pack 6.4TB NVMe SSD) and upgrade to Oracle Database Appliance release 12.2.1.4 or later. Six (6) 6.4TB NVMe drives populated in slots 2 to 7. Order Qty 2 - 7117431 (3-pack 6.4TB NVMe SSD) and upgrade to Oracle Database Appliance release 12.2.1.4 or later.

Configuration

Oracle Database Appliance X7-2S

Oracle Database Appliance X7-2M

Base Configuration

Two (2) 6.4TB NVMe drives populated in slots 0 and 1.

Expansion Options

None

Options:

Three (3) 6.4TB NVMe drives populated in slots 2 to 4. Order Qty 1 - 7117431 (3-pack 6.4TB NVMe SSD) and upgrade to Oracle Database Appliance release 12.2.1.4 or later.
Six (6) 6.4TB NVMe drives populated in slots 2 to 7. Order Qty 2 - 7117431 (3-pack 6.4TB NVMe SSD) and upgrade to Oracle Database Appliance release 12.2.1.4 or later.

Parent topic: Storage on Single Node Platforms

Adding NVMe Storage Disks

Depending on the available drives, you can expand Oracle Database Appliance X7-2M storage to add NVMe disks or replace existing NVMe disks.

Use the ODAADMCLI commands to perform appliance storage maintenance tasks, including perform storage diagnostics and collect diagnostic logs for storage components.

Preparing for a Storage Upgrade

Update Oracle Database Appliance to the latest Patch Bundle before expanding storage.
```
# odacli describe-component 
```
Check the disk health of the existing storage disks.

Use the default checks option to check the NetworkComponents, OSDiskStorage, SharedStorage, and SystemComponents
```
# odaadmcli validate -d
```
Run the odaadmcli show diskgroup command to display and review Oracle Automatic Storage Management (Oracle ASM) disk group information.
Use orachk to confirm Oracle ASM and CRS health.

Review and perform these best practices before adding storage.

Adding NVMe Storage Disks

The default configuration for Oracle Database Appliance X7-2S or X7-2M includes two (2) NVMe disks. You cannot expand storage for Oracle Database Appliance X7-2S.

For Oracle Database Appliance X7-2M, you can expand storage by adding three (3) additional disks for a total of five (5) NVMe disks or by adding six (6) additional disks for a total of eight (8) NVMe disks. When you expand storage, adding just one or two NVMe drives is not supported.

WARNING:

Follow these steps to add NVMe storage disks:

Before adding the NVMe disks, ensure that the current disks are online in oakd and Oracle ASM. Otherwise, the prechecks fail. For example, for 3-disks expansion from slots 2 to 4, the disks in slots 0 and 1 must be online in Oracle ASM and oakd. For 3-disks expansion from slots 5 to 7 when slots 0 to 4 are filled, then all disks in slots 0 to 4 must be online. For 6-disks expansion from slots 2 to 7, all disks in slots 0 and 1 must be online.
Insert disks one at a time in the slots and power on the device.
For example, to add three (3) NVMe disks, insert the disks in slots 2 to 4. To add six (6) NVMe drives, insert the disks in slots 2 to 7.
```
# odaadmcli power disk on slot_number
```
For example, when adding three (3) NVMe disks:
```
# odaadmcli power disk on pd_02 
# odaadmcli power disk on pd_03
# odaadmcli power disk on pd_04
```
Allow at least one minute between inserting each disk.
Run the odaadmcli expand storage command to add the new storage drives:
```
# odaadmcli expand storage -ndisk number_of_disks
```
For example, to add three (3) NVMe drives:
```
#odaadmcli expand storage -ndisk 3
Precheck passed. 
Check the progress of expansion of storage by executing 'odaadmcli show disk' 
Waiting for expansion to finish ...
```
Run the odaadmcli show disk command to ensure that all disks are listed, are online, and are in a good state.
```
# odaadmcli show disk
```

Parent topic: Storage on Single Node Platforms

Storage on Multi Node Platforms

Review storage options on Oracle Database Appliance multi node platforms.

About Expanding Storage on Multi-Node Systems
Oracle Database Appliance X7-2-HA platforms have options for high performance and high capacity storage configurations.
Preparing for a Storage Upgrade
Review and perform these best practices before adding storage to the base shelf or adding the expansion shelf.
Preparing for a Storage Upgrade for a Virtualized Platform
Review and perform these best practices before adding storage to the base shelf or adding the expansion shelf.
Adding Solid-State Drives (SSDs) for Data Storage
Add a pack of solid-state drives (SSDs) for data storage into the existing Oracle Database Appliance X7-2-HA base configuration to fully populate the base storage shelf.
Adding the Storage Expansion Shelf
After the base storage shelf is fully populated, you can add the storage expansion shelf to expand your data storage on your multi-node platform.

Parent topic: Managing Storage

About Expanding Storage on Multi-Node Systems

Oracle Database Appliance X7-2-HA platforms have options for high performance and high capacity storage configurations.

The base configuration has 16 TB SSD raw storage for DATA and 3.2 TB SSD raw storage for REDO, leaving 15 available slots to expand the storage. If you choose to expand the storage, you can fill the 15 slots with either SSD or HDD drives. For even more storage, you can add a storage expansion shelf to double the storage capacity of your appliance.

In all configurations, the base storage and the storage expansion shelf each have four (4) 800 GB SSDs for REDO disk group and five (5) 3.2TB SSDs (either for DATA/RECO in the SSD option or FLASH in the HDD option).

Note:

The base storage shelf must be fully populated before you can add an expansion shelf and the expansion shelf must have the same storage configuration as the base shelf. Once you select a base configuration, you cannot change the type of storage expansion.

High Performance

A high performance configuration uses solid state drives (SSDs) for DATA and REDO storage. The base configuration has 16 TB SSD raw storage for DATA and 3.2 TB SSD raw storage for REDO.

You can add up to three (3) 5-Pack SSDs on the base configuration, for a total of 64 TB SSD raw storage. If you need more storage, you can double the capacity by adding an expansion shelf of SSD drives. The expansion shelf provides an additional 64 TB SSD raw storage for DATA, 3.2 TB SSD raw storage for REDO, and 16 TB SDD raw storage for FLASH.

Adding an expansion shelf requires that the base storage shelf and expansion shelf are fully populated with SSD drives. When you expand the storage using only SSD, there is no downtime.

Description of the illustration ssd.png

High Capacity

A high capacity configuration uses a combination of SSD and HDD drives.

The base configuration has 16 TB SSD raw storage for DATA and 3.2 TB SSD raw storage for REDO.

The following expansion options are available:

Base shelf: additional 150 TB HDD raw storage for DATA (15 pack of 10 TB HDDs.)
HDD Expansion shelf: additional 150 TB HDD raw storage for DATA, 3.2 TB SSD for REDO, and 16 TB SSD for FLASH
Expansion Storage shelf: additional shelf storage configuration must be identical to the storage configuration of the base shelf. See the topic “Adding the Storage Expansion Shelf” in this chapter, for the procedure to add the storage expansion shelf.

Note:

When you expand storage to include HDD on the base storage shelf, you must reposition the drives to the correct slots and redeploy the appliance after adding the HDD drives.

A system fully configured for high capacity has 300 TB HDD raw storage for DATA, 6.4 TB SSD raw storage for REDO and 32 TB SSD for Flash.

Description of the illustration hdd.png

Table 10-2 Storage Options for Oracle Database Appliance X7-2-HA

Configuration	Oracle Database Appliance X7-2-HA Base Configuration	Oracle Database Appliance X7-2-HA SSD Only Configuration for High Performance	Oracle Database Appliance X7-2-HA SSD and HDD Configuration for High Capacity
Base Configuration	Four (4) 800GB SSDs populated in slots 20 to 23. Five (5) 3.2TB SSDs populated in slots 0 to 4.	JBOD: Four (4) 800GB SSD Twenty (20) 3.2TB SSD	JBOD: Four (4) 800GB SSD Five (5) 3.2TB SSD Fifteen (15) 10TB HDD
Base Shelf Expansion Options	Options: Options to fully populate the base configuration: Up to three 5-packs of 3.2 TB populated in slots 5 to 19 (part number 7117369; quantity 1, 2, or 3) If the base is not fully populated with SSD, you can order 15 x 10TB HDD to the base. (part number 7117704) Note: You must redeploy Oracle Database Appliance if you choose this option.	Up to three 5-packs of 3.2TB SSDs populated in slots 5 to 19. Order Qty 1, 2, or 3: 7117369: Five 3.2TB SSD drive-pack and upgrade to Oracle Database Appliance release 12.2.1.4 or later.	Five (5) 3.2TB SSDs populated in slots 15 to 19. Fifteen 10TB HDDs populated in slots 0 to 14. Order Qty 1: 7117704: Fifteen 10TB HDD drive-pack and upgrade to Oracle Database Appliance release 12.2.1.4 or later.
Storage Expansion Shelf	You can only add an expansion shelf with the same storage configuration as the base configuration	Expansion Shelf with four (4) 800GB SSDs and twenty (20) 3.2TB SSDs. Order Qty 1 - 7119675 - Twenty 3.2 TB SSDs and four 800 GB SSDs and upgrade to Oracle Database Appliance release 12.2.1.4 or later.	Expansion Shelf with four (4) 800GB SSDs, five (5) 3.2TB SSDs and fifteen (15) 10TB HDDs. Order Qty 1 - 7119676 - fifteen 10 TB HDDs, five 3.2 TB SSDs, and four 800 GB SSDs and upgrade to Oracle Database Appliance release 12.2.1.4 or later.

Configuration

Oracle Database Appliance X7-2-HA Base Configuration

Oracle Database Appliance X7-2-HA SSD Only Configuration for High Performance

Oracle Database Appliance X7-2-HA SSD and HDD Configuration for High Capacity

Base Configuration

Four (4) 800GB SSDs populated in slots 20 to 23. Five (5) 3.2TB SSDs populated in slots 0 to 4.

JBOD:

Four (4) 800GB SSD
Twenty (20) 3.2TB SSD

JBOD:

Four (4) 800GB SSD
Five (5) 3.2TB SSD
Fifteen (15) 10TB HDD

Base Shelf Expansion Options

Options:

Options to fully populate the base configuration:
- Up to three 5-packs of 3.2 TB populated in slots 5 to 19 (part number 7117369; quantity 1, 2, or 3)
- If the base is not fully populated with SSD, you can order 15 x 10TB HDD to the base. (part number 7117704)
  
  Note: You must redeploy Oracle Database Appliance if you choose this option.

Up to three 5-packs of 3.2TB SSDs populated in slots 5 to 19. Order Qty 1, 2, or 3: 7117369: Five 3.2TB SSD drive-pack and upgrade to Oracle Database Appliance release 12.2.1.4 or later.

Five (5) 3.2TB SSDs populated in slots 15 to 19. Fifteen 10TB HDDs populated in slots 0 to 14. Order Qty 1: 7117704: Fifteen 10TB HDD drive-pack and upgrade to Oracle Database Appliance release 12.2.1.4 or later.

Storage Expansion Shelf

You can only add an expansion shelf with the same storage configuration as the base configuration

Expansion Shelf with four (4) 800GB SSDs and twenty (20) 3.2TB SSDs. Order Qty 1 - 7119675 - Twenty 3.2 TB SSDs and four 800 GB SSDs and upgrade to Oracle Database Appliance release 12.2.1.4 or later.

Expansion Shelf with four (4) 800GB SSDs, five (5) 3.2TB SSDs and fifteen (15) 10TB HDDs. Order Qty 1 - 7119676 - fifteen 10 TB HDDs, five 3.2 TB SSDs, and four 800 GB SSDs and upgrade to Oracle Database Appliance release 12.2.1.4 or later.

Parent topic: Storage on Multi Node Platforms

Preparing for a Storage Upgrade

Review and perform these best practices before adding storage to the base shelf or adding the expansion shelf.

Update Oracle Database Appliance to the latest Patch Bundle before expanding storage.
Confirm both nodes are at the same version and patch bundle level for software and firmware.
```
# odacli describe-component  
```
Check the disk health of the existing storage disks.

Run the check on both nodes and use the default checks option to check the NetworkComponents, OSDiskStorage, SharedStorage, and SystemComponents.
```
# odaadmcli validate -d
```
Run the odaadmcli show diskgroup command on each node to display and review Oracle Automatic Storage Management (Oracle ASM) disk group information.
```
# odaadmcli show diskgroup DATA
```
```
# odaadmcli show diskgroup RECO
```
```
 # odaadmcli show diskgroup REDO 
```
Confirm Oracle ASM and CRS health on both nodes.
Run orachk on each node. If there is a problem connecting to either node, then check the /etc/bashrc file and remove (or remark out) any values in the profile for root ; oracle ; grid users
Confirm communications between the nodes and that SSH is working using the same password for oracle, root and grid.
From each node:
1. ssh to both nodes.
2. Ping both nodes.
Confirm there is at least 10 GB of space available on each node.
```
[root@oda]# df -h
```
```
[root@odb]# df -h 
```

Parent topic: Storage on Multi Node Platforms

Preparing for a Storage Upgrade for a Virtualized Platform

Review and perform these best practices before adding storage to the base shelf or adding the expansion shelf.

Update Oracle Database Appliance to the latest Patch Bundle before expanding storage.
Confirm both nodes are at the same version and patch bundle level for software and firmware.
```
# oakcli show version -detail 
```
```
#oakcli inventory -q 
```
Note:
If oakd is not running on either node, fix the problem before adding storage.
Check the disk health of the existing storage disks.

Run the check on both nodes and use the default checks option to check the NetworkComponents, OSDiskStorage, SharedStorage, and SystemComponents.
```
# oakcli validate -d
```
Run the command oakcli show diskgroup on each node to display and review Oracle Automatic Storage Management (Oracle ASM) disk group information.
```
# oakcli show diskgroup data
```
```
# oakcli show diskgroup reco
```
```
 # oakcli show diskgroup redo 
```

Confirm Oracle ASM and CRS health on both nodes.

Run the oakcli orachk command on each node. If there is a problem connecting to either node, then check the /etc/bashrc file and remove (or remark out) any values in the profile for root ; oracle ; grid users

Run oakcli orachk on Node 0:

# oakcli orachk
...

Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS

. . . . . . . . .
-------------------------------------------------------------------------------------------------------
Oracle Stack Status
-------------------------------------------------------------------------------------------------------
Host Name CRS Installed  ASM HOME   RDBMS Installed    CRS UP    ASM UP    RDBMS UP DB Instance Name
-------------------------------------------------------------------------------------------------------
odax3rm1       Yes           No          Yes              No        No        No          ........
-------------------------------------------------------------------------------------------------------

 ...

Run oakcli orachk on Node 1:

# oakcli orachk
...

Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
-------------------------------------------------------------------------------------------------------
Oracle Stack Status
-------------------------------------------------------------------------------------------------------
Host Name CRS Installed  ASM HOME   RDBMS Installed    CRS UP    ASM UP    RDBMS UP DB Instance Name
-------------------------------------------------------------------------------------------------------
odax3rm2      Yes           Yes           Yes            Yes       Yes        Yes      b22S2 b23S2 b24S2
-------------------------------------------------------------------------------------------------------

...

Confirm communications between the nodes and that SSH is working using the same password for oracle, root and grid.
From each node:
1. ssh to both nodes.
2. Ping both nodes.
Confirm that there is at least 10 GB of space available on each node.
```
[root@oda]# df -h
```
```
[root@odb]# df -h 
```

Parent topic: Storage on Multi Node Platforms

Adding Solid-State Drives (SSDs) for Data Storage

Add a pack of solid-state drives (SSDs) for data storage into the existing Oracle Database Appliance X7-2-HA base configuration to fully populate the base storage shelf.

If you need to add storage to the base configuration, you can order one, two, or three 5-pack of SSDs to complete the base configuration on Oracle Database Appliance X7-2-HA.

You must fully populate the base configuration before you can add an expansion shelf. If you add an expansion shelf, the shelf must have the same disk storage configuration.

Note:

You can only add SSDs to the base storage shelf, for a high-performance configuration. For high-capacity configuration, you can expand storage to use HDDs, or add a storage expansion shelf.

Before adding the disks to the system, ensure that Oracle Database Appliance is on the latest update version.

Insert disks one at a time in the slots.

To add one 5-pack of SSDs, insert the disks in slots 5 to 9. To add two 5-pack of SSDs, insert the disks in slots 5 to 14. To add three 5-pack of SSDs, insert the disks in slots 5 to 19.

Note:
Allow at least one minute between inserting each disk.

After all disks are added, go to Step 2.
Run the odaadmcli show ismaster command to determine which node is the master.
```
# odaadmcli show ismaster
```

Run the odaadmcli expand storage command on the master node.

#odaadmcli expand storage -ndisk number of disks to be added 
-enclosure enclosure number of the disks to be added, either 0 or 1

For example:


#odaadmcli expand storage -ndisk 5 -enclosure 0
Precheck passed. 
Check the progress of expansion of storage by executing 'odaadmcli show disk' 
Waiting for expansion to finish ...

It takes 10 to 12 minutes to add all of the disks to the configuration.

Run the odaadmcli show disk command to ensure that all disks are listed, are online, and are in a good state.
```
# odaadmcli show disk
```

Verify that the disks in slots 5 to 9 are added to Oracle Automatic Storage Management (Oracle ASM).

Run the asm_script to verify that the disks in slots 5 to 9 are added to Oracle Automatic Storage Management (Oracle ASM). If the 5 disks are successfully added (CACHED and MEMBER), then go to Step 7.

su grid user /opt/oracle/oak/bin/stordiag/asm_script.sh 1 6

For example:

#/opt/oracle/oak/bin/stordiag/asm_script.sh 1 6 | grep CACHED
.......
/dev/mapper/SSD_E0_S05_1399652120p1 SSD_E0_S05_1399652120P1 1 NORMAL ONLINE CACHED MEMBER
/dev/mapper/SSD_E0_S05_1399652120p2 SSD_E0_S05_1399652120P2 3 NORMAL ONLINE CACHED MEMBER
/dev/mapper/SSD_E0_S06_1399645200p1 SSD_E0_S06_1399645200P1 1 NORMAL ONLINE CACHED MEMBER
/dev/mapper/SSD_E0_S06_1399645200p2 SSD_E0_S06_1399645200P2 3 NORMAL ONLINE CACHED MEMBER
/dev/mapper/SSD_E0_S07_1399646692p1 SSD_E0_S07_1399646692P1 1 NORMAL ONLINE CACHED MEMBER
/dev/mapper/SSD_E0_S07_1399646692p2 SSD_E0_S07_1399646692P2 3 NORMAL ONLINE CACHED MEMBER
/dev/mapper/SSD_E0_S08_1399649840p1 SSD_E0_S08_1399649840P1 1 NORMAL ONLINE CACHED MEMBER
/dev/mapper/SSD_E0_S08_1399649840p2 SSD_E0_S08_1399649840P2 3 NORMAL ONLINE CACHED MEMBER
/dev/mapper/SSD_E0_S09_1399649424p1 SSD_E0_S09_1399649424P1 1 NORMAL ONLINE CACHED MEMBER
/dev/mapper/SSD_E0_S09_1399649424p2 SSD_E0_S09_1399649424P2 3 NORMAL ONLINE CACHED MEMBER

If the disks are not added to Oracle ASM, then add them manually. As grid user, execute the sqlplus '/as sysasm' command on the master node to add the disks to Oracle ASM.

For a system without Oracle Automatic Storage Management Filter Driver (Oracle ASM Filter Driver) configured, add the Oracle ASM disks as follows:


SQL> alter diskgroup /*+ _OAK_AsmCookie */ data add disk 
'/dev/mapper/SSD_E0_S05_1399764284p1' name SSD_E0_S05_1399764284p1, 
'/dev/mapper/SSD_E0_S06_1399765076p1' name SSD_E0_S06_1399765076p1, 
'/dev/mapper/SSD_E0_S07_1399765116p1' name SSD_E0_S07_1399765116p1, 
'/dev/mapper/SSD_E0_S08_1399765484p1' name SSD_E0_S08_1399765484p1, 
'/dev/mapper/SSD_E0_S09_1399765504p1' name SSD_E0_S09_1399765504p1;  

SQL> alter diskgroup /*+ _OAK_AsmCookie */ reco add disk 
'/dev/mapper/SSD_E0_S05_1399764284p2' name SSD_E0_S05_1399764284p2, 
'/dev/mapper/SSD_E0_S06_1399765076p2' name SSD_E0_S06_1399765076p2, 
'/dev/mapper/SSD_E0_S07_1399765116p2' name SSD_E0_S07_1399765116p2, 
'/dev/mapper/SSD_E0_S08_1399765484p2' name SSD_E0_S08_1399765484p2, 
'/dev/mapper/SSD_E0_S09_1399765504p2' name SSD_E0_S09_1399765504p2;

For a system with Oracle Automatic Storage Management Filter Driver (Oracle ASM Filter Driver) configured, add the Oracle ASM disks as follows:

SQL> alter diskgroup /*+ _OAK_AsmCookie */ data add disk 
'AFD:SSD_E0_S05_1399764284P1' name SSD_E0_S05_1399764284p1, 
'AFD:SSD_E0_S06_1399765076P1' name SSD_E0_S06_1399765076p1, 
'AFD:SSD_E0_S07_1399765116P1' name SSD_E0_S07_1399765116p1, 
'AFD:SSD_E0_S08_1399765484P1' name SSD_E0_S08_1399765484p1, 
'AFD:SSD_E0_S09_1399765504P1' name SSD_E0_S09_1399765504p1;  

SQL> alter diskgroup /*+ _OAK_AsmCookie */ reco add disk 
'AFD:SSD_E0_S05_1399764284P2' name SSD_E0_S05_1399764284p2, 
'AFD:SSD_E0_S06_1399765076P2' name SSD_E0_S06_1399765076p2, 
'AFD:SSD_E0_S07_1399765116P2' name SSD_E0_S07_1399765116p2, 
'AFD:SSD_E0_S08_1399765484P2' name SSD_E0_S08_1399765484p2, 
'AFD:SSD_E0_S09_1399765504P2' name SSD_E0_S09_1399765504p2;

Use the odaadmcli show validation storage errors command to show hard storage errors.
Hard errors include having the wrong type of disk inserted into a particular slot, an invalid disk model, or an incorrect disk size.

# odaadmcli show validation storage errors
Use the odaadmcli show validation storage failures command to show soft validation errors.
A typical soft disk error would be an invalid version of the disk firmware.

# odaadmcli show validation storage failures
Confirm that the oak_storage_conf.xml file shows 24 on both nodes.
```
#cat /opt/oracle/oak/conf/oak_storage_conf.xml
```

Parent topic: Storage on Multi Node Platforms

Adding the Storage Expansion Shelf

After the base storage shelf is fully populated, you can add the storage expansion shelf to expand your data storage on your multi-node platform.

The expansion shelf is available on Oracle Database Appliance multi-node platforms, such as Oracle Database Appliance X7-2-HA. The addition of the storage expansion shelf includes checks across both nodes. It is important to confirm that SSH does work across the nodes and all users can connect as expected using their shared password.

You must fully populate the base configuration before you can add an expansion shelf. If you add an expansion shelf, the shelf must have the same disk storage configuration as the base storage shelf.

Note:

Oracle recommends that you add a storage expansion shelf when you have relatively little activity on your databases. When the system discovers the new storage, Oracle Automatic Storage Management (Oracle ASM) automatically rebalances the disk groups. The rebalance operation may degrade database performance until the operation completes.

Install and cable the storage expansion shelf, but do not power on the expansion shelf.

Caution:

Review cabling instructions carefully to ensure that you have carried out cabling correctly. Incorrect connections can cause data loss when adding a storage expansion shelf to Oracle Database Appliance with existing databases.
If this is a new deployment or re-image of Oracle Database Appliance, perform the following steps in order:
1. Power on the base storage.
2. Power on Node 0.
3. Power on Node 1.
Caution:
Do not power on the expansion shelf yet.
Verify that both nodes plus the base storage shelf are up and running. Log into each server node and run the odacli validate-storagetopology command to confirm that the base configuration cabling is correct.
```
 odacli validate-storagetopology
 ...
      INFO  : Check if JBOD powered on
 SUCCESS    : JBOD : Powered-on                                          
      INFO  : Check for correct number of EBODS(2 or 4)
 SUCCESS    : EBOD found : 2                                                                                                                                                                     INFO       : Check for overall status of cable validation on Node0
  SUCCESS   : Overall Cable Validation on Node0            
 SUCCESS    : JBOD Nickname set correctly : Oracle Database Appliance - E0
```
The correct results will confirm if the two server nodes are properly cabled to the base storage shelf and all disks are online, with a good status, and added to the existing disk groups on both nodes. If there any failures, then fix the cabling before proceeding to the next step.

Note:
If the output shows that EBOD found is 2, then you only have the base storage shelf. If EBOD found is 4, then you have a base storage shelf and an expansion shelf.
Power on the storage expansion shelf.

Log in to each server node and run the odacli validate-storagetopology command to validate the storage cabling and confirm that the new storage shelf is recognized.


# odacli validate-storagetopology

  INFO    : Check if JBOD powered on
  SUCCESS : 2JBOD : Powered-on                                               
  INFO    : Check for correct number of EBODS(2 or 4)
  SUCCESS : EBOD found : 4                                                   
   ...
   ...

   INFO    : Check for overall status of cable validation on Node0
   SUCCESS : Overall Cable Validation on Node0            
   SUCCESS : JBOD0 Nickname set correctly : Oracle Database Appliance - E0
   SUCCESS : JBOD1 Nickname set correctly : Oracle Database Appliance - E1

Look for the following indicators that both storage shelves are recognized:

When there are two shelves, the JBOD (just a bunch of disks) is numbered. For example:
```
SUCCESS : 2JBOD : Powered-on
```
When both shelves are recognized, the EBOD found value is 4.
```
SUCCESS : EBOD found : 4
```

When the expansion shelf is cabled properly, the nickname is E1. For example:

        SUCCESS : JBOD0 Nickname set correctly : Oracle Database Appliance - E0
        SUCCESS : JBOD1 Nickname set correctly : Oracle Database Appliance - E1

Fix any errors before proceeding.

Run the odaadmcli show disk command to ensure that all disks in the expansion shelf are listed, are online, and are in a good state.
```
# odaadmcli show disk
```
When all disks are online and in a good state, proceed to the next step.
Run the odaadmcli show enclosure command to check the health of components in expansion shelf.
```
# odaadmcli show enclosure
```
Run the odaadmcli show ismaster command on Node 0 to confirm that Node 0 is the master.
```
# odaadmcli show ismaster
```

Run the odaadmcli expand storage command on the master node.

#odaadmcli expand storage -ndisk 24 -enclosure 1 

Precheck passed. 
Check the progress of expansion of storage by executing 'odaadmcli show disk' 
Waiting for expansion to finish ...

It takes approximately 10 to 12 minutes to add all of the disks to the configuration.

Use the odaadmcli show validation storage errors command to show hard storage errors.
Hard errors include having the wrong type of disk inserted into a particular slot, an invalid disk model, or an incorrect disk size.
```
# odaadmcli show validation storage errors
```
Use the odaadmcli show validation storage failures command to show soft validation errors.
A typical soft disk error would be an invalid version of the disk firmware.
```
# odaadmcli show validation storage failures
```
Run the odacli describe-component command to verify that all firmware components in the storage expansion are current.
```
# odaadmcli describe-component
```
If needed, update the storage shelf and then run the odacli describe-component command to confirm that the firmware is current.
```
# odaadmcli update
```
```
# odaadmcli describe-component
```

Parent topic: Storage on Multi Node Platforms