Managing Storage

10 Managing Storage

You can add storage to fully populate the base storage shelf and add a storage expansion shelf to your Oracle Database Appliance.

About Managing Storage
You can add storage at any time without shutting down your databases or applications.
Storage Addition Options for Oracle Database Appliance X6-2-HA
You can also add X7-2 storage disks to Oracle Database Appliance X6-2-HA, with some configuration changes.
Preparing for Storage Expansion
Review and perform these best practices before adding storage to the base shelf or adding the expansion shelf.
Adding Two 5-Pack SSDs
Add two 5-pack solid-state drives (SSDs) for data storage into the existing base configuration to fully populate the base storage shelf.
Adding the Storage Expansion Shelf
After the base storage shelf is fully populated, you can add the storage expansion shelf to expand the storage capacity.

About Managing Storage

You can add storage at any time without shutting down your databases or applications.

Oracle Database Appliance uses raw storage to protect data in the following ways:

Flash or Fast Recovery Area (FRA) backup. Flash or fast recovery area is a storage area (directory on disk or Oracle ASM diskgroup) that contains redo logs, control file, archived logs, backup pieces and copies, and flashback logs.
Mirroring. Double or triple mirroring provides protection against mechanical issues.

The amount of available storage is determined by the location of the FRA backup (external or internal) and if double or triple mirroring is used.

Oracle Database Appliance uses storage shelves, a base shelf and an optional storage expansion shelf. You can expand the base storage by adding a pack of solid-state drives (SSDs) to fully populate the base storage. You can further expand the storage by adding a second storage shelf. External NFS storage is supported for online backups, data staging, or additional database files.

Note:

You must fully populate the base storage shelf before adding the expansion shelf.

When you add storage, Oracle Automatic Storage Management (Oracle ASM) automatically rebalances the data across all of the storage including the new drives. Rebalancing a disk group moves data between disks to ensure that every file is evenly spread across all of the disks in a disk group and all of the disks are evenly filled to the same percentage. Oracle ASM automatically initiates a rebalance after storage configuration changes, such as when you add disks.

About Expanding Storage

If you need additional storage after fully populating the base shelf, you can add a storage expansion shelf. The expansion shelf is hot-pluggable, enabling you to expand storage without database downtime. After cabling and powering up the expansion shelf, the system automatically configures Oracle ASM storage and data is automatically distributed to the new shelf.

Note:

The process of rebalancing the data might impact performance until the new storage is correctly balanced across all drives. If possible, add a storage expansion shelf during a non-peak or non-production time period to minimize the performance impact of the automatic storage balancing.

The addition of the storage expansion shelf includes checks across both nodes. It is important to confirm that SSH does work across the nodes and all users can connect as expected using their shared password.

The following are the high level steps to expand storage:

Review the Oracle Database Appliance storage options.
Prepare for a storage upgrade by running checks to verify that the configuration is ready before adding storage to the base shelf or adding the expansion shelf.
Add storage if the base configuration is not full.
Add the storage expansion shelf, then log in to each server node and validate the cabling. After you confirm that the cabling is correct, power on the shelf and validate the storage.

Caution:

Review cabling instructions carefully to ensure that you have carried out cabling correctly. Incorrect connections can cause data loss when adding a storage expansion shelf to Oracle Database Appliance with existing databases.

Parent topic: Managing Storage

Storage Addition Options for Oracle Database Appliance X6-2-HA

You can also add X7-2 storage disks to Oracle Database Appliance X6-2-HA, with some configuration changes.

Oracle Database Appliance X6-2-HA uses 1.6 TB raw Solid-State Drives (SSDs), formatted to 1.2 TB for performance. When you order for Oracle Database Appliance X7-2 storage, you receive the default 3.2 TB raw Solid-State Drives, partitioned as per your existing storage capacity.

The following table explains the base configuration and storage expansion options for Oracle Database Appliance X6-2-HA.

Table 10-1 Storage Addition Options for Oracle Database Appliance X6-2-HA

Configuration	Storage Option	Total Storage
Base Configuration	Ten (10) 1.6TB SSDs in slots 0 - 9 and four (4) 200GB SSDs in slots 20 - 23	200 GB SSDs dedicated for database REDO and 1.6TB SDDs used for DATA.
Base Shelf Expansion	Additional Ten (10) 3.2TB SSD in slots 10 - 19 (Order Qty 2 -7117369 - Five (5) 3.2TB SSD drive pack and upgrade to Oracle Database Appliance release 12.2.1.4 or later)	3.2TB SSDs are partitioned to 1.2TB. The storage shelf can be expanded to be fully populated for a total of 24TB of raw storage.
Storage Shelf Expansion	Twenty 1.6TB/3.2TB SSDs in slots 0 - 19 and four 200GB SSD in slots 20 - 23 Oracle Database Appliance X7-2 fully populated expansion shelf with four 800GB and twenty 3.2TB SSDs (Order Qty 1 - 7119675 - Twenty 3.2 TB SSDs and four 800 GB SSDs and upgrade to Oracle Database Appliance release 12.2.1.4 or later)	800GB SSDs are partitioned to 200GB and 3.2TB SSDs are partitioned to 1.2TB. The expansion shelf provides an additional 24TB of raw storage.

The following options are available for storage addition:

About Storage Expansion

You can expand the base storage shelf by adding two 5-pack of solid-state drives (SSDs) to fully populate the base storage. In this case, partitions of 1.2 TB are created.

If you expand the base storage shelf by adding the 3.2 TB SSDs, then the storage is scaled to the existing configuration, that is, the usable disk capacity is 1.2 TB.

See the procedure Adding Two 5-Pack SSDs for steps to add the SSDs.

About Addition of Expansion Shelf

When the base storage shelf is fully populated, you can add a second storage shelf from the X7-2 storage. In this case, the usable disk capacity is 1.2 TB.

See the procedure Adding the Storage Expansion Shelf for steps to add the expansion shelf.

About Storage Disk Replacement

You can completely replace your 1.6 TB raw SSD storage with the X7-2 storage of 3.2 TB, and the 200 GB SSDs for REDO disk group with the 800 GB SSDs, respectively. oakd uses the same size as the disk replaced. In this case, the usable disk capacity is 3.2 TB. You must redeploy Oracle Database Appliance, if you choose this option.

See the procedure Preparing for Storage Expansion for steps to expand the storage.

Parent topic: Managing Storage

Preparing for Storage Expansion

Review and perform these best practices before adding storage to the base shelf or adding the expansion shelf.

Update Oracle Database Appliance to the latest Patch Bundle before expanding storage.
Confirm both nodes are at the same version and patch bundle level for software and firmware.
```
# odaadmcli show version -detail 
```
```
# odaadmcli inventory -q 
```
Note:
If oakd is not running in the foreground mode, on either node, fix the problem before adding storage.
Check the disk health of the existing storage disks.

Run the check on both nodes and use the default checks option to check the NetworkComponents, OSDiskStorage, SharedStorage, and SystemComponents.
```
#  odaadmcli validate -d
```
Run the odaadmcli show diskgroup command on each node to display and review Oracle Automatic Storage Management (Oracle ASM) disk group information. Verify that all disks are listed, are online, and are in a good state.
```
# odaadmcli show diskgroup data
```
```
# odaadmcli show diskgroup reco
```
```
 # odaadmcli show diskgroup redo 
```

Confirm Oracle ASM and CRS health on both nodes.

Run the odaadmcli orachk command on each node. If there is a problem connecting to either node, then check the /etc/bashrc file and remove (or remark out) any values in the profile for root, oracle, grid users.

Run odaadmcli orachk on Node 0:

# odaadmcli orachk
...

Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS

. . . . . . . . .
-------------------------------------------------------------------------------------------------------
Oracle Stack Status
-------------------------------------------------------------------------------------------------------
Host Name CRS Installed  ASM HOME   RDBMS Installed    CRS UP    ASM UP    RDBMS UP DB Instance Name
-------------------------------------------------------------------------------------------------------
odax3rm1       Yes           No          Yes              No        No        No          ........
-------------------------------------------------------------------------------------------------------

 ...

Run odaadmcli orachk on Node 1:

# odaadmcli orachk
...

Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
-------------------------------------------------------------------------------------------------------
Oracle Stack Status
-------------------------------------------------------------------------------------------------------
Host Name CRS Installed  ASM HOME   RDBMS Installed    CRS UP    ASM UP    RDBMS UP DB Instance Name
-------------------------------------------------------------------------------------------------------
odax3rm2      Yes           Yes           Yes            Yes       Yes        Yes      b22S2 b23S2 b24S2
-------------------------------------------------------------------------------------------------------

...

Confirm communications between the nodes and that SSH is working using the same password for oracle, root, and grid.
From each node:
1. ssh to both nodes.
2. Ping both nodes.
Confirm there is at least 10 GB of space available on each node.
```
[root@oda]# df -h
```
```
[root@odb]# df -h 
```

Parent topic: Managing Storage

Adding Two 5-Pack SSDs

Add two 5-pack solid-state drives (SSDs) for data storage into the existing base configuration to fully populate the base storage shelf.

Before adding the disks to the system, ensure that Oracle Database Appliance is on the latest update version.

Insert disks one at a time in slots 10 through 19.

Note:
Allow at least one minute between inserting each disk to avoid flooding of disk events into oakd.

After all 10 disks are added, go to Step 2.
Run the odaadmcli show ismaster command to determine which node is the master.
```
# odaadmcli show ismaster
```

Run the odaadmcli expand storage command on the master node.

#odaadmcli expand storage 
Precheck passed. 
Successfully formatted 1.6TB SSD disks... 
Check the progress of expansion of storage by executing 'odaadmcli show disk' 
Waiting for expansion to finish ...

Wait 30 seconds before proceeding to the next step.

Execute the odaadmcli expand storage command on the slave node.

#odaadmcli expand storage 
Precheck passed. 
Successfully formatted 1.6TB SSD disks... 
Check the progress of expansion of storage by executing 'odaadmcli show disk' 
Waiting for expansion to finish ...

It takes 10 to 12 minutes to add all of the disks to the configuration.

Run the odaadmcli show disk command to ensure that all disks are listed, are online, and are in a good state.
```
# odaadmcli show disk
```

Verify that the disks in slots 10 to 20 are added to Oracle Automatic Storage Management (Oracle ASM).

Run the asm_script to verify that the disks in slots 10 to 20 are added to Oracle Automatic Storage Management (Oracle ASM). If the 10 disks are successfully added (CACHED and MEMBER), then go to Step 7.
```
su grid user /opt/oracle/oak/bin/stordiag/asm_script.sh 1 6
```

If the disks are not added to Oracle ASM, then add them manually. As a grid user, execute the sqlplus '/as sysasm' command on the master node to add the disks to Oracle ASM.

#sqlplus '/as sysasm'
....
SQL> alter diskgroup /*+ _OAK_AsmCookie */ data add disk 
'/dev/mapper/SSD_E0_S11_1399764284p1' name SSD_E0_S11_1399764284p1, 
'/dev/mapper/SSD_E0_S12_1399765076p1' name SSD_E0_S12_1399765076p1, 
'/dev/mapper/SSD_E0_S13_1399765116p1' name SSD_E0_S13_1399765116p1, 
'/dev/mapper/SSD_E0_S14_1399765484p1' name SSD_E0_S14_1399765484p1, 
'/dev/mapper/SSD_E0_S15_1399765404p1' name SSD_E0_S15_1399765404p1, 
'/dev/mapper/SSD_E0_S16_1399766160p1' name SSD_E0_S16_1399766160p1, 
'/dev/mapper/SSD_E0_S17_1399765264p1' name SSD_E0_S17_1399765264p1, 
'/dev/mapper/SSD_E0_S18_1399763588p1' name SSD_E0_S18_1399763588p1, 
'/dev/mapper/SSD_E0_S19_1399765504p1' name SSD_E0_S19_1399765504p1;  

SQL> alter diskgroup /*+ _OAK_AsmCookie */ reco add disk 
'/dev/mapper/SSD_E0_S11_1399764284p2' name SSD_E0_S11_1399764284p2, 
'/dev/mapper/SSD_E0_S12_1399765076p2' name SSD_E0_S12_1399765076p2, 
'/dev/mapper/SSD_E0_S13_1399765116p2' name SSD_E0_S13_1399765116p2, 
'/dev/mapper/SSD_E0_S14_1399765484p2' name SSD_E0_S14_1399765484p2, 
'/dev/mapper/SSD_E0_S15_1399765404p2' name SSD_E0_S15_1399765404p2, 
'/dev/mapper/SSD_E0_S16_1399766160p2' name SSD_E0_S16_1399766160p2, 
'/dev/mapper/SSD_E0_S17_1399765264p2' name SSD_E0_S17_1399765264p2, 
'/dev/mapper/SSD_E0_S18_1399763588p2' name SSD_E0_S18_1399763588p2, 
'/dev/mapper/SSD_E0_S19_1399765504p2' name SSD_E0_S19_1399765504p2;

Use the odaadmcli show validation storage errors command to show hard storage errors.
Hard errors include having the wrong type of disk inserted into a particular slot, an invalid disk model, or an incorrect disk size.

# odaadmcli show validation storage errors
Use the odaadmcli show validation storage failures command to show soft validation errors.
A typical soft disk error would be an invalid version of the disk firmware.

# odaadmcli show validation storage failures
Confirm that the oak_storage_conf.xml file shows 24 on both nodes.
```
#cat /opt/oracle/oak/conf/oak_storage_conf.xml
```

Parent topic: Managing Storage

Adding the Storage Expansion Shelf

After the base storage shelf is fully populated, you can add the storage expansion shelf to expand the storage capacity.

Note:

Oracle recommends that you add a storage expansion shelf when you have relatively little activity on your databases. When the system discovers the new storage, Oracle ASM automatically rebalances the disk groups. The rebalance operation may degrade database performance until the operation completes.
Only offline addition of X7-2 storage expansion shelf to X6-2-HA Oracle Database Appliance is supported.

Install and cable the storage expansion shelf, but do not power on the expansion shelf.

Caution:

Review cabling instructions carefully to ensure that you have carried out cabling correctly. Incorrect connections can cause data loss when adding a storage expansion shelf to Oracle Database Appliance with existing databases.
If this is a new deployment or re-image of Oracle Database Appliance, perform the following steps in order:
1. Power on the base storage.
2. Power on Node 0.
3. Power on Node 1.
Caution:
Do not power on the expansion shelf yet.
Verify that both nodes plus the base storage shelf are up and running. Log into each server node and run the odaadmcli validate command to confirm that the base configuration cabling is correct.
```
 odaadmcli validate -c storagetopology
 ...
      INFO  : Check if JBOD powered on
 SUCCESS    : JBOD : Powered-on                                          
      INFO  : Check for correct number of EBODS(2 or 4)
 SUCCESS    : EBOD found : 2                                                                                                                                                                     INFO       : Check for overall status of cable validation on Node0
  SUCCESS   : Overall Cable Validation on Node0            
 SUCCESS    : JBOD Nickname set correctly : Oracle Database Appliance - E0
```
The correct results will confirm if the two server nodes are properly cabled to the base storage shelf and all disks are online, with a good status, and added to the existing diskgroups on both nodes. If there any failures, then fix the cabling before proceeding to the next step.

Note:
IF the output shows that EBOD found is 2, then you only have the base storage shelf. If EBOD found is 4, then you have a base storage shelf and a storage expansion shelf.
Power on the storage expansion shelf.

Log in to each server node and run the odaadmcli validate command to validate the storage cabling and confirm that the new storage shelf is recognized.

# odaadmcli validate -c storagetopology

  INFO    : Check if JBOD powered on
  SUCCESS : 2JBOD : Powered-on                                               
  INFO    : Check for correct number of EBODS(2 or 4)
  SUCCESS : EBOD found : 4                                                   
   ...
   ...

   INFO    : Check for overall status of cable validation on Node0
   SUCCESS : Overall Cable Validation on Node0            
   SUCCESS : JBOD0 Nickname set correctly : Oracle Database Appliance - E0
   SUCCESS : JBOD1 Nickname set correctly : Oracle Database Appliance - E1

Look for the following indicators that both storage shelves are recognized:

When there are two shelves, the JBOD (just a bunch of disks) is numbered. For example:
```
SUCCESS : 2JBOD : Powered-on
```
When both shelves are recognized, the EBOD found value is 4.
```
SUCCESS : EBOD found : 4
```

When the expansion shelf is cabled properly, the nickname is E1. For example:

        SUCCESS : JBOD0 Nickname set correctly : Oracle Database Appliance - E0
        SUCCESS : JBOD1 Nickname set correctly : Oracle Database Appliance - E1

Fix any errors before proceeding.

Run the odaadmcli show disk command to ensure that all disks in the expansion shelf are listed, are online, and are in a good state.
```
# odaadmcli show disk
```
When all disks are online and in a good state, proceed to the next step.
Run the odaadmcli show enclosure command to check the health of components in expansion shelf.
```
# odaadmcli show enclosure
```
Restart oakd on each node using the command.
```
odaadmcli restart oak
```
Run the odaadmcli show ismaster command on Node 0 to check the master and slave configuration.
```
# odaadmcli show ismaster
```
You can change the master and slave configuration by restarting oakd on the node.
When oakd restarts on both nodes, run the command odaadmcli show disk on both nodes.
All 24 disks of the expansion storage shelf e1_pd_xx are displayed with the status"UNKNOWN NewDiskInserted: in the command output on both nodes. For example:
```
e0_pd_23 /dev/sdab SSD ONLINE Good 
e1_pd_00 /dev/sdax SSD UNKNOWN 
NewDiskInserted
...
e1_pd_23 /dev/sdbu SSD UNKNOWN 
NewDiskInserted
```
Run the command odaadmcli add jbod disk on the master node first, and then on the slave node.
```
# odaadmcli add jbod disk 
Sat Jun 30 01:04:08 UTC 2018 
Successfully formatted 1.6TB SSD disks...  
Check addition of JBOD status by executing 
odaadmcli ' show jbod progress'
```
Wait for the command to finish. It takes about 10 to 12 minutes. Then verify that the expansion storage shelf disk status is ONLINE GOOD. If yes, proceed to next step.
Run the command odaadmcli add jbod asm on the master node first, and then on the slave node.

Wait for the command to finish. It takes about 2 to 3 minutes. Then verify that the expansion storage shelf disk status in ASM is ONLINE and CACHED.

Run the command show jbod status to check Oracle ASM disk group and disk status.

# odaadmcli show jbod status
INFO: 2018-06-30 01:32:46: Present status of Expansion storage shelf
 Disk Added    Disk Not Added     Status     
OAKD     24              0            SUCCESS     
ASM     
DATA      20              0            SUCCESS     
RECO     20              0            SUCCESS    
 REDO      4              0            SUCCESS  
INFO: 2018-06-30 01:32:46: DiskGroup Mount Status  
GROUP_NUMBER NAME       STATE      TYPE         TOTAL_MB    FREE_MB            
1 DATA       MOUNTED    HIGH         39321600   37272260            
2 REDO       MOUNTED    HIGH          1525760    1383896            
3 RECO       MOUNTED    HIGH          6461440    5984272

Use the odaadmcli show validation storage errors command to show hard storage errors.
Hard errors include having the wrong type of disk inserted into a particular slot, an invalid disk model, or an incorrect disk size.

# odaadmcli show validation storage errors
Use the odaadmcli show validation storage failures command to show soft validation errors.
A typical soft disk error would be an invalid version of the disk firmware.

# odaadmcli show validation storage failures
Run the odaadmcli show version command to verify that all firmware components in the storage expansion are current.
```
# odaadmcli show version -detail
```
If needed, update the storage shelf and then run the show version command to confirm that the firmware is current.
```
# odaadmcli update
```
```
# odaadmcli show version -detail
```

Parent topic: Managing Storage