Managing Storage Directory Sizes

Managing Disk Thresholds

It is very important to configure each storage directory with a specific amount of available disk space. The Oracle NoSQL Database uses the configured Storage Directory sizes to enforce disk space limits. Without configuring how much disk space is available, the store opportunistically uses all available space, less 5 GB free disk space. The system maintains 5 GB of free space to allow manual recovery if the Storage Node exceeds its configured disk limit. Be sure to monitor disk usage regularly using the statistics provided, as described in Monitoring Disk Usage.

Storage Nodes use their available disk space for two purposes:

To store your data.
To save reserved files.

Reserved files consist of data that has already been replicated to active replica nodes. The purpose of storing a copy of this data is to use for Replica Nodes that lose contact with the Master Node. Losing contact typically occurs because Replica nodes are shut down, or a network partition event occurs, or because another transient problem occurs. The Storage Node is primarily designed to consume the amount of disk space you assign it, and to use the remaining disk space to save the reserved files. Each Storage Node manages its available disk space, leaving 5 GB free for recovery purposes. Your intervention is typically not required in this disk management process, unless a storage node exceeds its available disk capacity.

Note:

If a Storage Node (SN) consumes more than what is assigned as storagedirsize, including leaving 5 GB of space free, the SN automatically attempts to free up disk space by deleting reserved files (not your data files), until more than 5 GB of space is available. If the Storage Node is unable to free up enough space, it suspends write operations to the node. Read operations continue as normal. Write operations resume automatically once the node obtains sufficient free disk space.

You can limit how much disk space the store consumes on a node by node basis, by explicitly specifying a storage directory size for each storage node, as described in Specifying Storage Directory Sizes. Storage nodes can then consume all of their configured disk space as needed, leaving free the required 5 GB. However, if you do not indicate a storage directory size, the Storage Node uses disk space until it consumes the disk, except for the required 5 GB for manual recovery.

Consider a storage node with a 200 GB disk. Without configuring a storagedirsize for that disk, the store keeps consuming up to 195 GB of disk space (leaving only the 5 GB for manual recovery). If your standard policy requires a minimum 20 GB available space on each disk, you must configure the storage node with a storagedirsize of 175 GB, leaving 20 GB available, and 5 GB for store recovery.

The most common reason a node's storage directory fills up is because of reserved files. If the Storage Node exceeds its disk threshold, it continues to delete the reserved files until the threshold is no longer exceeded.

Specifying Storage Directory Sizes

Use the makebootconfig storagedirsize parameter to specify Storage Node (SN) capacity when you initially install your store. See Configuring Your KVStore Installation and makebootconfig for details. Additionally, if your SN has the capacity to support more than one Replication Node, specify a storage directory location and storage directory size for each Replication Node.

To specify or change storage capacity after you have installed the store, use plan change-storagedir. When you use plan change-storagedir be sure to specify the -storagedirsize parameter to indicate how large the new storage directory is.

Note:

If you specify the -storagedir parameter, but not -storagedirsize, makebootconfig displays a warning. Always specify both parameters for control and tracking.

The value specified for the storagedirsize parameter must be a long, optionally followed by a unit string. Accepted unit strings are: KB, MB, GB, and TB, corresponding to 1024, 1024^2, 1024^3, 1024^4 respectively. Acceptable strings are case insensitive. Valid delimiters between the long value and the unit string are " ", "-", or "_". If you specify the delimiter as " ", your value should be enclosed in double quotes, For example "10 GB". If you have any other delimiter double quotes is not mandatory. For example 10_GB or 10-GB.

For example:

kv-> verbose
Verbose mode is now on
kv-> show topology
store=mystore  numPartitions=300 sequence=308
  zn: id=zn1 name=Manhattan repFactor=3 type=PRIMARY 
allowArbiters=false

  sn=[sn1] zn:[id=zn1 name=Manhattan] node1:9000 capacity=1 RUNNING
    [rg1-rn1] RUNNING  /storage-dir/sn1 0
              No performance info available
  sn=[sn2] zn:[id=zn1 name=Manhattan] node2:9000 capacity=1 RUNNING
    [rg1-rn2] RUNNING  /storage-dir/sn2 0
            single-op avg latency=0.0 ms   multi-op avg latency=0.0 ms
  sn=[sn3] zn:[id=zn1 name=Manhattan] node3:9000 capacity=1 RUNNING
    [rg1-rn3] RUNNING  /storage-dir/sn3 0
              No performance info available

  shard=[rg1] num partitions=300
    [rg1-rn1] sn=sn1 haPort=node1:9010
    [rg1-rn2] sn=sn2 haPort=node2:9010
    [rg1-rn3] sn=sn3 haPort=node3:9010
    partitions=1-300

kv-> plan change-storagedir -sn sn1 -storagedir /storage-dir/sn1 \
-storagedirsize "200 gb" -add -wait
Executed plan 7, waiting for completion...
Plan 7 ended successfully
kv-> plan change-storagedir -sn sn2 -storagedir /storage-dir/sn2 \
-storagedirsize "300 gb" -add -wait
Executed plan 8, waiting for completion...
Plan 8 ended successfully
kv-> plan change-storagedir -sn sn3 -storagedir /storage-dir/sn3 \
-storagedirsize "400 gb" -add -wait
Executed plan 9, waiting for completion...
Plan 9 ended successfully
kv-> show topology
store=mystore  numPartitions=300 sequence=308
  zn: id=zn1 name=Manhattan repFactor=3 type=PRIMARY 
allowArbiters=false

  sn=[sn1] zn:[id=zn1 name=Manhattan] node1:9000 capacity=1 RUNNING
    [rg1-rn1] RUNNING  /storage-dir/sn1 214748364800
              No performance info available
  sn=[sn2] zn:[id=zn1 name=Manhattan] node2:9000 capacity=1 RUNNING
    [rg1-rn2] RUNNING  /storage-dir/sn2 322122547200
            single-op avg latency=0.0 ms   multi-op avg latency=0.0 ms
  sn=[sn3] zn:[id=zn1 name=Manhattan] node3:9000 capacity=1 RUNNING
    [rg1-rn3] RUNNING  /storage-dir/sn3 429496729600
            single-op avg latency=0.0 ms   multi-op avg latency=0.0 ms

  shard=[rg1] num partitions=300
    [rg1-rn1] sn=sn1 haPort=node1:9010
    [rg1-rn2] sn=sn2 haPort=node2:9010
    [rg1-rn3] sn=sn3 haPort=node3:9010
    partitions=1-300

Note:

If any Storage Node stores its data in the root directory (not recommended), then instead of plan change-storagedir, set the rootDirSize parameter. For example:

kv-> plan change-parameters -service sn1 -params rootDirSize=200_gb

Specifying Differing Disk Capacities

By default, Oracle NoSQL Database evenly distributes data across all the Storage Nodes in your store. No check is made in advance. The store expects all of the hardware in your storee to be homogenous, and so all Storage Nodes would have the same disk capacity.

However, more likely, you are running a store in an environment where some Storage Nodes have more disk capacity than others. In this case, you must specify appropriate disk capacity for each storage node. Oracle NoSQL Database will then place more data on higher capacity Storage Nodes. Be aware that specifying greater disk capacity to a storage node can result in an increased workload. Storage Nodes with more capacity than others could then serve more read and/or write activity. Be sure to size your storage nodes accordingly to support additional workload, if any.

Monitoring Disk Usage

If a Storage Node exceeds its disk usage threshold value (storagedirsize - 5GB), then all write activity for that node is suspended until sufficient disk space is made available. The store makes disk space available by removing reserved files to satisfy the threshold requirement. No data files are removed. Read activity continues while reserved data is being removed.

To ensure that your Storage Node can continue to service write requests, monitor the availableLogSize JMX statistic. This represents the amount of space that can be used by write operations. This value is not necessarily representative of the amount of disk space currently in use, since quite a lot of disk space can, and is, used for reserved files, which are not included in the availableLogSize statistic.

Reserved files are data files that have already been replicated, but which are retained for replication to nodes that are out of contact with the master node. Because Oracle NoSQL Database liberally reserves files, all available storage will frequently be consumed by reserved data. However, reserved data is automatically deleted as necessary by the Storage Node to continue write operations. For this reason, monitoring the actual disk usage is not meaningful.

If availableLogSize reaches zero, writes are suspended for the Storage Node. Earlier, as availableLogSize approaches zero, the node has less and less space for reserved data files. The result is that the store becomes increasingly less resilient in the face of a prolonged but temporary node outage because there are increasingly fewer historical log files that the store can use to gracefully bring a node up to date once it is available again.

The following tables lists some other useful statistics about disk usage. These statistics are stored in the stats file, or you can monitor them using the JMX oracle.kv.repnode.envmetric type. (Xref)

Statistic	Description
`availableLogSize`	Disk space available (in bytes) for write operations. This value is calculated with consideration fo reserved data files, which are deleted automatically whenever space is required to perform write operations: `free space + reservedLogSize - protectedLogSize` In general, monitoring disk usage in the file system is not meaningful, because of the presence of reserved files that can be deleted automatically.
`activeLogSize`	Bytes used by all active data files: files required for basic operation.
`reservedLogSize`	Bytes used by all reserved data files: files that have been cleaned and can be deleted if they are not protected.
`protectedLogSize`	Bytes used by all protected data files: the subset of reserved files that are temporarily protected and cannot be deleted.
`ProtectedLogSizeMap`	A breakdown of protectedLogSize as a map of protecting entity name to protected size in bytes.
`TotalLogSize`	Total bytes used by data files on disk: activeLogSize + reservedLogSize.

The following list from part of some JMX output, shows an example of how you will see each statistic. All of these statistic names have a Cleaning_ prefix, indicating that they may be in the log cleaning statistics group (for garbage collection):

.
.
.
"Cleaning_nRepeatIteratorReads": 0,
"Cleaning_nLNsExpired": 0,
"Cleaning_nCleanerRuns": 0,
"Cleaning_nBINDeltasDead": 0,
"Cleaning_nCleanerDisksReads": 0,
"Cleaning_protectedLogSizeMap": "",
"Cleaning_nCleanerDeletions": 0,
"Cleaning_nCleanerEntriesRead": 0,
"Cleaning_availableLogSize": 48942137344,
"Cleaning_nLNsDead": 0,
"Cleaning_nINsObsolete": 0,
"Cleaning_activeLogSize": 112716,
"Cleaning_nINsDead": 0,
"Cleaning_nINsMigrated": 0,
"Cleaning_totalLogSize": 112716,
"Cleaning_nBINDeltasCleaned": 0,
"Cleaning_nLNsObsolete": 0,
"Cleaning_nLNsCleaned": 0,
"Cleaning_nLNQueueHits": 0,
"Cleaning_reservedLogSize": 0,
"Cleaning_protectedLogSize": 0,
"Cleaning_nClusterLNsProcessed": 0,
"Node Compression_processedBins": 0,
.
.
.

You can tell if writes have been suspended for a Storage Node using the ping command from the CLI. In the following sample output, the Shard Status shows read-only:1. This indicates that one of the Storage Nodes is in read-only mode. The likeliest reason for that is that it has exceeded its disk threshold.

kv-> ping
Pinging components of store istore based upon topology sequence #11
3 partitions and 3 storage nodes
Time: 2018-09-28 06:57:10 UTC   Version: 18.3.2
Shard Status: healthy:0 writable-degraded:0 read-only:1 offline:0
Admin Status: healthy
Zone [name=dc1 id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false] 
RN Status: online:1 offline:2
Storage Node [sn1] on sn1.example.com:5000 Zone: [name=dc1 
id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false] Status: RUNNING
Ver: 18.3.2 2018-09-17 09:33:45 UTC  Build id: a72484b8b33c
Admin [admin1] Status: RUNNING,MASTER
Rep Node [rg1-rn1] Status: RUNNING,MASTER (non-authoritative) 
sequenceNumber:39,177,477 haPort:5011 available storage size:6 GB
Storage Node [sn2] on sn2.example.com:5000 Zone: 
[name=dc1 id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false] Status: RUNNING 
Ver: 18.3.2 2018-09-17 09:33:45 UTC  Build id: a72484b8b33c
Rep Node [rg1-rn2] Status: RUNNING,UNKNOWN sequenceNumber:39,176,478
haPort:5010 available storage size:NOT AVAILABLE delayMillis:? catchupTimeSecs:?
Storage Node [sn3] on sn3.example.com:5000 Zone: [name=dc1 
id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false] Status: RUNNING
Ver: 18.3.2 2018-09-17 09:33:45 UTC  Build id: a72484b8b33c
Rep Node [rg1-rn3] Status: RUNNING,UNKNOWN sequenceNumber:39,166,804
haPort:5010 available storage size:NOT AVAILABLE delayMillis:? catchupTimeSecs:?

For information on JMX monitoring the store, see Java Management Extensions (JMX) Notifications.

Handling Disk Limit Exception

If a Storage Node exceeds its disk usage threshold value (<storagedirsize> - 5 GB), then the store suspends all write activities on that node, until sufficient data is removed to satisfy the threshold requirement. In such a situation, there are two ways to bring the store back to read and write availability, without deleting user data.

Increasing storagedirsize on one or more Replication Nodes if there is available disk space
Expanding the store by adding a new shard

If there is enough space left on the disk or if the complete disk size is not set as the size of storagedirsize, you can bring back the write availability (without any additional need of the hardware) by simply increasing the storage directory size for one or more Replication Nodes.

If there is not enough space left on disk or if the complete disk size is set as the size of the storage directory, then you should follow the store expansion procedure, where you will need additional hardware to increase the number of shards by one.

Note:

If you are following the store expansion procedure, it is important to check the performance files to see if the cleaner is working well, by monitoring the minUtilization statistics. If the minUtilization statistics is less than 30%, it may mean that the cleaner is not keeping up. In this case it is not possible to perform store expansion.

Store expansion can only be performed if the minUtilization statistics percentage is not less than 30%.

For example:

2018-03-07 16:07:12.499 UTC INFO [rg1-rn1] JE: Clean file 0x2b: 
predicted min util is below minUtilization, current util min: 39 max: 39, 
predicted util min: 39 max: 39, chose file with util min: 30 max: 30 avg: 30

2018-03-07 16:07:04.029 UTC INFO [rg1-rn2] JE: Clean file 0x27: 
predicted min util is below minUtilization, current util min: 39 max: 39, 
predicted util min: 39 max: 39, chose file with util min: 30 max: 30 avg: 30

2018-03-07 16:05:44.960 UTC INFO [rg1-rn3] JE: Clean file 0x27: 
predicted min util is below minUtilization, current util min: 39 max: 39, 
predicted util min: 39 max: 39, chose file with util min: 30 max: 30 avg: 30

Increasing Storage Directory Size

To increase the storage directory size in one or more Replication Nodes, open the CLI and execute the following commands:

Disable write operations on the store or on the failed shard.
```
plan enable-requests -request-type READONLY \
{-shard <shardId[,shardId]*> | -store}
```
Here, -request-type READONLY is the option which disables write operations on a shard. You can disable write operations on one or more shards by using the -shard option, or on the entire store by using the —store option.

Note:
Though Replication Nodes are already in non-write availability mode whenever they hit an out of disk limit exception, it is important to disable user write operations explicitly. Disabling the user write operations ensures that the Replication Nodes are brought back up in the correct manner.
Execute the PING command to analyze the state of one or more Replication Nodes.
```
kv-> ping
```
Usually, when Replication Nodes hit an out of disk limit exception, Replica Replication Nodes are in the RUNNING, UNKNOWN state, and Master Replication Nodes are in the RUNNING, MASTER (non-authoritative) state.
To display the current, deployed topology, execute the show topology —verbose command. Make note of the current storage directory size allocated to each Replication Node.
```
show topology -verbose [-zn] [-rn] [-an] [-sn] [-store] [-status] [-json]
```
To ensure that other Replication nodes in the store do not hit a disk limit exception while you increase the storagedirsize, reduce the JE free disk space on all Replication Nodes to 2 GB or 3 GB. You can use the -all-rns option to reduce the JE free disk space on all Replication Nodes at once, or the -service -rgx-rgy option to reduce the free disk space on a specific Replication Node.
```
kv-> plan change-parameters [-all-rns|-service -rgx-rgy] \
-params "configProperties=je.freeDisk=XXX"
```
After executing this command with either option, the system will stop the Replication Nodes, update parameters, and restart Replication Nodes with the JE free disk space parameter you specify.
To increase the storage directory size on one or more Replication Nodes.
```
kv-> plan change-storagedir -wait -sn snX \
-storagedir <storagedirpath> –add -storagedirsize X_GB
```
Here snX is the Storage Node whose directory size you want to increase, and X is the new storage size in GB.
After the plan change-parameters command executes successfully, verify the new storagedirsize value is assigned to one or more Replication Nodes in the store.
```
show topology -verbose [-zn] [-rn] [-an] [-sn] [-store] [-status] [-json]
```
Lastly, reset the JE free disk space back to 5 GB. Also, enable write operations back on the store or a specific shard.
```
kv-> plan change-parameters [-all-rns|-service -rgx-rgy] \
-params "configProperties=je.freeDisk=5368709120"

kv-> plan enable-requests –request-type ALL {-shard <shardId[,shardId]*> | -store}
```
The –request-type ALL option re-enables write operations on the store or on a specific shard.

Example

Let us consider a store with 1x3 topology, hitting a disk limit exception. Perform the following steps to increase the storage directory size of all Replication Nodes in the store from 16 GB to 25 GB.

Stop the write operations on the store level:

kv-> plan enable-requests –request-type READONLY –store;

Ping the store to analyze the state of one or more Replication Nodes.

kv-> ping
Pinging components of store istore based upon topology sequence #11
3 partitions and 3 storage nodes
Time: 2018-09-28 06:57:10 UTC   Version: 18.3.2
Shard Status: healthy:0 writable-degraded:0 read-only:1 offline:0 total:1
Admin Status: healthy
Zone [name=dc1 id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]
RN Status: online:1 offline:2   Storage Node [sn1] on node21:port1
Zone: [name=dc1 id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]
Status: RUNNING   Ver: 18.3.2 2018-09-17 09:33:45 UTC  Build id: a72484b8b33c
        Admin [admin1]          Status: RUNNING,MASTER
        Rep Node [rg1-rn1]      Status: RUNNING,MASTER (non-authoritative)
        sequenceNumber:27,447,667 haPort:5011 available storage size:12 GB
Storage Node [sn2] on node22:port1    
Zone: [name=dc1 id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]
Status: RUNNING   Ver: 18.3.2 2018-09-17 09:33:45 UTC  Build id: a72484b8b33c
        Rep Node [rg1-rn2]      Status: RUNNING,UNKNOWN 
        sequenceNumber:27,447,667 haPort:5010 available storage size:10 GB delayMillis:? catchupTimeSecs:?
Storage Node [sn3] on node23:port1    
Zone: [name=dc1 id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]
Status: RUNNING   Ver: 18.3.2 2018-09-17 09:33:45 UTC  Build id: a72484b8b33c
        Rep Node [rg1-rn3]      Status: RUNNING,UNKNOWN 
        sequenceNumber:27,447,667 haPort:5010 available storage size:9 GB delayMillis:? catchupTimeSecs:?

The example shows that the Replication Nodes are in RUNNING, UNKNOWN state and Master Replication Node is in RUNNING, MASTER(non-authoritative) state.

View the current, deployed topology.

kv-> show topology -verbose
store=istore  numPartitions=3 sequence=11
  zn: id=zn1 name=dc1 repFactor=3 type=PRIMARY allowArbiters=false \
  masterAffinity=false

  sn=[sn1] zn:[id=zn1 name=dc1] node21:port1 capacity=1 RUNNING
    [rg1-rn1] RUNNING  /scratch/kvroot 16 GB
                 single-op avg latency=36.866146 ms   multi-op avg latency=0.0 ms
    [rg1-rn1] RUNNING  /scratch/kvroot 16 GB
                 single-op avg latency=36.866146 ms   multi-op avg latency=0.0 ms
  sn=[sn2] zn:[id=zn1 name=dc1] node22:port1 capacity=1 RUNNING
    [rg1-rn2] RUNNING  /scratch/kvroot 16 GB
                 single-op avg latency=0.0 ms   multi-op avg latency=0.0 ms
    [rg1-rn2] RUNNING  /scratch/kvroot 16 GB
                 single-op avg latency=0.0 ms   multi-op avg latency=0.0 ms
  sn=[sn3] zn:[id=zn1 name=dc1] node23:port1 capacity=1 RUNNING
    [rg1-rn3] RUNNING  /scratch/kvroot 16 GB
                 single-op avg latency=0.0 ms   multi-op avg latency=0.0 ms
    [rg1-rn3] RUNNING  /scratch/kvroot 16 GB
                 single-op avg latency=0.0 ms   multi-op avg latency=0.0 ms

  numShards=1
  shard=[rg1] num partitions=3
    [rg1-rn1] sn=sn1 haPort=node21:port2
    [rg1-rn2] sn=sn2 haPort=node22:port3
    [rg1-rn3] sn=sn3 haPort=node23:port3
    partitions=1-3

You see that 16 GB of disk space is assigned as the storage directory size for each Replication Node.

Reduce the JE free disk space from 5 GB to 2 GB for all Replication Nodes in the store.

kv-> plan change-parameters -all-rns -params \
"configProperties=je.freeDisk=2147483648";
Started plan 70. Use show plan -id 70 to check status.
To wait for completion, use plan wait -id 70

For each Replication Node, increase the storage directory size to 25 GB.

kv-> plan change-storagedir -wait -sn sn1 -storagedir /scratch/kvroot \
-add -storagedirsize 25_GB -wait
Executed plan 72, waiting for completion...
Plan 72 ended successfully

kv-> plan change-storagedir -wait -sn sn2 -storagedir /scratch/kvroot \
-add -storagedirsize 25_GB -wait
Executed plan 73, waiting for completion...
Plan 73 ended successfully

kv-> plan change-storagedir -wait -sn sn3 -storagedir /scratch/kvroot \
-add -storagedirsize 25_GB -wait
Executed plan 74, waiting for completion...
Plan 74 ended successfully

View the topology again to verify that the new value is assigned to storagedirsize.

kv-> show topology -verbose
store=istore  numPartitions=3 sequence=11
  zn: id=zn1 name=dc1 repFactor=3 type=PRIMARY allowArbiters=false \
masterAffinity=false

  sn=[sn1] zn:[id=zn1 name=dc1] node21:port1 capacity=1 RUNNING
    [rg1-rn1] RUNNING  /scratch/kvroot 25 GB
                 single-op avg latency=0.0 ms   multi-op avg latency=0.0 ms
  sn=[sn2] zn:[id=zn1 name=dc1] node22:port1 capacity=1 RUNNING
    [rg1-rn2] RUNNING  /scratch/kvroot 25 GB
                 single-op avg latency=552.51996 ms   multi-op avg latency=0.0 ms
  sn=[sn3] zn:[id=zn1 name=dc1] node23:port1 capacity=1 RUNNING
    [rg1-rn3] RUNNING  /scratch/kvroot 25 GB
                 single-op avg latency=14.317171 ms   multi-op avg latency=0.0 ms

  numShards=1
  shard=[rg1] num partitions=3
    [rg1-rn1] sn=sn1 haPort=node21:port2
    [rg1-rn2] sn=sn2 haPort=node22:port3
    [rg1-rn3] sn=sn3 haPort=node23:port3
    partitions=1-3

The example now shows that 25 GB is assigned as the storage directory size for each Replication Node.

Reset the JE free disk space to 5 GB and enable write operations back on the store.

kv-> plan change-parameters [-all-rns|-service -rgx-rgy] \
-params "configProperties=je.freeDisk=5368709120"

kv-> plan enable-requests –request-type READONLY –store;

Adding a New Shard

Apart from increasing the storage directory size, you can also handle disk limit exceptions by adding a new shard and expanding your store.

The following example demonstrates adding three new Storage Nodes (Storage Nodes 21, 22, and 23) and deploying the new store to recover from disk limit exception:

Disable write operations on the store.
```
kv-> plan enable-requests -request-type READONLY -store;
```
Here, -request-type READONLY disables write operations on a store and allows only read operations.
Reduce the JE free disk space to 2 GB on all nodes and increase the je.cleaner.minUtilization configuration parameter from 40 (the default in a KVStore) to 60.
```
kv-> plan change-parameters -all-rns \
-params "configProperties=je.cleaner.minUtilization 60; \
je.freeDisk 2147483648";
```
Executing this command creates more free space for store expansion. Replication Nodes will be stopped, parameters will be updated, and the Replication Nodes will be restarted with the new parameters.

Create, start, and configure the new nodes for expanding the store.

Create the new node. Run the makebookconfig utility to configure each Storage Node in the store:

java -Xmx256m -Xms256m -jar KVHOME/kvstore.jar makebootconfig \
-root sn1/KVROOT \
-store-security none -capacity 1 \
-port port1 -host node21 \
-harange 5010,5020 \
-storagedir /scratch/sn1/u01 –storagedirsize 20-Gb

java -Xmx256m -Xms256m -jar KVHOME/kvstore.jar makebootconfig \
-root sn2/KVROOT \
-store-security none -capacity 1 \
-port port1 -host node22 \
-harange 5010,5020 \
-storagedir /scratch/sn2/u01 –storagedirsize 20-Gb

java -Xmx256m -Xms256m -jar KVHOME/kvstore.jar makebootconfig \
-root sn3/KVROOT \
-store-security none -capacity 1 \
-port port1 -host node23 \
-harange 5010,5020 \
-storagedir /scratch/sn3/u01 –storagedirsize 20-Gb

Restart the Storage Node Agent (SNA) on each of the Oracle NoSQL Database nodes using the start utility:
```
kv-> nohup java -Xmx256m -Xms256m -jar \
KVHOME/lib/kvstore.jar start -root KVROOT &
```

Configure the new store:

java -Xmx256m -Xms256m -jar KVHOME/lib/kvstore.jar runadmin \
-port port1 -host node21

java -Xmx256m -Xms256m -jar KVHOME/lib/kvstore.jar runadmin \
-port port1 -host node22

java -Xmx256m -Xms256m -jar KVHOME/lib/kvstore.jar runadmin \
-port port1 -host node23

Redistribute the store according to its new configuration.

kv-> java -Xmx256m -Xms256m -jar KVHOME/lib/kvstore.jar runadmin \
-port port1 -host host1

kv-> plan deploy-sn -zn zn1 -host node21 -port port1 -wait
Executed plan 7, waiting for completion...
Plan 7 ended successfully
kv-> plan deploy-sn -zn zn1 -host node22 -port port1 -wait
Executed plan 8, waiting for completion...
Plan 8 ended successfully
kv-> plan deploy-sn -zn zn1 -host node23 -port port1 -wait
Executed plan 9, waiting for completion...
Plan 9 ended successfully
Plan 11 ended successfully

kv-> pool join -name ExamplePool -sn sn4
Added Storage Node(s) [sn4] to pool ExamplePool
kv-> pool join -name ExamplePool -sn sn5
Added Storage Node(s) [sn5] to pool ExamplePool
kv-> pool join -name ExamplePool -sn sn6
Added Storage Node(s) [sn6] to pool ExamplePool

kv-> topology clone -current -name newTopo
Created newTopo

kv-> topology redistribute -name newTopo -pool ExamplePool
Redistributed: newTopo

kv-> plan deploy-topology -name newTopo -wait
Executed plan 11, waiting for completion...

Restore the Replication Nodes to its original configuration.

plan change-parameters -all-rns \
-params "configProperties=je.cleaner.minUtilization 40; \
je.freeDisk 5368709120";

Enable write operations back on the store.
```
kv-> plan enable-requests -request-type ALL -store;
```
Here, —request-type ALL enables both read and write operations on the store.