Redistributing Data in a Database

You can increase or decrease the number of elements in which your data is distributed. However, this requires more than just adding or removing data instances from the current version of the model; you must also add or remove the elements of the data instances from the distribution map of the database.

The different tasks for maintaining the distribution map of a database are:

Add a replica set to the distribution map. When you add a replica set to the distribution map (and the distribution map is applied), TimesTen Scaleout re-distributes a portion of the data in the elements of each replica set to the elements of the newly added replica set.
Remove a replica set without a replacement from the distribution map. If the removed replica set is not replaced with another replica set, when the distribution map is applied, the data stored in the elements of the removed replica set is evenly re-redistributed into the elements of the remaining replica sets.
Remove a data instance and replace it with another data instance that is not already defined in the distribution map. In this case, when the distribution map is applied, the data is copied from the element of the removed data instance to the element of the new data instance; the data stored in the elements of the other replica sets is not re-distributed.
Evict a replica set from the distribution map. If all elements in a replica set have unrecoverable failures, evict the replica set from the distribution map. Evicting a replica set results in data loss. When you evict a replica set from the distribution map, you can either:
- Evict the replica set without a replacement. If the evicted replica set is not replaced with another replica set, when the distribution map is applied, the data in the evicted replica set is lost and the data stored in the elements of the other replica sets is not re-distributed.
- Evict and replace the replica set with another replica set that is not already defined in the distribution map. When the distribution map is applied, since the data in the elements of the evicted replica set is lost, the element of the new replica set is empty and the data stored in the elements of the other replica sets is not re-distributed.
  
  See Recovering When the Replica Set Has a Permanently Failed Element for information on how to evict failed replica sets from the distribution map.

The ttGridAdmin dbDistribute command can add and remove elements and evict replica sets from the distribution map of a database, then redistribute existing data across the resulting replica sets. Your existing data is redistributed once you apply the change to distribution map with the ttGridAdmin dbDistribute -apply command.

Note:

Data distribution cannot run concurrently with DDL or DML statements. As a result, the ttGridAdmin dbDistribute -apply command terminates with an error if you are currently executing any DDL or DML statements that insert, update, or delete data. Any DML statements that insert, update or delete while data distribution is in process are blocked until data distribution completes. However, you can run any read-only statements while data distribution is in process.

Figure 8-1 shows the database schema and topology of the elements of the database1 database that the examples in the following topics use.

Figure 8-1 Data Spaces and Replica Sets

Description of "Figure 8-1 Data Spaces and Replica Sets"

Adding Elements to the Distribution Map

To increase the number of elements in which your data is distributed, you need to first increase the number of data instances associated with the grid. Also, you must ensure that you have the same number of data instances to each data space group. For example, in a grid with k set to 3, you must add an equal number of data instances to the three available data space groups.

If you are adding elements to the distribution map of the database with the intention of increasing the amount of memory available in the permanent memory region, consider increasing the size of the permanent memory region instead. You can accomplish this by modifying the value of the PermSize attribute.

Note:

Every host with a data instance must have enough physical memory available to support the value of the PermSize attribute. See Determining the Value of the PermSize Attribute and Modify the Connection Attributes in a Database Definition for more information on how to calculate and modify the value of the PermSize attribute.
Consider that even when rows are re-distributed to the elements of the new data instances, the memory previously used by these rows in their original elements is still in use by a table page and can only be used by new rows of the same table.

Add a data instance for each data space group available to the current version of the model.

% ttGridAdmin hostCreate -internalAddress int-host9.example.com -externalAddress ext-host9.example.com -like host3 -cascade -dataSpaceGroup 1
Host host9 created in Model
Installation installation1 created in Model
Instance instance1 created in Model
 
% ttGridAdmin hostCreate -internalAddress int-host10.example.com -externalAddress ext-host10.example.com -like host3 -cascade -dataSpaceGroup 2
Host host10 created in Model
Installation installation1 created in Model
Instance instance1 created in Model

% ttGridAdmin hostCreate -internalAddress int-host11.example.com -externalAddress ext-host11.example.com -like host3 -cascade -dataSpaceGroup 3
Host host11 created in Model
Installation installation1 created in Model
Instance instance1 created in Model

% ttGridAdmin modelApply
...
Verifying installations...............................................OK
Creating new installations............................................OK
Verifying instances...................................................OK
Creating new instances................................................OK
...
Checking ssh connectivity of new instances............................OK
Starting new data instances...........................................OK
ttGridAdmin modelApply complete

Note:

See Adding Data Instances and Applying the Changes Made to the Model for more information on how to add data instances to a grid.

Figure 8-2 shows an example of the hash distribution of the customers table in one data space of the database1 database. Notice that the element of the host9.instance1 data instance is empty. Even though the host9 host is assigned to data space group 1, its element is not considered part of data space 1 until the host9.instance1 data instance is added to the distribution map of the database1 database.

Figure 8-2 Data Distribution of a Table

Description of "Figure 8-2 Data Distribution of a Table"

Add the element of the host9.instance1 data instance to the distribution map of the database1 database.

% ttGridAdmin dbDistribute database1 -add host9.instance1
Element host9.instance1 has been marked to be added
Distribution map change enqueued

To ensure that the distribution map of the database remains balanced, add the element of the data instance that will hold the replicas of the element of the host9.instance1 data instance, host10.instance1 and host11.instance1, to the distribution map of the database1 database.

% ttGridAdmin dbDistribute database1 -add host10.instance1
Element host10.instance1 has been marked to be added
Distribution map change enqueued

% ttGridAdmin dbDistribute database1 -add host11.instance1 -apply
Element host11.instance1 has been marked to be added 
Distribution map updated

Note:

Ensure that you only use the -apply option when you are done adding all new elements to the distribution map of the database to avoid TimesTen Scaleout returning an error.

Figure 8-3 shows how some of the data stored in the elements inside data space 1 in Figure 8-2 is re-distributed into the element of the new data instance, host9.instance1.

Figure 8-3 Data Distribution After Adding an Element (and Its Replica)

Description of "Figure 8-3 Data Distribution After Adding an Element (and Its Replica)"

You can verify the progress of the redistribution operation from any element of the database with the ttDistributionProgress built-in procedure.

Command> call ttDistributionProgress();
< 2018-12-04 14:49:48.872975, 1, 2, 1, Data Checkpoint, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, 1910, 0, 176, 1910, 8, 8 >
1 row found.

For more information on the ttGridAdmin hostCreate or ttGridAdmin dbDistribute command, see Create a Host (hostCreate) or Set or Modify the Distribution Scheme of a Database (dbDistribute), respectively, in Oracle TimesTen In-Memory Database Reference.

For more information on the ttGridAdmin modelApply command, see Applying the Changes Made to the Model and Model Operations in Oracle TimesTen In-Memory Database Reference.

For more information on the ttDistributionProgress built-in procedure, see ttDistributionProgress in Oracle TimesTen In-Memory Database Reference.

Removing Elements from the Distribution Map

You can remove and replace elements from the distribution map with the following in mind:

Remove and replace a single element:
- If you have a grid where k is set to 1, you can remove and replace the element only if both the element and data instance are operational.
- If you have a grid where k is set to 2 or greater, you can remove and replace a single element within a replica set by removing the element and replacing it with another element as long as another element in the replica set is operational.
  
  See Replace an Element with Another Element and Remove a Replica Set for more information on how to use the ttGridAdmin dbDistribute command with the -remove option.
Note:

Remove and Replace a Failed Element in a Replica Set has more information on how to resolve failure issues of a single element within a replica set.
Evict an entire replica set:
- If all the elements of a replica set have failed, then the data stored in the replica set is unavailable. Recovering When the Replica Set Has a Permanently Failed Element describes what happens when a replica set fails, how TimesTen Scaleout recovers the replica set, or how you can evict the entire replica set if the elements in the replica set cannot be automatically recovered.

The ttGridAdmin dbDistribute command with the -remove option removes an element from the distribution map of a database. When you remove a an element from the distribution map of a database, you have these options:

Replace an Element with Another Element

If the removed element is replaced with the element of a new data instance and you apply this change to the distribution map of the database, the data in the replica set is copied to the element of the new data instance. The data stored in the other replica sets is not re-distributed. Consider doing this when you want to replace a host with another one or a host must be shut down, but you do not want to modify the way your data is being distributed.

Add a data instance to the current version of the model.

% ttGridAdmin hostCreate -internalAddress int-host9.example.com -externalAddress ext-host9.example.com -like host3 -cascade -dataSpaceGroup 1
Host host9 created in Model
Installation installation1 created in Model
Instance instance1 created in Model
 
% ttGridAdmin modelApply
...
Verifying installations...............................................OK
Creating new installations............................................OK
Verifying instances...................................................OK
Creating new instances................................................OK
...
Checking ssh connectivity of new instances............................OK
Starting new data instances...........................................OK
ttGridAdmin modelApply complete

Note:

See Adding Data Instances and Applying the Changes Made to the Model for more information on how to add data instances to a grid.

Figure 8-4 shows an example of the hash distribution of the customers table in one data space of the database1 database. Notice that the element of the host9.instance1 data instance is empty. Even though the host9 host is assigned to data space group 1, its element is not part of a replica set until it is added to the distribution map of the database1 database.

Figure 8-4 Data Distribution of a Table

Description of "Figure 8-4 Data Distribution of a Table"

Remove the element of the host7.instance1 data instance and replace it with the element of the host9.instance1 data instance in the distribution map of the database1 database.

% ttGridAdmin dbDistribute database1 -remove host6.instance1 -replaceWith host9.instance1 -apply
Element host6.instance1 has been marked to be removed and replaced by element host9.instance1 
Distribution map updated

Figure 8-5 shows how the data previously stored in the element of the host7.instance1 data instance is copied to its replacement.

Figure 8-5 Data Distribution After Replacing an Element

Description of "Figure 8-5 Data Distribution After Replacing an Element"

To destroy the checkpoints and transaction logs of the removed element, use the ttGridAdmin dbDestroy -instance command.

% ttGridAdmin dbDestroy database1 -instance host6.instance1
Database database1 instance host6 destroy started

For more information on the ttGridAdmin dbDistribute or ttGridAdmin dbDestroy command, see Set or Modify the Distribution Scheme of a Database (dbDistribute) or Destroy a Database (dbDestroy), respectively, in Oracle TimesTen In-Memory Database Reference.

Remove a Replica Set

If you remove the element of a data instance without a replacement from the distribution map of a database, you must also remove its replicas. In other words, you must remove the replica set in its entirety. When you remove a replica set, TimesTen Scaleout re-distributes the data stored in the replica set to the remaining replica sets. Consider doing this when you want to scale down the number of hosts in which your data is stored.

Note:

Consider that the database size is defined by the value of the PermSize attribute times the number of replica sets available. Removing one replica set from the distribution map of the database will remove as many MB from the database size as MB set in the PermSize attribute. See Determining the Value of the PermSize Attribute for more information on how to determine the database size of a database.

Before removing a replica set, ensure that the remaining replica sets will have enough space to store a portion of the data stored in the replica set you are about to remove. If necessary, increase the database size by increasing the value of the PermSize attribute. See Modify the Connection Attributes in a Database Definition for more information on how to increase the value of the PermSize attribute.

Figure 8-6 shows an example of the hash distribution of the customers table in the database1 database.

Figure 8-6 Data Distribution of a Table

Description of "Figure 8-6 Data Distribution of a Table"

Remove the element of the host6.instance1 data instance from the distribution map of the database1 database.

% ttGridAdmin dbDistribute database1 -remove host6.instance1
Element host6.instance1 has been marked to be removed 
Distribution map change enqueued

To ensure that the distribution map of the database remains balanced, remove the element of the data instance holding the replicas of the element of the host6.instance1 data instance from the distribution map of the database1 database.

% ttGridAdmin dbDistribute database1 -remove host7.instance1
Element host7.instance1 has been marked to be removed 
Distribution map change enqueued

% ttGridAdmin dbDistribute database1 -remove host8.instance1 -apply
Element host8.instance1 has been marked to be removed 
Distribution map updated

Note:

To find out which data instance holds the replica of the element of another data instance, use the ttGridAdmin dbStatus command while specifying the -replicaset option.
Ensure that you only use the -apply option when you are done removing all the necessary data instances from the distribution map of the database to avoid TimesTen Scaleout returning an error.

Figure 8-7 shows how removing a replica set from the distribution map of a database removes its elements from their previously assigned data spaces. The figure also shows how the data previously stored in the removed replica set is re-distributed to the replica sets still within each data space.

Figure 8-7 Data Distribution After Removing a Replica Set

Description of "Figure 8-7 Data Distribution After Removing a Replica Set"

To destroy the checkpoints and transaction logs of the removed replica set, use the ttGridAdmin dbDestroy -instance command.

% ttGridAdmin dbDestroy database1 -instance host6.instance1
Database database1 instance host6 destroy started

% ttGridAdmin dbDestroy database1 -instance host7.instance1
Database database1 instance host7 destroy started

% ttGridAdmin dbDestroy database1 -instance host8.instance1
Database database1 instance host8 destroy started