TimesTen Scaleout Architecture

One OS user creates and manages a grid. This user is called the instance administrator. See Instance Administrator in Oracle TimesTen In-Memory Database Installation, Migration, and Upgrade Guide. TimesTen Scaleout enables the instance administrator to:

Configure whether the grid creates one or more copies of your data by using K-safety.
Create one or two management instances through which the grid is managed.
Create multiple data instances in which data is contained and managed.
Set up a membership service to track which data instances are operational at any moment. The membership service consists of three or more membership servers.
Create one or more databases.
Create one or more repositories to store backups for your databases.

Figure 1-2 Grid Structure

Description of "Figure 1-2 Grid Structure"

A database consists of multiple elements, where each element stores a portion of data from its database. Each data instance contains one element of each database. If you create multiple databases in the grid, then each data instance contains multiple elements (one from each database).

For each database you create, you decide which elements participate in data distribution. Usually, all elements participate, but when you bring online new data instances, you decide when the elements of those new data instances begin to participate in database operations. You need to explicitly add elements into the distribution map of database for them to participate in database operations. Likewise, you need to remove elements from the distribution map (which stops them from participating in database operations) before you can remove their data instances from the grid.

Upon including an element into the distribution map, each element of a database is automatically placed into a replica set. Each replica set contains the same number of elements as the value set for K-safety. Elements in the same replica set hold the same data.

The following topics provide a more detailed description of these components and their responsibilities within a grid:

Instances

A grid uses instances to manage, contain, and distribute one or more copies of your data. An instance is a running copy of the TimesTen software. When you create an instance on a host, you associate it with a TimesTen installation. An installation can be used by a single instance or shared by multiple instances. Each instance usually resides on its own host to provide maximum data availability and as a safeguard against data loss should one host fail.

Each instance has an associated instance administrator (who created the instance) and an instance home. The instance home is the location for the instance on your host. The same instance administrator manages all instances in the grid.

TimesTen Scaleout supports two types of instances:

Management Instances

Management instances control a grid and maintain the model, which is the central configuration of a grid. To ensure that the administrator can easily control a grid, all management activity is processed through a single management instance using the ttGridAdmin utility.

Note:

See Central Configuration of the Grid for more details on the model.

TimesTen Scaleout enables you to create two management instances to provide for high availability and guard against a single management instance failure that could impede grid management. Consider having two management instances a best practice for a production environment. Once created, TimesTen Scaleout configures both management instances in an active standby configuration. You always run all management operations through the active management instance. The standby management instance exists purely as a safeguard against failure of the active management instance.

If you create two management instances, as shown in Figure 1-3, then all information used by the active management instance is automatically replicated to the standby management instance. Thus, if the active management instance fails, you can promote the standby management instance to become the new active management instance through which you continue to manage the grid.

Note:

See Managing Failover for the Management Instances for details on how TimesTen Scaleout replicates information for the management instances.

Figure 1-3 Administrator Manages the Grid Through Management Instances

Description of "Figure 1-3 Administrator Manages the Grid Through Management Instances"

Consider that:

You can manage a grid through a single management instance without a standby management instance. However, it is not recommended for production environments.
If both management instances fail, databases in the grid continue to operate. Some management operations are unavailable until you restart at least one of the management instances.

Data Instances

Data instances store one element per database in the grid. Data instances run SQL statements and PL/SQL blocks. A grid distributes the data within each database across data instances. You manage all data instances through the active management instance, as shown in Figure 1-4.

Figure 1-4 Management Instances Manage a Grid of Multiple Data Instances

Description of "Figure 1-4 Management Instances Manage a Grid of Multiple Data Instances"

Installations

Instances need an installation of a TimesTen distribution to operate. An installation is read-only and can be used locally or shared across multiple instances. You create the installation of the initial management instance by extracting a TimesTen distribution on any given location on the system defined as the host of the management instance. TimesTen Scaleout can locally create any subsequent installation on the rest of the hosts in the grid and associate the new installations with the instances run by those hosts. All instances that run on the same host may share the same installation.

As long as an installation can be accessed by multiple hosts that installation can be shared by instances in those hosts. However, sharing an installation on a shared file server, such as NFS, between multiple instances on separate hosts may reduce availability. If the shared network storage or the network connecting all of the hosts to the NFS server fails or has performance issues then all instances sharing that installation are impacted. Thus, while sharing an installation on a shared file server across instances may be a valid option for a development environment, you may want to evaluate whether this is advisable for a production environment.

K-Safety

You configure your grid to create either single or multiple copies of the data of each database within your grid. TimesTen Scaleout uses its implementation of K-safety (k) to manage one or multiple copies of your data. You specify the number of copies you want of your data by the value set for k when you create the grid.

You improve data availability and fault tolerance when you specify that the grid creates two or more copies of data located across separate hosts.

If you set k to 1, TimesTen Scaleout stores a single copy of the data (which is not recommended for production environments).

When k is set to 1, then the following may occur if one or more elements fail:
- Any data contained in the element is unavailable until the element recovers.
- Any data contained in the element is lost if the element does not recover.
Even though there is only a single copy of the data, the data is still distributed across separate elements to increase capacity and data accessibility.
If you set k to 2 (or greater), then TimesTen Scaleout stores k copies of the data. A grid can tolerate multiple faults when you have multiple copies of the data.

If one element fails, another copy of the data is accessed to provide the requested data. K-safety enables availability to your data as long as one of the copies of the data is available. Where possible, locate each copy of the data on distinct physical hardware for maximum data safety.

The following topics describe how multiple copies are managed and organized.

Understanding Replica Sets

Each element of a database is automatically placed into a replica set depending on the value of k, where:

If you set k to 1, then each replica set contains a single element.
If you set k to 2 (or greater), then each replica set contains k elements (where each element is an exact copy of the other elements in the replica set).

Thus, each replica set contains the same number of elements as the value set for k.

When k is set to 2 (or greater), any change made to the data in one element is also made to the other elements in the replica set to keep the data consistent on all elements in the replica set at all times. Because of the transparency capabilities of TimesTen Scaleout, you can initiate transactions on any element, even if the requested data is not contained in that element or if the requested data spans multiple replica sets. If an element fails, then one of the other elements in the replica set is accessed to provide the requested data. All data in the database is available as long as one element in each replica set is functioning.

Understanding Data Spaces

Each database consists of a set of elements, where each element stores a portion of data from its database. The grid organizes the elements for each database into data spaces.

Each database consists of either one or two data spaces. When k is set to 2 (or greater), the elements within each replica set are assigned to separate data spaces.

Figure 1-5 shows how three copies of the data are organized within three data spaces, where each data space contains the elements that make up a single copy of the data of the database. There are two replica sets and the elements of each replica set are assigned to a separate data space. Thus, each element in data space 1 is identical to its replicas in data space 2 and 3.

Figure 1-5 Three Copies, Each in Own Data Space

Description of "Figure 1-5 Three Copies, Each in Own Data Space"

As your needs grow or diminish, you may add or remove replica sets to a grid. When you add data instances, the grid automatically creates elements for each database. However, the data is not automatically redistributed when you add or remove a data instance. You decide when it is appropriate to assign an element to a replica set and redistribute the data across all the elements in each data space.

Assigning Hosts to Data Space Groups

You decide how the data is physically located by assigning hosts into data space groups that represents the physical organization of your grid. As discussed in Understanding Data Spaces, copies of the data are organized logically into data spaces. Each data space should use separate physical resources. Shared physical resources can include similar racks, the same power supply, or the same storage. Be aware that if all elements in a single replica set are stored on hosts that share a physical component, then data stored in that replica set becomes unavailable if that shared physical component fails.

TimesTen Scaleout requires you to assign all hosts that will run data instances into data space groups. When using K-safety, there are k copies of the data and the same number of data space groups (which are numbered from 1 to k). You should assign hosts that share the same physical resources into the same data space group. The elements in data instances running on hosts that are assigned to the same data space group are in the same data space. Each data space contains a full copy of all data in the database.

If you ensure that the hosts in one data space group do not physically share resources with the hosts in another data space group, then hosts in separate data space groups are less likely to fail simultaneously. This scenario makes it likely that all data in the database is available, even if a single hardware failure takes down multiple hosts. For example, you may ensure that all of the hosts in one data space group are plugged into a power supply that is separate from the power supply for the hosts in another data space group. If that is the case, pulling one plug does not power down all the hosts in a single replica set, thus making some data unavailable.

Figure 1-6 shows a grid configured where k is set to 3, so the grid contains three data space groups. There are three racks, each with independent power sources and two hosts. Two hosts have been assigned to each data space group. TimesTen Scaleout creates replica sets such that such that one element in each replica set is in each data space group.

Figure 1-6 Hosts Organized into Data Space Groups

Description of "Figure 1-6 Hosts Organized into Data Space Groups"

The process for assigning hosts to a data space group includes deciding how you will physically separate the hosts supporting the data spaces.

Data Distribution

You can create one or more databases within a grid. Each database is independent, with separate users, schemas, tables, persistence, and data distribution. TimesTen Scaleout manages the distribution of the data according to the defined distribution map and the distribution scheme for each table.

Defining the Distribution Map for a Database

You decide on the number of data instances in a grid, which dictates the maximum number of elements and replica sets for any one database. Each data instance hosts one element of each database in the grid. Thus, the data instances in a grid can manage one or more databases simultaneously. If you create multiple databases in the grid, then each data instance will contain multiple elements (one element from each database).

Each database consists of multiple replica sets, where each replica set stores a portion of data from its database. You define which elements of the available data instances store data of the database with a distribution map. Once the distribution map is defined and applied, TimesTen Scaleout automatically assigns each element to a replica set and distributes the data to its corresponding replica set, where each element communicates with other elements of different replica sets to provide a single database image. The details of how data is distributed may vary for each table of a database based on the distribution scheme of the table.

Note:

TimesTen Scaleout stores the composition of the distribution map, or how every data instance associates with each other, in a partition table that is managed by the ttGridAdmin utility.

Once you add the elements of the data instances that will manage and contain the data of each database to the distribution map, you can explicitly request that the data be distributed across the resulting replica sets.

As the needs of your business change, you can increase the capacity of a database by increasing the number of replica sets in the grid. You can accomplish this by:

Adding new hosts to the grid. The number of hosts you add must be proportional to the number of replica sets you want to add and the value of K-safety. For example, if you want to add another replica set to a database in a grid with k set to 3, you need to add a host for each of the three data space groups available.
Creating an installation to support data instances on each new host.
Creating a data instance on each new host.
Adding the elements of the new data instances to the distribution map of each database you want to increase its capacity. TimesTen Scaleout automatically creates new replica sets as appropriate.
Redistributing the data across all replica sets.

When you add new data instances or remove existing data instances to the grid, the grid does not automatically re-distribute the data stored in the database across the replica sets of those new or remaining data instances. Instead, you decide when is the appropriate time to re-distribute the data across the existing data instances. Redistribution can negatively impact your operational database. You should redistribute in small increments to minimize the impact. The larger the number of data instances that you have, the less of an impact it is to incrementally add or remove a single replica set.

If you need to replace a data instance with a new data instance in the same data space group, this action does not require a redistribution of all data.

To reduce your capacity, remove the data instances that manage a replica set from the distribution map and redistribute the data across the remaining data instances in the grid.

Defining the Distribution Scheme for Tables

TimesTen Scaleout distributes the data in a database across replica sets. All tables in a database are present in every replica set. You define the distribution scheme for each table in a database in the CREATE TABLE statement. The distribution scheme describes how the rows of the table are distributed across the grid.

How the data is distributed is defined by one of the following distribution schemes specified during table creation.

Hash: The data is distributed based on the hash of the primary key or a composite of multiple columns that are specified by the user. A given row is present in a replica set chosen by the grid. Rows are evenly distributed across the replica sets. This is the default method as it is appropriate for most tables.

See Figure 1-7 for an example of a table, terry.customers, with a hash distribution scheme. Each element belongs to a different replica set.

Figure 1-7 Table with Hash Distribution

Description of "Figure 1-7 Table with Hash Distribution"
Reference: Distributes the data of a child table based on the location of the parent table that is identified by the foreign key. That is, a given row of a child table is present in the same replica set as its parent table. This distribution scheme optimizes the performance of joins by distributing related data within a single replica set. Thus, this distribution scheme is appropriate for tables that are logically children of a single parent table as parent and child tables are often referenced together in queries.

See Figure 1-8 for an example of a child table, accounts, with a reference distribution scheme to a parent table, customers. Each element belongs to a different replica set.

Figure 1-8 Table with Reference Distribution

Description of "Figure 1-8 Table with Reference Distribution"
Duplicate: Distributes full identical copies of data to all the elements of a database. That is, all rows are present in every element. This distribution scheme optimizes the performance of reads by storing identical data in every data instance. This distribution scheme is appropriate for tables that are relatively small, frequently read, and infrequently modified.

See Figure 1-9 for an example of a table, account_type, with a duplicate distribution scheme. Each element belongs to a different replica set.

Figure 1-9 Table with Duplicate Distribution

Description of "Figure 1-9 Table with Duplicate Distribution"

Backups

TimesTen Scaleout enables you to create backups of the databases in your grid and restore them to the same grid or another grid with a similar topology. TimesTen Scaleout also enables you to export your databases to a grid with a different topology. You define a repository as a location for your database backups, exports, and collections of log files. Multiple grids may use the same repository.

Internal and External Networks

For most production environments, TimesTen Scaleout requires a single private internal network and at least one external network.

Internal network: Instances in a grid communicate with each other over a single internal network using the TCP protocol. In addition, instances communicate with membership servers through this network. Membership servers use this network to communicate among themselves.
External networks: Applications use the external network to connect to data instances to access a database. Applications do not need external network access to management instances or membership servers.

See Network Requirements.