Global Active Tables in NDCS
Oracle NoSQL Database Cloud Service supports a global active table architecture in which you can create tables, replicate them across multiple regions, and maintain synchronized data across the regional replicas.
Today's businesses need to provide faster and better services to their customers. Network latency is a crucial parameter for assessing the performance of any application. Network latency is the minimum time a data packet takes to travel across the network. Users expect to complete their online activities smoothly and quickly from anywhere. To meet such expectations, enterprises need to host applications and data in distributed regions closest to their users.
Oracle NoSQL Database Cloud Service provides a solution to these requirements through Global Active tables. This feature enables application data written in a region to be replicated transparently across multiple regions.
- Read and write locally, and access your data globally: An active-active configuration of Global Active tables in multiple regions ensures that an update performed on a table in one region is automatically replicated to the table's replicas in other replication regions. The data can be accessed from any replicated region.
- Performance: Global Active tables enable you to read and write your data locally, providing single-digit millisecond latency, which is characteristic of local access for your globally distributed application at any scale. This can boost the performance of massively scaled global applications and reduce the latency for application requests.
- Easy to set up and manage: Oracle NoSQL Database Cloud Service eliminates the complexity of deploying and managing multi-region replication using Global Active tables. Adding regional table replicas is simple and can be done using APIs, Terraform, or the OCI console.
- Multi-Region Resiliency: Global Active tables also enable fault tolerance in the rare case of a region failure. If a single region becomes unavailable or offline, the application requests can be redirected to a region where the table is replicated, and reads and writes can be performed against this table.
How to Choose a Global Active Setup in NDCS?
The Global Active tables feature enables application data written in a region to be replicated transparently across multiple regions.
A rare event of a single region failure should not impact the experience of the users. For instance, if a single region becomes offline, isolated, or degraded, your application should be redirected to a different region and continue to perform reads and writes as before. In short, your database needs to provide disaster recovery even when a region fails. You can use a Global Active table in Oracle NoSQL Database Cloud Service (NDCS) to provide disaster recovery through active-active configuration or to actively replicate data across multiple regions to achieve low latency using local data access.
Consider the needs of a traveling user who shops from a popular website. They may access the same shopping website from different parts of the world on the same day. You need to ensure the user experiences minimum latency and a seamless interaction no matter where they are.
The Global Active table feature in NDCS provides a solution to both the scenarios discussed above. Let us explore each of the two scenarios and understand how a Global Active table will be the best solution to provide high availability, low latency, and disaster recovery.
In the first scenario, assume your company uses NDCS in Phoenix (US -West), Frankfurt (Germany), and Tokyo (Japan) and you have a table called ShoppingCart, which stores the shopping information from customers who are shopping in different regions across the globe. In this example, your company is servicing its customers via data centers that are geographically closest to them. Such a setup involves multiple geographic locations where Oracle NoSQL Database Cloud Service is available. An architecture having two or more geographically distributed regions and NoSQL Database Cloud service available in each of these regions is known as a global active table architecture. The table ShoppingCart is a Global Active table and is replicated in multiple regions.
In an active-active configuration, you have NDCS available in multiple regions, and the data across all regions are synchronized. When a region fails, Global Active tables in the other replica regions will continue to work as usual and will not be affected by the failed region. When the failed region comes back, its regional table replica will be synchronized with the other regions. The Global Active table provides disaster recovery in that when a region is down, your application is connected to another replica.
In the second scenario, assume the user in Phoenix, shops online, and adds a mobile phone to their shopping cart. The west coast NDCS region serves this session, and the user experiences minimum read and write request latency from the region's local data store. This user then gets on a plane to Germany, lands 13 hours later, gets to the hotel, connects to the Wi-Fi network, goes to the mobile company's online store, and finds that there's another model of the phone that looks more appealing. So the user decides to update the shopping cart with this new model of the phone and continues to browse the mobile e-commerce store. The regional data store in Frankfurt, which is the most proximal data store, serves this session, and provides the user with the same low latency read and write experience as the one in the US. The user then travels to Japan and decides to visit a local mobile store to get more info on the latest mobile models. The user then updates the shopping cart with the latest model of the phone that is present at the mobile store. As NoSQL Database Cloud Service is available in three regions one in Phoenix, second in Germany, and third in Japan and there is more than one regional table replica, whenever the user updates the shopping cart or browses the mobile e-commerce store, the personalized search results, and other data get fetched from a local region closest to the user. This kind of local processing offers the lowest latencies, the best performance, and eliminates wide area network access.
Basic Concepts
Terminology used in Global Active tables:
- Region: An Oracle Cloud Infrastructure (OCI) region. It is a single localized geographic area where customers can deploy their applications.
- Sender Region: A region from where a table update is replicated to other regions.
- Receiver Region: A region that receives the table update from another region.
- Singleton table: A table that is not regionally replicated.
- Regional table replica: A replica of a table created in another region.
- Global Active table: A table that has one or more regional table replicas.
Basic rules for creating and managing Global Active tables:
- The singleton table must have at least one JSON column.
- The schema state of the table must be FROZEN.
- In the receiver region, a table with the same name must not already exist.
- In the receiver region, the compartment (with the same name as the one in the sender region) must already exist.
- Before dropping a table, all the regional table replicas of the table must be removed.
Note:
In NDCS the regional table replication is performed asynchronously in the background.You can create child tables on a Global Active Table. A child table of a Global Active table can be a singleton table or a Global Active Table. To make the child table a Global Active table, you need to freeze the child table's schema and add a regional replica. You can pick from one of the regional replicas of the parent table.
Global Active Table Workflow
What gets replicated in a Global Active table?
- Table definition/schema
- Indexes in the table - Number and definitions of secondary indexes.
- Data in the table.
- Read capacity and write capacity - By default, the read limit of a regional replica is the same as that of any other regional replicas of the Global Active Table. However, read operations are purely local, so you can optionally set each region's own read limits. By default, the write limit of a regional replica is the same as that of any other regional replicas of the Global Active table. However, write limits can be set to different values in each region.
- Storage Capacity - Since all regional replicas of the Global Active table store the same table data, the storage limit of a regional replica is copied to all other regional replicas of the Global Active table.
- Table's default Table Time to Live (TTL)
Adding a regional table replica
Note:
When you add a regional table replica, the table in the receiver region is placed in the same compartment with the same table name as the table in the sender region. Over the life of the Global Active table, you can move it to another compartment at any time.Capacity (Read Units, Write Units, and Storage) for regional table replicas
When you add a regional replica of a table, read capacity, write capacity and storage capacity fields automatically default to the corresponding values of the table in the sender region. However, you can edit and change the values of the read capacity and write capacity of the table in the receiver region. The storage capacity of the table cannot be changed. The table in the receiver region has the same storage capacity as the table in the sender region. The capacity mode of a Global Active table can be either on-demand or provisioned. You can also change the capacity mode of any regional replica for a Global Active table after it is created. The change in capacity mode is local to that regional replica and does not affect any other regional replica of the Global Active table.
TTL of records in regional table replicas
Table Time to Live (TTL) is expressed as the amount of time (number. of hours or days) the data is allowed to live in the table. Oracle NoSQL Database Cloud Service lets developers set an expiry time on table rows, after which the rows expire automatically, and are no longer available. After the specified duration, the rows expire automatically and are no longer available. The TTL in the regional table replica is the same value as the table's TTL in the sender region. When a row is replicated to other regions, its expiration time is replicated as an absolute timestamp. Therefore, this row will expire at the same time, irrespective of when it was replicated.
Once a regional table replica is created, it is initialized with the data from the sender region's table. During this initialization of table data, if the TTL value is reached, these rows will expire and a read operation will not see these records.
Scope of DDL operations after regional table replicas are created:
- Add Index
- Drop Index
- Change in Storage Capacity of the table
- Change in table TTL
- Change in Read Units
- Change in Write units
- Change in Capacity Mode from On-Demand to provisioned or vice-versa
Data Modeling Considerations for Global Active Tables
Why must you freeze a table’s schema?
In a Global Active table configuration, you have tables in NDCS deployed across multiple regions, and all the regions are simultaneously serving read and write requests. The application connecting to NDCS should be designed to connect to the closest replication region. The table's primary key, shard key, and data must be identical across replication regions as these three are an intrinsic part of how the application uses the table and how queries execute. In a Global Active table, as the table records can be written simultaneously to the table in multiple regions, it becomes necessary to reach a consensus on the schema for the table. You can do this by preventing the schema from changing, forcing all regions into immediate consensus about the schema for a table. For simplicity, performance, and predictability, the Oracle NoSQL Cloud Service requires a schema to be frozen for any table that is participating in regional replication. Thus, before converting a singleton table to a Global Active table, the table schema must be frozen and no further schema changes are allowed. Should you need to change a Global Active table's schema after creating regional replicas, you must first drop all regional replicas, alter the table's schema, and then re-add the regional replicas. Oracle NoSQL Cloud service will re-populate all regional replicas with the most current data from the sender region.
Why is at least one JSON column required in a table when freezing its schema?
Coordinating a schema alteration in regional table replicas is difficult and leads to potential error cases. Providing a JSON column provides more flexibility in the schema and prevents the need for a schema alteration.
Defining Identity in a Global Active Table
-
The identity of a record in a regional table replica should be unique across all of the table's regional replicas. Natural keys (globally unique identifiers that identify each record in a table) can be used as identity in Global Active tables only if they can guarantee uniqueness across all regional table replicas.
- While using system-generated identity in a Global Active table, UUID should be used. In most cases, the identity column should not be used because it is not guaranteed to be unique across regional table replicas.
Policies & User Permissions
A table's IAM policies are specific to the sender region.
When a user adds a replica of a singleton table or a Global Active table, no policy or user permission is added in the receiver region. The user in the source region, wanting to create and manage replicas must also have the necessary privileges in the receiver region. Otherwise, you get an error when the user adds a regional table replica.
Once the regional table replicas are created, replicating any data modification from a sender region to the receiver region does not require any user permission. That means data change in one replica table will be replicated in all the other table replicas regardless of user permission. The permissions needed for creating and managing the regional table replicas are listed below.
Add replica:
The users who want to manage replicas of a table must have NOSQL_TABLE_ALTER permission in the sender region and all the existing receiver regions. The users who want to create a new replica need to have NOSQL_TABLE_CREATE permission in the receiver region (where a replica has to be created). When you create a regional table replica, the existing read and write permission of the table in the sender region is not created in the receiver region. The users who want to create a new replica in the receiver region are responsible for creating the table read and write permissions for every regional table replica that they create.
Drop replica:
The users who want to manage replicas of a table must have NOSQL_TABLE_ALTER permission in the sender region and all the existing receiver regions. The users who want to drop a replica need to have NOSQL_TABLE_DROP permission in the receiver region (where a replica has to be dropped).
Create Index:
The users who want to create indexes in regional table replicas need to have NOSQL_INDEX_CREATE permission.
Drop Index:
The users who want to drop indexes in regional table replicas need to have NOSQL_INDEX_DROP permission.
Update table limits/TTL/:
The users who want to manage replicas of a table must have NOSQL_TABLE_ALTER permission in all regional table replicas.
Reads, Writes & ACID transactions
After creating a Global Active table, you can perform read or write operations on the table using the existing data access APIs or DML statements.
For the best latency, your application will typically read from a local regional replica. The data in the local regional replica will also include the data updates replicated from other regional table replicas. Whenever you execute a write operation (INSERT, UPDATE, or DELETE) on a Global Active table, the changes will be replicated across multiple regions asynchronously. That is, when you write a row in the sender region, the write operation is executed completely in the sender region without waiting for the replica regions to update. If multiple regions update a row with the same primary key, a rule is applied to decide which region's update is considered final. In all such cases, this built-in conflict resolution rule will cause the update with the latest timestamp to win and commit to the database. This rule also applies while updating the TTL value from multiple regions simultaneously.
ACID transactions are local to a region. When a transaction commits, it is only guaranteed to be atomic, consistent, isolated, and durable, in the region where the write occurred. The consistency model semantics of a regional table replica is the same as that of a non-replicated table. The consistency of the regional table replicas is not absolute. Absolute consistency is local to a single region where you perform the read operation. Reads on the data being replicated from the sender region to the receiver regions are always eventually consistent.
Writes from a sender region are replicated across all receiver regions within a time lag. This time lag to replicate the changes across multiple regions includes the time taken to receive the data from the regional table replica in the sender region and the time taken to complete the write operations at the receiver region. Eventually, the receiver region gets the update from the sender region and the receiver region never misses a transaction update that happened in the sender region. All the regional table replicas will eventually receive the write and become durable.
Overview of Replica Lag
Whenever you execute a write operation (INSERT, UPDATE, or DELETE) on a Global Active table, the changes will be replicated across multiple regions asynchronously.
That is, when you write a row in the sender region, the write operation is executed completely in the sender region without waiting for the replica regions to update. The network latency creates a time lag in replicating the changes to the receiver regions. The latency for replicating the changes across multiple regions includes the time taken to receive writes from the replica and apply them in the receiver region. If there has been no application writes for the table at the sender region, the service uses the ping mechanisms to calculate an approximation of the lag, and the lag statistic will still be available in the receiver region.
For more details on the Replica Lag metric see Details on Replica Lag.
Overview of creating a Global Active table
The process of creating a Global Active table starts with creating a singleton table and then converting it to a Global Active table
- Create a singleton table and make sure one column is JSON.
- Change the schema state of the table from Mutable to Frozen.
- Decide the region where you want to add a regional replica of the table. While adding a regional replica, the read capacity, write capacity, and storage capacity fields are automatically populated with the corresponding values of the sender region. You can change the read capacity and write capacity of the table.
- The cloud service creates the table in the receiver region. If the table in the
source region has data, it starts copying data from the sender region to the
receiver region. As the data is being copied from the sender region to the
receiver region, the initialization percentage increases from 0 to 100. You can
view the value of the initialization percentage under the table information in
the OCI console as shown below. No DML operations are allowed in the new
regional table replica until the initialization process is complete.
