Multi-Region Architecture

Oracle NoSQL Database applications read and write data by performing network requests against an Oracle NoSQL Database data store, referred to as the KVStore. Sometimes, organizations may need to set up multiple KVStores to maintain their NoSQL data. In more realistic situations, these KVStore clusters may even be geographically distributed. Oracle NoSQL Database multi-region architecture enables you to create tables in multiple KVStore clusters and maintain consistent data across these clusters.

For example, consider a use-case where an organization deploys three on-premise KVStores, one each at Frankfurt, London, and Dublin. In such a setup involving multiple KVStores, each independent Oracle NoSQL Database installation is referred to as a Region. Such an architecture having two or more independent, geographically distributed KVStore clusters bridged by bi-directional NoSQL Streams is known as Multi-Region Architecture.

Figure 1-3 Multi-Region Architecture

Description of "Figure 1-3 Multi-Region Architecture"

Suppose you want to collect and maintain similar data across multiple regions. You need a mechanism to create tables that can span across multiple regions and keep themselves updated with the inputs from all the participating regions. You can achieve these using Multi-Region tables. A Multi-Region Table or MR Table is a global logical table that is stored and maintained in different regions or installations. It is a read-anywhere and write-anywhere table that lives in multiple regions.

As you can see in the diagram, all the Multi-Region Tables defined in these regions are synchronized via NoSQL Streams. Essentially, all the distributed KVStores form a fully connected graph. For each distributed KVStore cluster, there is one inbound stream from each remote KVStore cluster. This inbound stream subscribes the local KVStore to all the Multi-Region Tables from the remote KVStore. Each region must be running a Cross-Region Service (XRegion Service) to receive the data from the subscribed tables in the remote regions.

In addition, note that in a Multi-Region Architecture are:

• The local and remote KVStores:
  • May have different topology
  • May experience elastic operations.
  • Are independently managed, that is, each KVStore has its own index, security credentials etc.
  • Have sufficient create and read table privileges to each other.
• The inbound and outbound streams are:
  • Completely symmetrical.
  • Independently managed without any coordination between the outbound and the inbound streams.

You can create child tables in the Multi-Region architecture. That means an existing Multi-Region table can have child tables. If you enable a top-level table in a Multi-Region architecture, the child tables created are automatically enabled in those regions. You need not explicitly specify the regions while creating the child tables. That is, the Multi-Region architecture is enabled for the whole hierarchy.

Cross Region Service

In a Multi-Region Oracle NoSQL Database setup, a Cross-Region Service or XRegion Service is a standalone service running on a separate node. In simple terms, this is also called an agent. The XRegion Service is deployed when you are connecting the local KVStore with a remote KVStore to create a Multi-Region Table.

Figure 1-4 Inbound Stream from Remote KVStore

Description of "Figure 1-4 Inbound Stream from Remote KVStore"

Consider an example where the remote KVStore has four shards and the local KVStore has three shards. The local KVStore deploys two NoSQL agents to stream two shards each from the remote KVStore, as depicted in the diagram above. These agents subscribe to updates from the remote KVStore and publish the new and modified rows to the local KVStore. All the agents connecting a remote KVStore to a local KVStore are referred as the Agent Group for the local KVStore.

Note:

Eventhough the above example demonstrates Agent Groups, please note that the current release of Oracle NoSQL Database does not support configuring multiple agents. As of the current release, you can configure only one agent for each local region.

As you can see in the diagram below, an agent enables streaming the data from a remote KVStore to a set of MR Tables in the local KVStore.

Figure 1-5 View of a Single Agent

Description of "Figure 1-5 View of a Single Agent"

Each inbound stream utilizes the subscriber group feature in the Streams API to create a group of subscribers to stream from a store. Each local agent is responsible for:

• Establishing the inbound subscription stream from a remote KVStore.
• Maintaining the connection and reconnect during any failures.
• Checkpointing the subscription stream in case of any failures.
• Subscribing writes to the MR Tables from a remote KVStore.
• Automatically dealing with elastic operations in the remote KVStore.

For a local KVStore, the overhead of inbound stream consists of:

• A group of threads in the NoSQL agent that streams the data from the remote KVStore. Please note that these threads run outside the local store as a standalone agent. Even though the agent serves the local KVStore, it does not add to the local KVStore's expense.
• Local PUT or DELETE resulting from the streamed data from the remote KVStore after conflict resolution.

For a local KVStore, the overhead of outbound stream involves the create, read, and write checkpoints for the remote KVStore.

Life Cycle of Multi-Region Tables

To create and use Multi-Region Tables (MR Tables) in Oracle NoSQL Database, you must be aware of the sequence of tasks to execute and the related concepts.

For clarity, let us discuss the life cycle of MR Tables with an example. Consider an Oracle NoSQL Database with three regions, Frankfurt, London, and Dublin. Assume that you want to create a table called Users to store the user details for all the three regions as depicted in the diagram.

Figure 1-6 Multi-Region Table

Description of "Figure 1-6 Multi-Region Table"

The sequence of tasks that you must perform to create and manage the Users table as a Multi-Region Table are:

• Deploy Independent KVStores: You must deploy KVStore in each region of the Multi-Region NoSQL Database independently. See Configuration Overview in the Administrator's Guide.
• Set Local Region Name: After deploying the KVStore and before creating the first MR Table in each participating region, you must set a local region name. You can change the local region name as long as no MR Tables are created in that region. After creating the first MR Table, the local region name becomes immutable. The local region name is completely independent of the KVStores created in that region. See Set Local Region Name in the Administrator's Guide.
• Configure XRegion Service: Before creating any MR Table, you must deploy an XRegion Service with one or more agents. The agent runs independently with the local KVStore and it is recommended to deploy it close to the local KVStore.
  While setting up a secure KVStore to support multi-region tables, the administrator has to grant the following permissions to the XRegion Service agent:

  • WRITE_SYSTEM_TABLE (or the equivalent writesystable role) to the local store.
  • Write permission on all the multi-region tables in the local store.
  • Read permission on all the multi-region tables in the remote stores.
  • Write permission for checkpoint table in the remote stores.

  To learn how to deploy an agent, see Configure XRegion Service in the Administrator's Guide.

• Start XRegion Service:You must start XRegion service in each region using the XRSTART command. As this service is a long-running process, it is recommended to invoke it as a background process by appending the & at the end of the command. see Start XRegion Service in the Administrator's Guide.

  Note:

  The local KVStore must be started before starting the XRegion Service. If the KVStore in the local region has not started or is not reachable, the XRegion Service will not start.
• Create Remote Regions: Before creating and operating on the MR table, you must define the remote regions. A remote region signifies a region different from the region where the command is executed. In this example, to create the MR Table called Users from the Frankfurt region, you must first define the other two regions, that is, London and Dublin using the CREATE REGION DDL command. To learn how to create remote regions, see Create Remote Regions in the Administrator's Guide.
• Create the MR Table: You must create an MR Table on each KVStore in the connected graph, and specify the list of regions that the table should span. In this example, to create the Users table as an MR Table at the Frankfurt and London regions, you need to execute the CREATE TABLE command specifying Frankfurt and London as the regions. The order in which you list the regions in the DDL command does not matter. After you create the Users MR Table, it will be included in the incoming stream from each remote KVStore specified. Symmetrically at the remote KVStore the Users table will be included in its own incoming stream too. To create the MR table successfully, you must:
  • Ensure that you acquire the necessary privileges to create the table in the specified regions, in advance. Otherwise, the MR Table creation will fail in all the regions. See KVStore Required Privileges in the Security Guide.
  • Specify at least one region in the CREATE TABLE DDL command. If you specify only one region, then the MR Table is created only in the specified region and no writes will be replicated to the other regions.

    Note:

    Even though a single-region MR table works similar to a local table, the difference between them is that the single-region MR Table can be expanded to multiple regions in future.
  To learn how to create an MR Table, see Create MR Tables in the Administrator's Guide.
• Perform Read/Write Operations on the MR Table (Optional): After creating the MR Table, you can perform read or write operations on the table using the existing data access APIs or DML statements. There is no change to any existing data access APIs or DML statements to work with the MR Tables. The following aspects applicable to the regular tables apply to the MR Tables also without any deviation:
  • Durability and consistency configurations and constraints: For any local writes to an MR table, the semantics of consistency model does not change. It is the same as any writes to a regular (non MR) table.

    Note:

    In case of MR Tables, absolute consistency is not global across the participating regions. It is only local to a single region where you perform the read and write operations.
  • Table index infrastructure: Creating primary and secondary indices on the MR Table in each region remains the same as with any regular (non MR) table. However, if you wish to drop an MR Table from any region, you must first drop all the indices defined on this table.
  Read Operations:
  • Each Read operation on an MR Table is a local read, that is, you read only the local copy of the data. However, this local copy may have rows that might have come from one of the other participating regions, as a result of a table sync-up via Oracle NoSQL Streams.
  Write Operations:
  • Whenever you execute a write operation (INSERT, UPDATE, or DELETE) on an MR Table, the changes will be replicated across multiple regions asynchronously. It means, when you write a row in the local region, the write operation is executed completely in the local region without waiting for the subscribing regions to update.
  • The latency for replicating the changes across multiple regions includes the time taken to:
    • Complete the write operations at the remote region, and
    • Receive the data from the subscribed tables.
  • If multiple regions update a row with the same primary key, a built-in conflict resolution rule is applied to decide which region's update is considered as final. In all such cases, this built-in conflict resolution rule will cause the update with the latest timestamp to win and commit to the database.
  • The above mentioned built-in conflict resolution rule applies to the TTL value updates too.
  • When you delete a row from an MR Table, the system allows time to ensure that the change is propagated to all the other regions where this table exists.
  • You can define a TTL value while inserting or updating a row in the MR Table. This value applies only to the row being added or updated. The row-level TTL overrides table level TTL if any exists.
  • An MR Table can have different table level TTL values in different regions.
  • When a row is replicated to other regions, its expiration time is replicated as an absolute timestamp to the replicated rows. This can be either the default table level TTL value or a row level override that is set by your application. Therefore, this row will expire in all the regions at the same time, irrespective of when they were replicated.
  • If a row expires before it makes it to one of the regions during replication, the rows that are already replicated to other regions will expire immediately after persistence.

  See Access and Manipulate MR Tables in the Administrator's Guide for examples.

• Add New Regions to the MR Table (Optional): Oracle NoSQL Database lets you expand an MR Table to new regions. It effectively means adding new regions to an existing MR Table. In the example being discussed, suppose you created the Users table only in two regions, Frankfurt and London. Later, if you want to expand this Users table to the Dublin region, you must:
  • Create the Users MR Table in the new region, that is, Dublin. Note that you must specify all the three regions while creating the MR Table in the new region. See Create MR Table in New Region in the Administrator's Guide.
  • Add the new region (Dublin) to the Users MR Table in existing regions, that is, Frankfurt and London. This is achieved with the help of the ALTER TABLE DDL command. See Add New Region to Existing Regions in the Administrator's Guide.

    Note:

    Depending on the volume of the data in the existing regions, it might take some time to initialize the MR Table in the new region with the data from the other regions. However, the MR Table in the new region is available for read/write operations immediately after its creation.

    To learn how to expand an MR Table with detailed code demonstrations, see Use Case 2: Expand a Multi-Region Table in the Administrator's Guide.

• Remove an Existing Region from the MR Table (Optional): Not only can you add new regions to an existing MR Table but also remove any regions linked to it. It effectively means that you disconnect the MR Table from a particular region so that MR Table is not synchronized with any writes from the removed region. This is called contracting an MR Table. To learn how to contract an MR Table with detailed code demonstrations, see Use Case 3: Contract a Multi-Region Table in the Administrator's Guide.
• Drop Remote Regions (Optional): You can drop one or more participating regions from a Multi-Region Oracle NoSQL Database setup as per your business requirement. However, before removing a region from a Multi-Region NoSQL Database, it is recommended to:
  • Stop writing to all the MR Tables linked to that region.
  • Ensure that all writes to the MR Tables in that region have synchronized with other regions. This helps in maintaining consistent data across the different regions.

  Note:

  Even though NoSQL Database lets you drop a region directly, it is a recommended practice to isolate that region from all the other regions before dropping it. This ensures that the existing regions are no longer linked with the region being dropped.

  To learn how to drop a region from a Multi-Region NoSQL Database, see Use Case 4: Drop a Region in the Administrator's Guide.

• Shut Down XRegion Service and KVstores: In a case where you want to relocate your XRegion Service to another host machine, you must shut it down in the current machine and then restart it in the new host machine. See Stop XRegion Service in the Administrator's Guide.

Using CRDT datatype in a multi-region table

Overview of the MR_COUNTER data type

MR_Counter data type is a counter CRDT. CRDT stands for Conflict-free Replicated Data Type. In a multi-region setup of an Oracle NoSQL Database, a CRDT is a data type that can be replicated across servers where regions can be updated independently and it converges on a correct common state. Changes in the regions are concurrent and not synchronized with one another. In short, CRDTs provide a way for concurrent modifications to be merged across regions without user intervention. Oracle NoSQL Database currently supports the counter CRDT which is called MR_Counter. The MR_COUNTER datatype is a subtype of the INTEGER or LONG or NUMBER data type. You can also use the MR_COUNTER data type in a schema-less JSON field, which means one or more fields in a JSON document can be of MR_COUNTER data type.

Why do you need MR_Counter in a multi-region table?

In a multi-region database configuration, copies of the same data need to be stored in multiple regions. This configuration needs to deal with the fact that the data may be concurrently modified in different regions.

Take an example of a multi-region table in three different regions (where data is stored in three different Oracle NoSQL Database stores). Concurrent updates of the same data in multiple regions, without coordination between the machines hosting the regions, can result in inconsistencies between the regions, which in the general case may not be resolvable. Restoring consistency and data integrity when there are conflicts between updates may require some or all of the updates to be entirely or partially dropped. For example, in the current configuration of a multi-region table in the Oracle NoSQL Database, if the same column (a counter) of a multi-region table is updated across two regions at the same time with different values, a conflict arises.

Currently, the conflict resolution is that the latest write overwrites the value across regions. For example, Region 1 updates column1 with a value R1, and region2 updates column1 with a value R2, and if the region2 update happens after region1, the value of the column (counter) in both the regions becomes R2. This is not what is actually desired. Rather every region should update the column (a counter) at their end and also the system internally needs to determine the sum of the column across regions.

One way to handle this conflict is making serializable/linearizable transactions (one transaction is completed and changes are synchronized in all regions and only then the next transaction happens). A significant problem of having serializable transactions is performance. This is where MR_COUNTER datatype comes in handy. With MR_COUNTER datatype, we don't need serializable transactions and the conflict resolution is taken care of. That is, MR_COUNTER datatype ensures that though data modifications can happen simultaneously on different regions, the data can always be merged into a consistent state. This merge is performed automatically by MR_COUNTER datatype, without requiring any special conflict resolution code or user intervention.

Use-case for MR_COUNTER datatype

Consider a Telecom provider providing different services and packages to its customers. One such service is a "Family Plan" option where a customer and their family share the Data Usage plan. The customer is allocated a free data usage limit for a month which your the customer's entire family collectively uses. When the total usage of customer's family reaches 90 percent of the data limit, the telecom provider sends the customer an alert. Say there are four members in customer's family plan who are spread across different physical regions. The customer needs to get an alert from the telecom provider once the total consumption of their family reaches 90 percent of the free usage. The data is replicated in different regions to cater to latency, throughput, and better performance. That means there are four regions and each has a kvstore containing the details of the customer's data usage. The usage of their family members needs to be updated in different regions and at any point in time, the total usage should be monitored and an alert should be sent if the data usage reaches the limit.

An MR_COUNTER data type is ideal in such a situation to do conflict-free tracking of the data usage across different regions. In the above example, an increment counter in every data region's data store will track the data usage in that region. The consolidated data usage for all regions can be determined by the system at any point without any user intervention. That is the total data usage at any point in time can be easily determined by the system using an MR_COUNTER datatype.

Types of MR_COUNTER Datatype

Currently, Oracle NoSQL Database supports only one type of MR_COUNTER data type. which is Positive-Negative (PN) counter.

Positive-Negative (PN) Counter

A PN counter can be incremented or decremented. Therefore, these can serve as a general-purpose counter. For example, you can use these counters to count the number of users active on a social media website at any point. When the users go offline you need to decrement the counter.

To create a multi-region table with an MR_COUNTER column, See Create multi-region table with an MR_COUNTER column section in the Administrator's Guide.