Multi-Region Architecture

Oracle NoSQL Database applications read and write data by performing network requests against an Oracle NoSQL Database data store, referred to as the KVStore. Sometimes, organizations may need to set up multiple KVStores to maintain their NoSQL data. In more realistic situations, these KVStore clusters may even be geographically distributed. Oracle NoSQL Database multi-region architecture enables you to create tables in multiple KVStore clusters and maintain consistent data across these clusters.

For example, consider a use-case where an organization deploys three on-premise KVStores, one each at Frankfurt, London, and Dublin. In such a setup involving multiple KVStores, each independent Oracle NoSQL Database installation is referred to as a Region. Such an architecture having two or more independent, geographically distributed KVStore clusters bridged by bi-directional NoSQL Streams is known as Multi-Region Architecture.

Figure 1-3 Multi-Region Architecture

Description of "Figure 1-3 Multi-Region Architecture"

Suppose you want to collect and maintain similar data across multiple regions. You need a mechanism to create tables that can span across multiple regions and keep themselves updated with the inputs from all the participating regions. You can achieve these using Multi-Region tables. A Multi-Region Table or MR Table is a global logical table that is stored and maintained in different regions or installations. It is a read-anywhere and write-anywhere table that lives in multiple regions.

As you can see in the diagram, all the Multi-Region Tables defined in these regions are synchronized via NoSQL Streams. Essentially, all the distributed KVStores form a fully connected graph. For each distributed KVStore cluster, there is one inbound stream from each remote KVStore cluster. This inbound stream subscribes the local KVStore to all the Multi-Region Tables from the remote KVStore. Each region must be running a Cross-Region Service (XRegion Service) to pull the data from the subscribed tables in the remote regions.

In addition, a list of points worth noting regarding the Multi-Region Architecture are:

• The local and remote KVStores:
  • May have different topology
  • May experience elastic operations.
  • Are independently managed, that is, each KVStore has its own index, security credentials etc.
  • Have sufficient create and read table privileges to each other.
• The inbound and outbound streams are:
  • Completely symmetrical.
  • Independently managed without any coordination between the outbound and the inbound streams.

Cross Region Service

In a Multi-Region Oracle NoSQL Database setup, a Cross-Region Service or XRegion Service is a standalone service running on a separate node. In simple terms, this is also called an agent. Based on the requirement, you can deploy one or more agents for a region. The XRegion Service is deployed when you are connecting the local KVStore with a remote KVStore to create a Multi-Region Table.

Figure 1-4 Inbound Stream from Remote KVStore

Description of "Figure 1-4 Inbound Stream from Remote KVStore"

Consider an example where the remote KVStore has four shards and the local KVStore has three shards. The local KVStore deploys two NoSQL agents to stream two shards each from the remote KVStore, as depicted in the diagram above. These agents subscribe to updates from the remote KVStore and publish the new and modified rows to the local KVStore. All the agents connecting a remote KVStore to a local KVStore are referred as the Agent Group for the local KVStore.

As you can see in the diagram below, an agent enables streaming the data from a remote KVStore to a set of MR Tables in the local KVStore.

Figure 1-5 View of a Single Agent

Description of "Figure 1-5 View of a Single Agent"

Each inbound stream utilizes the subscriber group feature in the Streams API to create a group of subscribers to stream from a store. Each local agent is responsible for:

• Establishing the inbound subscription stream from a remote KVStore.
• Maintaining the connection and reconnect during any failures.
• Checkpointing the subscription stream in case of any failures.
• Subscribing writes to the MR Tables from a remote KVStore.
• Automatically dealing with elastic operations in the remote KVStore.

For a local KVStore, the overhead of inbound stream consists of:

• A group of threads in the NoSQL agent that streams the data from the remote KVStore. Please note that these threads run outside the local store as a standalone agent. Even though the agent serves the local KVStore, it does not add to the local KVStore's expense.
• Local PUT or DELETE resulting from the streamed data from the remote KVStore after conflict resolution.

For a local KVStore, the overhead of outbound stream involves the create, read, and write checkpoints for the remote KVStore.

Life Cycle of Multi-Region Tables

To create and use Multi-Region Tables (MR Tables) in Oracle NoSQL Database, you must be aware of the sequence of tasks to execute and the related concepts.

For clarity, let us discuss the life cycle of MR Tables with an example. Consider an Oracle NoSQL Database with three regions, Frankfurt, London, and Dublin. Assume that you want to create a table called Users to store the user details for all the three regions as depicted in the diagram.

Figure 1-6 Multi-Region Table

Description of "Figure 1-6 Multi-Region Table"

The sequence of tasks that you must perform to create and manage the Users table as a Multi-Region Table are:

• Deploy Independent KVStores: You must deploy KVStore in each region of the Multi-Region NoSQL Database independently. See Configuration Overview in the Administrator's Guide.
• Set Local Region Name: After deploying the KVStore and before creating the first MR Table in each participating region, you must set a local region name. You can change the local region name as long as no MR Tables are created in that region. After creating the first MR Table, the local region name becomes immutable. The local region name is completely independent of the KVStores created in that region. See Set Local Region Name in the Administrator's Guide.
• Configure XRegion Service: Before creating any MR Table, you must deploy an XRegion Service with one or more agents. The agent runs independently with the local KVStore and it is recommended to deploy it close to the local KVStore.
  While setting up a secure KVStore to support multi-region tables, the administrator has to grant the following permissions to the XRegion Service agent:

  • WRITE_SYSTEM_TABLE (or the equivalent writesystable role) to the local store.
  • Write permission on all the multi-region tables in the local store.
  • Read permission on all the multi-region tables in the remote stores.
  • Write permission for checkpoint table in the remote stores.

  To learn how to deploy an agent, see Configure XRegion Service in the Administrator's Guide.

• Start XRegion Service:You must start XRegion service in each region using the XRSTART command. As this service is a long-running process, it is recommended to invoke it as a background process by appending the & at the end of the command. see Start XRegion Service in the Administrator's Guide.

  Note:

  The local KVStore must be started before starting the XRegion Service. If the KVStore in the local region has not started or is not reachable, the XRegion Service will not start.
• Create Remote Regions: Before creating and operating on the MR table, you must define the remote regions. A remote region signifies a region different from the region where the command is executed. In this example, to create the MR Table called Users from the Frankfurt region, you must first define the other two regions, that is, London and Dublin using the CREATE REGION DDL command. To learn how to create remote regions, see Create Remote Regions in the Administrator's Guide.
• Create the MR Table: You must create an MR Table on each KVStore in the connected graph, and specify the list of regions that the table should span. In this example, to create the Users table as an MR Table at the Frankfurt and London regions, you need to execute the CREATE TABLE command specifying Frankfurt and London as the regions. The order in which you list the regions in the DDL command does not matter. After you create the Users MR Table, it will be included in the incoming stream from each remote KVStore specified. Symmetrically at the remote KVStore the Users table will be included in its own incoming stream too. To create the MR table successfully, you must:
  • Ensure that you acquire the necessary privileges to create the table in the specified regions, in advance. Otherwise, the MR Table creation will fail in all the regions. See KVStore Required Privileges in the Security Guide.
  • Specify at least one region in the CREATE TABLE DDL command. If you specify only one region, then the MR Table is created only in the specified region and no writes will be replicated to the other regions.

    Note:

    Even though a single-region MR table works similar to a local table, the difference between them is that the single-region MR Table can be expanded to multiple regions in future.
  To learn how to create an MR Table, see Create MR Tables in the Administrator's Guide.
• Perform Read/Write Operations on the MR Table (Optional): After creating the MR Table, you can perform read or write operations on the table using the existing data access APIs or DML statements. There is no change to any existing data access APIs or DML statements to work with the MR Tables. The following aspects applicable to the regular tables apply to the MR Tables also without any deviation:
  • Durability and consistency configurations and constraints: For any local writes to an MR table, the semantics of consistency model does not change. It is the same as any writes to a regular (non MR) table.

    Note:

    In case of MR Tables, absolute consistency is not global across the participating regions. It is only local to a single region where you perform the read and write operations.
  • Table index infrastructure: Creating primary and secondary indices on the MR Table in each region remains the same as with any regular (non MR) table. However, if you wish to drop an MR Table from any region, you must first drop all the indices defined on this table.
  Read Operations:
  • Each Read operation on an MR Table is a local read, that is, you read only the local copy of the data. However, this local copy may have rows that might have come from one of the other participating regions, as a result of a table sync-up via Oracle NoSQL Streams.
  Write Operations:
  • Whenever you execute a write operation (INSERT, UPDATE, or DELETE) on an MR Table, the changes will be synchronized across multiple regions asynchronously. It means, when you write a row in the local region, the write operation is executed completely in the local region without waiting for the subscribing regions to update.
  • The latency for synchronizing the changes across multiple regions includes the time involved for the replication (PULL) and write operations.
  • In a case where multiple regions update a row with the same primary key, a conflict resolution rule is applied to decide which region's update is considered as final. In this case, the update with the latest timestamp wins and is committed to the database.
  • Whenever you delete a row from an MR Table, a TTL (Time-To-Live) value of seven days is applied to the deleted row automatically. This 7-day TTL applies only to the tombstone for the deleted key and not to the row itself. You cannot set or change this value manually.

  See Access and Manipulate MR Tables in the Administrator's Guide for examples.

• Add New Regions to the MR Table (Optional): Oracle NoSQL Database lets you expand an MR Table to new regions. It effectively means adding new regions to an existing MR Table. In the example being discussed, suppose you created the Users table only in two regions, Frankfurt and London. Later, if you want to expand this Users table to the Dublin region, you must:
  • Create the Users MR Table in the new region, that is, Dublin. Note that you must specify all the three regions while creating the MR Table in the new region. See Create MR Table in New Region in the Administrator's Guide.
  • Add the new region (Dublin) to the Users MR Table in existing regions, that is, Frankfurt and London. This is achieved with the help of the ALTER TABLE DDL command. See Add New Region to Existing Regions in the Administrator's Guide.

    Note:

    Depending on the volume of the data in the existing regions, it might take some time to initialize the MR Table in the new region with the data from the other regions. However, the MR Table in the new region is available for read/write operations immediately after its creation.

    To learn how to expand an MR Table with detailed code demonstrations, see Use Case 2: Expand a Multi-Region Table in the Administrator's Guide.

• Remove an Existing Region from the MR Table (Optional): Not only can you add new regions to an existing MR Table but also remove any regions linked to it. It effectively means that you disconnect the MR Table from a particular region so that MR Table is not synchronized with any writes from the removed region. This is called contracting an MR Table. To learn how to contract an MR Table with detailed code demonstrations, see Use Case 3: Contract a Multi-Region Table in the Administrator's Guide.
• Drop Remote Regions (Optional): You can drop one or more participating regions from a Multi-Region Oracle NoSQL Database setup as per your business requirement. However, before removing a region from a Multi-Region NoSQL Database, it is recommended to:
  • Stop writing to all the MR Tables linked to that region.
  • Ensure that all writes to the MR Tables in that region have synchronized with other regions. This helps in maintaining consistent data across the different regions.

  Note:

  Even though NoSQL Database lets you drop a region directly, it is a recommended practice to isolate that region from all the other regions before dropping it. This ensures that the existing regions are no longer linked with the region being dropped.

  To learn how to drop a region from a Multi-Region NoSQL Database, see Use Case 4: Drop a Region in the Administrator's Guide.

• Shut Down XRegion Service and KVstores: In a case where you want to relocate your XRegion Service to another host machine, you must shut it down in the current machine and then restart it in the new host machine. See Stop XRegion Service in the Administrator's Guide.