Database Sharding

Database Sharding is a way to organize large sets of data across multiple servers. Sharded data is stored completely separate from other shards and each individual shard has no knowledge of the other, but the Unified Assurance web interface seamlessly correlates and displays data collected from all shards allowing for a completely unified view. The Unified Assurance platform supports database sharding for Events & Historical, and Metric data that is simple to configure and transparent to users.

There are many benefits and use cases when considering sharding a database:

Load balancing - Unified Assurance applications only talk to the shards needed lightening the load across the databases.
Scaling - Big data can be stored in manageable pieces across the multiple database shards supporting larger scale environments.
Data Storage Compliance - Each shard's data is stored in a standalone fashion and has no knowledge of data from any other shard.

This section details the procedure for installing and configuring new database shards for Events, Historical Events, and Metrics databases within the Unified Assurance platform.

Pre-Planning Shard Data Organization

Event & Historical

Unified Assurance aggregators support different shards but must know at startup what shard it will be communicating with. Within the Unified Assurance platform, events can be sharded in the following ways:

Collection Point (most common) - Alarm data based on which server received it. This method allows for segregation of data based on geographical regions. For example: Shard 1 is in US and only handles collected events from the US Unified Assurance aggregation servers, while Shard 2 is in Europe and only handles collected events from the Europe Unified Assurance aggregation servers.
Alarm Type - Alarm data from a certain aggregator, such as SNMP traps, to a certain shard. This method allows for additional analysis or profiling of a certain type of data.

Metric

Unified Assurance metric collectors and pollers support multiple shards based on the device's and metric's shard definition. Shard information can be set at the device or metric level within the Unified Assurance platform and metrics can be sharded in a few different manners:

Device (most common) - All metrics for certain devices are stored on a specific shard, this includes separation by Device Zone or Device Group. This method allows for segregation of device data by geographical regions or polling clusters. For example: Shard 1 is in US and only handles collected metrics from the US Unified Assurance collection servers, while Shard 2 is in Europe and only handles collected metrics from the Europe Unified Assurance collection servers. (This is the easiest to configure and maintain)
Metric Type - Metric data of certain types, such as Latency & Packet Loss, are stored on a specific shard. This method allows for additional analysis or profiling of a certain type of data, but additional maintenance is required to keep accurate.

Installation

New Server(s)

Follow the steps in the Multi-Server Install to get the server ready for Unified Assurance.
Follow the steps in the Multi-Server Install to get Unified Assurance installed.

Note:

When asked what roles to install, the Database.Event and/or Database.Metric roles can be entered depending on the type of database being sharded. Additional roles can also be installed, if needed.
Wait for the installation to complete.

Existing Servers

Login to the command line of the server that will have the database added to it.
Use the Package Manager application to install additional roles with the new database functionality.
```
$A1BASEDIR/bin/Package install-role (Comma-separated list of roles)
```
Note:

See the Understanding Server Roles guide for a list of roles that can be installed.
Wait for the installation to complete.

Sharding Procedure

New Events Shard

Navigate to the Databases UI and verify a new shard entry exists for the Event database.

Configuration -> Databases -> Databases
Run the ApplyEventSchema application to update the database table schema, passing in the correct Event Shard ID:
```
$A1BASEDIR/bin/ApplyEventSchema --EventShard N
```
Note:

The --Dry-Run option can be used to log the queries that will be executed to change the database:
```
$A1BASEDIR/bin/ApplyEventSchema --Dry-Run
```
Then look at the log file to see the queries that will be executed:
```
lnav $A1BASEDIR/logs/ApplyEventSchema.log
```
Organize event aggregators into shards based on pre-planning or policies.
Clone necessary aggregators and respective application configurations, updating ShardID to appropriate shard.
Verify new events are being inserted into the correct database instance.

New Metric Shard

Navigate to the Databases UI and verify a new shard entry exists for the Metric database.

Configuration -> Databases -> Databases
Organize and set Devices' and Metrics' ShardIDs to appropriate shards based on pre-planning or policies.
Restart any affected polling clusters.
Verify new metrics are being inserted into the correct database instance.

Redundancy

Login to the command line of the server that will have the redundancy added to it.
Use the Package Manager application to install the Database.Redundancy role.
```
$A1BASEDIR/bin/Package install-role Database.Redundancy
```
Wait for the installation to complete.
Follow the steps in the Multi-Server Redundant Install to configure database redundancy.

Title and Copyright Information

Implementation Guide

F88952-01