Database Watchdog
The Database Watchdog microservice checks the availability of configured database sources and publishes the results as Prometheus metrics. It performs an initial check at startup and continues to check each source at the configured interval.
In redundant deployments, it enables other microservices to detect when a database source fails over from the primary source to the secondary source. When the primary source becomes available again, the microservices can switch back to it. This failover and failback behavior is not available in non-redundant deployments because they do not have a secondary source.
Database Watchdog supports MySQL, InfluxDB, Neo4j, and OpenSearch ingestion nodes. For MySQL replica, it also detects replication lag that exceeds the configured limit.
Database Watchdog runs as a Kubernetes DaemonSet, with one pod on each eligible Kubernetes node. This enables it to report source availability from the perspective of each node.
Database Watchdog Prerequisites
Before deploying the microservice, confirm that the following prerequisites are met:
-
A microservice cluster is set up. See Microservice Cluster Setup.
-
The database sources that you want to monitor are configured in /config/assure1.yaml and are reachable from the cluster nodes on which Database Watchdog runs.
Deploying Database Watchdog
To deploy the microservice, run the following commands:
su - assure1
export WEBFQDN=<WebFQDN>
a1helm install <microservice-release-name> assure1/database-watchdog -n a1-monitoring --set global.imageRegistry=$WEBFQDN
In the commands:
-
<WebFQDN> is the fully qualified domain name of the primary presentation server for the cluster.
-
<microservice-release-name> is the name to use for the microservice instance. Oracle recommends using the microservice name database-watchdog unless you are deploying multiple instances of the microservice to the same cluster.
Note that this microservice must be deployed to the a1-monitoring namespace.
You can also use the Unified Assurance UI to deploy microservices. See Deploying a Microservice by Using the UI for more information.
You can check whether all the pods are deployed and running properly using the following command:
a1k get pods -n a1-monitoring
To review Database Watchdog logs, run:
a1k logs -n a1-monitoring daemonset/<microservice-release-name>
Changing Database Watchdog Configuration Parameters
When running the install command, you can optionally change default configuration parameter values by including additional --set arguments. You can add as many additional --set arguments as you need.
For example, set a parameter described in Default Database Watchdog Configuration by adding --set configData.<parameter_name>=<parameter_value>. To change the polling interval to 60 seconds, add --set configData.POLL_INTERVAL=60.
Default Database Watchdog Configuration
The following table describes the default configuration parameters found in the Helm chart under configData for the microservice.
| Name | Default Value | Possible Values | Notes |
|---|---|---|---|
| LOG_LEVEL | INFO | FATAL, ERROR, WARN, INFO, DEBUG | The logging level used by the application. |
| POLL_INTERVAL | 30 | Integer | The interval, in seconds, between checks of configured database sources. |
| DB_CONNECTION_TIMEOUT | 10 | Integer | The MySQL database connection timeout, in seconds. |
| DB_SECONDS_BEHIND_SOURCE_LIMIT | 10 | Integer | The maximum allowed MySQL replication lag, in seconds, before the database is reported as degraded. |
Monitoring Database Availability Metrics
Database Watchdog exposes database availability metrics on port 9092 at the /metrics endpoint.
To view the metrics directly for troubleshooting, forward the service port and query the endpoint:
a1k port-forward -n a1-monitoring service/<microservice-release-name> 9092:9092
curl http://localhost:9092/metrics
The output includes the database_availability metric for each configured database source in the following format:
database_availability{dsn="<dsn-key>",sourceNode="<kubernetes-node>"} <status>
For example:
database_availability{dsn="DBI:mysql:database=Event;host=<host-name>;port=3306",sourceNode="<kubernetes-node>"} 1
The output includes the dsn label, which identifies the database source, and the sourceNode label, which identifies the Kubernetes node that performed the check.
A value of 1 indicates that the database source is available and it responded successfully. A value of 0 indicates that the connection or health check failed and the database source is unavailable. A value of 2 indicates that a MySQL database is reachable, but its replication lag exceeds the DB_SECONDS_BEHIND_SOURCE_LIMIT value and the database is marked as degraded.
Updating Database Watchdog After a Configuration Change
After you add a database node, database shard, or OpenSearch ingestion node, update the configuration and restart the Database Watchdog pods.
-
As the root user, run the following command to update the a1-configmap:
$A1BASEDIR/bin/cluster/clusterctl update-config -
List the Database Watchdog pods:
a1k get pods -n a1-monitoring -
For each Database Watchdog pod, run the following command:
a1k delete pod -n a1-monitoring database-watchdog-<id>Replace <id> with the suffix of the Database Watchdog pod name that you want to restart.
After the pod is deleted, the DaemonSet automatically creates a replacement pod. The replacement pod loads the updated configuration and begins monitoring the new node or shard.