4 cnDBTier APIs

This chapter describes the APIs used in cnDBTier and their parameters.

Note:

The "occne-cndbtier" namespace name used in this section is only an example. Ensure that you configure the namespace name according to your environment.

4.1 cnDBTier APIs for CNC Console

cnDBTier APIs facilitate CNC Console to determine the status of cnDBTier and the associated cross-site replication. The DB replication service hosts these APIs.

Table 4-1 cnDBTier APIs for CNC Console

Resource Rest URL HTTP Method Request Payload Response Code Response Payload
Site specific real time replication status http://<base-uri>/ocdbtier/status/replication/realtime/{remoteSiteName} GET NA
  • 200 OK - Fetched the list of backups present in cnDBTier.
  • 400 Bad Request
  • 404 Not Found
  • 500 Internal Server Error.
  • 503 Service Unavailable - Cross-site replication is not enabled.
  • 200 OK:
    {
         "localSiteName":"<current-site-name>",
         "remoteSiteName":"<remote-site-name>",
         "replicationStatus":"UP/DOWN/INITIALIZING"
         "secondsBehindRemote": "<greater_secondsBehindRemote_of_multiplegroups_withremotesite>"
      "replicationGroupDelay": [
        {
          "replchannel_group_id": "1",
          "secondsBehindRemote": <seconds>,
          "channelDetails": [
            {
              "remote_replication_ip": "127.0.0.1",
              "role": "ACTIVE"
            },
            {
              "remote_replication_ip": "127.0.0.2",
              "role": "STANDBY"
            }
          ]
        },
        {
          "replchannel_group_id": "2",
          "secondsBehindRemote": <seconds>,
          "channelDetails": [
            {
              "remote_replication_ip": "127.0.0.3",
              "role": "ACTIVE"
            },
            {
              "remote_replication_ip": "127.0.0.4",
              "role": "STANDBY"
            }
          ]
        }
      ]
    }
  • 400 Bad Request:
    {
      "title": "Bad Request",
      "status": 400,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 404 Not Found:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
       "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 503 Service Unavailable - Cross-site replication is not enabled:
    {
      "title": "Service Unavailable",
      "status": 503,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
Real time overall replication status http://<base-uri>/ocdbtier/status/replication/realtime GET NA
  • 200 OK - Cross-site replication is active
  • 500 Internal Server Error
  • 503 Service Unavailable - Cross-site replication is not enabled
  • 200 OK - Cross-site replication is active:
    {
     [
       {
         "localSiteName":"<current-site-name>",
         "remoteSiteName":"<remote-site-name>",
         "replicationStatus":"UP/DOWN/INITIALIZING"
         "secondsBehindRemote": "<greater_secondsBehindRemote_of_multiplegroups_withremotesite>"
          "replicationGroupDelay":
         [{
             replchannel_group_id: 1,
             secondsBehindRemote: <seconds>
          },
          {
             replchannel_group_id: 2,
             secondsBehindRemote: <seconds>
         }]
       },
       {
         "localSiteName":"<current-site-name>",
         "remoteSiteName":"<remote-site-name>",
         "replicationStatus":"UP/DOWN/INITIALIZING"
         "secondsBehindRemote": "<greater_secondsBehindRemote_of_multiplegroups_withremotesite>"
          "replicationGroupDelay":
         [{
             replchannel_group_id: 1,
             secondsBehindRemote: <seconds>
          },
          {
             replchannel_group_id: 2,
             secondsBehindRemote: <seconds>
         }]
       }
     ]
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
       "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 503 Service Unavailable - Cross-site replication is not enabled:
    {
      "title": "Service Unavailable",
      "status": 503,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
Real time local cluster status http://<base-uri>/ocdbtier/status/cluster/local/realtime GET NA
  • 200 OK- The local NDB cluster is UP or DOWN
  • 500 Internal Server Error
  • 200 OK- The local NDB cluster is UP or DOWN:
    {
      "clusterName":"<current-cluster-name>",
      "clusterStatus":"UP/DOWN"
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
     "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
cnDBTier version http://<base-uri>/ocdbtier/version GET NA
  • 200 OK
  • 500 Internal Server Error
  • 200 OK:
    {
      "cndbtier_version": "24.2.6",
      "ndb_version": "ndb-8.0.37"
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
     "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
List completed backups of the current site http://<base-uri>/ocdbtier/list/backups GET NA
  • 200 OK - Fetched the Heart Beat Status of every replication service running on the current cluster
  • 404 NOT FOUND Table Missing
  • 500 Internal Server Error
  • 200 OK:
    {
      "localSiteName": "<current_site_name>",
      "backupDetails": [
        {
          "backupId": "<backup identifier>",
          "backupSize": "<size of backup>",
          "creationTimeStamp": "<timestamp at which backup was initiated>"
        },
        {
          "backupId": "<backup identifier>",
          "backupSize": "<size of backup>",
          "creationTimeStamp": "<timestamp at which backup was initiated>"
        },
        {
          "backupId": "<backup identifier>",
          "backupSize": "<size of backup>",
          "creationTimeStamp": "<timestamp at which backup was initiated>"
        }
      ]
    }
  • 404 NOT FOUND Table Missing:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "DBTIER_BACKUP_INFO table doesn't exist in the current site so cannot list the backup",
      "cause": null,
      "invalidParams": null
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
      "detail": "Could not list the backups",
      "cause": null,
      "invalidParams": null
    }
Database Statistic Reports http://<base-uri>/ocdbtier/statistics/report/dbinfo GET NA
  • 200 OK
  • 500 INTERNAL_SERVER_ERROR
  • 503 SERVICE_UNAVAILABLE
  • 200 OK:
    {
      "databaseCount": <database count>,
      "databaseTablesCount": [
        {
          "dbName": "<database name>",
          "tableCount": <table count>
        },
        {
          "dbName": "<database name>",
          "tableCount": <table count>
        }
    
      ],
      "databaseTableRowsCount": [
        {
          "dbName": "<database name>",
          "tables": [
            {
              "tableName": "<table name>",
              "rowCount": <row count>
            }
          ]
        },
        {
          "dbName": "<database name>",
          "tables": [
            {
              "tableName": "<table name>",
              "rowCount": <row count>
            },
            {
              "tableName": "<table name>",
              "rowCount": <row count>
            }
          ]
        }
      ]
    
    }
  • 500 INTERNAL_SERVER_ERROR:
    {
      "title": "Internal Server Error",
      "status": 500,
      "detail": "<Exception Details>",
      "cause": null,
      "invalidParams": null
    }
  • 503 SERVICE_UNAVAILABLE:
    {
      "title": "Service Unavailable",
      "status": 503,
      "detail": "<Exception details>",
      "cause": null,
      "invalidParams": null
    }
Overall HeartBeat Status http://<base-uri>/ocdbtier/heartbeat/status GET NA
  • 200 OK
  • 503 SERVICE_UNAVAILABLE - Not able to query the Database
  • 500 INTERNAL_SERVER_ERROR - Could Not Check The HeartBeat Status
  • 200 OK:
    {
      "heartBeatDetails": [
        {
          "remoteSiteName": "<remote_site_name>",
          "heartBeatStatus": "<SUCCESS/FAILURE>",
          "heartBeatLag": "<Heart Beat Lag in Seconds>",
          "replicationChannelGroupId": <replication channel group id>
        },
        {
          "remoteSiteName": "<remote_site_name>",
          "heartBeatStatus": "<SUCCESS/FAILURE>",
          "heartBeatLag": "<Heart Beat Lag in Seconds>",
          "replicationChannelGroupId": <replication channel group id>
        }
      ],
      "siteName": "<current_site_name>", }
  • 503 SERVICE_UNAVAILABLE - Not able to query the Data Base:
    {
      "title": "Service Unavailable",
      "status": 503,
      "detail": "Not able to query the Data Base",
      "cause": null,
      "invalidParams": null
    }
  • 500 INTERNAL_SERVER_ERROR - Could Not Check The HeartBeat Status:
    {
      "title": "Internal Server Error",
      "status": 500,
      "detail": "Could Not Check The HeartBeat Status",
      "cause": null,
      "invalidParams": null
    }
On-demand Backup http://<base-uri>/ocdbtier/on-demand/backup/initiate GET NA
  • 200 OK
  • 404 NOT FOUND
  • 500 INTERNAL_SERVER_ERROR
  • 200 OK:
    {
        "siteName": "<current_site_name>",
        "drStatus": "<drStatus>",
        "backupId": "<backupId>",
        "backupStatus": "<backupStatus>",
        "remoteTransferStatus": "<remoteTransferStatus>",
        "initiateBackup": false/true,
        "transferStatus": "<transferStatus>"
    }
  • 404 NOT FOUND:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Exception message>",
      "cause": null,
      "invalidParams": null
    }
  • 500 INTERNAL_SERVER_ERROR:
    {
      "title": "Internal Server Error",
      "status": 500,
      "detail": "<Exception message>",
      "cause": null,
      "invalidParams": null
    }
Initiate on-demand backup http://<base-uri>/ocdbtier/on-demand/backup/initiate POST { "siteName": "<current_site_name>", "drStatus": "<drStatus>", "backupId": "<backupId>", "backupStatus": "<backupStatus>", "remoteTransferStatus": "<remoteTransferStatus>", "initiateBackup":false/true, "transferStatus": "<transferStatus>" }
  • 200 OK
  • 400 BAD REQUEST
  • 404 NOT FOUND
  • 500 INTERNAL_SERVER_ERROR
  • 200 OK:
    {
      "siteName": "<current_site_name>",
      "drStatus": "<drStatus>",
      "backupId": "<backupId>",
      "backupStatus": "<backupStatus>",
      "remoteTransferStatus": "<remoteTransferStatus>",
      "initiateBackup": false/true,
      "transferStatus": "<transferStatus>"
    }
  • 404 BAD REQUEST:
    {
      "title": "Bad Request",
      "status": 400,
      "detail": "Couldn't initiate the backup when initiateBackup flag is set to false",
      "cause": null,
      "invalidParams": [
        {
          "param": "initiateBackup",
          "reason": "Couldn't initiate the backup when initiateBackup flag is set to false"
        }
      ]
    }
  • 404 NOT FOUND:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Exception message>",
      "cause": null,
      "invalidParams": null
    }
  • 500 INTERNAL_SERVER_ERROR:
    {
      "title": "Internal Server Error",
      "status": 500,
      "detail": "BACKUP_INITIATION_FAILED",
      "cause": null,
      "invalidParams": null
    }
Health status of replication service http://<base-uri>/ocdbtier/health-info/replication-svc/status GET NA
  • 200 OK
  • 503 SERVICE_UNAVAILABLE - Replication Service IP is empty
  • 500 INTERNAL_SERVER_ERROR - Could Not Check The Replication Health Status
  • 200 OK:
    {
        "localSiteName": "<siteName>",
        "healthStatusDetails": [
            {
                "serviceName": "<serviceName>",
                "serviceStatus": "UP/DOWN",
                "connectionToDbStatus": "UP/DOWN",
                "overAllReplicationServiceHealth": "UP/DOWN"
            },
            {
                "serviceName": "<serviceName>",
                "serviceStatus": "UP/DOWN",
                "connectionToDbStatus": "UP/DOWN",
                "overAllReplicationServiceHealth": "UP/DOWN"
            }.......
        ]
    }
  • 503 SERVICE_UNAVAILABLE - Replication Service IP is empty:
    {
      "title": "Service Unavailable",
      "status": 503,
      "detail": "Replication Service IP is empty",
      "cause": null,
      "invalidParams": null
    }
  • 500 INTERNAL_SERVER_ERROR - Could Not Check The Replication Health Status:
    {
      "title": "Internal Server Error",
      "status": 500,
      "detail": "Could Not Check The Health Status",
      "cause": null,
      "invalidParams": null
    }
Health status of monitor service http://<base-uri>/ocdbtier/health-info/monitor-svc/status GET NA 200 OK (Monitor Service is healthy) 200 OK:
{
  "serviceName": "mysql-cluster-db-monitor-svc",
  "connectionToDbStatus": "UP",
  "metricScrapeStatus": "UP",
  "overAllMonitorServiceHealth": "UP"
}
Overall health status of NDB data service http://<base-uri>/ocdbtier/health-info/ndb-svc/status GET NA
  • 200 OK
  • 404 NOT FOUND - sitename not found
  • 500 INTERNAL_SERVER_ERROR - Could Not Check The Health Status
  • 200 OK:
    {
        "localSiteName": "<siteName>",
        "ndbHealthStatusDetails": [
            {
                "serviceName": "<serviceName>",
                "serviceStatus": "UP/DOWN",
                "pvcHealthStatus": "UP/DOWN"
            },
            {
                "serviceName": "<serviceName>",
                "serviceStatus": "UP/DOWN",
                "pvcHealthStatus": "UP/DOWN"
            }.......
        ]
    }
  • 404 NOT FOUND - sitename not found:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "sitename not found",
      "cause": null,
      "invalidParams": null
    }
  • 500 INTERNAL_SERVER_ERROR - Could Not Check The Health Status:
    {
      "title": "Internal Server Error",
      "status": 500,
      "detail": "Could Not Check The Health Status",
      "cause": null,
      "invalidParams": null
    }
Health status of backup manager service http://<base-uri>/ocdbtier/health-info/backup-mgr-svc/status GET NA
  • 200 OK
  • 500 INTERNAL_SERVER_ERROR - Could Not Check The Health Status
  • 200 OK:
    {
      "serviceName": "mysql-cluster-db-backup-manager-svc",
      "connectionToDbStatus": "UP",
      "serviceStatus": "UP",
      "backupExecutorHealthStatus": [
        {
          "nodeId": 2,
          "connectionToDbStatus": "UP"
        },
        {
          "nodeId": 1,
          "connectionToDbStatus": "UP"
        }
      ],
      "overAllBackupManagerHealth": "UP"
    }
  • 500 INTERNAL_SERVER_ERROR - Could Not Check The Health Status:
    {
      "title": "Internal Server Error",
      "status": 500,
      "detail": "Could Not Check The Health Status",
      "cause": null,
      "invalidParams": null
    }
cnDBTier Cluster Details http://<base-uri>/ocdbtier/list/clusterdetails GET NA
  • 200 OK
  • 404 NOT FOUND
  • 500 INTERNAL SERVER ERROR
  • 200 OK:
    [
      {
        "clusterName": "<clusterName>",
        "clusterId": "1",
        "status": "ACTIVE/FAILED"
      },
      {
        "clusterName": "<clusterName>",
        "clusterId": "2",
        "status": "ACTIVE/FAILED"
      },
      {
        "clusterName": "<clusterName>",
        "clusterId": "3",
        "status": "ACTIVE/FAILED"
      }
    ]
  • 404 NOT FOUND:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
  • 500 INTERNAL SERVER ERROR:
    {
      "title": "INTERNAL_SERVER_ERROR",
      "status": 500,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
Get Failed cnDBTier Clusters http://<base-uri>/ocdbtier/markcluster/failed GET NA
  • 200 OK
  • 404 NOT FOUND
  • 500 INTERNAL SERVER ERROR
  • 200 OK:
    {
      "clusterNameTobeMarkedAsFailed": "",
      "failedClusterNames": [
        "<failedcluster1>",
        "<failedcluster2>"
      ]
    }
  • 404 NOT FOUND:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
  • 500 INTERNAL SERVER ERROR:
    {
      "title": "INTERNAL_SERVER_ERROR",
      "status": 500,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
Mark cnDBTier Clusters As Failed http://<base-uri>/ocdbtier/markcluster/failed PUT
{
  "clusterNameTobeMarkedAsFailed": "<failedcluster>",
  "failedClusterNames": []
}
  • 200 OK
  • 400 BAD REQUEST
  • 404 NOT FOUND
  • 500 INTERNAL SERVER ERROR
  • 503 SERVICE_UNAVAILABLE
  • 200 OK:
    {
      "clusterNameTobeMarkedAsFailed": "",
      "failedClusterNames": [
        "<failedcluster1>",
        "<failedcluster2>"
      ]
    }
  • 400 BAD REQUEST:
    {
      "title": "BAD_REQUEST",
      "status": 400,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
  • 404 NOT FOUND:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
  • 500 INTERNAL SERVER ERROR:
    {
      "title": "INTERNAL_SERVER_ERROR",
      "status": 500,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
  • 503 SERVICE_UNAVAILABLE:
    {
      "title": "Service Unavailable",
      "status": 503,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
Monitor Georeplication Recovery Status http://<base-uri>/ocdbtier/faultrecovery/status GET NA
  • 200 OK
  • 404 NOT FOUND
  • 500 INTERNAL SERVER ERROR
  • 200 OK:
    {
      "localClusterName": "<local cluster name>",
      "faultRecoverStatus": [
        {
          "clusterName": "<clusterName>",
          "faultRecoverState": "<faultRecoverState>"
        },
        {
          "clusterName": "<clusterName>",
          "faultRecoverState": "<faultRecoverState>"
        },
        {
          "clusterName": "<clusterName>",
          "faultRecoverState": "<faultRecoverState>"
        }
      ]
    }
  • 404 NOT FOUND:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
  • 500 INTERNAL SERVER ERROR:
    {
      "title": "INTERNAL_SERVER_ERROR",
      "status": 500,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
Start Georeplication Recovery http://<base-uri>/ocdbtier/faultrecovery/start PUT { "backupClusterName": "<backup cluster name>", "failedClusterName": "<failed cluster name>" }
  • 200 OK
  • 400 BAD REQUEST
  • 404 NOT FOUND
  • 409 CONFLICT
  • 500 INTERNAL SERVER ERROR
  • 503 SERVICE_UNAVAILABLE
  • 200 OK:
    {
        "backupClusterName": "<backup cluster name>",
        "failedClusterName": "<failed cluster name>"
    }
  • 400 BAD REQUEST:
    {
      "title": "INVALID INPUT",
      "status": 400,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
  • 404 NOT FOUND:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
  • 409 CONFLICT:
    {
      "title": "Conflict",
      "status": 409,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
  • 500 INTERNAL SERVER ERROR:
    {
      "title": "INTERNAL_SERVER_ERROR",
      "status": 500,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }
  • 503 SERVICE_UNAVAILABLE:
    {
      "title": "Service Unavailable",
      "status": 503,
      "detail": "<Error message>",
      "cause": null,
      "invalidParams": null
    }

Note:

Replace the value of <base-uri> in the URL with <db-monitor-svc service name>.<namespace>:8080.

For example: curl http://mysql-cluster-db-monitor-svc.occne-cndbtier:8080/ocdbtier/status/replication/realtime

4.2 cnDBTier Backup APIs

cnDBTier backup APIs are used to create on-demand database backups and check the status of the backups in cnDBTier. The cnDBTier backup API is hosted by the DB backup manager service.

Table 4-2 cnDBTier Backup API

Resource Rest URL HTTP Method Request Payload Response Code Response Payload
Site Specific On-demand Backup Creation http://<base-uri>/db-tier/on-demand/backup/initiate POST NA
  • 200 OK - Successfully started the on-demand backup
  • 400 BAD REQUEST request is invalid
  • 409 CONFLICT backup is already going on
  • 500 INTERNAL SERVER ERROR - Could not initiate on-demand backup.
  • 503 SERVICE UNAVAILABLE data node is down
  • 200 OK:
    {
      "backup_id":"<Backup id>",
      "status":"<Backup Initiation Status>"
      "backup_encryption_flag": "<backup encryption enabled/disabled>"
    }
  • 400 BAD REQUEST request is invalid:
    {
      "backup_id":"",
      "status":"BACKUP_INITIATION_FAILED"
      "backup_encryption_flag": "true/false"
    }
  • 409 CONFLICT backup is already going on:
    {
      "backup_id":"",
      "status":"BACKUP_INITIATION_FAILED"
      "backup_encryption_flag": "true/false"
    }
  • 500 INTERNAL SERVER ERROR:
    {
      "error_type":"<error type>",
      "error_message":"<error message>"
    }
  • 503 SERVICE UNAVAILABLE data node is down:
    {
      "backup_id":"",
      "status":"BACKUP_INITIATION_FAILED"
      "backup_encryption_flag": "true/false"
    }
Site Specific On-demand Backup Status http://<base-uri>/db-tier/on-demand/backup/<BACKUP_ID_INITIATED>/status GET NA
  • 200 OK - Successfully started the on-demand backup
  • 400 BAD REQUEST Request is invalid
  • 404 NOT FOUND backup_id does not exist
  • 500 INTERNAL SERVER ERROR - Could not check the status for the specified on-demand backup ID
  • 200 OK:

    Response payload when .Values.global.remotetransfer.enable is set to true:

    
    {
      "backup_status":"<Backup status>",
      "transfer_status":"<Transfer Status>"
    }

    Note: The "transfer_status" parameter in the response payload indicates the backup transfer status from db-backup-manager-svc to db-replication-svc and not to the remote server.

    Response payload when .Values.global.remotetransfer.enable is set to false:
    {
       "status": "<Backup status>"
    }
  • 400 BAD REQUEST Request is invalid:
    {
      "error_description": "<error_description>",
      "error_type": "<error type>"
    }
  • 404 NOT FOUND backup_id does not exist:
    {
      "error_description": "<error_description>",
      "error_type": "<error type>"
    }
  • 500 INTERNAL SERVER ERROR:
    {
      "error_type":"<error type>",
      "error_message":"<error message>"
    }
Health Check http://<base-uri>/db-tier/health GET NA
  • 200 OK
  • 500 INTERNAL SERVER ERROR
  • 200 OK:
    {
        "is_connected_to_db": "<true|false>", 
        "overall_status": "<true|false>",
        "service_name": "<mysql-cluster-db-backup-manager-svc>"
        "executor_status_list": 
        [
            {
                "_node_id": "<node_id>",
                "_is_connected_to_db": "<true|false>",
                "_is_running": "<true|false>"
            },
            ...
        ]
    }
  • 500 INTERNAL SERVER ERROR:
    {
      "error_type":"<error type>",
      "error_message":"<error message>"
    }

Note:

Replace the value of <base-uri> in the URL with <db-backup-manager-svc svc>: 8080.
For example,
  • Run the following command to get the db-backup-manager-svc pod name from cnDBTier cluster:
    $ kubectl -n <namespace of cnDBTier cluster> get pods | grep "db-backup-manager-svc" | awk  '{print $1}'
  • Run the following command get the db-backup-manager-svc svc name from cnDBTier cluster:
    $ kubectl -n <namespace of cnDBTier cluster> get svc | grep "db-backup-manager-svc" | awk  '{print $1}'
    
  • Run the following commands to login to the db-backup-manager-svc pod, use the REST APIs to create an on-demand backup, and check the backup status:
    $ kubectl -n <namespace of cnDBTier cluster> exec -it <db-backup-manager-svc pod> -- bash
    $ curl -i -X POST http://<db-backup-manager-svc svc>:8080/db-tier/on-demand/backup/initiate
    $ curl -i -X GET http://<db-backup-manager-svc svc>:8080/db-tier/on-demand/backup/<BACKUP_ID_INITIATED>/status

4.3 cnDBTier Replication Service APIs

This section provides details about cnDBTier replication service APIs.

4.3.1 cnDBTier Switchover APIs

cnDBTier Switchover APIs are used to start or stop switchovers on replication services.

Table 4-3 cnDBTier Switchover APIs

Resource Rest URL HTTP Method Request Payload Response Code Response Payload
Start replication switchover for a site with respect to remote site and group ID http://<base-uri>/ocdbtier/georeplication/switchover/start/sitename/{siteName}/remotesitename/{remoteSiteName}/replchannelgroupid/{replChannelGroupId} PUT NA
  • 200 OK
  • 404 Not Found
  • 500 Internal Server Error
  • 200 OK:
    {
       "replicationSwitchOver":"start"
    } 
  • 404 Not Found:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
       "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
Stop replication switchover for a site with respect to remote site and group ID http://<base-uri>/ocdbtier/georeplication/switchover/stop/sitename/{siteName}/remotesitename/{remoteSiteName}/replchannelgroupid/{replChannelGroupId} PUT NA
  • 200 OK
  • 404 Not Found
  • 500 Internal Server Error
  • 200 OK:
    {
       "replicationSwitchOver":"stop"
    } 
  • 404 Not Found:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
       "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
Start replication switchover for a site with respect to remote site name http://<base-uri>/ocdbtier/georeplication/switchover/start/sitename/{siteName}/remotesitename/{remoteSiteName} PUT NA
  • 200 OK
  • 404 Not Found
  • 500 Internal Server Error
  • 200 OK:
    {
       "replicationSwitchOver":"start"
    } 
  • 404 Not Found:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
       "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
Stop replication switchover for a site with respect to remote site name http://<base-uri>/ocdbtier/georeplication/switchover/stop/sitename/{siteName}/remotesitename/{remoteSiteName} PUT NA
  • 200 OK
  • 404 Not Found
  • 500 Internal Server Error
  • 200 OK:
    {
       "replicationSwitchOver":"stop"
    } 
  • 404 Not Found:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
       "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
Start replication switchover with respect to site name http://<base-uri>/ocdbtier/georeplication/switchover/start/sitename/{siteName} PUT NA
  • 200 OK
  • 404 Not Found
  • 500 Internal Server Error
  • 200 OK:
    {
       "replicationSwitchOver":"start"
    } 
  • 404 Not Found:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
       "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
Stop replication switchover for a site with respect to site name http://<base-uri>/ocdbtier/georeplication/switchover/stop/sitename/{siteName} PUT NA
  • 200 OK
  • 404 Not Found
  • 500 Internal Server Error
  • 200 OK:
    {
       "replicationSwitchOver":"stop"
    } 
  • 404 Not Found:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
       "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }

Note:

The value of <base-uri> in the REST URL is <db-replication-svc Loadbalancer IP>: <db-replication-svc Loadbalancer PORT>.
Depending on your scenario, choose one of the following methods to obtain the LoadBalancer IP and LoadBalancer Port of the replication service in a cnDBTier cluster:
  • When HTTP is enabled in a cnDBTier cluster:
    • Run the following command to get the LoadBalancer IP of the replication service for cluster1:
      $ IP=$(kubectl get svc -n cluster1 | grep repl | awk '{print $4}' | head -n 1 )
    • Run the following commands to get the LoadBalancer Port of the replication service for cluster1:
      $ PORT=$(kubectl get svc -n cluster1 | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1)
    • Sample command to call the REST API:
      $ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/start/sitename/{siteName}
  • When HTTPS is enabled in a cnDBTier cluster:
    • Create the key PEM file (file.key.pem) and cert PEM file (file.crt.pem) using the p12 certificate (replicationcertificate.p12). Use the same p12 certificate that was used to enable the HTTPS while installing cnDBTier.
    • Run the following command to get the LoadBalancer IP of the replication service for cluster1:
      $ IP=$(kubectl get svc -n cluster1 | grep repl | awk '{print $4}' | head -n 1 )
    • Run the following commands to get the LoadBalancer Port of the replication service for cluster1:
      $ PORT=$(kubectl get svc -n cluster1 | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1) 
    • Sample command to call the REST API:
      $ curl --cert file.crt.pem --cert-type PEM --key file.key.pem --key-type PEM --pass PUT NextGenCne https://$IP:$PORT/ocdbtier/georeplication/switchover/start/sitename/{siteName}

4.3.2 cnDBTier Stop Replica APIs

cnDBTier Stop Replica APIs are used to stop a replication channel.

Table 4-4 cnDBTier Stop Replica APIs

Resource Rest URL HTTP Method Request Payload Response Code Response Payload
Stop replication on a site with respect to remote site name and replication channel group ID http://<base-uri>/ocdbtier/georeplication/stopreplica/sitename/{siteName}/remotesitename/{remoteSiteName}/replchannelgroupid/{replChannelGroupId} PUT NA
  • 200 OK
  • 404 Not Found
  • 500 Internal Server Error
  • 200 OK:
    {
       "stopReplica":"stop"
    }  
  • 404 Not Found:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
       "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
Stop replication on a site with respect to remote site names http://<base-uri>/ocdbtier/georeplication/stopreplica/sitename/{siteName}/remotesitename/{remoteSiteName} PUT NA
  • 200 OK
  • 404 Not Found
  • 500 Internal Server Error
  • 200 OK:
    {
       "stopReplica":"stop"
    } 
  • 404 Not Found:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
       "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
Stop replica on a site with respect to site name http://<base-uri>/ocdbtier/georeplication/stopreplica/sitename/{siteName} PUT NA
  • 200 OK
  • 404 Not Found
  • 500 Internal Server Error
  • 200 OK:
    {
       "stopReplica":"stop"
    } 
  • 404 Not Found:
    {
      "title": "Not Found",
      "status": 404,
      "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }
  • 500 Internal Server Error:
    {
      "title": "Internal Server Error",
      "status": 500,
     "detail": "<Exception details>",
      "cause": <cause>,
      "invalidParams": <params>
    }

Note:

The value of <base-uri> in the REST URL is <db-replication-svc Loadbalancer IP>: <db-replication-svc Loadbalancer PORT>.
Depending on your scenario, choose one of the following methods to obtain the LoadBalancer IP and LoadBalancer Port of the replication service in a cnDBTier cluster:
  • When HTTP is enabled in a cnDBTier cluster:
    • Run the following command to get the LoadBalancer IP of the replication service for cluster1:
      $ IP=$(kubectl get svc -n cluster1 | grep repl | awk '{print $4}' | head -n 1 )
    • Run the following commands to get the LoadBalancer Port of the replication service for cluster1:
      $ PORT=$(kubectl get svc -n cluster1 | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1) 
    • Sample command to call the REST API:
      $ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/stopreplica/sitename/{siteName}
  • When HTTPS is enabled in a cnDBTier cluster:
    • Create the key PEM file (file.key.pem) and cert PEM file (file.crt.pem) using the p12 certificate (replicationcertificate.p12). Use the same p12 certificate that was used to enable the HTTPS while installing cnDBTier.
    • Run the following command to get the LoadBalancer IP of the replication service for cluster1:
      $ IP=$(kubectl get svc -n cluster1 | grep repl | awk '{print $4}' | head -n 1 )
    • Run the following commands to get the LoadBalancer Port of the replication service for cluster1:
      $ PORT=$(kubectl get svc -n cluster1 | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1) 
    • Sample command to call the REST API:
      $ curl --cert file.crt.pem --cert-type PEM --key file.key.pem --key-type PEM --pass NextGenCne PUT https://$IP:$PORT/ocdbtier/georeplication/stopreplica/sitename/{siteName}

4.3.3 cnDBTier Listbackups API

cnDBTier Listbackups API is used to determine the backups present in the cnDBTier. The cnDBTier Listbackups API is hosted by the DB Replication service.

Table 4-5 cnDBTier Listbackups API

cnDBTier Listbackups API Rest URL HTTP Method Request Payload Response Code Response Payload
Site Specific Listing the Backups http://<base-uri>/db-tier/backup/site/{siteName}/listbackups GET NA
  • 200 OK - Fetched the list of backups present in cnDBTier.
  • 404 Not Found - Improper Site details or missing table.
  • 503 INTERNAL SERVER ERROR - Could not list the backups.
  • 200 OK:
    {
      "localSiteName":"<current_site_name>",
      "backupDetails":[
        {
          "backupId":"<backup>",
          "backupSize":"<size of backup>",
          "creationTimeStamp":"<timestamp at which backup was initiated>"
        },
        {
          "backupId":"<backup_id>",
          "backupSize":"<size of backup>",
          "creationTimeStamp":"<timestamp at which backup was initiated>"
        },
        {
          "backupId":"<backup_id>",
          "backupSize":"<size of backup>",
          "creationTimeStamp":"<timestamp at which backup was initiated>"
        }
      ]
    }
  • 404 Not Found: The response payload can be one of the following:
    • Site Name is null so cannot list the backup
    • Site: <siteName> doesn't match with the current site Name so cannot list the backup
    • DBTIER_BACKUP_INFO table doesn't exist in the current site so cannot list the backup
  • 503 INTERNAL SERVER ERROR: Could not list the backups, <Exception Message>

Note:

The value of <base-uri> in the REST URL is <db-replication-svc Loadbalancer IP>: <db-replication-svc Loadbalancer PORT>.
Depending on your scenario, refer to the following points to obtain the LoadBalancer IP and LoadBalancer Port of a cnDBTier cluster:
  • When HTTP is enabled in a cnDBTier cluster:
    • Run the following command to get the replication service LoadBalancer IP for cluster1:
      $ IP=$(kubectl get svc -n cluster1 | grep repl | awk '{print $4}' | head -n 1 )
    • Run the following commands to get the replication service LoadBalancer Port for cluster1:
      $ PORT=$(kubectl get svc -n cluster1 | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1) 
    • Sample command to call the REST API:
      $ curl -i -X GET http://$IP:$PORT/db-tier/backup/site/cluster1/listbackups
  • When HTTPS is enabled in a cnDBTier cluster:
    • Create the key PEM file (file.key.pem) and cert PEM file (file.crt.pem) using the p12 certificate (replicationcertificate.p12). Use the same p12 certificate that was used to enable the HTTPS while installing cnDBTier.
    • Run the following command to get the replication service LoadBalancer IP for cluster1:
      $ IP=$(kubectl get svc -n cluster1 | grep repl | awk '{print $4}' | head -n 1 )
    • Run the following commands to get the replication service LoadBalancer Port for cluster1:
      $ PORT=$(kubectl get svc -n cluster1 | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1) 
    • Sample command to call the REST API:
      $ curl -k --cert file.crt.pem --cert-type PEM --key file.key.pem --key-type PEM --pass NextGenCne https://$IP:$PORT/db-tier/backup/site/cluster1/listbackups

4.3.4 cnDBTier Replication Service Heartbeat API

cnDBTier Replication Service Heartbeat API is used to provide the Heartbeat statuses of all replication services running on the current cluster.

Table 4-6 cnDBTier Replication Service Heartbeat API

cnDBTier Replication Service Heartbeat API Rest URL HTTP Method Request Payload Response Code Response Payload
Site Specific Listing the Heartbeat Status http://<base-uri>/db-tier/status/db-replication-svc/realtime GET NA
  • 200 OK - Fetched the heartbeat status of each replication service running on the current cluster.
  • 503 SERVICE_UNAVAILABLE - Not able to query the database
  • 503 INTERNAL SERVER ERROR - Could not check the replication service status.
  • 200 OK:
    {
      "localSiteName":"<current_site_name>",
      "heartBeatDetails":[
        {
          "remoteSiteName":"<remote_site_name>",
          "heartBeatStatus":"<SUCCESS/FAILURE>",
          "heartBeatLag":"<Heart Beat Lag in Seconds>",
          "replicationChannelGroupId":<replication channel group id>
        },
        {
          "remoteSiteName":"<remote_site_name>",
          "heartBeatStatus":"<SUCCESS/FAILURE>",
          "heartBeatLag":"<Heart Beat Lag in Seconds>",
          "replicationChannelGroupId":<replication channel group id>
        }
      ]
    }
  • 503 SERVICE_UNAVAILABLE:

    Not able to query the Data Base

  • 500 INTERNAL_SERVER_ERROR:

    Could not check the replication service status, <Exception Message>

Note:

The value of <base-uri> in the REST URL is <db-replication-svc Loadbalancer IP>: <db-replication-svc Loadbalancer PORT>.
Depending on your scenario, refer to the following points to obtain the LoadBalancer IP and LoadBalancer Port of a cnDBTier cluster:
  • When HTTP is enabled in a cnDBTier cluster:
    • Run the following command to get the replication service LoadBalancer IP for cluster1:
      $ IP=$(kubectl get svc -n cluster1 | grep repl | awk '{print $4}' | head -n 1 )
    • Run the following commands to get the replication service LoadBalancer Port for cluster1:
      $ PORT=$(kubectl get svc -n cluster1 | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1) 
    • Sample command to call the REST API:
      $ curl -i -X GET http://$IP:$PORT/db-tier/status/db-replication-svc/realtime
  • When HTTPS is enabled in a cnDBTier cluster:
    • Create the key PEM file (file.key.pem) and cert PEM file (file.crt.pem) using the p12 certificate (replicationcertificate.p12). Use the same p12 certificate that was used to enable the HTTPS while installing cnDBTier.
    • Run the following command to get the replication service LoadBalancer IP for cluster1:
      $ IP=$(kubectl get svc -n cluster1 | grep repl | awk '{print $4}' | head -n 1 )
    • Run the following commands to get the replication service LoadBalancer Port for cluster1:
      $ PORT=$(kubectl get svc -n cluster1 | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1) 
    • Sample command to call the REST API:
      $ curl -k --cert file.crt.pem --cert-type PEM --key file.key.pem --key-type PEM --pass NextGenCne https://$IP:$PORT/db-tier/status/db-replication-svc/realtime

4.4 cnDBTier Status APIs

cnDBTier Status APIs enable the NF applications to determine the status of cnDBTier and the associated cross-site replication. The DB Monitor service hosts the cnDBTier Status APIs.

Table 4-7 cnDBTier Status API

cnDBTier Status API Rest URL HTTP Method Request Payload Response Code Response Payload
Site Specific Real Time Replication Status http://<base-uri>/db-tier/status/replication/{mate-site-name}/realtime GET NA
  • 200 OK - Cross-site replication is enabled.
  • 400 Bad Request.
  • 404 Not Found - The indicated mate site has not been configured.
  • 503 Service Unavailable - Cross-site replication is not enabled and/or DOWN.
  • 200 OK:
    {
         "replicationStatus": "UP"    
         "secondsBehindRemote": "<greater_secondsBehindRemote_of_multiplegroups_withremotesite>"
         "replicationGroupDelay":
         [{
             replchannel_group_id: 1,
             secondsBehindRemote: <seconds>
          },
          {
             replchannel_group_id: 2,
             secondsBehindRemote: <seconds>
         }]
         "siteDetails":
         [{
             remote_replication_ip: 127.0.0.1,
             role: ACTIVE
          },
          {
             remote_replication_ip: 127.0.0.2,
             role: STANDBY
          }, 
          {
             remote_replication_ip: 127.0.0.2,
             role: ACTIVE
          },
          {
             remote_replication_ip: 127.0.0.3,
             role: STANDBY
         }]
    }
  • 404 Not Found: NA
  • 503 Service Unavailable: NA
Real Time Overall Replication Status http://<base-uri>/db-tier/status/replication/realtime GET NA
  • 200 OK - Cross-site replication is enabled.
  • 503 Service Unavailable - Cross-site replication is not enabled and/or DOWN.
  • 200 OK:
    {
     [
       {
         "localSiteName":"<current-site-name>",
         "remoteSiteName":"<remote-site-name>",
         "replicationStatus":"UP/DOWN/INITIALIZING"
         "secondsBehindRemote": "<greater_secondsBehindRemote_of_multiplegroups_withremotesite>"
          "replicationGroupDelay":
         [{
             replchannel_group_id: 1,
             secondsBehindRemote: <seconds>
          },
          {
             replchannel_group_id: 2,
             secondsBehindRemote: <seconds>
         }]
       },
       {
         "localSiteName":"<current-site-name>",
         "remoteSiteName":"<remote-site-name>",
         "replicationStatus":"UP/DOWN/INITIALIZING"
         "secondsBehindRemote": "<greater_secondsBehindRemote_of_multiplegroups_withremotesite>"
          "replicationGroupDelay":
         [{
             replchannel_group_id: 1,
             secondsBehindRemote: <seconds>
          },
          {
             replchannel_group_id: 2,
             secondsBehindRemote: <seconds>
         }]
       }
     ]
    }
    
    
  • 503 Service Unavailable: NA
Local Cluster Status http://<base-uri>/db-tier/status/local GET NA
  • 200 OK - The local NDB cluster is available.
  • 503 Service Unavailable - The local NDB cluster is not available.
NA
Site Specific Replication Status http://<base-uri>/db-tier/status/replication/{mate-site-name} GET NA
  • 200 OK - Cross-site replication is enabled.
  • 400 Bad Request.
  • 404 Not Found - The indicated mate site has not been configured.
  • 503 Service Unavailable - Cross-site replication is not enabled and/or DOWN.
  • 200 OK:
    {
      "replicationStatus": "UP/DOWN/INITIALIZING",
    "siteDetails":[
    {
    "remote_replication_ip":"<ip>",
    "role":"ACTIVE/STANDBY/FAILED"
    },
    {
           "remote_replication_ip":"<ip>",
           "role":"STANDBY/STANDBY/FAILED"
         }
       ]
    }
  • 404 Not Found: NA
  • 503 Service Unavailable: NA
Overall Replication Status http://<base-uri>/db-tier/status/replication GET NA
  • 200 OK - Cross-site replication is enabled.
  • 503 Service Unavailable - Cross-site replication is not enabled and/or DOWN.
  • 200 OK:
    {
     [
       {
         "localSiteName":"<current-site-name>",
         "remoteSiteName":"<remote-site-name>",
         "replicationStatus":"UP/DOWN/INITIALIZING"
       },
       {
         "localSiteName":"<current-site-name>",
         "remoteSiteName":"<remote-site-name>",
         "replicationStatus":"UP/DOWN/INITIALIZING"
       }
     ]
    }
  • 503 Service Unavailable: NA
Real Time Local Cluster Status http://<base-uri>/db-tier/status/cluster/local/realtime GET NA
  • 200 OK - The local NDB cluster is available.
  • 503 Service Unavailable - The local NDB cluster is not available.
NA
Database Statistics Reports API http://<base-uri>/db-tier/statistics/dbinfo GET NA
  • 200 OK - Data is available.
  • 503 Service Unavailable - DB statistics report is empty.
  • 500 INTERNAL SERVER ERROR - Could not list DB statistics reports
  • 200 OK:
    {
      "databaseCount": <database count>,
      "databaseTablesCount": [
        {
          "dbName": "<database name>",
          "tableCount": <table count>
        },
        {
          "dbName": "<database name>",
          "tableCount": <table count>
        }
    
      ],
      "databaseTableRowsCount": [
        {
          "dbName": "<database name>",
          "tables": [
            {
              "tableName": "<table name>",
              "rowCount": <row count>
            }
          ]
        },
        {
          "dbName": "<database name>",
          "tables": [
            {
              "tableName": "<table name>",
              "rowCount": <row count>
            },
            {
              "tableName": "<table name>",
              "rowCount": <row count>
            }
          ]
        }
      ]
    }
  • 503 Service Unavailable: NA
  • 500 INTERNAL SERVER ERROR: NA
cnDBTier Version http://<base-uri>/db-tier/version GET NA
  • 200 OK - cnDBTier version retrieved successfully.
  • 500 INTERNAL SERVER ERROR - failed to get the cnDBTier version.
  • 200 OK:
    {
      "cndbtier_version": "23.2.0",
      "ndb_version": "ndb-8.0.37"
    }
  • 500 INTERNAL SERVER ERROR:

    "failed to get version"

Note:

The value of base-uri in the table is <db-monitor-svc service name>.<namespace>:8080.

For example: curl http://mysql-cluster-db-monitor-svc.occne-cndbtier:8080/db-tier/status/replication/realtime