Graph Sink

Overview

The Unified Assurance Graph Sink microservice is part of the microservice topology pipeline. The Graph Sink is responsible for topology data that has been normalized. It and inserts the normalized topology data into the Neo4j database.

Prerequisites

  1. A microservices cluster must be setup. Refer to Microservice Cluster Setup.

  2. Apache Pulsar must be installed. Refer to Apache Pulsar microservice.

Setup

su - assure1
export NAMESPACE=a1-zone1-pri
export WEBFQDN=<Primary Presentation Web FQDN> 
a1helm install graph-sink assure1/graph-sink -n $NAMESPACE --set global.imageRegistry=$WEBFQDN

Default Configuration

Name Value Possible Values Notes
LOG_LEVEL INFO FATAL, ERROR, WARN, INFO, DEBUG Logging level used by application.
STREAM_INPUT persistent://assure1/graph/sink Text, 255 characters Apache Pulsar topic path. Topic at end of path may be any text value.
BATCH_DELAY 0 Integer Time to wait in seconds before processing a batch. If set to 0, messages will be processed one-by-one.
MAX_BATCH_SIZE 1000 Integer Maximum number of messages that can be present in a batch.

Configurations can be changed by passing the values to the a1helm install prefixed with the configData parent key.

Example of setting the log level to DEBUG

a1helm install ... --set configData.LOG_LEVEL=DEBUG

Autoscaling

For microservice scaling options, please refer to the autoscaling docs.

On top of the standard autoscaling options, this microservice supports the following additional configurations:

Name Value Possible Values Notes
thresholds.backlogSize 1000 Integer The number of items that need to be in the backlog before the auto-scaling starts additional processes.
thresholds.totalItemsProcessed 400 Integer Total items processed by the microservice. If the average of total items processed in five minutes exceeds the threshold, pods will be scaled.

Example of changing the auto-scaling value

To change the thresholds used for this functionality, pass the following to the installation command:

a1helm install ... --set autoscaling.thresholds.backlogSize=2000

Schema

Below is an example of JSON file formats with the accepted schema.

Single Vertex Schema

{
  "@timestamp": "2020-03-18T18:41:00.000Z",
  "_agents": [
    {
      "@timestamp": "2020-03-18T18:40:00.000Z",
      "host": "host.example.com",
      "id": "d5a346c8-d675-48a1-a42f-4a8db7c1552b",
      "app": "sample-integration",
      "version": "5.0.0",
      "input": "unknown://",
      "output": "pulsar+ssl://proxy.a1-messaging.svc.cluster.local/assure1/graph/sink"
    }
  ],
  "_domain": "topology",
  "_type": "vertex",
  "_version": "5.0.0",
  "vertex": {
    "@meta": false,
    "@method": "upsert",
    "@type": "Device",
    "keys": {
      "Name": "device-a.example.com",
      "ZoneID": 1
    },
    "properties": {
      "TimestampModified": "123456789000"
    }
  }
}

Edge Tuple Schema

{
  "@timestamp": "2021-03-30T15:32:21.690779663Z",
  "_agents": [
    {
      "@timestamp": "2021-03-30T15:32:21.690778749Z",
      "host": "pg-example.example.com",
      "id": "255c83bd-ff1e-2e10-6d92-dfecce7fba21",
      "app": "virtuora-connector",
      "version": "1.0.1",
      "input": "pg-example.example.com",
      "output": "persistent://assure1/graph/sink"
    }
  ],
  "_domain": "topology",
  "_type": "edgeTuple",
  "_version": "1.0.1",
  "edgeTuple": {
    "outVertex": {
      "@meta": false,
      "@method": "upsert",
      "@type": "Device",
      "keys": {
        "Name": "L100-107-TEST",
        "ZoneID": 1
      },
      "properties": {
        "TimestampModified": 1617118341
      }
    },
    "inVertex": {
      "@meta": false,
      "@method": "upsert",
      "@type": "Device",
      "keys": {
        "Name": "L100-107-TEST",
        "ZoneID": 0
      },
      "properties": {
        "TimestampModified": 1617118341
      }
    },
    "edge": {
      "@meta": false,
      "@method": "upsert",
      "@type": "ConnectsInterface",
      "keys": null,
      "properties": {
        "TimestampModified": 1617118341
      }
    }
  }
}

Deletion by single vertex example, type and at least one key is required:

{
  "@timestamp": "2020-03-18T18:41:00.000Z",
  "_agents": [
    {
      "@timestamp": "2020-03-18T18:40:00.000Z",
      "host": "host.example.com",
      "id": "d5a346c8-d675-48a1-a42f-4a8db7c1552b",
      "app": "sample-integration",
      "version": "5.0.0",
      "input": "unknown://",
      "output": "pulsar+ssl://proxy.a1-messaging.svc.cluster.local/assure1/graph/sink"
    }
  ],
  "_domain": "topology",
  "_type": "vertex",
  "_version": "5.0.0",
  "vertex": {
    "@meta": false,
    "@method": "delete",
    "@type": "Device",
    "keys": {
      "Name": "device-x.example.com",
      "ZoneID": 1
    }
  }
}

Deletion by single edge, type and at least one unique property is required:

{
  "@timestamp": "2020-03-18T18:41:00.000Z",
  "_agents": [
    {
      "@timestamp": "2020-03-18T18:40:00.000Z",
      "host": "host.example.com",
      "id": "d5a346c8-d675-48a1-a42f-4a8db7c1552b",
      "app": "sample-integration",
      "version": "5.0.0",
      "input": "unknown://",
      "output": "pulsar+ssl://proxy.a1-messaging.svc.cluster.local/assure1/graph/sink"
    }
  ],
  "_domain": "topology",
  "_type": "edge",
  "_version": "5.0.0",
  "edge": {
    "@meta": false,
    "@method": "delete",
    "@type": "HasInterface",
    "properties": {
      "connectionID": "1023"
    }
  }
}

Microservice self-metrics

The Unified Assurance Graph Sink microservice exposes the following self-metrics to Prometheus.

Metric Name Type Description
graph_sink_backlog_size Gauge Number of items pending in the backlog
total_items_processed Counter Number of items processed
processing_time_per_item Gauge Processing time per item from receiving it to sending it to acknowledging it