SupervisedGraphWise

Overview

SupervisedGraphWise is an inductive vertex representation learning algorithm which is able to leverage vertex feature information. It can be applied to a wide variety of tasks, including vertex classification and link prediction.

SupervisedGraphWise is based on GraphSAGE by Hamilton et al.

Model Structure

A SupervisedGraphWise model consists of graph convolutional layers followed by several prediction layers. The forward pass through a convolutional layer for a vertex proceeds as follows:

  1. A set of neighbors of the vertex is sampled.

  2. The previous layer representations of the neighbors are mean-aggregated, and the aggregated features are concatenated with the previous layer representation of the vertex.

  3. This concatenated vector is multiplied with weights, and a bias vector is added.

  4. The result is normalized such that the layer output has unit norm.

The prediction layers are standard neural network layers.

Functionalities

We describe here the usage of the main functionalities of our implementation of GraphSAGE in PGX using the Cora graph as an example.

Loading a graph

First, we create a session and an analyst:

1session = pypgx.get_session()
2analyst = session.analyst

 1from pypgx.api.filters import VertexFilter
 2full_graph = session.read_graph_with_properties(
 3    self.cora_cfg, graph_name="cora")
 4vertex_filter = VertexFilter.from_pgql_result_set(
 5    session.query_pgql(
 6        "SELECT v FROM MATCH (v) ON cora WHERE ID(v) % 4 > 0"), "v"
 7)
 8train_graph = full_graph.filter(vertex_filter)
 9
10test_vertices = []
11for v in full_graph.get_vertices():
12    if not train_graph.has_vertex(v.id):
13        test_vertices.append(v)

Building a GraphWise Model (minimal)

We build a GraphWise model using the minimal configuration and default hyper-parameters. Note that even though only one feature property is specified in this example, you can specify arbitrarily many.

1params = dict(
2    vertex_target_property_name="label",
3    vertex_input_property_names=["features"]
4)
5model = analyst.supervised_graphwise_builder(**params)

Advanced hyperparameter customization

The implementation allows for very rich hyperparameter customization. Internally, GraphWise for each node it applies an aggregation of the representations of neighbors, this operation can be configured through a sub-config class: either GraphWiseConvLayerConfig or GraphWiseAttentionLayerConfig.

Prediction layer config is implemented through pypgx.api.mllib.GraphWisePredictionLayerConfig class. In the following, we build such configurations and use them in a model. We specify a weight decay of 0.001 and dropout with dropping probability 0.5 to counteract overfitting.

To enable or disable GPU, we can use the parameter enable_accelerator. By default this feature is enabled, however if there’s no GPU device and the cuda toolkit is not installed, the feature will be disabled and CPU will be the device used for all mllib operations.

 1weight_property = analyst.pagerank(train_graph).name
 2conv_layer_config = dict(
 3    num_sampled_neighbors=25,
 4    activation_fn='tanh',
 5    weight_init_scheme='xavier',
 6    neighbor_weight_property_name=weight_property,
 7    dropout_rate=0.5
 8)
 9
10conv_layer = analyst.graphwise_conv_layer_config(**conv_layer_config)
11pred_layer_config = dict(
12    hidden_dim=32,
13    activation_fn='relu',
14    weight_init_scheme='he',
15    dropout_rate=0.5
16)
17
18pred_layer = analyst.graphwise_pred_layer_config(**pred_layer_config)
19params = dict(
20    vertex_target_property_name="labels",
21    conv_layer_config=[conv_layer],
22    pred_layer_config=[pred_layer],
23    vertex_input_property_names=["vertex_features"],
24    edge_input_property_names=["edge_features"],
25    seed=17,
26    weight_decay=0.001
27)
28
29model = analyst.supervised_graphwise_builder(**params)

The above code uses GraphWiseConvLayerConfig for the convolutional layer configuration. It can be replaced with GraphWiseAttentionLayerConfig if a graph attention network model is desired. If the number of sampled neighbors is set to -1 using setNumSampledNeighbors, all neighboring nodes will be sampled.

1conv_layer_config = dict(
2    num_sampled_neighbors=25,
3    activation_fn='leaky_relu',
4    weight_init_scheme='xavier_uniform',
5    num_heads=4,
6    dropout_rate=0.5
7)
8
9conv_layer = analyst.graphwise_attention_layer_config(**conv_layer_config)

For a full description of all available hyperparameters and their default values, see the pypgx.api.mllib.SupervisedGraphWiseModelBuilder, pypgx.api.mllib.GraphWiseConvLayerConfig, pypgx.api.mllib.GraphWiseAttentionLayerConfig and pypgx.api.mllib.GraphWisePredictionLayerConfig docs.

Property types supported

The model supports two types of properties for both vertices and edges:

  • continuous properties (boolean, double, float, integer, long)

  • categorical properties (string)

For categorical properties, two categorical configurations are possible:

  • one-hot-encoding: each category is mapped to a vector, that is concatenated to other features (default)

  • embedding table: each category is mapped to an embedding that is concatenated to other features and is trained along with the model

One-hot-encoding converts each category into an independent vector. Therefore, it is suitable if we want each category to be interpreted as an equally independent group. For instance, if there are categories ranging from A to E without meaning anything by each alphabet, one-hot-encoding can be a good fit.

Embedding table is recommended if the semantics of the properties matter, and we want certain categories to be closer to each other than the others. For example, let’s assume there is a “day” property with values ranging from Monday to Sunday and we want to preserve our intuition that “Tuesday” is closer to “Wednesday” than “Saturday”. Then by choosing the embedding table configuration, we can let the vectors that represent the categories to be learned during training so that the vector that is mapped to “Tuesday” becomes close to that of “Wednesday”.

Although the embedding table approach has an advantage over one-hot-encoding that we can learn more suitable vectors to represent each category, this also means that a good amount of data is required to train the embedding table properly. The one-hot-encoding approach might be better for use-cases with limited training data.

When using the embedding table, we let users set the out-of-vocabulary probability. With the given probability, the embedding will be set to the out-of-vocabulary embedding randomly during training, in order to make the model more robust to unseen categories during inference.

 1vertex_input_property_configs = [
 2    analyst.one_hot_encoding_categorical_property_config(
 3        property_name="vertex_str_feature_1",
 4        max_vocabulary_size=100,
 5    ),
 6    analyst.learned_embedding_categorical_property_config(
 7        property_name="vertex_str_feature_2",
 8        embedding_dim=4,
 9        shared=False, # set whether to share the vocabulary or not when several  types have a property with the same name
10        oov_probability=0.001 # probability to set the word embedding to the out-of-vocabulary embedding
11    ),
12]
13
14model_params = dict(
15    vertex_input_property_names=[
16        "vertex_int_feature_1", # continuous feature
17        "vertex_str_feature_1", # string feature using one-hot-encoding
18        "vertex_str_feature_2", # string feature using embedding table
19        "vertex_str_feature_3", # string feature using one-hot-encoding (default)
20    ],
21    vertex_input_property_configs=vertex_input_property_configs,
22    vertex_target_property_name="labels",
23    enable_accelerator=True # Enable or Disable GPU
24)
25
26model = analyst.supervised_graphwise_builder(**model_params)

Classification vs Regression models

Whatever the type of the property you’re trying to predict, the default task that the model addresses is classification. Even if this property is a number, the model will assign one label for each value found and classify on it.

In some cases, you may prefer to infer continuous values for your property when it is an integer or a float. This is called the regression mode, and to enable it, you need to provide the MSELoss loss function object.

Setting a custom Loss Function and Batch Generator (for Anomaly Detection)

It is possible to select different loss functions for the supervised model by providing a loss function object, and different batch generators by providing a batch generator type. This is useful for applications such as Anomaly Detection, which can be cast into the standard supervised framework but require different loss functions and batch generators.

SupervisedGraphWise model can use the DevNetLoss and the StratifiedOversamplingBatchGenerator. Where the DevNetLoss takes two parameters: the confidence margin and the value the anomaly takes in the target property. In the following example, we assume the convLayerConfig has already been defined:

 1from pypgx.api.mllib import DevNetLoss
 2
 3pred_layer_config = dict(
 4    hidden_dim=32,
 5    activation_fn='linear'
 6)
 7
 8pred_layer = analyst.graphwise_pred_layer_config(**pred_layer_config)
 9params = dict(
10    vertex_target_property_name="labels",
11    conv_layer_config=[conv_layer],
12    pred_layer_config=[pred_layer],
13    vertex_input_property_names=["vertex_features"],
14    edge_input_property_names=["edge_features"],
15    loss_fn=DevNetLoss(5.0, True),
16    batch_gen="stratified_oversampling",
17    seed=17,
18    enable_accelerator=True # Enable or Disable GPU
19)
20
21model = analyst.supervised_graphwise_builder(**params)

Training the SupervisedGraphWiseModel

We can train a SupervisedGraphWiseModel model on a graph:

1model.fit(train_graph)

Getting Loss value

We can fetch the training loss value:

1loss = model.get_training_loss()

Inferring vertex labels

We can infer the labels for vertices on any graph (including vertices or graphs that were not seen during training):

1labels = model.infer(full_graph, test_vertices)
2labels.print()

If the model is a classification model, it’s also possible to set the decision threshold applied to the logits by adding it as an extra parameter, which is by default 0:

1labels = model.infer(
2    full_graph,
3    full_graph.get_vertices(),
4    6
5)
6labels.print()

The output will be similar to the following example output:

vertexId

label

2

Neural Networks

6

Theory

7

Case Based

22

Rule Learning

30

Theory

34

Neural Networks

47

Case Based

48

Probabalistic Methods

50

Theory

52

Theory

In a similar fashion, if the model is a classification model, you can get the model confidence for each class by inferring the prediction logits:

1logits = model.infer_logits(graph, test_vertices)
2logits.print()

If the model is a classification model, the infer_labels method is also available and equivalent to infer.

Evaluating model performance

evaluate() is a convenience method to evaluate various metrics for the model:

1model.evaluate(full_graph, test_vertices).print()

Similar to inferring labels, we can add the decision threshold as an extra parameter:

1model.evaluate(full_graph, test_vertices, 6).print()

The output will be similar to the following examples. For a classification model:

Accuracy

Precision

Recall

F1-Score

0.8488

0.8523

0.831

0.8367

For a regression model:

MSE

0.9573243436116953

If the model is a classification model, the evaluate_labels method is also available and equivalent to evaluate.

Inferring embeddings

We can use a trained model to infer embeddings for unseen nodes and store in a CSV file:

1vertex_vectors = model.infer_embeddings(full_graph, test_vertices).flatten_all()
2vertex_vectors.store("<path>/vertex_vectors.csv", file_format="csv", overwrite=True)

The schema for the vertex_vectors would be as follows without flattening (flatten_all splits the vector column into separate double-valued columns):

vertexId

embedding

Storing a trained model

Models can be stored either to the server file system, or to a database.

The following shows how to store a trained SupervisedGraphWise model to a specified file path:

1model.export().file("<path>/<model_name>", key)

When storing models in database, they are stored as a row inside a model store table. The following shows how to store a trained SupervisedGraphWise model in database in a specific model store table:

1model.export().db(
2    "modeltablename",
3    "model_name",
4    username="user",
5    password="password",
6    jdbc_url="jdbcUrl"
7)

Loading a pre-trained model

Similarly to storing, models can be loaded from a file in the server file system, or from a database. We can load a pre-trained SupervisedGraphWise model from a specified file path as follows:

1model = analyst.load_supervised_graphwise_model("<path>/<model>", "key")

We can load a pre-trained SupervisedGraphWise model from a model store table in database as follows:

1model = analyst.get_supervised_graphwise_model_loader().db(
2    "modeltablename",
3    "model_name",
4    username="user",
5    password="password",
6    jdbc_url="jdbcUrl"
7)

Explaining a Prediction

In order to understand which features and vertices were important for a prediction of the model, we can generate a SupervisedGnnExplanation using a technique similar to the GNNExplainer by Ying et al..

The explanation holds information related to

  • graph structure: an importance score for each vertex

  • features: an importance score for each graph property

Note that the vertex being explained is always assigned importance 1. Further, the feature importances are scaled such that the most important feature has importance 1.

Additionally, a SupervisedGnnExplanation contains the inferred embedding, logits, and label.

To get explanations for a model’s predictions, its SupervisedGnnExplainer object can be obtained using the gnn_explainer() method. After obtaining the SupervisedGnnExplainer, its inferAndExplain() method can be used to request and explanation for a vertex.

The parameters of the explainer can be configured while the explainer is being created or afterwards using the relevant setter functions. The configurable parameters for the SupervisedGnnExplainer are:

  • numOptimizationSteps: the number of optimization steps used by the explainer

  • learningRate: the learning rate of the explainer

  • marginalize: whether the explainer loss is marginalized over features. This can help in cases where there are important features that take values close to zero. Without marginalization the explainer can learn to mask such features out even if they are important, marginalization solves this by instead learning a mask for the deviation from the estimated input distribution.

Note that, in order to achieve best results, the features should be centered around 0.

We can generate an explanation on a simple graph as follows. The graph contains a feature that correlates with the label and one that does not. We hence expect the importance of the features to differ significantly (with the feature correlating with the label being more important), whereas structural importance does not play a big role here.

 1simple_graph = session.create_graph_builder() \
 2    .add_vertex(0) \
 3    .set_property("label_feature", 0.5) \
 4    .set_property("const_feature", 0.5) \
 5    .set_property("label", True) \
 6    .add_vertex(1) \
 7    .set_property("label_feature", -0.5) \
 8    .set_property("const_feature", 0.5) \
 9    .set_property("label", False) \
10    .add_edge(0, 1) \
11    .build()
12# build and train model as described above
13params = dict(
14    vertex_target_property_name="label",
15    vertex_input_property_names=["label_feature", "const_feature"]
16)
17
18model = analyst.supervised_graphwise_builder(**params)
19model.fit(simple_graph)
20
21# obtain the explainer
22explainer = model.gnn_explainer(learning_rate=0.05)
23explainer.num_optimization_steps = 200
24
25# explain prediction of vertex 0
26explanation = explainer.infer_and_explain(
27    simple_graph, graph.get_vertex(0))
28# if we used the devNet loss, we can add the decision threshold as an extra parameter:
29# explanation = explainer.inferAndExplain(simple_graph, simple_graph.get_vertex(0), 6)
30
31const_property = simple_graph.get_vertex_property("const_feature")
32label_property = simple_graph.get_vertex_property("label_feature")
33
34# retrieve feature importances
35feature_importances = explanation.get_vertex_feature_importance()
36# small as unimportant
37importance_const_prop = feature_importances[const_property]
38# large (1) as important
39importance_label_prop = feature_importances[label_property]
40
41# retrieve computation graph with importances
42importance_graph = explanation.get_importance_graph()
43
44# retrieve importance of vertices
45importance_property = explanation.get_vertex_importance_property()
46# has importance 1
47importance_vertex_0 = importance_property[0]
48# available if vertex 1 part of computation
49importance_vertex_1 = importance_property[1]

Destroying a model

We can destroy a model as follows:

1model.destroy()