************************************* UnsupervisedAnomalyDetectionGraphWise ************************************* Overview -------- :class:`UnsupervisedAnomalyDetectionGraphWise` is an inductive vertex representation learning and anomaly detection algorithm which is able to leverage vertex and edge feature information. While it can be applied to a wide variety of tasks, it is particularly suitable for unsupervised learning of vertex embeddings for anomaly detection. After training this model, it is possible to infer anomaly scores or labels for unseen nodes. :class:`UnsupervisedAnomalyDetectionGraphWise` is based on `Deep Anomaly Detection on Attributed Networks (Dominant) by Ding, Kaize, et al. `_ Model Structure --------------- A :class:`UnsupervisedAnomalyDetectionGraphWise` model consists of graph convolutional layers followed by an embedding layer. There are two types of embedding layer available: DGI layer and Dominant layer. Both of the layers are for inductive vertex representation learning, with different loss functions. The embedding layer defaults to the DGI layer. The forward pass through a convolutional layer for a vertex proceeds as follows: 1. A set of neighbors of the vertex is sampled. 2. The previous layer representations of the neighbors are mean-aggregated, and the aggregated features are concatenated with the previous layer representation of the vertex. 3. This concatenated vector is multiplied with weights, and a bias vector is added. 4. The result is normalized to such that the layer output has unit norm. The DGI Layer, which is based on [`Deep Graph Infomax (DGI)` by Velickovic et al.](https://arxiv.org/pdf/1809.10341.pdf) consists of three parts enabling unsupervised learning using embeddings produced by the convolution layers. 1. Corruption function: Shuffles the node features while preserving the graph structure to produce negative embedding samples using the convolution layers. 2. Readout function: Sigmoid activated mean of embeddings, used as summary of a graph 3. Discriminator: Measures the similarity of positive (unshuffled) embeddings with the summary as well as the similarity of negative samples with the summary from which the loss function is computed. Since none of these contains mutable hyperparameters, the default DGI layer is always used and cannot be adjusted. The Dominant layer enables unsupervised learning using a deep autoencoder. It uses GCNs to reconstruct the features in the autoencoder setting, together with the reconstructed structure that is estimated using the dot products of the embeddings. The loss function is computed from the feature reconstruction loss and the structure reconstruction loss. The importance given to features or to the structure can be tuned with the alpha hyperparameter. The Dominant layer is based on [`Deep Anomaly Detection on Attributed Networks (Dominant)` by Ding, Kaize, et al.](https://www.public.asu.edu/~kding9/pdf/SDM2019_Deep.pdf) Functionalities --------------- We describe here the usage of the main functionalities of our implementation of ``Dominant`` in PGX. The following example demonstrates a scenario where we want to detect fraudulent vertices based on their features. Loading a graph ~~~~~~~~~~~~~~~ First, we create a session and an analyst: .. code-block:: python :linenos: session = pypgx.get_session() analyst = session.analyst Since we train the model unsupervised, we do not have to use a test graph or test vertices .. code-block:: python :linenos: graph = session.read_graph_with_properties(cpath) Building an UnsupervisedGraphWise Model (minimal) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We build an :class:`UnsupervisedAnomalyDetectionGraphWise` model using the minimal configuration and default hyper-parameters. Note that even though only one feature property is specified in this example, you can specify arbitrarily many. .. code-block:: python :linenos: model = analyst.unsupervised_anomaly_detection_graphwise_builder( vertex_input_property_names=["features"] ) Advanced hyperparameter customization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The implementation allows for very rich hyperparameter customization. This is done through a sub-config class: :class:`GraphWiseConvLayerConfig` and :class:`GraphWiseEmbeddingConfig`. In the following, we build such a configuration and use it in a model. We specify a weight decay of ``0.001`` and dropout with dropping probability ``0.5`` to counteract overfitting. We also specify the Dominant embedding layer's alpha value to 0.6 to slightly increase the importance of the feature reconstruction. To enable or disable GPU, we can use the parameter `enable_accelerator`. By default this feature is enabled, however if there's no GPU device and the cuda toolkit is not installed, the feature will be disabled and CPU will be the device used for all mllib operations. .. code-block:: python :linenos: # customize convolutional layer config weight_property = analyst.pagerank(train_graph).name conv_layer_config = dict( num_sampled_neighbors=25, activation_fn='tanh', weight_init_scheme='xavier', neighbor_weight_property_name=weight_property, dropout_rate=0.5, # set dropout rate to prevent overfitting ) conv_layer = analyst.graphwise_conv_layer_config(**conv_layer_config) # customize embedding layer config dominant_config = dict(alpha=0.6) dominant_layer = analyst.graphwise_dominant_layer_config(**dominant_config) params = dict( conv_layer_config=[conv_layer], embedding_config=dominant_layer, vertex_input_property_names=["vertex_features"], edge_input_property_names=["edge_features"], weight_decay=0.001, # set weight decay to prevent overfitting enable_accelerator=True # Enable or Disable GPU ) model = analyst.unsupervised_anomaly_detection_graphwise_builder(**params) For a full description of all available hyperparameters and their default values, see the :class:`pypgx.api.mllib.UnsupervisedAnomalyDetectionGraphWiseModel`, :class:`pypgx.api.mllib.GraphWiseConvLayerConfig`, :class:`pypgx.api.mllib.GraphWiseDgiLayerConfig` and :class:`pypgx.api.mllib.GraphWiseDominantLayerConfig` docs. Training the :class:`UnsupervisedAnomalyDetectionGraphWiseModel` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We can train a ``UnsupervisedAnomalyDetectionGraphWiseModel`` model on a graph: .. code-block:: python :linenos: model.fit(graph) Getting Loss value ~~~~~~~~~~~~~~~~~~ We can fetch the training loss value: .. code-block:: python :linenos: loss = model.get_training_loss() Inferring embeddings ~~~~~~~~~~~~~~~~~~~~ We can use a trained model to infer embeddings for unseen nodes and store in a ``CSV`` file: .. code-block:: python :linenos: vertex_vectors = model.infer_embeddings(full_graph, cora.get_vertices()).flatten_all() vertex_vectors.store("/vertex_vectors.csv", file_format="csv", overwrite=True) The schema for the :meth:`vertex_vectors` would be as follows without flattening (:meth:`flatten_all` splits the vector column into separate double-valued columns): +-----------------------------------------+---------------------+ | vertexId | embedding | +-----------------------------------------+---------------------+ Inferring anomalies ~~~~~~~~~~~~~~~~~~~~ We can use a trained model to infer anomaly scores or labels for unseen nodes and store in a ``CSV`` file: .. code-block:: python :linenos: vertex_scores = model.infer_anomaly_scores(full_graph, fullGraph.get_vertices()).flatten_all() vertex_scores.store("/vertex_scores.csv", file_format="csv", overwrite=True) If we know the contamination factor of our data we can use it to find a good threshold: .. code-block:: python :linenos: contamination_factor = 0.2 threshold = model.find_anomaly_threshold(full_graph, fullGraph.get_vertices(), contamination_factor) If we have a threshold value we can directly infer labels: .. code-block:: python :linenos: threshold = 0.9 vertex_labels = model.infer_anomaly_labels(full_graph, fullGraph.get_vertices(), threshold).flatten_all() vertex_labels.store("/vertex_labels.csv", file_format="csv", overwrite=True) Storing a trained model ~~~~~~~~~~~~~~~~~~~~~~~ Models can be stored either to the server file system, or to a database. The following shows how to store a trained :class:`UnsupervisedAnomalyDetectionGraphWise` model to a specified file path: .. code-block:: python :linenos: model.export().file("/", key) When storing models in database, they are stored as a row inside a model store table. The following shows how to store a trained :class:`UnsupervisedAnomalyDetectionGraphWise` model in database in a specific model store table: .. code-block:: python :linenos: model.export().db( "modeltablename", "model_name", username="user", password="password", jdbc_url="jdbcUrl" ) Loading a pre-trained model ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Similarly to storing, models can be loaded from a file in the server file system, or from a database. We can load a pre-trained :class:`UnsupervisedAnomalyDetectionGraphWise` model from a specified file path as follows: .. code-block:: python :linenos: model = analyst.load_unsupervised_anomaly_detection_graphwise_model( "/", key ) We can load a pre-trained :class:`UnsupervisedAnomalyDetectionGraphWise` model from a model store table in database as follows: .. code-block:: python :linenos: model = analyst.get_unsupervised_anomaly_detection_graphwise_model_loader().db( "modeltablename", "model_name", username="user", password="password", jdbc_url="jdbcUrl" ) Destroying a model ~~~~~~~~~~~~~~~~~~ We can destroy a model as follows: .. code-block:: python :linenos: model.destroy()