cdt.metrics

The CDT package implements some metrics to evaluate the output of a algorithm given a ground truth. All these metrics are in the form metric(target, prediction), where any of those arguments are either numpy matrixes that represent the adjacency matrix or networkx.DiGraph instances.

Warning

in the case of heterogeneous types of arguments target and prediction, special care has to be given to the order of the nodes, as the type networkx.DiGraph does not retain node order.

cdt.metrics.precision_recall(target, prediction, low_confidence_undirected=False)[source]

Compute precision-recall statistics for directed graphs.

Precision recall statistics are useful to compare algorithms that make predictions with a confidence score. Using these statistics, performance of an algorithms given a set threshold (confidence score) can be approximated. Area under the precision-recall curve, as well as the coordinates of the precision recall curve are computed, using the scikit-learn library tools. Note that unlike the AUROC metric, this metric does not account for class imbalance.

Precision is defined by: \(Pr=tp/(tp+fp)\) and directly denotes the total classification accuracy given a confidence threshold. On the other hand, Recall is defined by: \(Re=tp/(tp+fn)\) and denotes misclassification given a threshold.

Parameters

  • target (numpy.ndarray or networkx.DiGraph) – Target graph, must be of ones and zeros.

  • prediction (numpy.ndarray or networkx.DiGraph) – Prediction made by the algorithm to evaluate.

  • low_confidence_undirected – Put the lowest confidence possible to undirected edges (edges that are symmetric in the confidence score). Default: False

Returns

tuple containing:

  • Area under the precision recall curve (float)

  • Tuple of data points of the precision-recall curve used in the computation of the score (tuple).

Return type

tuple

Examples

>>> from cdt.metrics import precision_recall
>>> import numpy as np
>>> tar, pred = np.random.randint(2, size=(10, 10)), np.random.randn(10, 10)
>>> # adjacency matrixes of size 10x10
>>> aupr, curve = precision_recall(target, input)
>>> # leave low_confidence_undirected to False as the predictions are continuous
cdt.metrics.SHD(target, pred, double_for_anticausal=True)[source]

Compute the Structural Hamming Distance.

The Structural Hamming Distance (SHD) is a standard distance to compare graphs by their adjacency matrix. It consists in computing the difference between the two (binary) adjacency matrixes: every edge that is either missing or not in the target graph is counted as a mistake. Note that for directed graph, two mistakes can be counted as the edge in the wrong direction is false and the edge in the good direction is missing ; the double_for_anticausal argument accounts for this remark. Setting it to False will count this as a single mistake.

Parameters

  • target (numpy.ndarray or networkx.DiGraph) – Target graph, must be of ones and zeros.

  • prediction (numpy.ndarray or networkx.DiGraph) – Prediction made by the algorithm to evaluate.

  • double_for_anticausal (bool) – Count the badly oriented edges as two mistakes. Default: True

Returns

Structural Hamming Distance (int).

The value tends to zero as the graphs tend to be identical.

Return type

int

Examples

>>> from cdt.metrics import SHD
>>> from numpy.random import randint
>>> tar, pred = randint(2, size=(10, 10)), randint(2, size=(10, 10))
>>> SHD(tar, pred, double_for_anticausal=False)
cdt.metrics.SID(target, pred)[source]

Compute the Strutural Intervention Distance.

[R wrapper] The Structural Intervention Distance (SID) is a new distance for graphs introduced by Peters and Bühlmann (2013). This distance was created to account for the shortcomings of the SHD metric for a causal sense. It consists in computing the path between all the pairs of variables, and checks if the causal relationship between the variables is respected. The given graphs have to be DAGs for the SID metric to make sense.

Required R packages: SID

Parameters

  • target (numpy.ndarray or networkx.DiGraph) – Target graph, must be of ones and zeros, and instance of either numpy.ndarray or networkx.DiGraph. Must be a DAG.

  • prediction (numpy.ndarray or networkx.DiGraph) – Prediction made by the algorithm to evaluate. Must be a DAG.

Returns

Structural Intervention Distance.

The value tends to zero as the graphs tends to be identical.

Return type

int

Note

Ref: Structural Intervention Distance (SID) for Evaluating Causal Graphs, Jonas Peters, Peter Bühlmann: https://arxiv.org/abs/1306.1043

Examples

>>> from cdt.metrics import SID
>>> from numpy.random import randint
>>> tar = np.triu(randint(2, size=(10, 10)))
>>> pred = np.triu(randint(2, size=(10, 10)))
>>> SID(tar, pred)