Metrics in Data Science

Metrics in Data Science

Evaluation metrics for classification

A classification model is evaluated on the test data. The common metrics used to evaluate the model are listed as

• Accuracy,
• Confusion matrix,
• Precision and Recall,
• Sensitivity and Specificity,
• F1Score and weighted F1,
• ROC curve and AUC ROC.

Accuracy is the ratio of correctly classified examples to the total number of examples in the test set. Defining true examples (TE) and false examples (FE) the number of correctly and incorrectly classified examples in the test set, accuracy can be given as

$$\mathrm {Accuracy} ={\frac {TE}{TE+FE}}.$$

The confusion matrix is a metric to visualize the classification performance. The rows of the matrix represent the true classes and the columns shows the predicted classes or vice versa. Both versions are used and can be found in the literature. Here we use the first version. The element (i,j) of the matrix shows the number of test examples belonging to class ${\textstyle i}$ and classified as class ${\textstyle j}$. Thus besides showing the number of correctly classified examples for each class, the confusion matrix gives an idea of typical missclassifications the model makes. An example confusion matrix can be seen in Figure 16.

Figure 16: An example confusion matrix for a classification model with 3 classes

The metrics precision and recall, sensitivity and specificity as well as F1Score and weighted F1 were originally introduced for binary classification models, but their usage can be extended to multiclass case. Therefore they will be defined and explained first for binary classification model.

In terms of binary classification with positive and negative classes, the outcome of the classification task can fall in one of four cases: correctly/incorrectly classified examples belonging to positive/negative class. The prediction is true positive (TP) when a positive example is correctly classified, like e.g. presence of a disease. A false negative (FN) prediction occurs when a positive example is classified by the model as would belong to negative class. Similarly a true negative (TN) prediction occurs when a negative example is correctly classified. Finally the prediction is false positive (FP) when a negative example is classified as would belong to positive class. These cases are summarized in Table 8.

Possible cases of the outcome of the classification task (=prediction)

predicted ${\textstyle \backslash }$ true	positive	negative
positive	true positive (TP)	false positive (FP)
negative	false negative (FN)	true negative (TN)

For the case of binary classification the definition of Accuracy can be given alternatively as

$$\mathrm {Accuracy} ={\frac {TP+TN}{TP+TN+FP+FN}}.$$

The terms TP, FN, TN and FP are also used to denote the number of corresponding cases, e.g. TP also denotes the number of true positives.

Precision is the ratio of the correctly classified positive examples to the total number of examples classified as positive:

$$\mathrm {Precision} ={\frac {TP}{TP+FP}}.$$

In contrast to that Recall is the ratio of the correctly classified positive examples to the total number of positive examples:

$$\mathrm {Recall} ={\frac {TP}{TP+FN}}.$$

The measures sensitivity and specificity are also commonly used, especially in the healthcare.

Sensitivity is the True Positive Rate (TPR), i.e. the proportion of captured trues and hence it equals to recall.

$$\mathrm {Sensitivity} =\mathrm {TPR} ={\frac {TP}{TP+FN}}=\mathrm {Recall} .$$

Specificity is the proportion of the captured negatives, i.e. True Negative Rate (TNR).

$$\mathrm {Specificity} ={\frac {TN}{TN+FP}}.$$

It follows that ${\textstyle 1-Specificity}$ is the False Positive Rate (FPR), i.e. proportion of not captured negatives, which is needed to define the ROC curve (see below).

The F1Score is a harmonic mean of precision and recall and therefore it is a number between ${\textstyle 0}$ and ${\textstyle 1}$.

$$\mathrm {F1Score} =F_{1}=2*{\frac {Precision*Recall}{Precision+Recall}}.$$

The usage of the different metrics.

Metric	When to use ?
Accuracy	In case of classification problem
	with balanced classes.
Precision	When it is important to be sure about the
	positive prediction to avoid any negative
	consequences, like e.g in case of decrease
	of credit limit to avoid customer dissatisfaction.
Recall	When it is important to capture positive even
	with low probability, like e.g. to predict
	whether a person has illness or not.
Sensitivity	If the question of interest is TPR,
	i.e. the proportion of the captured positives.
Specificity	If the question of interest is TNR,
	i.e. the proportion of the captured negatives.
F1Score	When both Precision and Recall are important.
weighted F1 metric	When importance of Precision and Recall
	against each other can be characterized
	by weights explicitly .
ROC curve	It is used for determining probability threshold
	for deciding the output class of the task,
	see Figure x.
AUC ROC	It is used to determine how well the positive class
	is separated from the negative class.

The weighted F1 metric is a refined version of F1Score, in which Precision and Recall can have different weights.

$$\mathrm {weightedF1} =F_{\beta }=(1+\beta ^{2})*{\frac {Precision*Recall}{\beta ^{2}*Precision+Recall}},$$

where Recall has weight ${\textstyle 1}$ and ${\textstyle \beta ^{2}}$ is the weight of Precision.

In a multi-class setting the metrics precision and recall, sensitivity and specificity as well as F1Score and weighted F1 metric are calculated first for each class individually and then averaged. This way they quantify the overall classification performance.

The metrics ROC curve and AUC ROC are defined for binary classification task. The Receiver Operating Characteristic (ROC) curve is the True Positive Rate (=Sensitivity) as a function of the False Positive Rate (= ${\textstyle 1-Specificity}$). The Area Under Curve ROC is called AUC ROC. It indicates how well the positive class is separated from the negative class.

The usage of the different metrics are summarized in the Table 9.

Figure 17: Illustrating the determination of probability threshold on ROC curve (Source:\url{https://www.kdnuggets.com/2019/10/5-classification-evaluation-metrics-every-data-scientist-must-know.html}

Depending on the use case strict, optimal or lenient (= moderate or high) probability threshold can be selected on the ROC curve. This is illustrated in Figure 17.

Evaluation metrics for regression

The common metrics used for evaluating regression models are are listed here.

• Mean Squared Error (MSE),
• Root Mean Squared Error (RMSE)
• Mean Absolute Error (MAE),
• Mean Absolute Percentage Error (MAPE),
• Coefficient of Determination (COD), R-squared (${\textstyle R^{2}}$),
• modified R-squared,

The Mean Squared Error (MSE) is one of the basic statistic used to evaluate the quality of a regression model. It is the average of the squares of the difference between the real and predicted values, in other words:

$$\mathrm {MSE} ={\frac {1}{K}}\sum _{k}({\hat {y}}_{k}-y_{k})^{2}.$$

Less the MSE, better the regression model fits the real values.

A similar measure is the Root Mean Squared Error (RMSE), which is the square root of MSE and thus it is biven as

$$\mathrm {RMSE} ={\sqrt {{\frac {1}{K}}\sum _{k}({\hat {y}}_{k}-y_{k})^{2}}}.$$

RMSE is a measure in the same units as the considered variable, which makes it a commonly used measure.

The Mean Absolute Error (MAE) is the average absolute difference between the real and predicted values.

$$\mathrm {MAE} ={\frac {1}{K}}\sum _{k}|y_{k}-{\hat {y}}_{k}|$$

A nice property of MAE, that it is less likely influenced by extreme values. It is a common measure used in time series analysis as forecast error.

The mean absolute percentage error (MAPE) quantifies the average of the ratio of the average absolute difference between the real and predicted values to the real value as a percentage. Hence the formula of MAPE can be given as

$$\mathrm {MAPE} =100{\frac {1}{K}}\sum _{k}|{\frac {y_{k}-{\hat {y}}_{k}}{y_{k}}}|$$

MAPE is is commonly used for evaluating regression models and it is an appropriate metric where the scale of the considered value varies in a broad range. Intuitively it can be interpreted as a kind of relative error. This makes it also suitable to be used as a loss function as an objective in the optimization in regression problems.

The metric Coefficient of Determination (COD) is also referred as R-squared and it is denoted by ${\textstyle R^{2}}$ or ${\textstyle r^{2}}$ and pronounced as "R-squared". The coefficient of determination determines the predictable proportion of the variation in the dependent variable, ${\textstyle {\bf {y}}}$. Let ${\textstyle {\bar {y}}}$ denote the mean of the output values, in other words

$${\bar {y}}={\frac {1}{K}}\sum _{k}y_{k}).$$

The coefficient of determination is defined in terms of residual sum of squares, ${\textstyle SS_{\mathrm {res} }}$ and total sum of squares (related to the variance of ${\textstyle {\bf {y}}}$), ${\textstyle SS_{\mathrm {tot} }}$

$${\begin{aligned}SS_{\mathrm {res} }&=\sum _{k}(y_{k}-{\hat {y}}_{k})^{2},\\SS_{\mathrm {tot} }&=\sum _{k}(y_{k}-{\bar {y}}_{k})^{2}\end{aligned}}$$

as

$$R^{2}=1-{\frac {SS_{\mathrm {res} }}{SS_{\mathrm {tot} }}}.$$

${\textstyle R^{2}}$ as metric quantifies the predictable proportion of the variation in the dependent variable. Its value falls between 0 and 1 with greater values indicating better regression fit.

The metric modified (or adjusted) R-squared is introduced to compensate that ${\textstyle R^{2}}$ increases when the dimension ${\textstyle {bfy}}$ becomes higher. Denoting the dimension of ${\textstyle {bfy}}$ by ${\textstyle N}$, the modified R-squared, ${\textstyle {\bar {R}}^{2}}$ is defined as

$${\bar {R}}^{2}=1-{\frac {SS_{\mathrm {res} }}{SS_{\mathrm {tot} }}}{\frac {K-1}{K-N-1}}=1-(1-R^{2}){\frac {K-1}{K-N-1}}.$$

Evaluation metricS for KG

Quality of KG

The two most important quality measures of a KG are

• completeness and
• accuracy.

The completeness refers to evaluate the amount of existing triplets in the KG, while accuracy targets to measure the amounts of correct and incorrect triplets in the KG. After KG completion the resulted extraction graph is considered to be not yet a ready KG. Therefore quality measurement of KG is relevant only after KG refinement.

Evaluation metric for KG refinement

Usually completeness is measured in recall, precision and F-measure.

The accuracy of the KG, i.e. the amounts of correct and incorrect triplets is evaluated in terms of accuracy and alternatively, or in addition by means of AUC (i.e. the area under the ROC curve).

The accuracy of the KG, ${\textstyle \mu ({\mathcal {G}})}$ can be also based on the correctness of the individual triplets ${\textstyle t\in {\mathcal {T}}}$, which can be assigned by human e.g. based on random sampling. Then ${\textstyle \mu ({\mathcal {G}})}$ is given by

$$\mu ({\mathcal {G}})={\frac {1}{|{\mathcal {T}}|}}\sum _{t\in {\mathcal {T}}}\mathbf {1} _{\{t\}},$$

where ${\textstyle \mathbf {1} _{\{t\}}}$ is the indicator variable indicating the correctness of the individual triplets ${\textstyle t\in {\mathcal {T}}}$ by ${\textstyle 1}$ otherwise taking the value ${\textstyle 0}$.

In case of human judging, the usual evaluation metric is accuracy or precision together with the total number of judged triplets and errors found.

Evaluation metric for link prediction with KG embeddings

The used evaluation metric for link prediction with KG embeddings depends on the subtask of link prediction, see in Table 10.

Evaluation metric for subtasks of link prediction

Subtask	Evaluation metrics
Entity prediction	rank based measures
Entity type prediction	Macro-${\textstyle F_{1}}$ and Micro-${\textstyle F_{1}}$
Triple classification	accuracy

The rank based evaluation metrics for entity prediction include

• Mean Reciprocal Rank (MRR) and
• Hits@K.

Mean reciprocal rank is the average of the reciprocal ranks of the correct entities:

$$MRR={\frac {1}{|{\mathcal {Q}}_{c}|}}\sum _{q\in {\mathcal {Q}}_{c}}{\frac {1}{R(q)}},$$

where ${\textstyle {\mathcal {Q}}_{c}}$ is set of correct entries and ${\textstyle R(q)}$ is the rank of the entry q. Larger the MRR, better the entity prediction and hence also the used KG embedding model.

Hits@k is the proportion of the correct entities in the best k predictions:
Hits@k${\textstyle ={\frac {|q\in {\mathcal {Q}}_{c}|:R(q)<k}{|{\mathcal {Q}}_{c}|}}}$.

Larger the Hits@k better the entity prediction and hence also the used KG embedding model.