SAFFER Zsolt am 13. März 2025 um 21:14 Uhr

2025-03-13T21:14:17Z

← Nächstältere Version		Version vom 13. März 2025, 21:14 Uhr
Zeile 22:		Zeile 22:
	<div id="fig:exa_conf_matrix" class="figure">		<div id="fig:exa_conf_matrix" class="figure">

	[[~~File~~:~~./figs/~~EvalMetric_ConfMatrExa.~~pdf~~]]		[[Datei:EvalMetric_ConfMatrExa.jpg\|460px\|thumb\|center\|Figure 16: An example confusion matrix for a classification model with 3 classes]]

	</div>		</div>
Zeile 164:		Zeile 164:
	<div id="fig:ill_ROC_curve" class="figure">		<div id="fig:ill_ROC_curve" class="figure">

	[[~~File~~:./~~figs~~/~~ROC_curve~~.~~pdf~~]]		[[Datei:ROC_curve.jpg\|460px\|thumb\|center\|Figure 17: Illustrating the determination of probability threshold on ROC curve (Source:\url{https://www.kdnuggets.com/2019/10/5-classification-evaluation-metrics-every-data-scientist-must-know.html}]]

	</div>		</div>

SAFFER Zsolt am 11. März 2025 um 22:21 Uhr

2025-03-11T22:21:17Z

← Nächstältere Version		Version vom 11. März 2025, 22:21 Uhr
Zeile 230:		Zeile 230:
	The accuracy of the KG, <math display="inline">\mu(\mathcal{G})</math> can be also based on the correctness of the individual triplets <math display="inline">t \in \mathcal{T}</math>, which can be assigned by human e.g. based on random sampling. Then <math display="inline">\mu(\mathcal{G})</math> is given by		The accuracy of the KG, <math display="inline">\mu(\mathcal{G})</math> can be also based on the correctness of the individual triplets <math display="inline">t \in \mathcal{T}</math>, which can be assigned by human e.g. based on random sampling. Then <math display="inline">\mu(\mathcal{G})</math> is given by

	<math display="block">\mu(\mathcal{G}) = \frac{1}{\|\mathcal{T}\|} \sum_{t \in \mathcal{T}} \~~mathbbm~~{1}_{\{t\}},</math> where <math display="inline">\~~mathbbm~~{1}_{\{t\}}</math> is the indicator variable indicating the correctness of the individual triplets <math display="inline">t \in \mathcal{T}</math> by <math display="inline">1</math> otherwise taking the value <math display="inline">0</math>.		<math display="block">\mu(\mathcal{G}) = \frac{1}{\|\mathcal{T}\|} \sum_{t \in \mathcal{T}} \mathbf{1}_{\{t\}},</math> where <math display="inline">\mathbf{1}_{\{t\}}</math> is the indicator variable indicating the correctness of the individual triplets <math display="inline">t \in \mathcal{T}</math> by <math display="inline">1</math> otherwise taking the value <math display="inline">0</math>.

	In case of human judging, the usual evaluation metric is accuracy or precision together with the total number of judged triplets and errors found.		In case of human judging, the usual evaluation metric is accuracy or precision together with the total number of judged triplets and errors found.
Zeile 270:		Zeile 270:
	Mean reciprocal rank is the average of the reciprocal ranks of the correct entities: <math display="block">MRR = \frac{1}{\|\mathcal{Q}_c\|}\sum_{q \in \mathcal{Q}_c} \frac{1}{R(q)},</math> where <math display="inline">\mathcal{Q}_c</math> is set of correct entries and <math display="inline">R(q)</math> is the rank of the entry q. Larger the MRR, better the entity prediction and hence also the used KG embedding model.		Mean reciprocal rank is the average of the reciprocal ranks of the correct entities: <math display="block">MRR = \frac{1}{\|\mathcal{Q}_c\|}\sum_{q \in \mathcal{Q}_c} \frac{1}{R(q)},</math> where <math display="inline">\mathcal{Q}_c</math> is set of correct entries and <math display="inline">R(q)</math> is the rank of the entry q. Larger the MRR, better the entity prediction and hence also the used KG embedding model.

	Hits@k is the proportion of the correct entities in the best k predictions: <math display="~~block~~">~~Hits@k~~ = \frac{\|q \in \mathcal{Q}_c\|: R(q) < k}{\|\mathcal{Q}_c\|}.</math> Larger the Hits@k better the entity prediction and hence also the used KG embedding model.		Hits@k is the proportion of the correct entities in the best k predictions:<br />
			Hits@k<math display="inline">= \frac{\|q \in \mathcal{Q}_c\|: R(q) < k}{\|\mathcal{Q}_c\|}</math>.

			Larger the Hits@k better the entity prediction and hence also the used KG embedding model.

SAFFER Zsolt: Die Seite wurde neu angelegt: „ = Metrics in Data Science = == Evaluation metrics for classification == A classification model is evaluated on the test data. The common metrics used to evaluate the model are listed as * Accuracy, * Confusion matrix, * Precision and Recall, * Sensitivity and Specificity, * F1Score and weighted F1, * ROC curve and AUC ROC. Accuracy is the ratio of corr…“

2025-03-04T23:09:23Z

Die Seite wurde neu angelegt: „<span id="metrics-in-data-science"></span> = Metrics in Data Science = <span id="evaluation-metrics-for-classification"></span> == Evaluation metrics for classification == A classification model is evaluated on the test data. The common metrics used to evaluate the model are listed as * Accuracy, * Confusion matrix, * Precision and Recall, * Sensitivity and Specificity, * F1Score and weighted F1, * ROC curve and AUC ROC. Accuracy is the ratio of corr…“

Neue Seite

<span id="metrics-in-data-science"></span>
= Metrics in Data Science =

<span id="evaluation-metrics-for-classification"></span>
== Evaluation metrics for classification ==

A classification model is evaluated on the test data. The common metrics used to evaluate the model are listed as

* Accuracy,
* Confusion matrix,
* Precision and Recall,
* Sensitivity and Specificity,
* F1Score and weighted F1,
* ROC curve and AUC ROC.

Accuracy is the ratio of correctly classified examples to the total number of examples in the test set. Defining true examples (TE) and false examples (FE) the number of correctly and incorrectly classified examples in the test set, accuracy can be given as

<math display="block">\mathrm{Accuracy} = \frac{TE}{TE+FE}.</math>

The confusion matrix is a metric to visualize the classification performance. The rows of the matrix represent the true classes and the columns shows the predicted classes or vice versa. Both versions are used and can be found in the literature. Here we use the first version. The element (i,j) of the matrix shows the number of test examples belonging to class <math display="inline">i</math> and classified as class <math display="inline">j</math>. Thus besides showing the number of correctly classified examples for each class, the confusion matrix gives an idea of typical missclassifications the model makes. An example confusion matrix can be seen in Figure [[#fig:exa_conf_matrix|16]].

<div id="fig:exa_conf_matrix" class="figure">

[[File:./figs/EvalMetric_ConfMatrExa.pdf]]

</div>
The metrics precision and recall, sensitivity and specificity as well as F1Score and weighted F1 were originally introduced for binary classification models, but their usage can be extended to multiclass case. Therefore they will be defined and explained first for binary classification model.

In terms of binary classification with positive and negative classes, the outcome of the classification task can fall in one of four cases: correctly/incorrectly classified examples belonging to positive/negative class. The prediction is true positive (TP) when a positive example is correctly classified, like e.g. presence of a disease. A false negative (FN) prediction occurs when a positive example is classified by the model as would belong to negative class. Similarly a true negative (TN) prediction occurs when a negative example is correctly classified. Finally the prediction is false positive (FP) when a negative example is classified as would belong to positive class. These cases are summarized in Table [[#tab:poss_predictions|8]].

<div class="center">

<div id="tab:poss_predictions">

{| class="wikitable"
|+ Possible cases of the outcome of the classification task (=prediction)
|-
! style="text-align: left;"| predicted <math display="inline">\backslash</math> true
! style="text-align: center;"| positive
! style="text-align: center;"| negative
|-
| style="text-align: left;"| positive
| style="text-align: center;"| <span>true positive (TP)</span>
| style="text-align: center;"| <span>false positive (FP)</span>
|-
| style="text-align: left;"| negative
| style="text-align: center;"| <span>false negative (FN)</span>
| style="text-align: center;"| <span>true negative (TN)</span>
|}

</div>

</div>
For the case of binary classification the definition of Accuracy can be given alternatively as <math display="block">\mathrm{Accuracy} = \frac{TP+TN}{TP+TN+FP+FN}.</math>

The terms TP, FN, TN and FP are also used to denote the number of corresponding cases, e.g. TP also denotes the number of true positives.

Precision is the ratio of the correctly classified positive examples to the total number of examples classified as positive:

<math display="block">\mathrm{Precision} = \frac{TP}{TP+FP}.</math>

In contrast to that Recall is the ratio of the correctly classified positive examples to the total number of positive examples: <math display="block">\mathrm{Recall} = \frac{TP}{TP+FN}.</math>

The measures sensitivity and specificity are also commonly used, especially in the healthcare.

Sensitivity is the True Positive Rate (TPR), i.e. the proportion of captured trues and hence it equals to recall. <math display="block">\mathrm{Sensitivity} = \mathrm{TPR} = \frac{TP}{TP+FN} = \mathrm{Recall}.</math>

Specificity is the proportion of the captured negatives, i.e. True Negative Rate (TNR). <math display="block">\mathrm{Specificity} = \frac{TN}{TN+FP}.</math> It follows that <math display="inline">1-Specificity</math> is the False Positive Rate (FPR), i.e. proportion of not captured negatives, which is needed to define the ROC curve (see below).

The F1Score is a harmonic mean of precision and recall and therefore it is a number between <math display="inline">0</math> and <math display="inline">1</math>. <math display="block">\mathrm{F1Score} = F_1 = 2*\frac{Precision*Recall}{Precision+Recall}.</math>

<div class="center">

<div id="tab:use_metrics">

{| class="wikitable"
|+ The usage of the different metrics.
|-
! style="text-align: left;"| Metric
! style="text-align: center;"| When to use ?
|-
| style="text-align: left;"| Accuracy
| style="text-align: center;"| In case of classification problem
|-
| style="text-align: left;"|
| style="text-align: center;"| with balanced classes.
|-
| style="text-align: left;"| Precision
| style="text-align: center;"| When it is important to be sure about the
|-
| style="text-align: left;"|
| style="text-align: center;"| positive prediction to avoid any negative
|-
| style="text-align: left;"|
| style="text-align: center;"| consequences, like e.g in case of decrease
|-
| style="text-align: left;"|
| style="text-align: center;"| of credit limit to avoid customer dissatisfaction.
|-
| style="text-align: left;"| Recall
| style="text-align: center;"| When it is important to capture positive even
|-
| style="text-align: left;"|
| style="text-align: center;"| with low probability, like e.g. to predict
|-
| style="text-align: left;"|
| style="text-align: center;"| whether a person has illness or not.
|-
| style="text-align: left;"| Sensitivity
| style="text-align: center;"| If the question of interest is TPR,
|-
| style="text-align: left;"|
| style="text-align: center;"| i.e. the proportion of the captured positives.
|-
| style="text-align: left;"| Specificity
| style="text-align: center;"| If the question of interest is TNR,
|-
| style="text-align: left;"|
| style="text-align: center;"| i.e. the proportion of the captured negatives.
|-
| style="text-align: left;"| F1Score
| style="text-align: center;"| When both Precision and Recall are important.
|-
| style="text-align: left;"| weighted F1 metric
| style="text-align: center;"| When importance of Precision and Recall
|-
| style="text-align: left;"|
| style="text-align: center;"| against each other can be characterized
|-
| style="text-align: left;"|
| style="text-align: center;"| by weights explicitly .
|-
| style="text-align: left;"| ROC curve
| style="text-align: center;"| It is used for determining probability threshold
|-
| style="text-align: left;"|
| style="text-align: center;"| for deciding the output class of the task,
|-
| style="text-align: left;"|
| style="text-align: center;"| see Figure x.
|-
| style="text-align: left;"| AUC ROC
| style="text-align: center;"| It is used to determine how well the positive class
|-
| style="text-align: left;"|
| style="text-align: center;"| is separated from the negative class.
|}

</div>

</div>
The weighted F1 metric is a refined version of F1Score, in which Precision and Recall can have different weights. <math display="block">\mathrm{ weighted F1} = F_{\beta} = (1+ \beta^2)*\frac{Precision*Recall}{\beta^2 *Precision+Recall},</math>

where Recall has weight <math display="inline">1</math> and <math display="inline">\beta^2</math> is the weight of Precision.

In a multi-class setting the metrics precision and recall, sensitivity and specificity as well as F1Score and weighted F1 metric are calculated first for each class individually and then averaged. This way they quantify the overall classification performance.

The metrics ROC curve and AUC ROC are defined for binary classification task. The Receiver Operating Characteristic (ROC) curve is the True Positive Rate (=Sensitivity) as a function of the False Positive Rate (= <math display="inline">1-Specificity</math>). The Area Under Curve ROC is called AUC ROC. It indicates how well the positive class is separated from the negative class.

The usage of the different metrics are summarized in the Table [[#tab:use_metrics|9]].

<div id="fig:ill_ROC_curve" class="figure">

[[File:./figs/ROC_curve.pdf]]

</div>
Depending on the use case strict, optimal or lenient (= moderate or high) probability threshold can be selected on the ROC curve. This is illustrated in Figure [[#fig:ill_ROC_curve|17]].

<span id="evaluation-metrics-for-regression"></span>
== Evaluation metrics for regression ==

The common metrics used for evaluating regression models are are listed here.

* Mean Squared Error (MSE),
* Root Mean Squared Error (RMSE)
* Mean Absolute Error (MAE),
* Mean Absolute Percentage Error (MAPE),
* Coefficient of Determination (COD), R-squared (<math display="inline">R^2</math>),
* modified R-squared,

The Mean Squared Error (MSE) is one of the basic statistic used to evaluate the quality of a regression model. It is the average of the squares of the difference between the real and predicted values, in other words:

<math display="block">\mathrm{MSE} = \frac{1}{K} \sum_{k} (\hat{y}_k - y_k)^2.</math> Less the MSE, better the regression model fits the real values.

A similar measure is the Root Mean Squared Error (RMSE), which is the square root of MSE and thus it is biven as <math display="block">\mathrm{RMSE} = \sqrt{\frac{1}{K} \sum_{k} (\hat{y}_k - y_k)^2}.</math> RMSE is a measure in the same units as the considered variable, which makes it a commonly used measure.

The Mean Absolute Error (MAE) is the average absolute difference between the real and predicted values. <math display="block">\mathrm{MAE} = \frac{1}{K}\sum_{k}|y_k - \hat{y}_k|</math> A nice property of MAE, that it is less likely influenced by extreme values. It is a common measure used in time series analysis as forecast error.

The mean absolute percentage error (MAPE) quantifies the average of the ratio of the average absolute difference between the real and predicted values to the real value as a percentage. Hence the formula of MAPE can be given as <math display="block">\mathrm{MAPE} = 100 \frac{1}{K}\sum_{k}|\frac{y_k - \hat{y}_k}{y_k}|</math> MAPE is is commonly used for evaluating regression models and it is an appropriate metric where the scale of the considered value varies in a broad range. Intuitively it can be interpreted as a kind of relative error. This makes it also suitable to be used as a loss function as an objective in the optimization in regression problems.

The metric Coefficient of Determination (COD) is also referred as R-squared and it is denoted by <math display="inline">R^2</math> or <math display="inline">r^2</math> and pronounced as "R-squared". The coefficient of determination determines the predictable proportion of the variation in the dependent variable, <math display="inline">{\bf y}</math>. Let <math display="inline">\bar{y}</math> denote the mean of the output values, in other words <math display="block">\bar{y} = \frac{1}{K} \sum_{k} y_k).</math>

The coefficient of determination is defined in terms of residual sum of squares, <math display="inline">SS_{\mathrm{res}}</math> and total sum of squares (related to the variance of <math display="inline">{\bf y}</math>), <math display="inline">SS_{\mathrm{tot}}</math>

<math display="block">\begin{aligned}
SS_{\mathrm{res}} &= \sum_{k} ( y_k - \hat{y}_k)^2, \\
SS_{\mathrm{tot}} &= \sum_{k} ( y_k - \bar{y}_k)^2
\end{aligned}</math>

as

<math display="block">R^2 = 1 -\frac{SS_{\mathrm{res}}}{SS_{\mathrm{tot}}}.</math> <math display="inline">R^2</math> as metric quantifies the predictable proportion of the variation in the dependent variable. Its value falls between 0 and 1 with greater values indicating better regression fit.

The metric modified (or adjusted) R-squared is introduced to compensate that <math display="inline">R^2</math> increases when the dimension <math display="inline">{bf y}</math> becomes higher. Denoting the dimension of <math display="inline">{bf y}</math> by <math display="inline">N</math>, the modified R-squared, <math display="inline">\bar{R}^2</math> is defined as

<math display="block">\bar{R}^2 = 1 -\frac{SS_{\mathrm{res}}}{SS_{\mathrm{tot}}}\frac{K-1}{K-N-1} = 1 -(1-R^2)\frac{K-1}{K-N-1}.</math>

<span id="evaluation-metrics-for-kg"></span>
== Evaluation metricS for KG ==

<span id="quality-of-kg"></span>
=== Quality of KG ===

The two most important quality measures of a KG are

* completeness and
* accuracy.

The completeness refers to evaluate the amount of existing triplets in the KG, while accuracy targets to measure the amounts of correct and incorrect triplets in the KG. After KG completion the resulted extraction graph is considered to be not yet a ready KG. Therefore quality measurement of KG is relevant only after KG refinement.

<span id="evaluation-metric-for-kg-refinement"></span>
=== Evaluation metric for KG refinement ===

Usually completeness is measured in recall, precision and F-measure.

The accuracy of the KG, i.e. the amounts of correct and incorrect triplets is evaluated in terms of accuracy and alternatively, or in addition by means of AUC (i.e. the area under the ROC curve).

The accuracy of the KG, <math display="inline">\mu(\mathcal{G})</math> can be also based on the correctness of the individual triplets <math display="inline">t \in \mathcal{T}</math>, which can be assigned by human e.g. based on random sampling. Then <math display="inline">\mu(\mathcal{G})</math> is given by

<math display="block">\mu(\mathcal{G}) = \frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} \mathbbm{1}_{\{t\}},</math> where <math display="inline">\mathbbm{1}_{\{t\}}</math> is the indicator variable indicating the correctness of the individual triplets <math display="inline">t \in \mathcal{T}</math> by <math display="inline">1</math> otherwise taking the value <math display="inline">0</math>.

In case of human judging, the usual evaluation metric is accuracy or precision together with the total number of judged triplets and errors found.

<span id="evaluation-metric-for-link-prediction-with-kg-embeddings"></span>
=== Evaluation metric for link prediction with KG embeddings ===

The used evaluation metric for link prediction with KG embeddings depends on the subtask of link prediction, see in Table [[#tab:Eval_metr_subt_LP|10]].

<div class="center">

<div id="tab:Eval_metr_subt_LP">

{| class="wikitable"
|+ Evaluation metric for subtasks of link prediction
|-
! style="text-align: left;"| <span>'''Subtask'''</span>
! style="text-align: center;"| <span>'''Evaluation metrics'''</span>
|-
| style="text-align: left;"| Entity prediction
| style="text-align: center;"| rank based measures
|-
| style="text-align: left;"| Entity type prediction
| style="text-align: center;"| Macro-<math display="inline">F_1</math> and Micro-<math display="inline">F_1</math>
|-
| style="text-align: left;"| Triple classification
| style="text-align: center;"| accuracy
|}

</div>

</div>
The rank based evaluation metrics for entity prediction include

* Mean Reciprocal Rank (MRR) and
* Hits@K.

Mean reciprocal rank is the average of the reciprocal ranks of the correct entities: <math display="block">MRR = \frac{1}{|\mathcal{Q}_c|}\sum_{q \in \mathcal{Q}_c} \frac{1}{R(q)},</math> where <math display="inline">\mathcal{Q}_c</math> is set of correct entries and <math display="inline">R(q)</math> is the rank of the entry q. Larger the MRR, better the entity prediction and hence also the used KG embedding model.

Hits@k is the proportion of the correct entities in the best k predictions: <math display="block">Hits@k = \frac{|q \in \mathcal{Q}_c|: R(q) < k}{|\mathcal{Q}_c|}.</math> Larger the Hits@k better the entity prediction and hence also the used KG embedding model.

Metrics in Data Science - Versionsgeschichte

SAFFER Zsolt am 13. März 2025 um 21:14 Uhr

SAFFER Zsolt am 11. März 2025 um 22:21 Uhr