Creating and Analyzing Multiclass Confusion Matrix

Creating a multiclass confusion matrix is a crucial step in evaluating the performance of a multiclass classification model. This matrix provides a clear and concise way to visualize the number of correct and incorrect predictions made by the model.

A multiclass confusion matrix is a table that displays the number of true positives, false positives, true negatives, and false negatives for each class in the dataset. Each row represents the predicted class, and each column represents the actual class.

To create a multiclass confusion matrix, you can use the confusion_matrix function in Python's scikit-learn library, which takes the true labels and predicted labels as input. This function returns a 2D array representing the confusion matrix.

The diagonal elements of the confusion matrix represent the number of correct predictions, while the off-diagonal elements represent the number of incorrect predictions.

A multiclass confusion matrix is a tool used to evaluate the performance of a multiclass classification model. It compares actual data values with predicted data values, making it easy to see if any mislabeling has occurred.

In a multiclass confusion matrix, you'll find an overview of every class found for the selected target, which is exactly what you need to assess how well your model is performing.

The confusion matrix is available for multiclass problems, and it's accessible after building your models and selecting the Confusion Matrix tab from the Evaluate division. This tab displays two confusion matrix tables for each multiclass model: the Multiclass Confusion Matrix and the Selected Class Confusion Matrix.

The Multiclass Confusion Matrix provides an overview of every class found for the selected target, while the Selected Class Confusion Matrix analyzes a specific class. From these comparisons, you can determine how well DataRobot models are performing.

Here's a breakdown of the components available in the Confusion Matrix tab:

By using a multiclass confusion matrix, you can identify areas where your model needs improvement and make data-driven decisions to optimize its performance.

To compute the confusion matrix for multiclass tasks, we need to understand its components. The confusion matrix is a NxN matrix where C_{i, i} represents the number of true positives for class i.

The confusion matrix also represents the number of false negatives for class i, which is the sum of the remaining cells in the matrix. This sum is calculated as \(\sum_{j=1, j

eq i}^N C_{i, j}\).

The number of false positives for class i is represented by \(\sum_{j=1, j

eq i}^N C_{j, i}\).

To build a multiclass confusion matrix, we need to provide the predictions and true labels as tensors. The predictions tensor should have the shape (N,...) if it's an int tensor, or (N,C,..) if it's a float tensor.

The true labels tensor should have the shape (N,...). If the predictions tensor is a floating point tensor, we apply torch.argmax along the C dimension to automatically convert probabilities/logits into an int tensor.

The number of classes is an essential parameter for building a multiclass confusion matrix. It's an integer that specifies the number of classes.

A multiclass confusion matrix is a [num_classes,num_classes] tensor that can be normalized in different ways. The normalization mode can be 'true', 'pred', 'all', or 'none'.

Micro-averaging is a good choice when you don't care about dataset imbalance and just want to see overall performance.

You should use macro-averaging to assess performance on a per-class basis and gain insights into the model's behavior across different categories.

A multiclass confusion matrix is a powerful tool for evaluating the performance of a classification model. It's a NxN matrix where each cell represents the number of true positives, false negatives, and false positives for a specific class.

Precision and recall are two popular metrics used to evaluate the performance of a classification model. Precision measures how many of the items predicted as positive were actually positive, while recall measures how many of the actual positive items were correctly identified. The formula for precision is TP / (TP + FP), where TP is the number of true positives and FP is the number of false positives. The formula for recall is TP / (TP + FN), where FN is the number of false negatives.

Here are the formulas for precision and recall in a concise table:

Micro-averaging and macro-averaging are two ways to calculate precision and recall for an entire model. Micro-averaging calculates the overall performance of the model across all classes, while macro-averaging calculates precision and recall for each class independently and then averages these values.

The Multiclass Confusion Matrix has three mode options: Global, Actual, and Predicted. These modes provide detailed information about each class within the target column.

The Global mode offers F1 Score, Recall, and Precision metrics for each selected class.

In the Actual mode, you'll find details about the Recall score, as well as a partial list of classes that the model confused with the selected class. Clicking Full List opens the Feature Misclassification popup, which lists scores for all classes.

The Predicted mode provides details about the Precision score, or how often the model accurately predicted the selected class. Clicking Full List again opens the Feature Misclassification popup, this time listing Precision scores for all confused classes.

Here's a quick rundown of the modes:

Classification Metrics are a crucial aspect of evaluating the performance of a multiclass classification model. Precision and Recall are two popular metrics in classification.

Precision measures how many of the items our model predicted as positive were actually positive. It's calculated by dividing the true positives by the sum of true positives and false positives.

Recall, on the other hand, measures how many of the actual positive items our model correctly identified. The value of recall is determined by dividing the true positives by the sum of true positives and false negatives.

There are two ways to calculate the average Precision and Recall of our model's entire predictions: Micro-averaging and Macro-averaging. Micro-averaging is like taking the overall performance of our model across all classes, while Macro-averaging calculates precision and recall for each class independently and then averages these values.

Precision is a useful metric in cases where False Positive is a higher concern than False Negatives, such as in music or video recommendation systems.

Recall is a useful metric in cases where False Negative trumps False Positive, such as in medical cases where it doesn't matter whether we raise a false alarm, but the actual positive cases should not go undetected.

The F1-score is a harmonic mean of Precision and Recall, giving a combined idea about these two metrics. It is maximum when Precision is equal to Recall.

Here's a summary of the four quadrants of the Selected Class Confusion Matrix:

In the context of multiclass confusion matrix metrics, selecting the best match across all classes is crucial for capturing errors that occur when the model confuses one class for another.

The multiclass confusion matrix (MCM) is designed to handle this by considering predictions from all classes, not just within the same class. This allows the MCM to capture errors that might otherwise be overlooked.

The MCM defines 4 types of predictions: Mispredicted, Ghost Prediction, True Positive, and True Negative. Mispredicted and Ghost Prediction are particularly relevant when considering the best match across all classes.

A Mispredicted prediction occurs when the model incorrectly predicts a class that is different from the true label. This would have been graded as a False Positive.

A Ghost Prediction is an incorrect prediction that is not matched with any annotation, also graded as a False Positive.

The MCM is a [num_classes,num_classes] tensor, where the number of classes is specified by the num_classes parameter.

The Multiclass Confusion Matrix (MCM) offers a more granular view into your model errors, including observing how errors are distributed across class combinations.

With the MCM, you can gain a deeper understanding into why and where your model is failing. For instance, a high number of mispredictions between two classes can indicate a poor class definition.

A high number of undetected objects could mean a large number of outliers and edge cases, such as cars of an unusual color or old models of cars that are not frequently-present in the dataset.

The MCM helps you identify undetected objects, ghost predictions, and mispredictions in a more intuitive way than simply using TP, FP, and FN.

Here are some common issues that the MCM can help you identify:

When choosing data for your Multiclass Confusion Matrix (MCM), you have various options depending on your project type. For non time-aware projects, data is sourced from the validation, cross-validation, or holdout partitions.

You can select from different data subsets in the Data Selection dropdown, which changes the display to reflect the chosen subset of your project's historical data. The option you choose is crucial in determining the performance of your model.

Here are the specific data sources you can choose from for non time-aware projects:

For time-aware projects, you can select from individual backtests, all backtests, or holdout (if unlocked). This allows you to evaluate your model's performance over time.

You can also add an external test dataset to help evaluate your model's performance, providing a more accurate picture of its capabilities.

The Multiclass Confusion Matrix (MCM) offers a more granular view into your model errors, allowing you to observe how errors are distributed across class combinations.

This level of detail is invaluable in gaining a deeper understanding into why and where your model is failing. For instance, a high number of mispredictions between two classes can indicate a poor class definition.

A high number of undetected objects could mean a large number of outliers and edge cases, such as cars of an unusual color or old models of cars that are not frequently-present in the dataset.

A high number of ghost predictions could indicate the need to set a higher confidence threshold for the model or a low representativity of certain cases in your data.

The MCM helps you quickly spot undetected, ghost predictions, and mispredicted instances, making it impossible to conduct on aggregate metrics, such as mAP or mAR.

Here's a breakdown of the different types of errors you can identify with the MCM:

To implement a multiclass confusion matrix, you'll need to use a classification algorithm that can handle multiple classes.

The accuracy score of a model can be misleading, as it only considers the overall correct predictions, not the individual class performance.

For a multiclass classification problem, you can use the one-vs-all approach, where you train a separate model for each class to predict the probability of the target class.

This approach can be computationally expensive and may not be feasible for large datasets.

You can use libraries like scikit-learn in Python to implement a multiclass confusion matrix and classification algorithms.

Displaying your Multiclass Confusion matrix just got a whole lot easier with the gear icon menu. This menu allows you to customize the matrix to your liking.

You can set the axis for the Actual values display to either rows or columns using the Orientation of Actuals option. This is a simple but effective way to tailor the matrix to your specific needs.

Sorting and ordering options are also available. You can sort the matrix by actual or predicted frequency, alphabetically, or by F1 Score using the Sort by option. This helps you quickly identify trends and patterns in your data.

The Order option allows you to choose whether the matrix is displayed in ascending or descending order. This is especially useful when you want to view the lowest or highest values in the matrix.

For example, to view the lowest Predicted Frequency values, select the Predicted Frequency and Ascending order options. This will display those values at the top of the matrix, making it easy to identify and analyze them.

Scikit-learn in Python is a powerful tool for machine learning tasks. It has two great functions: confusion_matrix() and classification_report().

The confusion_matrix() function returns the values of the Confusion matrix, with the rows as Actual values and the columns as Predicted values. This output is slightly different from what we have studied so far.

The classification_report() function outputs precision, recall, and f1-score for each target class. It also includes some extra values: micro avg, macro avg, and weighted avg.

Here's a breakdown of these extra values:

Once you've calculated your confusion matrix, you're ready to decipher it. The matrix is a 3 x 3 grid, with each class represented by a row and a column.

To calculate the true positive, true negative, false positive, and false negative for each class, simply add the cell values. This is done by looking at the cell where the row and column intersect.

A true positive is represented by the cell where the row and column are the same, and the value is the number of correct predictions.