guides/yolo-performance-metrics/

[giscus[bot]]

guides/yolo-performance-metrics/

A comprehensive guide on various performance metrics related to YOLOv8, their significance, and how to interpret them.

https://docs.ultralytics.com/guides/yolo-performance-metrics/

[HaldunMatar]

We wish there were illustrative charts for each scale.

[pderrenger]

@HaldunMatar thank you for your suggestion! 🌟 We're always looking to improve our documentation and provide more value to our users. Adding illustrative charts for each scale is a great idea to enhance understanding. While we work on incorporating this into our documentation, you might find our Performance Metrics Deep Dive helpful. It covers various metrics in detail, which could complement your understanding as we update our resources. Stay tuned for updates, and feel free to share any more feedback or ideas you have!

[StevanCakic]

Maybe I have a stupid question because I can't find direct answer, but here it is :)

The metric mAP50-95 observed during the training phase refers to the mean average precision on the train dataset, or on validation dataset? It is clear for metrics like loss because we see both training (train/box_loss) and validation (val/box_loss) losses reported, but there might be some uncertainty regarding the mAP metric.
If it represents mAP on the training dataset, is it possible to have mAP calculated for both the training and validation datasets concurrently during training, or if a separate validation mode is required with model.val()

[pderrenger]

@StevanCakic discussion 📊 Great question! The mAP50-95 metric you see during the training phase actually refers to the performance on the validation dataset, not the training dataset. This distinction is significant because evaluating on the validation set provides an unbiased measure of how well the model is expected to perform on unseen data. 👍

If you're interested in calculating mAP for both the training and validation datasets concurrently, you typically would focus on the validation set for performance evaluation during training. However, if you specifically want to assess overfitting or analyze the model's performance on the training set, you can separately run a validation using your training set by specifying it with the data argument in model.val(data='your_train_dataset.yaml').

Here's a quick example for the validation mode:

from ultralytics import YOLO

# Load your model
model = YOLO('path/to/your/model.pt')  # replace with your model path

# Validate on custom dataset
results = model.val(data='your_custom_dataset.yaml')  # Ensure your YAML points to your desired dataset

Hope this helps clear up the confusion! If you have more questions or need further clarifications, feel free to ask. 🌟

[lungger]

Hello, I've noticed that in the precision curve, beyond the maximum confidence predicted by the model, precision is set to 1. I'm curious as to why it is not set to 0, as it seems more logical to me that when the threshold is too high, the model does not make any predictions, hence setting it to 0 would be more reasonable. Is there any literature discussing the definition of the precision curve? Thank you.

[glenn-jocher]

@lungger hello! Great question! 🌟 Precision is defined as the number of correct positive results divided by the number of positive results predicted by the classifier. In the context you've described, when the confidence threshold is beyond the maximum predicted by the model and leads to zero predictions, setting precision to 1 instead of 0 can seem counterintuitive.

However, this approach prevents the precision from unfairly penalizing the model at extremely high confidence thresholds where no predictions are made. The idea is that if we make no predictions (and presumably, there are no objects to detect), we should not penalize the model's precision. This is a convention used in some implementations to ensure the curve starts from a "perfect precision" point when no false positives are introduced due to an overly strict threshold. It's more about the convention used in the calculation rather than a strict rule.

For literature discussing precision and other performance metrics, the COCO detection challenge and Pascal VOC are good starting points, as they are foundational to many object detection benchmarks and their metric definitions.

If you're working with a specific dataset or benchmark, it might be worth exploring their documentation or associated publications for further details on how they define and utilize precision curves.

Here's a simplistic graphical way to think about it:

# Assuming an overly simplified scenario
thresholds = [0.5, 0.9, 1.0]
predictions = [1, 0] # At th=0.5, we predict correctly; at th=0.9, no predictions
precisions = [1, 1] # Precision stays 1 as no false positives are introduced

# This visualization is over-simplified but illustrates why precision might not drop to 0 immediately.

Hope that sheds some light on the reasoning! If you have more queries or need clarification on other aspects, feel free to ask.

[Wang-taoshuo]

The train/dfl_loss is also a training result chart that was generated. What is the meaning of this chart?

[pderrenger]

The dfl_loss (Distribution Focal Loss) chart you're inquiring about shows the performance related to the Distribution Focal Loss over the training period. This loss function aims to enhance the model's accuracy, especially in complex object detection scenarios by focusing more on hard-to-detect objects. Essentially, it adjusts how the model weights errors differently depending on their difficulty, helping to improve overall performance. This chart helps you monitor how well the model is learning to manage these challenging cases over time. 📊

[Chinmoy-Nath]

does it provide plotting for accuracy, sensitivity and specificity ?

[glenn-jocher]

Yes, YOLOv8 provides extensive performance metrics including precision and recall which can be used to derive sensitivity (recall) and specificity. However, accuracy is directly provided, but sensitivity and specificity require a bit of calculation. Here's a brief on how you can interpret these from YOLOv8's output:

• Precision (P): Directly reported per class and can be used as is.
• Recall (R) or Sensitivity: Also directly reported per class. This is your sensitivity.
• Specificity: While not directly reported, specificity depends on the true negatives (TN), which isn't directly provided but can be inferred from the confusion matrix.

To calculate specificity, you'd typically need to know the number of true negatives (TN), which isn't a direct output for many detection models, including YOLO. However, you can get a good understanding of the model's performance on false positives and true negatives by looking at the Precision-Recall (PR) curve and the confusion matrix included in the validation results. These can give insights into how well the model is distinguishing between classes (including background as a class for negatives).

Keep in mind, for object detection, where the focus is often on identifying objects rather than classifying entire images, these metrics might need to be interpreted with the specific context of your application in mind.

For numerical calculations of specificity from YOLOv8 metrics or any custom needs, it might require a bit of post-processing of the results. If you're looking for a code snippet to plot these metrics, the documentation and output files from model.val() provide a good starting point, alongside tools like matplotlib for custom plotting.

Hope this answers your question! 😊

[DanyelR3199]

Anyone have an idea how i get from the results.csv-file a PR and a F1-Score plot? (YOLOv8)

[glenn-jocher]

Hello! To generate Precision-Recall (PR) and F1-Score plots from your results.csv file using YOLOv8, you can leverage Python libraries like Matplotlib for plotting. Here’s a quick example to help you visualize these metrics:

import pandas as pd
import matplotlib.pyplot as plt
from sklearn.metrics import precision_recall_curve, f1_score

# Load your results
data = pd.read_csv('results.csv')
y_true = data['true_label'].values  # Your actual labels
y_scores = data['model_scores'].values  # Your model's scores

# Calculate Precision-Recall values
precision, recall, _ = precision_recall_curve(y_true, y_scores)

# Calculate F1-Score
f1 = f1_score(y_true, (y_scores > 0.5).astype(int))  # Threshold = 0.5

# Plotting
plt.figure(figsize=(10, 5))

# Subplot 1: PR Curve
plt.subplot(1, 2, 1)
plt.plot(recall, precision, marker='.')
plt.title('Precision-Recall curve')
plt.xlabel('Recall')
plt.ylabel('Precision')

# Subplot 2: F1 Score
plt.subplot(1, 2, 2)
plt.bar(['F1 Score'], [f1])
plt.ylim(0, 1)
plt.ylabel('F1 Score')

plt.tight_layout()
plt.show()

This script first calculates the precision and recall from the scores and then plots them alongside a bar chart for the F1-Score. Adjust the path to your results.csv file accordingly, and ensure it contains the columns true_label and model_scores. Happy plotting! 😊📊

[GenieV19]

Firstly, thank you so much for providing so much comprehensive documentation, its is honestly so so helpful.
I have a question on how to change the weighting of the different metrics. I would like to reduce the importance of exactly predicting the correct coordinates for bounding boxes, and instead prioritise correct classification of my objects. Please could you point me in the right direction with this? Thank you so much in advance :)

[pderrenger]

Hi there! I'm glad you find the documentation helpful! 👍 To adjust the importance of different metrics like reducing the weight on exact bounding box coordinates and prioritizing object classification, you can modify the loss weights in your YOLOv8 model configuration.

Here's a basic outline for customizing these weights:

1. Locate your model's YAML file (e.g., yolov8n.yaml).
2. Adjust the loss weights:
   • Decrease box weight to reduce the emphasis on bounding box coordinates.
   • Increase cls (classification) weight to prioritize correct classification.

Here's an example snippet from a YOLOv8 YAML configuration file:

# Example modifying weights in yolov8n.yaml
nc: 80  # number of classes
depth_multiple: 0.33  # model depth multiple
width_multiple: 0.50  # layer channel multiple

# Loss weights (box, obj, cls)
box: 0.5  # Lower value reduces emphasis on box coordinate accuracy
obj: 1.0
cls: 2.0  # Higher value increases emphasis on classification accuracy

Adjust these weights according to your specific needs. The changes will direct the training to focus more on classification accuracy than the precise localization of bounding boxes.

For more guidance on how to apply these changes, you might check out the sections on model configuration and training adjustments in our documentation. Let me know if you need further assistance! 😊

[GuenKainto]

Hi, I have a question as to why when confidence increases, precision increases but recall decreases. Can I understand confident here like putting conf=0.2 to reduce reliable results below 0.2?
And one more thing I learned is that the Confusion Matrix is calculated with iou = 0.25 while defaulting when val iou = 0.6. Does that make the confusion matrix any more effective? As I understand it, it will reduce duplicate results with iou > 0.25 with NMS.

[pderrenger]

@GuenKainto hello! Great questions, let's dive into them! 😊

1. Precision and Recall Trade-off: Increasing the confidence threshold generally increases precision but can decrease recall. This happens because a higher threshold means only the predictions with higher certainty are considered, reducing false positives (increasing precision) but potentially missing true positive detections (decreasing recall). Setting conf=0.2 means you are considering detections with a confidence score above 0.2, indeed filtering out less reliable detections.

2. Confusion Matrix with Different IoU: Using an IoU of 0.25 for the confusion matrix versus the default IoU of 0.6 for other evaluations can indeed affect how the confusion matrix appears. A lower IoU threshold might count overlaps that are not considered in more stringent IoU settings, helping to reduce duplicates due to the application of Non-Max Suppression (NMS). This approach confirms detections more inclusively, which may offer a broader view of how many objects the model detects, though it might lessen the precision in some cases.

If you are aiming to refine how your model handles the balance between detecting as many relevant objects as possible versus ensuring the detections are correct, you might consider adjusting these parameters based on what fits best for your specific scenario and dataset. Test with different configurations to find the balance that meets your objectives.

Hope this clears your doubts! Let me know if you have more questions. 🌟

[GuenKainto]

Thank you for this information ^.^

[pderrenger]

Hey, thanks for reaching out and for the kind words! 😊

To focus more on classification accuracy over precise bounding box coordinates, you can adjust the loss weights in your model configuration. Generally, these changes involve reducing the box weight and increasing the cls weight. Here’s how you can do it:

1. Locate the .yaml configuration file for your model setup.
2. Modify the weights:

   # Example from a YOLOv8 YAML configuration
   box: 0.5  # lower this value
   cls: 2.0  # increase this value

This directs the model's learning to emphasize correct classification over bounding box precision.

Just remember to balance these adjustments to avoid potential trade-offs in overall model accuracy!

If you need further details, this kind of adjustment is discussed in the "Performance Metrics Deep Dive" section in our documentation. Keep experimenting to find the optimal settings that meet your specific needs! 🚀

[lishihong-1]

I want to know when I use the val command, does the output of mAP50 for a single class in fact refer to AP50 for a single class? Because mAP50 theoretically evaluate all classes.

[pderrenger]

Yes, you are correct! When you use the val command in YOLOv8 and see the output for mAP50 for a single class, it indeed refers to the AP50 for that specific class. In the context of mAP50, the metric is averaged over all classes, but when it's reported for a single class, it represents just the AP50, detailing how well the model predicts that particular class at an IoU threshold of 0.50. This can be especially useful for understanding performance across different classes in your dataset. If you have any more questions or need further clarification, feel free to ask! 🌟

[lishihong-1]

I noticed that when I used the val command, the data printed contained a Precision and a Recall value for all classes. I want to know that what's the meaning of the two values and how to calculate them in the official code? For the reason that the P and R value are usually used for a single class.

[lishihong-1]

the printed data like this:

[pderrenger]

Hello! In the YOLOv8 validation output, the Precision (P) and Recall (R) values you see for all classes are aggregated metrics. They provide an overall view of the model's performance across all classes, not just for a single class.

• Precision for all classes is calculated as the total number of correct positive predictions (true positives) divided by the total number of positive predictions made (true positives + false positives).
• Recall for all classes is the total number of correct positive predictions (true positives) divided by the total number of actual positives (true positives + false negatives).

These metrics help you understand the model's ability to correctly identify objects (precision) and its ability to find all the relevant cases (recall) across all classes. If you need a breakdown by individual class, you can look at the class-wise metrics section of the output.

If you're looking into the code, these metrics are typically calculated during the evaluation phase where predictions are compared against the ground truth data. The specific implementation details can be found in the validation script where metrics are computed using confusion matrices or similar methods.

Hope this helps! If you have more questions or need further clarification, feel free to ask. 😊