Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

low results on nusenes-mini #26

Open
zizo1111 opened this issue Apr 25, 2023 · 3 comments
Open

low results on nusenes-mini #26

zizo1111 opened this issue Apr 25, 2023 · 3 comments

Comments

@zizo1111
Copy link

zizo1111 commented Apr 25, 2023 •
edited

Hi,
I was trying to evaluate the pretrained models on the nuscenes mini data, but I am getting very bad results which is unexpected.
I have mmdet3d v1.0.0.rc4 installed

Do you maybe know what the problem is?

Thanks :)

here is the log

user@gpu-cluster:~/src/mmdet$ python3 extra_tools/test.py configs/uvtr/multi_modality/uvtr_m_v0075_r101_h5.py models/uvtr/uvtr_m_v0075_r101_h5.pth --eval=bbox --out mini_test.pkl
/home/user/.local/lib/python3.8/site-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  warnings.warn(
/home/user/.local/lib/python3.8/site-packages/mmdet/utils/setup_env.py:38: UserWarning: Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
  warnings.warn(
/home/user/.local/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
load checkpoint from local path: models/uvtr/uvtr_m_v0075_r101_h5.pth
2023-04-25 16:42:17,693 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.0.conv2 is upgraded to version 2.
2023-04-25 16:42:17,695 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.1.conv2 is upgraded to version 2.
2023-04-25 16:42:17,697 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.2.conv2 is upgraded to version 2.
2023-04-25 16:42:17,699 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.3.conv2 is upgraded to version 2.
2023-04-25 16:42:17,701 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.4.conv2 is upgraded to version 2.
2023-04-25 16:42:17,703 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.5.conv2 is upgraded to version 2.
2023-04-25 16:42:17,704 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.6.conv2 is upgraded to version 2.
2023-04-25 16:42:17,706 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.7.conv2 is upgraded to version 2.
2023-04-25 16:42:17,708 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.8.conv2 is upgraded to version 2.
2023-04-25 16:42:17,710 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.9.conv2 is upgraded to version 2.
2023-04-25 16:42:17,712 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.10.conv2 is upgraded to version 2.
2023-04-25 16:42:17,714 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.11.conv2 is upgraded to version 2.
2023-04-25 16:42:17,716 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.12.conv2 is upgraded to version 2.
2023-04-25 16:42:17,717 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.13.conv2 is upgraded to version 2.
2023-04-25 16:42:17,719 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.14.conv2 is upgraded to version 2.
2023-04-25 16:42:17,721 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.15.conv2 is upgraded to version 2.
2023-04-25 16:42:17,723 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.16.conv2 is upgraded to version 2.
2023-04-25 16:42:17,725 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.17.conv2 is upgraded to version 2.
2023-04-25 16:42:17,728 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.18.conv2 is upgraded to version 2.
2023-04-25 16:42:17,730 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.19.conv2 is upgraded to version 2.
2023-04-25 16:42:17,732 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.20.conv2 is upgraded to version 2.
2023-04-25 16:42:17,733 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.21.conv2 is upgraded to version 2.
2023-04-25 16:42:17,735 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.22.conv2 is upgraded to version 2.
2023-04-25 16:42:17,737 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.0.conv2 is upgraded to version 2.
2023-04-25 16:42:17,741 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.1.conv2 is upgraded to version 2.
2023-04-25 16:42:17,744 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.2.conv2 is upgraded to version 2.
[                                                  ] 0/81, elapsed: 0s, ETA:/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:4227: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  warnings.warn(
/src/mmdet3d/core/bbox/coders/nms_free_coder.py:67: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  self.post_center_range = torch.tensor(
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 81/81, 2.6 task/s, elapsed: 32s, ETA:     0s
writing results to mini_test.pkl

Formating bboxes of pts_bbox
Start to convert detection format...
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 81/81, 23.1 task/s, elapsed: 3s, ETA:     0s
Results writes to /tmp/tmps9zh56ms/results/pts_bbox/results_nusc.json
Evaluating bboxes of pts_bbox
mAP: 0.1188                                                                                                                                             
mATE: 0.8707
mASE: 0.5113
mAOE: 0.6999
mAVE: 1.0449
mAAE: 0.4827
NDS: 0.2029
Eval time: 4.8s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.204   0.872   0.186   0.334   0.212   0.065
truck   0.075   0.673   0.290   0.355   0.216   0.118
bus     0.017   1.250   0.216   0.406   3.521   1.000
trailer 0.000   1.000   1.000   1.000   1.000   1.000
construction_vehicle    0.000   1.000   1.000   1.000   1.000   1.000
pedestrian      0.319   0.582   0.308   0.673   0.572   0.367
motorcycle      0.212   0.869   0.317   0.922   0.070   0.011
bicycle 0.095   0.636   0.372   0.609   1.767   0.301
traffic_cone    0.265   0.825   0.425   nan     nan     nan
barrier 0.000   1.000   1.000   1.000   nan     nan
{'pts_bbox_NuScenes/car_AP_dist_0.5': 0.006, 'pts_bbox_NuScenes/car_AP_dist_1.0': 0.1039, 'pts_bbox_NuScenes/car_AP_dist_2.0': 0.2847, 'pts_bbox_NuScenes/car_AP_dist_4.0': 0.4234, 'pts_bbox_NuScenes/car_trans_err': 0.8717, 'pts_bbox_NuScenes/car_scale_err': 0.1856, 'pts_bbox_NuScenes/car_orient_err': 0.3339, 'pts_bbox_NuScenes/car_vel_err': 0.2123, 'pts_bbox_NuScenes/car_attr_err': 0.0646, 'pts_bbox_NuScenes/mATE': 0.8707, 'pts_bbox_NuScenes/mASE': 0.5113, 'pts_bbox_NuScenes/mAOE': 0.6999, 'pts_bbox_NuScenes/mAVE': 1.0449, 'pts_bbox_NuScenes/mAAE': 0.4827, 'pts_bbox_NuScenes/truck_AP_dist_0.5': 0.0037, 'pts_bbox_NuScenes/truck_AP_dist_1.0': 0.0371, 'pts_bbox_NuScenes/truck_AP_dist_2.0': 0.0762, 'pts_bbox_NuScenes/truck_AP_dist_4.0': 0.1825, 'pts_bbox_NuScenes/truck_trans_err': 0.673, 'pts_bbox_NuScenes/truck_scale_err': 0.2899, 'pts_bbox_NuScenes/truck_orient_err': 0.3548, 'pts_bbox_NuScenes/truck_vel_err': 0.2161, 'pts_bbox_NuScenes/truck_attr_err': 0.118, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_trans_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_scale_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_orient_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_vel_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_attr_err': 1.0, 'pts_bbox_NuScenes/bus_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_2.0': 0.0003, 'pts_bbox_NuScenes/bus_AP_dist_4.0': 0.0676, 'pts_bbox_NuScenes/bus_trans_err': 1.2499, 'pts_bbox_NuScenes/bus_scale_err': 0.216, 'pts_bbox_NuScenes/bus_orient_err': 0.4058, 'pts_bbox_NuScenes/bus_vel_err': 3.5214, 'pts_bbox_NuScenes/bus_attr_err': 1.0, 'pts_bbox_NuScenes/trailer_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/trailer_trans_err': 1.0, 'pts_bbox_NuScenes/trailer_scale_err': 1.0, 'pts_bbox_NuScenes/trailer_orient_err': 1.0, 'pts_bbox_NuScenes/trailer_vel_err': 1.0, 'pts_bbox_NuScenes/trailer_attr_err': 1.0, 'pts_bbox_NuScenes/barrier_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/barrier_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/barrier_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/barrier_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/barrier_trans_err': 1.0, 'pts_bbox_NuScenes/barrier_scale_err': 1.0, 'pts_bbox_NuScenes/barrier_orient_err': 1.0, 'pts_bbox_NuScenes/barrier_vel_err': nan, 'pts_bbox_NuScenes/barrier_attr_err': nan, 'pts_bbox_NuScenes/motorcycle_AP_dist_0.5': 0.0019, 'pts_bbox_NuScenes/motorcycle_AP_dist_1.0': 0.1057, 'pts_bbox_NuScenes/motorcycle_AP_dist_2.0': 0.2772, 'pts_bbox_NuScenes/motorcycle_AP_dist_4.0': 0.4626, 'pts_bbox_NuScenes/motorcycle_trans_err': 0.8689, 'pts_bbox_NuScenes/motorcycle_scale_err': 0.3167, 'pts_bbox_NuScenes/motorcycle_orient_err': 0.9223, 'pts_bbox_NuScenes/motorcycle_vel_err': 0.0699, 'pts_bbox_NuScenes/motorcycle_attr_err': 0.0111, 'pts_bbox_NuScenes/bicycle_AP_dist_0.5': 0.0086, 'pts_bbox_NuScenes/bicycle_AP_dist_1.0': 0.0796, 'pts_bbox_NuScenes/bicycle_AP_dist_2.0': 0.1092, 'pts_bbox_NuScenes/bicycle_AP_dist_4.0': 0.1837, 'pts_bbox_NuScenes/bicycle_trans_err': 0.636, 'pts_bbox_NuScenes/bicycle_scale_err': 0.3723, 'pts_bbox_NuScenes/bicycle_orient_err': 0.6093, 'pts_bbox_NuScenes/bicycle_vel_err': 1.7671, 'pts_bbox_NuScenes/bicycle_attr_err': 0.3007, 'pts_bbox_NuScenes/pedestrian_AP_dist_0.5': 0.1008, 'pts_bbox_NuScenes/pedestrian_AP_dist_1.0': 0.2702, 'pts_bbox_NuScenes/pedestrian_AP_dist_2.0': 0.3986, 'pts_bbox_NuScenes/pedestrian_AP_dist_4.0': 0.5082, 'pts_bbox_NuScenes/pedestrian_trans_err': 0.5822, 'pts_bbox_NuScenes/pedestrian_scale_err': 0.3078, 'pts_bbox_NuScenes/pedestrian_orient_err': 0.6728, 'pts_bbox_NuScenes/pedestrian_vel_err': 0.572, 'pts_bbox_NuScenes/pedestrian_attr_err': 0.3673, 'pts_bbox_NuScenes/traffic_cone_AP_dist_0.5': 0.0334, 'pts_bbox_NuScenes/traffic_cone_AP_dist_1.0': 0.2129, 'pts_bbox_NuScenes/traffic_cone_AP_dist_2.0': 0.3517, 'pts_bbox_NuScenes/traffic_cone_AP_dist_4.0': 0.4602, 'pts_bbox_NuScenes/traffic_cone_trans_err': 0.8253, 'pts_bbox_NuScenes/traffic_cone_scale_err': 0.4245, 'pts_bbox_NuScenes/traffic_cone_orient_err': nan, 'pts_bbox_NuScenes/traffic_cone_vel_err': nan, 'pts_bbox_NuScenes/traffic_cone_attr_err': nan, 'pts_bbox_NuScenes/NDS': 0.20291704194649415, 'pts_bbox_NuScenes/mAP': 0.1187504058313846}
@yanwei-li
Copy link
Member

Hi, I never tested the result on nuscenes mini data. Could you reproduce the results on the nuscenes val set?

@zizo1111
Copy link
Author

Hi, I managed to free up enough space to download the whole dataset and I got similar results.

user@gpu-cluster:~/src/mmdet$ python3 extra_tools/test.py configs/uvtr/multi_modality/uvtr_m_v0075_r101_h5.py ckpts/uvtr/uvtr_m_v0075_r101_h5.pth --eval=bbox
/usr/local/lib/python3.8/dist-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
load checkpoint from local path: ckpts/uvtr/uvtr_m_v0075_r101_h5.pth
2023-04-30 12:26:43,335 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.0.conv2 is upgraded to version 2.
2023-04-30 12:26:43,337 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.1.conv2 is upgraded to version 2.
2023-04-30 12:26:43,338 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.2.conv2 is upgraded to version 2.
2023-04-30 12:26:43,339 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.3.conv2 is upgraded to version 2.
2023-04-30 12:26:43,340 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.4.conv2 is upgraded to version 2.
2023-04-30 12:26:43,341 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.5.conv2 is upgraded to version 2.
2023-04-30 12:26:43,342 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.6.conv2 is upgraded to version 2.
2023-04-30 12:26:43,343 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.7.conv2 is upgraded to version 2.
2023-04-30 12:26:43,344 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.8.conv2 is upgraded to version 2.
2023-04-30 12:26:43,345 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.9.conv2 is upgraded to version 2.
2023-04-30 12:26:43,346 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.10.conv2 is upgraded to version 2.
2023-04-30 12:26:43,347 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.11.conv2 is upgraded to version 2.
2023-04-30 12:26:43,349 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.12.conv2 is upgraded to version 2.
2023-04-30 12:26:43,350 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.13.conv2 is upgraded to version 2.
2023-04-30 12:26:43,351 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.14.conv2 is upgraded to version 2.
2023-04-30 12:26:43,352 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.15.conv2 is upgraded to version 2.
2023-04-30 12:26:43,353 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.16.conv2 is upgraded to version 2.
2023-04-30 12:26:43,354 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.17.conv2 is upgraded to version 2.
2023-04-30 12:26:43,355 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.18.conv2 is upgraded to version 2.
2023-04-30 12:26:43,356 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.19.conv2 is upgraded to version 2.
2023-04-30 12:26:43,357 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.20.conv2 is upgraded to version 2.
2023-04-30 12:26:43,358 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.21.conv2 is upgraded to version 2.
2023-04-30 12:26:43,359 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.22.conv2 is upgraded to version 2.
2023-04-30 12:26:43,360 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.0.conv2 is upgraded to version 2.
2023-04-30 12:26:43,362 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.1.conv2 is upgraded to version 2.
2023-04-30 12:26:43,364 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.2.conv2 is upgraded to version 2.
[                                                  ] 0/6019, elapsed: 0s, ETA:/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:4227: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  warnings.warn(
[                                                  ] 1/6019, 0.3 task/s, elapsed: 3s, ETA: 18116s ~/src/mmdet/mmdet3d/core/bbox/coders/nms_free_coder.py:67: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  self.post_center_range = torch.tensor(
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 6019/6019, 2.1 task/s, elapsed: 2800s, ETA:     0s
Formating bboxes of pts_bbox
Start to convert detection format...
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 6019/6019, 40.5 task/s, elapsed: 149s, ETA:     0s
Results writes to /tmp/tmp2zhts1f2/results/pts_bbox/results_nusc.json
Evaluating bboxes of pts_bbox
mAP: 0.1348                                                                                               
mATE: 0.8543
mASE: 0.3209
mAOE: 0.6170
mAVE: 0.8633
mAAE: 0.2407
NDS: 0.2778
Eval time: 83.0s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.142   0.875   0.180   0.249   0.971   0.183
truck   0.049   0.992   0.265   0.324   0.876   0.229
bus     0.017   1.112   0.248   0.555   2.468   0.604
trailer 0.021   1.100   0.283   0.908   0.330   0.135
construction_vehicle    0.002   1.001   0.529   1.144   0.130   0.228
pedestrian      0.295   0.541   0.323   0.683   0.649   0.316
motorcycle      0.177   0.681   0.296   0.629   1.075   0.194
bicycle 0.253   0.663   0.311   0.703   0.408   0.038
traffic_cone    0.225   0.773   0.394   nan     nan     nan
barrier 0.166   0.803   0.380   0.359   nan     nan
{'pts_bbox_NuScenes/car_AP_dist_0.5': 0.0019, 'pts_bbox_NuScenes/car_AP_dist_1.0': 0.0591, 'pts_bbox_NuScenes/car_AP_dist_2.0': 0.191, 'pts_bbox_NuScenes/car_AP_dist_4.0': 0.316, 'pts_bbox_NuScenes/car_trans_err': 0.8745, 'pts_bbox_NuScenes/car_scale_err': 0.1798, 'pts_bbox_NuScenes/car_orient_err': 0.2486, 'pts_bbox_NuScenes/car_vel_err': 0.9711, 'pts_bbox_NuScenes/car_attr_err': 0.1828, 'pts_bbox_NuScenes/mATE': 0.8543, 'pts_bbox_NuScenes/mASE': 0.3209, 'pts_bbox_NuScenes/mAOE': 0.617, 'pts_bbox_NuScenes/mAVE': 0.8633, 'pts_bbox_NuScenes/mAAE': 0.2407, 'pts_bbox_NuScenes/truck_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/truck_AP_dist_1.0': 0.0057, 'pts_bbox_NuScenes/truck_AP_dist_2.0': 0.0585, 'pts_bbox_NuScenes/truck_AP_dist_4.0': 0.1328, 'pts_bbox_NuScenes/truck_trans_err': 0.9924, 'pts_bbox_NuScenes/truck_scale_err': 0.2653, 'pts_bbox_NuScenes/truck_orient_err': 0.3236, 'pts_bbox_NuScenes/truck_vel_err': 0.8758, 'pts_bbox_NuScenes/truck_attr_err': 0.2292, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_2.0': 0.0002, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_4.0': 0.0097, 'pts_bbox_NuScenes/construction_vehicle_trans_err': 1.0013, 'pts_bbox_NuScenes/construction_vehicle_scale_err': 0.5295, 'pts_bbox_NuScenes/construction_vehicle_orient_err': 1.144, 'pts_bbox_NuScenes/construction_vehicle_vel_err': 0.1305, 'pts_bbox_NuScenes/construction_vehicle_attr_err': 0.2276, 'pts_bbox_NuScenes/bus_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_2.0': 0.0076, 'pts_bbox_NuScenes/bus_AP_dist_4.0': 0.0606, 'pts_bbox_NuScenes/bus_trans_err': 1.1117, 'pts_bbox_NuScenes/bus_scale_err': 0.2478, 'pts_bbox_NuScenes/bus_orient_err': 0.5546, 'pts_bbox_NuScenes/bus_vel_err': 2.4679, 'pts_bbox_NuScenes/bus_attr_err': 0.6041, 'pts_bbox_NuScenes/trailer_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_2.0': 0.0116, 'pts_bbox_NuScenes/trailer_AP_dist_4.0': 0.072, 'pts_bbox_NuScenes/trailer_trans_err': 1.1001, 'pts_bbox_NuScenes/trailer_scale_err': 0.283, 'pts_bbox_NuScenes/trailer_orient_err': 0.908, 'pts_bbox_NuScenes/trailer_vel_err': 0.3297, 'pts_bbox_NuScenes/trailer_attr_err': 0.1348, 'pts_bbox_NuScenes/barrier_AP_dist_0.5': 0.011, 'pts_bbox_NuScenes/barrier_AP_dist_1.0': 0.0915, 'pts_bbox_NuScenes/barrier_AP_dist_2.0': 0.2187, 'pts_bbox_NuScenes/barrier_AP_dist_4.0': 0.3434, 'pts_bbox_NuScenes/barrier_trans_err': 0.8034, 'pts_bbox_NuScenes/barrier_scale_err': 0.3798, 'pts_bbox_NuScenes/barrier_orient_err': 0.3595, 'pts_bbox_NuScenes/barrier_vel_err': nan, 'pts_bbox_NuScenes/barrier_attr_err': nan, 'pts_bbox_NuScenes/motorcycle_AP_dist_0.5': 0.0253, 'pts_bbox_NuScenes/motorcycle_AP_dist_1.0': 0.133, 'pts_bbox_NuScenes/motorcycle_AP_dist_2.0': 0.2365, 'pts_bbox_NuScenes/motorcycle_AP_dist_4.0': 0.3129, 'pts_bbox_NuScenes/motorcycle_trans_err': 0.6813, 'pts_bbox_NuScenes/motorcycle_scale_err': 0.2961, 'pts_bbox_NuScenes/motorcycle_orient_err': 0.629, 'pts_bbox_NuScenes/motorcycle_vel_err': 1.0745, 'pts_bbox_NuScenes/motorcycle_attr_err': 0.1937, 'pts_bbox_NuScenes/bicycle_AP_dist_0.5': 0.0458, 'pts_bbox_NuScenes/bicycle_AP_dist_1.0': 0.1917, 'pts_bbox_NuScenes/bicycle_AP_dist_2.0': 0.3496, 'pts_bbox_NuScenes/bicycle_AP_dist_4.0': 0.4252, 'pts_bbox_NuScenes/bicycle_trans_err': 0.6635, 'pts_bbox_NuScenes/bicycle_scale_err': 0.311, 'pts_bbox_NuScenes/bicycle_orient_err': 0.7029, 'pts_bbox_NuScenes/bicycle_vel_err': 0.4081, 'pts_bbox_NuScenes/bicycle_attr_err': 0.0378, 'pts_bbox_NuScenes/pedestrian_AP_dist_0.5': 0.1057, 'pts_bbox_NuScenes/pedestrian_AP_dist_1.0': 0.264, 'pts_bbox_NuScenes/pedestrian_AP_dist_2.0': 0.3596, 'pts_bbox_NuScenes/pedestrian_AP_dist_4.0': 0.4512, 'pts_bbox_NuScenes/pedestrian_trans_err': 0.5412, 'pts_bbox_NuScenes/pedestrian_scale_err': 0.3233, 'pts_bbox_NuScenes/pedestrian_orient_err': 0.6827, 'pts_bbox_NuScenes/pedestrian_vel_err': 0.649, 'pts_bbox_NuScenes/pedestrian_attr_err': 0.3156, 'pts_bbox_NuScenes/traffic_cone_AP_dist_0.5': 0.0235, 'pts_bbox_NuScenes/traffic_cone_AP_dist_1.0': 0.1444, 'pts_bbox_NuScenes/traffic_cone_AP_dist_2.0': 0.2996, 'pts_bbox_NuScenes/traffic_cone_AP_dist_4.0': 0.4316, 'pts_bbox_NuScenes/traffic_cone_trans_err': 0.7734, 'pts_bbox_NuScenes/traffic_cone_scale_err': 0.3937, 'pts_bbox_NuScenes/traffic_cone_orient_err': nan, 'pts_bbox_NuScenes/traffic_cone_vel_err': nan, 'pts_bbox_NuScenes/traffic_cone_attr_err': nan, 'pts_bbox_NuScenes/NDS': 0.27776341220635314, 'pts_bbox_NuScenes/mAP': 0.13476739579111732}

Thanks in advance :)

@yanwei-li
Copy link
Member

Hi, You can try v0.17.3 for the inference. I guess the mismatch is caused by the coordinate change in the new version. Of course, if you train it using v1.0.0rc5, it should achieve the correct results. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yanwei-li @zizo1111