UnboundLocalError: local variable 'assign_result' referenced before assignment

[Simeon340703]

I was reproducing the output, but I got UnboundLocalError: local variable 'assign_result' referenced before assignment error. The problem happens after 10 epochs.
Here is the error /.../ .../projects/mmdet3d_plugin/models/dense_heads/uvtr_head.py", line 263, in _get_target_single
sampling_result = self.sampler.sample(assign_result, bbox_pred,
UnboundLocalError: local variable 'assign_result' referenced before assignment.

[yanwei-li]

Hi, it seems some variables are close to inf in the assignment. Could you provide the specific config you are using? Maybe you should lower the learning rate.

[Simeon340703]

Hi, it seems some variables are close to inf in the assignment. Could you provide the specific config you are using? Maybe you should lower the learning rate.

@yanwei-li Thank you. I will try with a lower learning rate. This error should not happen after running for some epochs. So, the problem is not directly variable reference.

[Simeon340703]

@yanwei-li Lowering the learning did not help. I am using the 'projects/configs/uvtr/multi_modality/uvtr_m_v0075_r101_h5.py' configuration'. This is the output before the error generated " 2022-10-24 17:14:06,234 - mmdet - INFO - Epoch [11][1100/7033] lr: 3.690e-05, eta: 2 days, 0:06:58, time: 2.516, data_time: 0.028, memory: 43804, loss_cls: 0.1731, loss_bbox: 0.4040, d0.loss_cls: 0.3295, d0.loss_bbox: 0.4937, d1.loss_cls: 0.3259, d1.loss_bbox: 0.4215, d2.loss_cls: 0.2206, d2.loss_bbox: 0.4173, d3.loss_cls: 0.1774, d3.loss_bbox: 0.4104, d4.loss_cls: 0.1732, d4.loss_bbox: 0.4058, loss: 3.9525, grad_norm: 158.9638
bbox_pred:(tensor(nan, device='cuda:1', grad_fn=), tensor(nan, device='cuda:1', grad_fn=)), cls_score:(tensor(nan, device='cuda:1', grad_fn=), tensor(nan, device='cuda:1', grad_fn=)), gt_bboxes:(tensor(45.8547, device='cuda:1'), tensor(-53.1617, device='cuda:1')), gt_labels:tensor([0, 1, 3, 0, 8, 0, 8, 0, 0, 8, 8, 0, 1, 3, 0, 0, 3, 0, 2, 2, 2, 2, 2, 4,
4, 5, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 9, 9], device='cuda:1'), gt_bboxes_ignore:None". It seems the tensor becomes inf. But, why at this epoch? I did not change any configs.

[JohnFuguiWang]

@Simeon340703 Hi, I met the same issue when running the code. Did you finally solve it?