Unable to reproduce numbers on ImagenetLT from the given GPaco Resnet50 checkpoint

[pradipto111]

We used the code below to load gpaco_r50_imagenetlt.pth.tar onto model and evaluated it on ImagenetLT. We ensured to use the correct moco builder files and appropriate parameters given in this repo. However, the model gives near 0 accuracy on ImagenetLT.

We were able to load the parameters successfully from the checkpoint to the model. We are unable to pinpoint the reason for the reduced accuracy, and seek your help for the same.

        if 'paco' in args.path:

            model = moco.builder.MoCo(
                models.__dict__[args.model],
                args.moco_dim, args.moco_k, args.moco_m, args.moco_t, args.mlp, args.feat_dim, num_classes=1000)
        else:
            model = models.__dict__[args.model](num_classes=args.nb_classes, use_norm=args.use_norm)


        if args.path.startswith('https'):
            checkpoint = torch.hub.load_state_dict_from_url(
                args.path, map_location='cpu', check_hash=True)
        else:
            print("[INFORMATION] Loading teacher model from path ", args.path)
            checkpoint = torch.load(args.path, map_location='cuda:0')


        if 'paco' in args.path:
            
            model.to(device)
            model = torch.nn.parallel.DistributedDataParallel(model, device_ids = [0], find_unused_parameters = True)
            model.load_state_dict(checkpoint['state_dict'])
        else:
            model.load_state_dict(checkpoint['model'] if 'model' in checkpoint.keys() else checkpoint['state_dict'])
        
        model.to(device)
        model.eval()

[jiequancui]

Hi,

Thanks for your interest in our work.
Have you tried our evaluation script (GPaCo/LT/sh/LT)/ImageNetLT_eval_R50.sh)?

Best,

[jiequancui]

Hi,

I have loaded the checkpoint "gpaco_r50_imagenetlt.pth.tar" and evaluated it on ImageNet-LT with our provided evaluation script. It works well and achieves 58.548 top-1 accuracy.

Is it possible that your downloaded checkpoint is corrupted during network transfer?

I'm very glad to discuss questions about our work if needed further.

Best,
Jiequan