CAS测评数值异常

[1339354001]

作者你好，这份工作很有意思。我现在用这个库的代码去测评了Wan-2.1-T2V-1.3B和Wan-2.1-T2V-14B，以及我们基于wan微调的一个模型。其中PAS和CAS和文章中的数值相差较大，其中CAS的测评，我发现1050个视频的数值都稳定在0.499多一点点，不知道是哪里有问题。想知道作者当时在测评的时候有什么值得注意的点吗

[h6kplus]

I also encountered this when doing some experiment, all the output seems to be 49.98~. I found they didn't load the checkpoint they trained (in the hf repo).

I found adding the loading script can give differentiable scores:

model.to(device)
if args.finetune:
    checkpoint = torch.load(args.finetune, map_location='cpu')
    
    # Handle different checkpoint formats
    if 'model' in checkpoint:
        checkpoint_model = checkpoint['model']
    elif 'module' in checkpoint:
        checkpoint_model = checkpoint['module']
    else:
        checkpoint_model = checkpoint
    
    # Load the state dict
    msg = model.load_state_dict(checkpoint_model, strict=False)
    print(f"Loaded checkpoint from {args.finetune}")
    print(msg)

I'm not sure if this scroe is the final CAS score. Can the authors verify that? @Ran0618 @chenzhu01

[Thorin215]

I also encountered this. commonsense_adherence_score.py lose torch.load(args.finetune) and model.load_state_dict(...).