Possible mismatch in RGBT234 blackwomen text annotation and question about pretrained-model reproduction

[yingros]

Hi @IdolLab,

Thank you for releasing RAGTrack and the pretrained models. I am trying to reproduce the RGBT234 results and found a possible issue in the test text annotations.

In the RGBT234 test sequence blackwomen, the provided text annotation seems inconsistent with the actual image content:

• class.txt: A bicycle.
• visible_description.txt: The object appears to be a small bicycle in the distance, possibly parked near the roadside, among a group of other vehicles and trees, blending cleverly with the surroundings due to its size and color.

However, from the original RGBT234 frames, the target appears to be a woman in black holding an umbrella and walking by the roadside, rather than a bicycle. Could you please confirm whether this annotation is correct, or whether the uploaded test text files may contain a mismatch for this sequence?

To check whether this was an isolated case or a more general issue, I first used AI to filter potentially suspicious test sequences, and then manually inspected those candidates. In RGBT234, blackwomen is the only sequence for which I found an obvious annotation mismatch.

I also encountered some difficulty reproducing the reported RGBT234 performance using the released pretrained model and text data. My current observations are:

• Evaluating the released author_results/RGBT234/RAGTrack with the official RGBT evaluation toolkit gives results close to the paper:

  • MPR ≈ 0.938
  • MSR ≈ 0.695

• But when I run inference myself using the released RGBT234 pretrained checkpoint and the provided RGBT234 text annotations, the result is much lower:

  • MPR ≈ 0.815
  • MSR ≈ 0.506

• For example, on the sequence manwithbasketball, my generated result starts to differ from the released author_results from the second frame.

I have checked the evaluation toolkit and the released result files, and the evaluation of author_results can reproduce the paper-level numbers. Therefore, I wonder whether there may be a version mismatch in the released RGBT234 test text annotations, pretrained checkpoint, or testing configuration.

Could you please help confirm:

1. Is the blackwomen text annotation in RGBT234 correct?
2. Are the currently released RGBT234 pretrained model and test text files exactly the same version used to generate the reported author_results?
3. Is there any additional testing detail or configuration needed to reproduce the RGBT234 numbers from the pretrained model?
   If Chinese is more convenient, I can also describe the issue in Chinese.

Thank you very much for your help.
Best regards,
Shirun Yao
Anhui University, China

[IdolLab]

Hi @yingros,

Thank you very much for your detailed feedback.

1. Regarding the blackwomen annotation in RGBT234

You are absolutely right. The text annotation for the blackwomen sequence was uploaded incorrectly — the file we previously released was actually used for robustness testing. We have re-uploaded the corrected text annotations; please download the latest version. If you discover any other inconsistencies, please feel free to reach out.

2. Regarding the reproduction issue on RGBT234

We re-ran the evaluation today and the results are consistent with the paper. Minor differences across different environments are expected, but a gap of that magnitude usually indicates that the model weights were not loaded properly. To be safe, we have re-uploaded the model weights — please try again with the latest files. If you still encounter issues, please contact me (WeChat: Mr_Lee_777).

Thanks again for bringing this to our attention!

Best regards,
Hao Li

[wwwshengbo]

Hi @IdolLab,

Thank you for releasing RAGTrack and the pretrained models. I am trying to reproduce the RGBT234 results and found a possible issue in the test text annotations.

In the RGBT234 test sequence blackwomen, the provided text annotation seems inconsistent with the actual image content:

• class.txt: A bicycle.
• visible_description.txt: The object appears to be a small bicycle in the distance, possibly parked near the roadside, among a group of other vehicles and trees, blending cleverly with the surroundings due to its size and color.

However, from the original RGBT234 frames, the target appears to be a woman in black holding an umbrella and walking by the roadside, rather than a bicycle. Could you please confirm whether this annotation is correct, or whether the uploaded test text files may contain a mismatch for this sequence?

To check whether this was an isolated case or a more general issue, I first used AI to filter potentially suspicious test sequences, and then manually inspected those candidates. In RGBT234, blackwomen is the only sequence for which I found an obvious annotation mismatch.

I also encountered some difficulty reproducing the reported RGBT234 performance using the released pretrained model and text data. My current observations are:

• Evaluating the released author_results/RGBT234/RAGTrack with the official RGBT evaluation toolkit gives results close to the paper:

  • MPR ≈ 0.938
  • MSR ≈ 0.695

• But when I run inference myself using the released RGBT234 pretrained checkpoint and the provided RGBT234 text annotations, the result is much lower:

  • MPR ≈ 0.815
  • MSR ≈ 0.506

• For example, on the sequence manwithbasketball, my generated result starts to differ from the released author_results from the second frame.

I have checked the evaluation toolkit and the released result files, and the evaluation of author_results can reproduce the paper-level numbers. Therefore, I wonder whether there may be a version mismatch in the released RGBT234 test text annotations, pretrained checkpoint, or testing configuration.

Could you please help confirm:

1. Is the blackwomen text annotation in RGBT234 correct?
2. Are the currently released RGBT234 pretrained model and test text files exactly the same version used to generate the reported author_results?
3. Is there any additional testing detail or configuration needed to reproduce the RGBT234 numbers from the pretrained model?
   If Chinese is more convenient, I can also describe the issue in Chinese.

Thank you very much for your help. Best regards, Shirun Yao Anhui University, China

Hi @IdolLab,

Thank you for releasing RAGTrack and the pretrained models. I am trying to reproduce the RGBT234 results and found a possible issue in the test text annotations.

In the RGBT234 test sequence blackwomen, the provided text annotation seems inconsistent with the actual image content:

• class.txt: A bicycle.
• visible_description.txt: The object appears to be a small bicycle in the distance, possibly parked near the roadside, among a group of other vehicles and trees, blending cleverly with the surroundings due to its size and color.

However, from the original RGBT234 frames, the target appears to be a woman in black holding an umbrella and walking by the roadside, rather than a bicycle. Could you please confirm whether this annotation is correct, or whether the uploaded test text files may contain a mismatch for this sequence?

To check whether this was an isolated case or a more general issue, I first used AI to filter potentially suspicious test sequences, and then manually inspected those candidates. In RGBT234, blackwomen is the only sequence for which I found an obvious annotation mismatch.

I also encountered some difficulty reproducing the reported RGBT234 performance using the released pretrained model and text data. My current observations are:

• Evaluating the released author_results/RGBT234/RAGTrack with the official RGBT evaluation toolkit gives results close to the paper:

  • MPR ≈ 0.938
  • MSR ≈ 0.695

• But when I run inference myself using the released RGBT234 pretrained checkpoint and the provided RGBT234 text annotations, the result is much lower:

  • MPR ≈ 0.815
  • MSR ≈ 0.506

• For example, on the sequence manwithbasketball, my generated result starts to differ from the released author_results from the second frame.

I have checked the evaluation toolkit and the released result files, and the evaluation of author_results can reproduce the paper-level numbers. Therefore, I wonder whether there may be a version mismatch in the released RGBT234 test text annotations, pretrained checkpoint, or testing configuration.

Could you please help confirm:

1. Is the blackwomen text annotation in RGBT234 correct?
2. Are the currently released RGBT234 pretrained model and test text files exactly the same version used to generate the reported author_results?
3. Is there any additional testing detail or configuration needed to reproduce the RGBT234 numbers from the pretrained model?
   If Chinese is more convenient, I can also describe the issue in Chinese.

Thank you very much for your help. Best regards, Shirun Yao Anhui University, China

@yingros 你好，想问一下您，遇到过这种Segmentation fault (core dumped)
问题吗？在训练脚本运行时

[wwwshengbo]

在训练脚本运行时遇到这种Segmentation fault (core dumped)
问题？ 可以尝试，直接运行脚本，传递超参数，试试，不要运行，train.sh 文件。