H,
Thank you for open-sourcing code and dataset.
We are currently using V2Xverse to evaluate a collaborative-perception model and are trying to reproduce the published Town05 perception evaluation.
While preparing the test set, we found a possible mismatch between the paper, the test manifest, and the public Hugging Face release. Using Hugging Face revision 41b3d639676bba921e04b5b493af0136aa9304e8, we found that:
The paper reports 108 routes and 4,306 test samples.
test.json references 19 Town05 routes.
The public archive appears to contain only seven of these Town05 routes: 6, 11, 12, 27, 28, 29, and 31.
The other 12 routes—0, 1, 3, 4, 5, 7, 17, 19, 25, 30, long, and short—appear to be absent.
In route 29, ego_vehicle_1 appears to lack LiDAR, measurements, and labels, as also discussed in GitHub issue #46.
Two questions, if you have a moment:
- Is the Hugging Face release an intentional subset, or are the remaining Town05 archives still pending upload? If the complete set is available elsewhere, we would be grateful for a pointer.
- Which sample selection produced the reported Town05 perception metrics — was it test.json with its selected_frames ranges? If you could share the canonical dataset_index.txt or frame manifest along with the evaluation configuration or command, that would let us match your protocol exactly.
We currently have results on the seven available routes, but we do not want to present them as official V2Xverse test-set results without confirming the evaluation protocol. I am happy to send our full file inventory, or to open a GitHub issue documenting the findings if that is more useful to you.
Thank you very much for your help.
Best regards,
Melih
H,
Thank you for open-sourcing code and dataset.
We are currently using V2Xverse to evaluate a collaborative-perception model and are trying to reproduce the published Town05 perception evaluation.
While preparing the test set, we found a possible mismatch between the paper, the test manifest, and the public Hugging Face release. Using Hugging Face revision 41b3d639676bba921e04b5b493af0136aa9304e8, we found that:
Two questions, if you have a moment:
We currently have results on the seven available routes, but we do not want to present them as official V2Xverse test-set results without confirming the evaluation protocol. I am happy to send our full file inventory, or to open a GitHub issue documenting the findings if that is more useful to you.
Thank you very much for your help.
Best regards,
Melih