Fix scene flow ground truth generation in Argoverse 2

Dockerfile

@@ -32,4 +32,3 @@ RUN apt-get update && apt-get install libgl1 -y
 RUN cd /home/kin/workspace/OpenSceneFlow && /opt/conda/bin/mamba env create -f environment.yaml
 RUN cd /home/kin/workspace/OpenSceneFlow/assets/cuda/mmcv && /opt/conda/envs/opensf/bin/python ./setup.py install
 RUN cd /home/kin/workspace/OpenSceneFlow/assets/cuda/chamfer3D && /opt/conda/envs/opensf/bin/python ./setup.py install
-

README.md

@@ -5,7 +5,7 @@
 </p>
 
 OpenSceneFlow is a codebase for point cloud scene flow estimation. 
-It is also an official implementation of the following paper (sored by the time of publication):
+It is also an official implementation of the following papers (sored by the time of publication):
 
 - **Flow4D: Leveraging 4D Voxel Network for LiDAR Scene Flow Estimation**  
 *Jaeyeul Kim, Jungwan Woo, Ukcheol Shin, Jean Oh, Sunghoon Im*  
@@ -28,9 +28,10 @@ International Conference on Robotics and Automation (**ICRA**) 2024
 [ Backbone ] [ Supervised ] - [ [arXiv](https://arxiv.org/abs/2401.16122) ] [ [Project](https://github.com/KTH-RPL/DeFlow) ] &rarr; [here](#deflow)
 
 
-💞 If you find *OpenSceneFlow* useful to your research, please cite [our works 📖](#cite-us) and give a star 🌟 as encouragement. (੭ˊ꒳​ˋ)੭✧
+💞 If you find *OpenSceneFlow* useful to your research, please cite [**our works** 📖](#cite-us) and give a star 🌟 as encouragement. (੭ˊ꒳​ˋ)੭✧
 
-🎁 <b>One repository, All methods!</b>. Additionally, *OpenSceneFlow* integrates the following excellent work: [ICLR'24 ZeroFlow](https://arxiv.org/abs/2305.10424), [ICCV'23 FastNSF](https://arxiv.org/abs/2304.09121), [RA-L'21 FastFlow](https://arxiv.org/abs/2103.01306), [NeurIPS'21 NSFP](https://arxiv.org/abs/2111.01253), 
+🎁 <b>One repository, All methods!</b>
+Additionally, *OpenSceneFlow* integrates following excellent works: [ICLR'24 ZeroFlow](https://arxiv.org/abs/2305.10424), [ICCV'23 FastNSF](https://arxiv.org/abs/2304.09121), [RA-L'21 FastFlow](https://arxiv.org/abs/2103.01306), [NeurIPS'21 NSFP](https://arxiv.org/abs/2111.01253). (More on the way...)
 
 <details> <summary> Summary of them:</summary>
 
@@ -42,7 +43,7 @@ International Conference on Robotics and Automation (**ICRA**) 2024
 
 </details>
 
-💡: Want to learn how to add your own network in this structure? Check [Contribute section] and know more about the code. Fee free to pull request and your bibtex [here](#cite-us) by pull request.
+💡: Want to learn how to add your own network in this structure? Check [Contribute section](assets/README.md#contribute) and know more about the code. Fee free to pull request and your bibtex [here](#cite-us) by pull request.
 
 ---
 
@@ -56,23 +57,29 @@ International Conference on Robotics and Automation (**ICRA**) 2024
 
 ## 0. Installation
 
-**Environment**: Setup
+There are two ways to install the codebase: directly on your [local machine](#environment-setup) or in a [Docker container](#docker-recommended-for-isolation).
+
+### Environment Setup
 
 ```bash
 git clone --recursive https://github.com/KTH-RPL/OpenSceneFlow.git
 cd OpenSceneFlow && mamba env create -f environment.yaml
+
+# You may need export your LD_LIBRARY_PATH with env lib
+# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/kin/mambaforge/lib
 ```
 
-CUDA package (need install nvcc compiler), the compile time is around 1-5 minutes:
+CUDA package (we already install nvcc compiler inside conda env), the compile time is around 1-5 minutes:
 ```bash
 mamba activate opensf
 # CUDA already install in python environment. I also tested others version like 11.3, 11.4, 11.7, 11.8 all works
 cd assets/cuda/mmcv && python ./setup.py install && cd ../../..
 cd assets/cuda/chamfer3D && python ./setup.py install && cd ../../..
 ```
 
-Or you always can choose [Docker](https://en.wikipedia.org/wiki/Docker_(software)) which isolated environment and free yourself from installation, you can pull it by. 
-If you have different arch, please build it by yourself `cd OpenSceneFlow && docker build -t zhangkin/opensf` by going through [build-docker-image](assets/README.md#build-docker-image) section.
+### Docker (Recommended for Isolation)
+
+You always can choose [Docker](https://en.wikipedia.org/wiki/Docker_(software)) which isolated environment and free yourself from installation. Pull the pre-built Docker image or build manually.
 
 ```bash
 # option 1: pull from docker hub
@@ -83,25 +90,36 @@ docker run -it --gpus all -v /dev/shm:/dev/shm -v /home/kin/data:/home/kin/data
 # and better to read your own gpu device info to compile the cuda extension again:
 cd /home/kin/workspace/OpenSceneFlow/assets/cuda/mmcv && /opt/conda/envs/opensf/bin/python ./setup.py install
 cd /home/kin/workspace/OpenSceneFlow/assets/cuda/chamfer3D && /opt/conda/envs/opensf/bin/python ./setup.py install
+
+mamba activate opensf
 ```
 
+If you prefer to build the Docker image by yourself, Check [build-docker-image](assets/README.md#build-docker-image) section for more details.
 
 ## 1. Data Preparation
 
 Refer to [dataprocess/README.md](dataprocess/README.md) for dataset download instructions. Currently, we support **Argoverse 2**, **Waymo**, and **custom datasets** (more datasets will be added in the future). 
 
-After downloading, convert the raw data to `.h5` format for easy training, evaluation, and visualization. Follow the steps in [dataprocess/README.md#process](dataprocess/README.md#process). For a quick start, use our **mini processed dataset**, which includes one scene in `train` and `val`. It is pre-converted to `.h5` format with label data ([Zenodo](https://zenodo.org/records/13744999/files/demo_data.zip)/[HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow/blob/main/demo_data.zip)).
+After downloading, convert the raw data to `.h5` format for easy training, evaluation, and visualization. Follow the steps in [dataprocess/README.md#process](dataprocess/README.md#process). 
+
+For a quick start, use our **mini processed dataset**, which includes one scene in `train` and `val`. It is pre-converted to `.h5` format with label data ([Zenodo](https://zenodo.org/records/13744999/files/demo_data.zip)/[HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow/blob/main/demo_data.zip)).
 
 
 ```bash
 wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/demo_data.zip
-unzip demo_data.zip -p /home/kin/data/av2
+unzip demo_data.zip -d /home/kin/data/av2/h5py
 ```
 
 Once extracted, you can directly use this dataset to run the [training script](#2-quick-start) without further processing.
 
 ## 2. Quick Start
 
+Don't forget to active Python environment before running the code.
+
+```bash
+mamba activate opensf
+```
+
 ### Flow4D
 
 Train Flow4D with the leaderboard submit config. [Runtime: Around 18 hours in 4x RTX 3090 GPUs.]
@@ -225,6 +243,12 @@ https://github.com/user-attachments/assets/07e8d430-a867-42b7-900a-11755949de21
   pages={2105-2111},
   doi={10.1109/ICRA57147.2024.10610278}
 }
+@article{zhang2025himu,
+    title={HiMo: High-Speed Objects Motion Compensation in Point Cloud},
+    author={Zhang, Qingwen and Khoche, Ajinkya and Yang, Yi and Ling, Li and Sina, Sharif Mansouri and Andersson, Olov and Jensfelt, Patric},
+    year={2025},
+    journal={arXiv preprint arXiv:2503.00803},
+}
 ```
 
 And our excellent collaborators works as followings:

conf/save.yaml

@@ -1,4 +1,4 @@
-dataset_path: /home/kin/data/av2/preprocess_v2/demo/sensor/val
+dataset_path: /home/kin/data/av2/h5py/demo/sensor/val
 checkpoint: /home/kin/model_zoo/seflow_best.ckpt
 res_name: # if None will directly be the `model_name.ckpt` in checkpoint path
 

dataprocess/extract_av2.py

@@ -35,6 +35,7 @@
 import pickle
 from zipfile import ZipFile
 import pandas as pd
+from copy import deepcopy
 
 import os, sys
 BASE_DIR = os.path.abspath(os.path.join( os.path.dirname( __file__ ), '..' ))
@@ -132,23 +133,49 @@ def compute_flow(sweeps, cuboids, poses):
         valid = np.ones(len(sweeps[0].xyz), dtype=np.bool_)
         # classes = -np.ones(len(sweeps[0].xyz), dtype=np.int8)
         classes = np.zeros(len(sweeps[0].xyz), dtype=np.uint8)
+
+        # # old version
+        # for id in cuboids[0]:
+        #     c0 = cuboids[0][id]
+        #     c0.length_m += BOUNDING_BOX_EXPANSION # the bounding boxes are a little too tight and some points are missed
+        #     c0.width_m += BOUNDING_BOX_EXPANSION
+        #     obj_pts, obj_mask = c0.compute_interior_points(sweeps[0].xyz)
+        #     classes[obj_mask] = CATEGORY_TO_INDEX[str(c0.category)]
 
+        #     if id in cuboids[1]:
+        #         c1 = cuboids[1][id]
+        #         c1_SE3_c0 = c1.dst_SE3_object.compose(c0.dst_SE3_object.inverse())
+        #         obj_flow = c1_SE3_c0.transform_point_cloud(obj_pts) - obj_pts
+        #         flow[obj_mask] = obj_flow.astype(np.float32)
+        #     else:
+        #         valid[obj_mask] = 0
+
+        # NOTE(HiMo): box expansion based on the object velocity
+        # check more detail: https://kin-zhang.github.io/HiMo
         for id in cuboids[0]:
-            c0 = cuboids[0][id]
-            c0.length_m += BOUNDING_BOX_EXPANSION # the bounding boxes are a little too tight and some points are missed
-            c0.width_m += BOUNDING_BOX_EXPANSION
+            c0 = deepcopy(cuboids[0][id])
             obj_pts, obj_mask = c0.compute_interior_points(sweeps[0].xyz)
-            classes[obj_mask] = CATEGORY_TO_INDEX[str(c0.category)]
-
             if id in cuboids[1]:
                 c1 = cuboids[1][id]
+                c1_SE3_c0_ego_frame = ego1_SE3_ego0.inverse().compose(c1.dst_SE3_object.compose(c0.dst_SE3_object.inverse()))
+                rel_obj_flow = c1_SE3_c0_ego_frame.transform_point_cloud(obj_pts) - obj_pts
+                delta_move = abs(np.linalg.norm(rel_obj_flow, axis=0).mean())
+
+                if delta_move > 0.04: # only when it's moving
+                    c0 = cuboids[0][id]
+                    c0.length_m += (BOUNDING_BOX_EXPANSION + min(delta_move/2, 2)) # since 180/360 for two LiDARs orientation
+                    c0.width_m += BOUNDING_BOX_EXPANSION
+                    c0.height_m += BOUNDING_BOX_EXPANSION
+                obj_pts, obj_mask = c0.compute_interior_points(sweeps[0].xyz)
+
+                # NOTE(Qingwen): after expansion, we need to recompute the flow
                 c1_SE3_c0 = c1.dst_SE3_object.compose(c0.dst_SE3_object.inverse())
                 obj_flow = c1_SE3_c0.transform_point_cloud(obj_pts) - obj_pts
+                classes[obj_mask] = CATEGORY_TO_INDEX[str(c0.category)]
                 flow[obj_mask] = obj_flow.astype(np.float32)
             else:
                 valid[obj_mask] = 0
         return flow, classes, valid, ego1_SE3_ego0
-
     sweeps = [Sweep.from_feather(data_dir / log_id / "sensors" / "lidar" / f"{ts}.feather") for ts in timestamps]
 
     # ================== Load annotations ==================

environment.yaml

@@ -26,6 +26,8 @@ dependencies:
   - scikit-learn==1.3.2
   - hdbscan
   - setuptools==69.5.1
+  - gxx_linux-64==11.4.0
+  - pathtools
   - pip:
     - open3d==0.18.0
     - dztimer

src/models/__init__.py

@@ -7,6 +7,8 @@
 # If you find this repo helpful, please cite the respective publication as 
 # listed on the above website.
 """
+import warnings
+warnings.simplefilter(action="ignore", category=FutureWarning)
 
 from .deflow import DeFlow
 from .fastflow3d import FastFlow3D

tools/visualization.py

@@ -69,7 +69,7 @@ def check_flow(
 
 def vis(
     data_dir: str ="/home/kin/data/av2/preprocess/sensor/mini",
-    res_name: str = "flow", # "flow", "flow_est"
+    res_name: str = "flow", # any res_name we write before in HDF5Data
     start_id: int = 0,
     point_size: float = 2.0,
     mode: str = "vis",