From 4f1dbbde1ce6a09a3f0443a83715bc862a303fbc Mon Sep 17 00:00:00 2001 From: Qingwen Date: Sat, 6 Sep 2025 09:33:55 +0100 Subject: [PATCH 1/5] docs(truckscene): starting update truckscene dataprocess. No radar involved at this stage. will do it in DoGFlow pull request. --- README.md | 8 +++++- dataprocess/README.md | 47 +++++++++++++++++++++++++++++-- dataprocess/extract_truckscene.py | 13 +++++++++ dataprocess/misc_data.py | 31 ++++++++++++++++++++ 4 files changed, 96 insertions(+), 3 deletions(-) create mode 100644 dataprocess/extract_truckscene.py diff --git a/README.md b/README.md index ad357a0..e50ca0b 100644 --- a/README.md +++ b/README.md @@ -102,7 +102,7 @@ If you prefer to build the Docker image by yourself, Check [build-docker-image]( ## 1. Data Preparation -Refer to [dataprocess/README.md](dataprocess/README.md) for dataset download instructions. Currently, we support **Argoverse 2**, **Waymo**, **nuScenes**, **ZOD** and **custom datasets** (more datasets will be added in the future). +Refer to [dataprocess/README.md](dataprocess/README.md) for dataset download instructions. Currently, we support **Argoverse 2**, **Waymo**, **nuScenes**, [**MAN-TruckScene**](https://github.com/TUMFTM/truckscenes-devkit), [**ZOD**](https://github.com/zenseact/zod) and **custom datasets** (more datasets will be added in the future). After downloading, convert the raw data to `.h5` format for easy training, evaluation, and visualization. Follow the steps in [dataprocess/README.md#process](dataprocess/README.md#process). @@ -337,6 +337,12 @@ If you find it useful, please cite our works: year={2025}, journal={arXiv preprint arXiv:2503.00803}, } +@article{zhang2025deltaflow, + title={{DeltaFlow}: An Efficient Multi-frame Scene Flow Estimation Method}, + author={Zhang, Qingwen and Zhu, Xiaomeng and Zhang, Yushan and Cai, Yixi and Andersson, Olov and Jensfelt, Patric}, + year={2025}, + journal={arXiv preprint arXiv:2508.17054}, +} ``` And our excellent collaborators works contributed to this codebase also: diff --git a/dataprocess/README.md b/dataprocess/README.md index 2651afa..07e41c1 100644 --- a/dataprocess/README.md +++ b/dataprocess/README.md @@ -6,13 +6,13 @@ README for downloading and preprocessing the dataset. We includes waymo, argover - [Download](#download): includes how to download the dataset. - [Process](#process): run script to preprocess the dataset. -We've updated the process dataset for: +We've updated the process dataset for (Please cite the original dataset paper and involved work if you use them): - [x] Argoverse 2.0: check [here](#argoverse-20). The process script Involved from [DeFlow](https://github.com/KTH-RPL/DeFlow). - [x] Waymo: check [here](#waymo-dataset). The process script was involved from [SeFlow](https://github.com/KTH-RPL/SeFlow). - [x] nuScenes: check [here](#nuscenes), The process script was involved from [DeltaFlow](https://github.com/Kin-Zhang/DeltaFlow). - [x] ZOD (w/o gt): check [here](#zod-dataset). The process script was involved from [HiMo](https://kin-zhang.github.io/HiMo). (It could be a good first reference for users to extract other datasets in the future.) -- [ ] TruckScene: done coding, public after review. Will be involved later by another paper. +- [x] TruckScene: check [here](#truckscene). The process script was involved from [DoGFlow](https://github.com/ajinkyakhoche/DoGFlow). If you want to **use all datasets above**, there is a **specific environment** in [envsftool.yaml](../envsftool.yaml) to install all the necessary packages. As Waymo package have different configuration and conflict with the main environment. Setup through the following command: @@ -134,6 +134,45 @@ For HiMo, we only downloaded [drives-set](https://zod.zenseact.com/drives/) for Please check the scripts: [dataprocess/extract_zod.py](./extract_zod.py) in detail, current we only process one scene while feel free to comment out for all scene etc. +### TruckScene + +Please visit the [TruckScene dataset](https://brandportal.man/d/QSf8mPdU5Hgj/downloads#/-/dataset) page for privacy policy. You can download the dataset by following command: + +```bash +# mini set, ~11G recommended for debugging purpose +cd /home/kin/data/truckscene/mini +wget https://man-truckscenes.s3.eu-central-1.amazonaws.com/release/mini/man-truckscenes_metadata_v1.0-mini.zip +wget https://man-truckscenes.s3.eu-central-1.amazonaws.com/release/mini/man-truckscenes_sensordata_v1.0-mini.zip +unzip "man-truckscenes_*.zip" + +# full trainval set, ~630G +cd /home/kin/data/truckscene/trainval +wget https://man-truckscenes.s3.eu-central-1.amazonaws.com/release/trainval/man-truckscenes_metadata_v1.0-trainval.zip +wget https://man-truckscenes.s3.eu-central-1.amazonaws.com/release/trainval/man-truckscenes_sensordata01_v1.0-trainval.zip +wget https://man-truckscenes.s3.eu-central-1.amazonaws.com/release/trainval/man-truckscenes_sensordata02_v1.0-trainval.zip +wget https://man-truckscenes.s3.eu-central-1.amazonaws.com/release/trainval/man-truckscenes_sensordata03_v1.0-trainval.zip +wget https://man-truckscenes.s3.eu-central-1.amazonaws.com/release/trainval/man-truckscenes_sensordata04_v1.0-trainval.zip +wget https://man-truckscenes.s3.eu-central-1.amazonaws.com/release/trainval/man-truckscenes_sensordata05_v1.0-trainval.zip +wget https://man-truckscenes.s3.eu-central-1.amazonaws.com/release/trainval/man-truckscenes_sensordata06_v1.0-trainval.zip +wget https://man-truckscenes.s3.eu-central-1.amazonaws.com/release/trainval/man-truckscenes_sensordata07_v1.0-trainval.zip +unzip "man-truckscenes_*.zip" +``` + +Folder structure: +``` +truckscene/ + — samples/ + — sweeps/ + — v1.0-mini/ +``` + +#### Dataset frames + +| Dataset | # Total Scene | # Total Frames | +| ------- | ------------- | -------------- | +| train | - | - | +| val | - | - | + ## Process @@ -144,6 +183,7 @@ This directory contains the scripts to preprocess the datasets into `.h5` files. - `extract_nus.py`: Process the datasets in nuScenes. - `extract_waymo.py`: Process the datasets in Waymo. - `extract_zod.py`: Process the datasets in ZOD. +- `extract_truckscene.py`: Process the datasets in TruckScene. Example Running command: ```bash @@ -155,6 +195,9 @@ python dataprocess/extract_waymo.py --mode train --flow_data_dir /home/kin/data/ # nus: python dataprocess/extract_nus.py --mode v1.0-trainval --output_dir /home/kin/data/nus/h5py/full --nproc 24 + +# truckscene: +python dataprocess/extract_truckscene.py TODO ``` diff --git a/dataprocess/extract_truckscene.py b/dataprocess/extract_truckscene.py new file mode 100644 index 0000000..8592d6e --- /dev/null +++ b/dataprocess/extract_truckscene.py @@ -0,0 +1,13 @@ +""" +# +# Created: 2025-09-06 09:16 +# Copyright (C) 2025-now, RPL, KTH Royal Institute of Technology +# Author: Qingwen Zhang (https://kin-zhang.github.io/), Ajinkya Khoche (https://ajinkyakhoche.github.io/) +# +# Description: Preprocess Data, save as h5df format for faster loading +# This one is for MAN TruckScenes dataset +# +# NOTE: truckscene follow really similar structure with nuscenes format. That's why +# **NOT ALL** frames are annotated! So the SL training might be not that effective. +# +""" \ No newline at end of file diff --git a/dataprocess/misc_data.py b/dataprocess/misc_data.py index b749a3c..e7727e2 100644 --- a/dataprocess/misc_data.py +++ b/dataprocess/misc_data.py @@ -155,4 +155,35 @@ def compose(self, right_se2: "SE2") -> "SE2": 'static.other': 'NONE', 'static.vegetation': 'NONE', 'vehicle.ego': 'NONE' +} + +## ====> MAN to Argoverse Mapping +ManNamMap = { + "animal": 'NONE', + "human.pedestrian.adult": 'PEDESTRIAN', + "human.pedestrian.child": 'PEDESTRIAN', + "human.pedestrian.construction_worker": 'PEDESTRIAN', + "human.pedestrian.personal_mobility": 'PEDESTRIAN', + "human.pedestrian.police_officer": 'PEDESTRIAN', + "human.pedestrian.stroller": 'STROLLER', + "human.pedestrian.wheelchair": 'WHEELCHAIR', + "movable_object.barrier": 'NONE', + "movable_object.debris": 'NONE', + "movable_object.pushable_pullable": 'NONE', + "movable_object.trafficcone": 'CONSTRUCTION_CONE', + "static_object.bicycle_rack": 'NONE', + "static_object.traffic_sign": 'SIGN', + "vehicle.bicycle": 'BICYCLE', + "vehicle.bus.bendy": 'ARTICULATED_BUS', + "vehicle.bus.rigid": 'BUS', + "vehicle.car": 'REGULAR_VEHICLE', + "vehicle.construction": 'LARGE_VEHICLE', + "vehicle.emergency.ambulance": 'LARGE_VEHICLE', + "vehicle.emergency.police": 'REGULAR_VEHICLE', + "vehicle.motorcycle": 'MOTORCYCLE', + "vehicle.trailer": 'VEHICULAR_TRAILER', + "vehicle.truck": 'TRUCK', + "vehicle.train": 'NONE', + "vehicle.other": 'NONE', + "vehicle.ego_trailer": 'NONE', } \ No newline at end of file From bf3f75a6f102cc5c4f5333d0146a7acc1fb2e8d0 Mon Sep 17 00:00:00 2001 From: Qingwen Date: Sat, 6 Sep 2025 19:20:33 +0100 Subject: [PATCH 2/5] feat(truckscene): debug-mdoe for the file, will clean it up. --- conf/others/truckscenes.toml | 31 +++ conf/{ground => others}/zod.toml | 0 dataprocess/extract_nus.py | 32 +-- dataprocess/extract_truckscene.py | 13 -- dataprocess/extract_truckscenes.py | 301 +++++++++++++++++++++++++++++ dataprocess/extract_zod.py | 2 +- dataprocess/misc_data.py | 30 +++ envsftool.yaml | 2 +- 8 files changed, 381 insertions(+), 30 deletions(-) create mode 100644 conf/others/truckscenes.toml rename conf/{ground => others}/zod.toml (100%) delete mode 100644 dataprocess/extract_truckscene.py create mode 100644 dataprocess/extract_truckscenes.py diff --git a/conf/others/truckscenes.toml b/conf/others/truckscenes.toml new file mode 100644 index 0000000..9ebb7e1 --- /dev/null +++ b/conf/others/truckscenes.toml @@ -0,0 +1,31 @@ + +[important] +height = 0 # ego height. since truckscene have multiple LiDARs, we set pc to ego-coord already. + +# 整个雷点点云以自己为中心 分为多少个segment，每个segment又分成多少个bin +[segments] +r_min = 0.1 # minimum point distance. 感兴趣的区域 +r_max = 100 # maximum point distance. +n_segments = 360 # number of radial segments. +n_bins = 160 # number of radial bins. + + +[ground] +# 直线的 slope 斜率，一个seg里拟合的线斜率 +min_slope = 0.0 # minimum slope of a ground line. [T_m_small] +max_slope = 0.1 # maximum slope of a ground line. [T_m] +# 如果拟合点超过这个 long threshold，判断其 height 变化是否超过 max_long_height +long_threshold = 2.0 # Distance at which points are considered far from each other. [T_d_prev] +max_long_height = 0.3 # maximum height change to previous point in long line. [T_b] +# 如果没有超过 long threshold，判断现在拟合的点与fitline高度变化是否超过 max_start_height +max_start_height = 0.2 # Maximum heigh of starting line to be labelled ground. +# 最大的拟合误差，超过这个误差的点不会被加入 line 拟合 +max_fit_error = 0.1 # maximum error of a point during line fit. [T_RMSE: will sqaure inside code.] + +# 判断是否是地面点的时候，需要满足的条件： +max_dist_to_line = 0.2 # maximum vertical distance of point to line to be considered ground. [T_d_ground] +line_search_angle = 2.0 # How far to search for a line in angular direction [rad]. + +[general] +n_threads = 8 # number of threads for parallel processing. +verbose = false # if you don't want to see every output, set this to false. \ No newline at end of file diff --git a/conf/ground/zod.toml b/conf/others/zod.toml similarity index 100% rename from conf/ground/zod.toml rename to conf/others/zod.toml diff --git a/dataprocess/extract_nus.py b/dataprocess/extract_nus.py index 4fcd2ad..28bb677 100644 --- a/dataprocess/extract_nus.py +++ b/dataprocess/extract_nus.py @@ -18,7 +18,7 @@ import multiprocessing from pathlib import Path from multiprocessing import Pool, current_process -from typing import Optional, Final +from typing import Optional from tqdm import tqdm import numpy as np import fire, time, h5py @@ -69,10 +69,13 @@ def _load_points_from_file(filename: str) -> np.ndarray: pc = pc.reshape((-1, 5))[:, :4] return pc -def get_pose(nusc, sweep_data): - # pose from lidar to world - ego2lidar = nusc.get('calibrated_sensor', sweep_data['calibrated_sensor_token']) - world2ego = nusc.get('ego_pose', sweep_data['ego_pose_token']) +def get_pose(data_fn, sweep_data, w2stf=True): + world2ego = data_fn.get('ego_pose', sweep_data['ego_pose_token']) + # without considering the sensor to ego transform, we will do it outside if multiple sensors are used. + if not w2stf: + return transform_matrix(world2ego['translation'], Quaternion(world2ego['rotation'])) + + ego2lidar = data_fn.get('calibrated_sensor', sweep_data['calibrated_sensor_token']) ego2lidar_np = transform_matrix(ego2lidar['translation'], Quaternion(ego2lidar['rotation'])) world2ego_np = transform_matrix(world2ego['translation'], Quaternion(world2ego['rotation'])) return np.dot(world2ego_np, ego2lidar_np) @@ -85,13 +88,13 @@ def if_annotated_frame(sample_ann_dict, ts0): break return gt_flow_flag -def _resample_data(nusc, sample_data, sample_ann_dict, resample2frequency=10): +def _resample_data(nusc, sample_data, sample_ann_dict, datafrequency=20, resample2frequency=10): """ NOTE(Qingwen) - 2025-05-18: We always want to start from the first GT frame, and then resample the data! So we have as many GT frames as possible... """ sweep_data_lst, timestamps_lst = [], [] - skipFrame = int(20 / resample2frequency) # since nuscenes sweep at 20Hz, we want to resample to 10Hz + skipFrame = int(datafrequency / resample2frequency) # since nuscenes sweep at 20Hz, we want to resample to 10Hz cnt = 0 # Find the first GT frame @@ -268,18 +271,17 @@ def proc(x, ignore_current_process=False): pos = 1 process_log(*x, n=pos) -def process_logs(nusc_mode, data_dir: Path, scene_list: list, output_dir: Path, nproc: int): - """Compute sceneflow for all logs in the dataset. Logs are processed in parallel. - Args: - data_dir: Argoverse 2.0 directory - output_dir: Output directory. +def process_logs(data_mode, data_dir: Path, scene_list: list, output_dir: Path, nproc: int): """ - + Compute sceneflow for all logs in the dataset. + Logs are processed in parallel. + """ + if not (data_dir).exists(): print(f'{data_dir} not found') return - args = sorted([(nusc_mode, data_dir, scene_num_id, output_dir) for scene_num_id in range(len(scene_list))]) + args = sorted([(data_mode, data_dir, scene_num_id, output_dir) for scene_num_id in range(len(scene_list))]) print(f'Using {nproc} processes') # # for debug @@ -295,7 +297,7 @@ def process_logs(nusc_mode, data_dir: Path, scene_list: list, output_dir: Path, res = list(tqdm(p.imap_unordered(proc, args), total=len(scene_list), ncols=100)) def main( - data_dir: str = "/home/kin/data/nus/v1.0", + data_dir: str = "/home/kin/data/nus/raw", mode: str = "v1.0-mini", output_dir: str ="/home/kin/data/nus/h5py/demo", nproc: int = (multiprocessing.cpu_count() - 1), diff --git a/dataprocess/extract_truckscene.py b/dataprocess/extract_truckscene.py deleted file mode 100644 index 8592d6e..0000000 --- a/dataprocess/extract_truckscene.py +++ /dev/null @@ -1,13 +0,0 @@ -""" -# -# Created: 2025-09-06 09:16 -# Copyright (C) 2025-now, RPL, KTH Royal Institute of Technology -# Author: Qingwen Zhang (https://kin-zhang.github.io/), Ajinkya Khoche (https://ajinkyakhoche.github.io/) -# -# Description: Preprocess Data, save as h5df format for faster loading -# This one is for MAN TruckScenes dataset -# -# NOTE: truckscene follow really similar structure with nuscenes format. That's why -# **NOT ALL** frames are annotated! So the SL training might be not that effective. -# -""" \ No newline at end of file diff --git a/dataprocess/extract_truckscenes.py b/dataprocess/extract_truckscenes.py new file mode 100644 index 0000000..33caaf9 --- /dev/null +++ b/dataprocess/extract_truckscenes.py @@ -0,0 +1,301 @@ +""" +# +# Created: 2025-09-06 09:16 +# Copyright (C) 2025-now, RPL, KTH Royal Institute of Technology +# Author: Qingwen Zhang (https://kin-zhang.github.io/), Ajinkya Khoche (https://ajinkyakhoche.github.io/) +# +# Description: Preprocess Data, save as h5df format for faster loading +# This one is for MAN TruckScenes dataset +# +# NOTE: truckscene follow really similar structure with TruckScenes format. That's why +# **NOT ALL** frames are annotated! So the SL training might be not that effective. +# Since truckscenes LiDAR is at 10Hz, we keep annotated flow to 10Hz also. +# +""" + +from collections import defaultdict +import os +# os.environ["OMP_NUM_THREADS"] = "1" + +from pathlib import Path +import multiprocessing +from multiprocessing import Pool, current_process +from typing import Optional +from tqdm import tqdm +import numpy as np +import fire, time, h5py + +from truckscenes import TruckScenes +from truckscenes.utils import splits +from truckscenes.utils.geometry_utils import transform_matrix +from truckscenes.utils.geometry_utils import points_in_box +from truckscenes.utils.data_classes import LidarPointCloud +from pyquaternion import Quaternion + +import os, sys +PARENT_DIR = os.path.abspath(os.path.join( os.path.dirname( __file__ ), '..')) +sys.path.append(PARENT_DIR) +from dataprocess.misc_data import create_reading_index, check_h5py_file_exists, ManNamMap, find_closest_integer_in_ref_arr +from dataprocess.extract_nus import remove_ego_points, get_pose +from src.utils import npcal_pose0to1 +from src.utils.av2_eval import CATEGORY_TO_INDEX +from linefit import ground_seg + + +GROUNDSEG_config = f"{PARENT_DIR}/conf/others/truckscenes.toml" +# NOTE(Qingwen): we only select 2x64 long-range LiDARs for processing, you can add more sensors if needed +# The timestamp reference will always be the last item in this list +SelectedSensor = ['LIDAR_LEFT', 'LIDAR_RIGHT'] # 2x64 +# SelectedSensor = ['LIDAR_TOP_FRONT', 'LIDAR_TOP_LEFT', 'LIDAR_TOP_RIGHT', 'LIDAR_REAR'] # 4x32 +# SelectedSensor = ['LIDAR_LEFT', 'LIDAR_RIGHT', 'LIDAR_TOP_FRONT', 'LIDAR_TOP_LEFT', 'LIDAR_TOP_RIGHT', 'LIDAR_REAR'] # all 6 LiDARs + +def process_log(data_mode, data_dir: Path, scene_num_id: int, output_dir: Path, n: Optional[int] = None) : + def create_group_data(group, pc, pose, lidar_id, lidar_center, gm = None, flow_0to1=None, flow_valid=None, flow_category=None, flow_instance=None, ego_motion=None): + group.create_dataset('lidar', data=pc.astype(np.float32)) + group.create_dataset('pose', data=pose.astype(np.float64)) + group.create_dataset('ground_mask', data=gm.astype(bool)) + group.create_dataset('lidar_id', data=lidar_id.astype(np.float32)) + group.create_dataset('lidar_center', data=lidar_center.astype(np.float32)) # shape: [LiDAR_num, 3] (x, y, z) + if ego_motion is not None: + group.create_dataset('ego_motion', data=ego_motion.astype(np.float32)) + if flow_0to1 is not None: + # ground truth flow information + group.create_dataset('flow', data=flow_0to1.astype(np.float32)) + group.create_dataset('flow_is_valid', data=flow_valid.astype(bool)) + group.create_dataset('flow_category_indices', data=flow_category.astype(np.uint8)) + group.create_dataset('flow_instance_id', data=flow_instance.astype(np.int16)) + + def compute_flow_simple(data_fn, pc0, pose0, pose1, ts0, ts1, sample_ann_list, dclass, DataNameMap=ManNamMap): + # compute delta transform between pose0 and pose1 + ego1_SE3_ego0 = npcal_pose0to1(pose0, pose1) + # flow due to ego motion + flow = np.zeros_like(pc0[:,:3]) + flow = pc0[:,:3] @ ego1_SE3_ego0[:3,:3].T + ego1_SE3_ego0[:3,3] - pc0[:,:3] # pose flow + + valid = np.ones(len(pc0), dtype=np.bool_) + classes = np.zeros(len(pc0), dtype=np.uint8) + instances = np.zeros(len(pc0), dtype=np.int16) + + delta_t = (ts1 - ts0) * 1e-6 + world_pc0 = pc0[:,:3] @ pose0[:3,:3].T + pose0[:3,3] + id_ = 0 + for ann in sample_ann_list: + # ann_vel = ann.velocity # it's nan in truckscenes, we need to compute... don't know why + ann_vel = data_fn.box_velocity(ann.token) # in world frame + if np.isnan(ann_vel).any(): + continue + + # previous x/y/z velocity is on world frame, need to transform to ego frame + ann_vel = ann_vel @ pose0[:3,:3] + cls = ann.name + + points_in_box_mask = points_in_box(ann, world_pc0[:,:3].T, wlh_factor=1.2) + classes[points_in_box_mask] = CATEGORY_TO_INDEX[DataNameMap[cls]] + if np.sum(points_in_box_mask) > 5: + obj_flow = np.ones_like(pc0[points_in_box_mask,:3]) * ann_vel * delta_t + flow[points_in_box_mask] += obj_flow + instances[points_in_box_mask] = (dclass[id_]+1) + id_ += 1 + else: + valid[points_in_box_mask] = False + + return {'flow_0_1': flow, 'valid_0': valid, 'classes_0': classes, + 'ego_motion': ego1_SE3_ego0, 'flow_instance_id': instances} + + mants = TruckScenes(dataroot=data_dir, version=data_mode, verbose=False) + scene = mants.scene[scene_num_id] + log_id = scene['name'] + + # In man-truckscenes, samples are annotated at 2 Hz and sweeps at 10 Hz. + sample_data_lst = [] + for sample in mants.sample: + if sample['scene_token'] != scene['token']: + continue + else: + sample_data_lst.append(sample) + + now_sample_token_str = scene['first_sample_token'] + sample = mants.get('sample', now_sample_token_str) + + # initialize full sweep data dict + full_sweep_data_dict, timestamps = {}, {} + for sensor_name in SelectedSensor: + full_sweep_data_dict[sensor_name] = [] + timestamps[sensor_name] = [] + + for channel, token in sample['data'].items(): + if channel in SelectedSensor: + sample_data = mants.get('sample_data', token) + while sample_data['next'] != '': + ts0 = sample_data['timestamp'] + full_sweep_data_dict[channel].append(sample_data) + timestamps[channel].append(ts0) + sample_data = mants.get('sample_data', sample_data['next']) + + if check_h5py_file_exists(output_dir/f'{log_id}.h5', timestamps[SelectedSensor[-1]]): + print(f'{log_id} already exists and all timestamps are , skip...') + return + + dclass = defaultdict(lambda: len(dclass)) + mygroundseg = ground_seg(GROUNDSEG_config) + + + with h5py.File(output_dir/f'{log_id}.h5', 'a') as f: + for cnt, sweep_data in enumerate(full_sweep_data_dict[SelectedSensor[-1]]): + ts0 = sweep_data['timestamp'] + pose0 = get_pose(mants, sweep_data, w2stf=False) + + lidar_list, lidar_center, lidar_dt, lidar_id = [], [], [], []; lidar_id_cnt = 0 + for single_sensor_name in full_sweep_data_dict.keys(): + # load closest sensor point cloud in reference ego frame + sensor_sweep_list = full_sweep_data_dict[single_sensor_name] + closest_ch_ind, closest_ch_timestamp, timestamp_diff = find_closest_integer_in_ref_arr( + ts0, np.array([t['timestamp'] for t in sensor_sweep_list]) + ) + sensor_sweep = sensor_sweep_list[closest_ch_ind] + + ego2lidar = mants.get('calibrated_sensor', sensor_sweep['calibrated_sensor_token']) + ego2lidar_np = transform_matrix(ego2lidar['translation'], Quaternion(ego2lidar['rotation'])) + + if 'LIDAR' in single_sensor_name: + pc = LidarPointCloud.from_file(os.path.join(str(data_dir), sensor_sweep['filename'])).points.T + # NOTE(Qingwen): we need all points in base_link (ego) coordinate, but we save sensor center also + pc[:,:3] = pc[:,:3] @ ego2lidar_np[:3,:3].T + ego2lidar_np[:3,3].T + lidar_list.append(pc) + + # lidar_dt.append(np.ones(pc.shape[0]) * timestamp_diff * 1e-6) # microsecond to s + lidar_id.append(np.ones(pc.shape[0]) * lidar_id_cnt) + # lidar_center.append(ego2lidar_np[:3,3].T) # x, y, z + lidar_center.append(ego2lidar_np) + lidar_id_cnt += 1 + + points = np.vstack(lidar_list) + # lidar_dt = np.hstack(lidar_dt) + lidar_id = np.hstack(lidar_id) + lidar_center = np.array(lidar_center) # shape Num_LiDAR x 4x4 + + points = remove_ego_points(points, length_threshold=2.0, width_threshold=7.0) + is_ground_0 = np.array(mygroundseg.run(points[:, :3])) + + # # debug visualization + # import open3d as o3d + # pcd = o3d.geometry.PointCloud() + # pcd.points = o3d.utility.Vector3dVector(points[:,:3]) + # # pcd.colors = o3d.utility.Vector3dVector(np.zeros((points.shape[0], 3))) + # show_lits = [pcd, o3d.geometry.TriangleMesh.create_coordinate_frame(size=1)] # ego frame + # for lidar_id_ in range(lidar_center.shape[0]): + # show_lits.append(o3d.geometry.TriangleMesh.create_coordinate_frame(size=1).transform(lidar_center[lidar_id_])) + # o3d.visualization.draw_geometries(show_lits) + # exit(0) + + if cnt == len(full_sweep_data_dict[SelectedSensor[-1]]) - 1: + group = f.create_group(str(ts0)) + create_group_data(group=group, pc=points, gm=is_ground_0.astype(np.bool_), pose=pose0, \ + lidar_id=lidar_id, lidar_center=lidar_center) + else: + sweep_data_next = full_sweep_data_dict[SelectedSensor[-1]][cnt+1] + ts1 = sweep_data_next['timestamp'] + pose1 = get_pose(mants, sweep_data_next, w2stf=False) + + group = f.create_group(str(ts0)) + # annotated frame, compute flow + if sweep_data['is_key_frame'] and sweep_data['prev'] != "": + curr_scene_ann = mants.get_boxes(sweep_data['token']) + scene_flow = compute_flow_simple(mants, points, pose0, pose1, ts0, ts1, curr_scene_ann, dclass, DataNameMap=ManNamMap) + + create_group_data(group=group, pc=points, gm=is_ground_0.astype(np.bool_), pose=pose0.astype(np.float32), \ + lidar_id=lidar_id, lidar_center=lidar_center, \ + flow_0to1=scene_flow['flow_0_1'], flow_valid=scene_flow['valid_0'], flow_category=scene_flow['classes_0'], \ + flow_instance=scene_flow['flow_instance_id'], + ego_motion=scene_flow['ego_motion'].astype(np.float32)) + else: # no annotations, only save data + create_group_data(group=group, pc=points, gm=is_ground_0.astype(np.bool_), pose=pose0.astype(np.float32), \ + lidar_id=lidar_id, lidar_center=lidar_center, \ + ego_motion=(np.linalg.inv(pose1) @ pose0).astype(np.float32)) + +def proc(x, ignore_current_process=False): + if not ignore_current_process: + current=current_process() + pos = current._identity[0] + else: + pos = 1 + process_log(*x, n=pos) + +def process_logs(data_mode, data_dir: Path, scene_list: list, output_dir: Path, nproc: int): + """ + Compute sceneflow for all logs in the dataset. + Logs are processed in parallel. + """ + + if not (data_dir).exists(): + print(f'{data_dir} not found') + return + + args = sorted([(data_mode, data_dir, scene_num_id, output_dir) for scene_num_id in range(len(scene_list))]) + print(f'Using {nproc} processes') + + # for debug + for x in tqdm(args[1:]): + proc(x, ignore_current_process=True) + break + + # if nproc <= 1: + # for x in tqdm(args): + # proc(x, ignore_current_process=True) + # else: + # with Pool(processes=nproc) as p: + # res = list(tqdm(p.imap_unordered(proc, args), total=len(scene_list), ncols=100)) + +def main( + data_dir: str = "/home/kin/data/truckscenes/man-truckscenes", + mode: str = "v1.0-mini", + output_dir: str ="/home/kin/data/truckscenes/h5py", + nproc: int = 1, + only_index: bool = False, + split_name = None +): + available_vers = ['v1.0-trainval', 'v1.0-test', 'v1.0-mini'] # defined by nus. + assert mode in available_vers + # man = TruckScenes(dataroot=data_dir, version=mode, verbose=False) + # print(f"Processing {mode} dataset with {len(man.scene)} scenes...") + + if mode == 'v1.0-trainval': + train_scenes = splits.train + val_scenes = splits.val + if split_name is not None and split_name == 'train': + input_dict = {'train': train_scenes} + elif split_name is not None and split_name == 'val': + input_dict = {'val': val_scenes} + else: + input_dict = {'train': train_scenes, 'val': val_scenes} + elif mode == 'v1.0-test': + test_scenes = splits.test + input_dict = {'test': test_scenes} + elif mode == 'v1.0-mini': + train_scenes = splits.mini_train + val_scenes = splits.mini_val + # input_dict = { + # 'train': train_scenes, + # 'val': val_scenes + # } + # NOTE(Qingwen): or if you don't want to split mini, use below + input_dict = {'mini': train_scenes + val_scenes} + else: + raise ValueError('unknown') + + for input_key, input_val in input_dict.items(): + output_dir_ = Path(output_dir) / input_key + print("[INFO] We are processing data to ", output_dir_) + if only_index: + create_reading_index(Path(output_dir_)) + create_reading_index(Path(output_dir_), flow_inside_check=True) + return + output_dir_.mkdir(exist_ok=True, parents=True) + process_logs(mode, Path(data_dir), input_val, output_dir_, nproc) + create_reading_index(output_dir_) + create_reading_index(Path(output_dir_), flow_inside_check=True) + +if __name__ == '__main__': + start_time = time.time() + fire.Fire(main) + print(f"\nTime used: {(time.time() - start_time)/60:.2f} mins") \ No newline at end of file diff --git a/dataprocess/extract_zod.py b/dataprocess/extract_zod.py index d60ecdd..295c0c7 100644 --- a/dataprocess/extract_zod.py +++ b/dataprocess/extract_zod.py @@ -25,7 +25,7 @@ from dataprocess.misc_data import create_reading_index from linefit import ground_seg -GROUNDSEG_config = f"{BASE_DIR}/conf/ground/zod.toml" +GROUNDSEG_config = f"{BASE_DIR}/conf/others/zod.toml" def process_log(data_dir: Path, log_id: str, output_dir: Path, n: Optional[int] = None) : diff --git a/dataprocess/misc_data.py b/dataprocess/misc_data.py index e7727e2..b292941 100644 --- a/dataprocess/misc_data.py +++ b/dataprocess/misc_data.py @@ -3,6 +3,10 @@ from pathlib import Path from tqdm import tqdm +from typing import Tuple, cast + +NDArrayInt = np.typing.NDArray[np.int64] + def check_h5py_file_exists(h5py_file: Path, timestamps: list, verbose: bool = False) -> bool: if not h5py_file.exists(): return False @@ -52,6 +56,32 @@ def create_reading_index(data_dir: Path, flow_inside_check=False): pickle.dump(data_index, f) print(f"Create {pkl_file_name} index Successfully, cost: {time.time() - start_time:.2f} s") +def find_closest_integer_in_ref_arr( + query_int: int, ref_arr: NDArrayInt +) -> Tuple[int, int, int]: + """Find the closest integer to any integer inside a reference array, and the corresponding difference. + + In our use case, the query integer represents a nanosecond-discretized timestamp, and the + reference array represents a numpy array of nanosecond-discretized timestamps. + + Instead of sorting the whole array of timestamp differences, we just + take the minimum value (to speed up this function). + + Args: + query_int: query integer, + ref_arr: Numpy array of integers + + Returns: + integer, representing the closest integer found in a reference array to a query + integer, representing the integer difference between the match and query integers + """ + closest_ind = np.argmin(np.absolute(ref_arr - query_int)) + closest_int = cast( + int, ref_arr[closest_ind] + ) # mypy does not understand numpy arrays + int_diff = np.absolute(query_int - closest_int) + return closest_ind, closest_int, int_diff + class SE2: def __init__(self, rotation: np.ndarray, translation: np.ndarray) -> None: diff --git a/envsftool.yaml b/envsftool.yaml index 887457a..4dc1f09 100644 --- a/envsftool.yaml +++ b/envsftool.yaml @@ -24,7 +24,7 @@ dependencies: - av2==0.2.1 - waymo-open-dataset-tf-2.11.0==1.5.0 - zod==0.5.0 - - truckscenes-devkit + - truckscenes-devkit==1.0.0 # Qingwen's public pkg: - dztimer - dufomap==1.1.0 From 472e62d917cd07fbc2f2835c63baa39fbc4fc187 Mon Sep 17 00:00:00 2001 From: Qingwen Date: Sat, 6 Sep 2025 20:28:24 +0100 Subject: [PATCH 3/5] feat(truckscenes): finished the version and test successfully. style: remove/comment all debug check, style to the final run version. --- assets/tests/hdbscan_speed.py | 2 +- assets/view/av2.json | 19 ----------- assets/view/demo.json | 19 +++++++++++ dataprocess/extract_truckscenes.py | 53 +++++++++++++----------------- tools/visualization.py | 2 +- 5 files changed, 44 insertions(+), 51 deletions(-) delete mode 100644 assets/view/av2.json create mode 100644 assets/view/demo.json diff --git a/assets/tests/hdbscan_speed.py b/assets/tests/hdbscan_speed.py index f92580d..b839b2c 100644 --- a/assets/tests/hdbscan_speed.py +++ b/assets/tests/hdbscan_speed.py @@ -25,7 +25,7 @@ def vis(pc0, labels, title='HDBSCAN'): # visualize - vis = MyVisualizer(view_file=f'{BASEF_DIR}/assets/view/av2.json', window_title=title) + vis = MyVisualizer(view_file=f'{BASEF_DIR}/assets/view/demo.json', window_title=title) pcd = o3d.geometry.PointCloud() num_points = pc0.shape[0] pcd.points = o3d.utility.Vector3dVector(pc0) diff --git a/assets/view/av2.json b/assets/view/av2.json deleted file mode 100644 index b1bf1bb..0000000 --- a/assets/view/av2.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "class_name" : "ViewTrajectory", - "interval" : 29, - "is_loop" : false, - "trajectory" : - [ - { - "boundingbox_max" : [ 211.125, 117.1875, 20.53125 ], - "boundingbox_min" : [ -215.25, -166.625, -3.392578125 ], - "field_of_view" : 90.0, - "front" : [ -0.78264077936294429, -0.0063155949007044129, 0.62244158259166249 ], - "lookat" : [ 16.764474584958553, 0.042979235705968843, -6.3527807404873249 ], - "up" : [ 0.62244710369604039, 0.0012897206826678895, 0.78266080757948497 ], - "zoom" : 0.080000000000000002 - } - ], - "version_major" : 1, - "version_minor" : 0 -} \ No newline at end of file diff --git a/assets/view/demo.json b/assets/view/demo.json new file mode 100644 index 0000000..ddead1e --- /dev/null +++ b/assets/view/demo.json @@ -0,0 +1,19 @@ +{ + "class_name" : "ViewTrajectory", + "interval" : 29, + "is_loop" : false, + "trajectory" : + [ + { + "boundingbox_max" : [ 215.05035400390625, 80.301795959472656, 34.890830993652344 ], + "boundingbox_min" : [ -211.76950073242188, -125.19444274902344, -4.2829980850219727 ], + "field_of_view" : 90.0, + "front" : [ -0.58999680455306347, -0.15524870408634167, 0.79233932787454509 ], + "lookat" : [ 9.9369846496341676, 13.322769196068615, -16.164870152242376 ], + "up" : [ 0.79499677526385637, 0.059685607254847638, 0.60367023746967163 ], + "zoom" : 0.17999999999999999 + } + ], + "version_major" : 1, + "version_minor" : 0 +} \ No newline at end of file diff --git a/dataprocess/extract_truckscenes.py b/dataprocess/extract_truckscenes.py index 33caaf9..d856213 100644 --- a/dataprocess/extract_truckscenes.py +++ b/dataprocess/extract_truckscenes.py @@ -45,7 +45,7 @@ GROUNDSEG_config = f"{PARENT_DIR}/conf/others/truckscenes.toml" # NOTE(Qingwen): we only select 2x64 long-range LiDARs for processing, you can add more sensors if needed # The timestamp reference will always be the last item in this list -SelectedSensor = ['LIDAR_LEFT', 'LIDAR_RIGHT'] # 2x64 +SelectedSensor = ['LIDAR_RIGHT', 'LIDAR_LEFT'] # 2x64 # SelectedSensor = ['LIDAR_TOP_FRONT', 'LIDAR_TOP_LEFT', 'LIDAR_TOP_RIGHT', 'LIDAR_REAR'] # 4x32 # SelectedSensor = ['LIDAR_LEFT', 'LIDAR_RIGHT', 'LIDAR_TOP_FRONT', 'LIDAR_TOP_LEFT', 'LIDAR_TOP_RIGHT', 'LIDAR_REAR'] # all 6 LiDARs @@ -89,8 +89,12 @@ def compute_flow_simple(data_fn, pc0, pose0, pose1, ts0, ts1, sample_ann_list, d ann_vel = ann_vel @ pose0[:3,:3] cls = ann.name - points_in_box_mask = points_in_box(ann, world_pc0[:,:3].T, wlh_factor=1.2) + # extend box length according to velocity, ref HiMo. + ann.wlh[1] = ann.wlh[1] + (np.linalg.norm(ann_vel) * delta_t / 2) + ann.wlh[2] = ann.wlh[2] + 0.2 # some truck top missing points, extend a bit + points_in_box_mask = points_in_box(ann, world_pc0[:,:3].T, wlh_factor=1.1) classes[points_in_box_mask] = CATEGORY_TO_INDEX[DataNameMap[cls]] + if np.sum(points_in_box_mask) > 5: obj_flow = np.ones_like(pc0[points_in_box_mask,:3]) * ann_vel * delta_t flow[points_in_box_mask] += obj_flow @@ -177,17 +181,6 @@ def compute_flow_simple(data_fn, pc0, pose0, pose1, ts0, ts1, sample_ann_list, d points = remove_ego_points(points, length_threshold=2.0, width_threshold=7.0) is_ground_0 = np.array(mygroundseg.run(points[:, :3])) - # # debug visualization - # import open3d as o3d - # pcd = o3d.geometry.PointCloud() - # pcd.points = o3d.utility.Vector3dVector(points[:,:3]) - # # pcd.colors = o3d.utility.Vector3dVector(np.zeros((points.shape[0], 3))) - # show_lits = [pcd, o3d.geometry.TriangleMesh.create_coordinate_frame(size=1)] # ego frame - # for lidar_id_ in range(lidar_center.shape[0]): - # show_lits.append(o3d.geometry.TriangleMesh.create_coordinate_frame(size=1).transform(lidar_center[lidar_id_])) - # o3d.visualization.draw_geometries(show_lits) - # exit(0) - if cnt == len(full_sweep_data_dict[SelectedSensor[-1]]) - 1: group = f.create_group(str(ts0)) create_group_data(group=group, pc=points, gm=is_ground_0.astype(np.bool_), pose=pose0, \ @@ -234,23 +227,23 @@ def process_logs(data_mode, data_dir: Path, scene_list: list, output_dir: Path, args = sorted([(data_mode, data_dir, scene_num_id, output_dir) for scene_num_id in range(len(scene_list))]) print(f'Using {nproc} processes') - # for debug - for x in tqdm(args[1:]): - proc(x, ignore_current_process=True) - break - - # if nproc <= 1: - # for x in tqdm(args): - # proc(x, ignore_current_process=True) - # else: - # with Pool(processes=nproc) as p: - # res = list(tqdm(p.imap_unordered(proc, args), total=len(scene_list), ncols=100)) + # # for debug + # for x in tqdm(args[1:]): + # proc(x, ignore_current_process=True) + # break + + if nproc <= 1: + for x in tqdm(args): + proc(x, ignore_current_process=True) + else: + with Pool(processes=nproc) as p: + res = list(tqdm(p.imap_unordered(proc, args), total=len(scene_list), ncols=100)) def main( data_dir: str = "/home/kin/data/truckscenes/man-truckscenes", mode: str = "v1.0-mini", output_dir: str ="/home/kin/data/truckscenes/h5py", - nproc: int = 1, + nproc: int = (multiprocessing.cpu_count() - 1), only_index: bool = False, split_name = None ): @@ -274,12 +267,12 @@ def main( elif mode == 'v1.0-mini': train_scenes = splits.mini_train val_scenes = splits.mini_val - # input_dict = { - # 'train': train_scenes, - # 'val': val_scenes - # } + input_dict = { + 'train': train_scenes, + 'val': val_scenes + } # NOTE(Qingwen): or if you don't want to split mini, use below - input_dict = {'mini': train_scenes + val_scenes} + # input_dict = {'mini': train_scenes + val_scenes} else: raise ValueError('unknown') diff --git a/tools/visualization.py b/tools/visualization.py index e1d5e41..67a3e43 100644 --- a/tools/visualization.py +++ b/tools/visualization.py @@ -30,7 +30,7 @@ from src.utils.o3d_view import MyVisualizer, MyMultiVisualizer, color_map, create_bev_square -VIEW_FILE = f"{BASE_DIR}/assets/view/av2.json" +VIEW_FILE = f"{BASE_DIR}/assets/view/demo.json" def check_flow( data_dir: str ="/home/kin/data/av2/preprocess/sensor/mini", From a49615228281a23d759e6f0d94eb5c202e54e632 Mon Sep 17 00:00:00 2001 From: Qingwen Date: Sat, 6 Sep 2025 22:06:47 +0100 Subject: [PATCH 4/5] !hotfix(pose): the pose save type must be float64! do not astype 32 before you finished the ego_motion compute. * spend half-hour to find... what's wrong with pose_flow.... outside extraction. --- conf/others/zod.toml | 2 +- dataprocess/extract_nus.py | 2 +- dataprocess/extract_truckscenes.py | 12 ++++++------ 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/conf/others/zod.toml b/conf/others/zod.toml index ab5e85b..3dfdcc4 100644 --- a/conf/others/zod.toml +++ b/conf/others/zod.toml @@ -23,7 +23,7 @@ max_start_height = 0.2 # Maximum heigh of starting line to be labelled ground. max_fit_error = 0.1 # maximum error of a point during line fit. [T_RMSE: will sqaure inside code.] # 判断是否是地面点的时候，需要满足的条件： -max_dist_to_line = 0.2 # maximum vertical distance of point to line to be considered ground. [T_d_ground] +max_dist_to_line = 0.25 # maximum vertical distance of point to line to be considered ground. [T_d_ground] line_search_angle = 2.0 # How far to search for a line in angular direction [rad]. [general] diff --git a/dataprocess/extract_nus.py b/dataprocess/extract_nus.py index 28bb677..1718176 100644 --- a/dataprocess/extract_nus.py +++ b/dataprocess/extract_nus.py @@ -73,7 +73,7 @@ def get_pose(data_fn, sweep_data, w2stf=True): world2ego = data_fn.get('ego_pose', sweep_data['ego_pose_token']) # without considering the sensor to ego transform, we will do it outside if multiple sensors are used. if not w2stf: - return transform_matrix(world2ego['translation'], Quaternion(world2ego['rotation'])) + return transform_matrix(world2ego['translation'], Quaternion(world2ego['rotation'])).astype(np.float64) ego2lidar = data_fn.get('calibrated_sensor', sweep_data['calibrated_sensor_token']) ego2lidar_np = transform_matrix(ego2lidar['translation'], Quaternion(ego2lidar['rotation'])) diff --git a/dataprocess/extract_truckscenes.py b/dataprocess/extract_truckscenes.py index d856213..dcc3008 100644 --- a/dataprocess/extract_truckscenes.py +++ b/dataprocess/extract_truckscenes.py @@ -196,15 +196,15 @@ def compute_flow_simple(data_fn, pc0, pose0, pose1, ts0, ts1, sample_ann_list, d curr_scene_ann = mants.get_boxes(sweep_data['token']) scene_flow = compute_flow_simple(mants, points, pose0, pose1, ts0, ts1, curr_scene_ann, dclass, DataNameMap=ManNamMap) - create_group_data(group=group, pc=points, gm=is_ground_0.astype(np.bool_), pose=pose0.astype(np.float32), \ + create_group_data(group=group, pc=points, gm=is_ground_0.astype(np.bool_), pose=pose0, \ lidar_id=lidar_id, lidar_center=lidar_center, \ flow_0to1=scene_flow['flow_0_1'], flow_valid=scene_flow['valid_0'], flow_category=scene_flow['classes_0'], \ flow_instance=scene_flow['flow_instance_id'], - ego_motion=scene_flow['ego_motion'].astype(np.float32)) + ego_motion=scene_flow['ego_motion']) else: # no annotations, only save data - create_group_data(group=group, pc=points, gm=is_ground_0.astype(np.bool_), pose=pose0.astype(np.float32), \ + create_group_data(group=group, pc=points, gm=is_ground_0.astype(np.bool_), pose=pose0, \ lidar_id=lidar_id, lidar_center=lidar_center, \ - ego_motion=(np.linalg.inv(pose1) @ pose0).astype(np.float32)) + ego_motion=npcal_pose0to1(pose0, pose1)) def proc(x, ignore_current_process=False): if not ignore_current_process: @@ -228,9 +228,9 @@ def process_logs(data_mode, data_dir: Path, scene_list: list, output_dir: Path, print(f'Using {nproc} processes') # # for debug - # for x in tqdm(args[1:]): + # for x in tqdm(args[:2]): # proc(x, ignore_current_process=True) - # break + # # break if nproc <= 1: for x in tqdm(args): From 4c3d0da8ef9966d35cb9b4f0d843f1c06f461396 Mon Sep 17 00:00:00 2001 From: Qingwen Date: Sun, 7 Sep 2025 08:46:40 +0100 Subject: [PATCH 5/5] docs(README): update readme and slurm process script for users . todo need check on the dufo label for truckscene later since it have multi-lidar. --- assets/slurm/0_process.sh | 37 ------------------------------ assets/slurm/data-process.sh | 44 ++++++++++++++++++++++++++++++++++++ dataprocess/README.md | 9 ++++---- 3 files changed, 49 insertions(+), 41 deletions(-) delete mode 100644 assets/slurm/0_process.sh create mode 100644 assets/slurm/data-process.sh diff --git a/assets/slurm/0_process.sh b/assets/slurm/0_process.sh deleted file mode 100644 index 44481f2..0000000 --- a/assets/slurm/0_process.sh +++ /dev/null @@ -1,37 +0,0 @@ -#!/bin/bash -#SBATCH -J pack_data -#SBATCH --gpus 0 -#SBATCH --cpus-per-task 64 -#SBATCH --mem 256G -#SBATCH --mincpus=64 -#SBATCH -t 1-00:00:00 -#SBATCH --mail-type=END,FAIL -#SBATCH --mail-user=qingwen@kth.se -#SBATCH --output /proj/berzelius-2023-154/users/x_qinzh/workspace/OpenSceneFlow/logs/slurm/%J_data.out -#SBATCH --error /proj/berzelius-2023-154/users/x_qinzh/workspace/OpenSceneFlow/logs/slurm/%J_data.err - -PYTHON=/proj/berzelius-2023-154/users/x_qinzh/mambaforge/envs/sftool/bin/python -export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/proj/berzelius-2023-154/users/x_qinzh/mambaforge/lib -cd /proj/berzelius-2023-364/users/x_qinzh/workspace/OpenSceneFlow -# export HYDRA_FULL_ERROR=1 - - -$PYTHON dataprocess/extract_av2.py --nproc 64 \ - --av2_type sensor \ - --data_mode train \ - --argo_dir /proj/berzelius-2023-154/users/x_qinzh/av2 \ - --output_dir /proj/berzelius-2023-364/users/x_qinzh/data/av2/h5py - -$PYTHON dataprocess/extract_av2.py --nproc 64 \ - --av2_type sensor \ - --data_mode val \ - --argo_dir /proj/berzelius-2023-154/users/x_qinzh/av2 \ - --output_dir /proj/berzelius-2023-364/users/x_qinzh/data/av2/h5py \ - --mask_dir /proj/berzelius-2023-154/users/x_qinzh/av2/3d_scene_flow - -$PYTHON dataprocess/extract_av2.py --nproc 64 \ - --av2_type sensor \ - --data_mode test \ - --argo_dir /proj/berzelius-2023-154/users/x_qinzh/av2 \ - --output_dir /proj/berzelius-2023-364/users/x_qinzh/data/av2/h5py \ - --mask_dir /proj/berzelius-2023-154/users/x_qinzh/av2/3d_scene_flow \ No newline at end of file diff --git a/assets/slurm/data-process.sh b/assets/slurm/data-process.sh new file mode 100644 index 0000000..d132e91 --- /dev/null +++ b/assets/slurm/data-process.sh @@ -0,0 +1,44 @@ +#!/bin/bash +#SBATCH -J dataprocess +#SBATCH -p berzelius-cpu +#SBATCH --cpus-per-task 64 +#SBATCH --mem 128G +#SBATCH --mincpus=64 +#SBATCH -t 2-00:00:00 +#SBATCH --mail-type=END,FAIL +#SBATCH --mail-user=qingwen@kth.se +#SBATCH --output /proj/berzelius-2023-154/users/x_qinzh/OpenSceneFlow/logs/slurm/%J_data.out +#SBATCH --error /proj/berzelius-2023-154/users/x_qinzh/OpenSceneFlow/logs/slurm/%J_data.err + +PYTHON=/proj/berzelius-2023-154/users/x_qinzh/mambaforge/envs/sftool/bin/python +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/proj/berzelius-2023-154/users/x_qinzh/mambaforge/lib +cd /proj/berzelius-2023-154/users/x_qinzh/OpenSceneFlow/ +# export HYDRA_FULL_ERROR=1 + + +# =============== Argoverse2 +# $PYTHON dataprocess/extract_av2.py --nproc 64 \ +# --av2_type sensor \ +# --data_mode train \ +# --argo_dir /proj/berzelius-2023-154/users/x_qinzh/av2 \ +# --output_dir /proj/berzelius-2023-364/users/x_qinzh/data/av2/h5py + +# $PYTHON dataprocess/extract_av2.py --nproc 64 \ +# --av2_type sensor \ +# --data_mode val \ +# --argo_dir /proj/berzelius-2023-154/users/x_qinzh/av2 \ +# --output_dir /proj/berzelius-2023-364/users/x_qinzh/data/av2/h5py \ +# --mask_dir /proj/berzelius-2023-154/users/x_qinzh/av2/3d_scene_flow + +# $PYTHON dataprocess/extract_av2.py --nproc 64 \ +# --av2_type sensor \ +# --data_mode test \ +# --argo_dir /proj/berzelius-2023-154/users/x_qinzh/av2 \ +# --output_dir /proj/berzelius-2023-364/users/x_qinzh/data/av2/h5py \ +# --mask_dir /proj/berzelius-2023-154/users/x_qinzh/av2/3d_scene_flow + +# =============== MAN-Truckscenes +$PYTHON dataprocess/extract_truckscenes.py --nproc 12 \ + --data_dir /proj/berzelius-2023-364/users/x_qinzh/data/truckscenes/raw_data/trainval/man-truckscenes \ + --mode v1.0-trainval \ + --output_dir /proj/berzelius-2023-364/data/truckscenes/h5py \ No newline at end of file diff --git a/dataprocess/README.md b/dataprocess/README.md index 07e41c1..419098b 100644 --- a/dataprocess/README.md +++ b/dataprocess/README.md @@ -164,14 +164,15 @@ truckscene/ — samples/ — sweeps/ — v1.0-mini/ + — v1.0-trainval/ ``` #### Dataset frames | Dataset | # Total Scene | # Total Frames | | ------- | ------------- | -------------- | -| train | - | - | -| val | - | - | +| train | 524 | 101902 / 20380 (w. gt) | +| val | 75 | 14625 / 2925 (w. gt) | @@ -185,7 +186,7 @@ This directory contains the scripts to preprocess the datasets into `.h5` files. - `extract_zod.py`: Process the datasets in ZOD. - `extract_truckscene.py`: Process the datasets in TruckScene. -Example Running command: +Example Running command, you can also check our [slurm data-process script](../assets/slurm/data-process.sh) for more details.: ```bash # av2: python dataprocess/extract_av2.py --av2_type sensor --data_mode train --argo_dir /home/kin/data/av2 --output_dir /home/kin/data/av2/h5py @@ -197,7 +198,7 @@ python dataprocess/extract_waymo.py --mode train --flow_data_dir /home/kin/data/ python dataprocess/extract_nus.py --mode v1.0-trainval --output_dir /home/kin/data/nus/h5py/full --nproc 24 # truckscene: -python dataprocess/extract_truckscene.py TODO +python dataprocess/extract_truckscene.py --data_dir /home/kin/data/man-truckscenes --mode v1.0-mini --output_dir /home/kin/data/nus/h5py ```