MarS3D: A Plug-and-Play Motion-Aware Model for Semantic Segmentation on Multi-Scan 3D Point Clouds

Jiahui Liu^1*, Chirui Chang^1*, Jianhui Liu¹, Xiaoyang Wu¹, Lan Ma², Xiaojuan Qi^1†,

¹The University of Hong Kong  ²TCL AI Lab

*equal contribution  ⁺corresponding author

CVPR 2023

MarS3D is a plug-and-play motion-aware module for semantic segmentation on multi-scan 3D point clouds. Extensive experiments show that MarS3D can improve the performance of the baseline model by a large margin.

video | arXiv | paper

• A plug-and-play module, which can be flexibly integrated with mainstream single-scan segmentation models.
• A Cross-Frame Feature Embedding for temporal information preservation.
• A BEV-based Motion-Aware Feature Learning module to exploit temporal information and enhance the model’s motion awareness.

• Performance improvement over baselines on SemanticKITTI public multi-scan validation set.

  Method	mIoU	#param
  SPVCNN	49.70%	21.8M
  SPVCNN+MarS3D	54.66%	21.9M
  SparseUNet	48.99%	39.2M
  SparseUNet+MarS3D	54.64%	39.3M
  MinkUNet	48.47%	37.9M
  MinkUNet+MarS3D	54.71%	38.0M

• Comparison with the state-of-the-art models on SemanticKITTI multi-scan benchmark.

  Method	mIoU
  SpSequenceNet	43.1%
  TemporalLidarSeg	47.0%
  TemporalLatticeNet	47.1%
  Meta-RangeSeg	49.5%
  KPConv	51.2%
  SPVCNN	49.2%
  SPVCNN+MarS3D	52.7%

• Ubuntu: 18.04+
• CUDA: 11.x

conda create -n mars3d python=3.9 -y conda activate mars3d conda install ninja -y conda install pytorch torchvision torchaudio cudatoolkit -c pytorch -y # follow the instructions from the official pytorch website conda install -c anaconda h5py pyyaml -y conda install -c conda-forge sharedarray tensorboardx yapf addict einops scipy plyfile -y pip install tqdm pip install numpy==1.23 # for fixing version conflict, should < 1.25

# Please follow the official instructions from the official repo for different backbones # Minkowski Engine: https://github.com/NVIDIA/MinkowskiEngine # spconv: https://github.com/traveller59/spconv # torchsparse: https://github.com/mit-han-lab/torchsparse # torchsparse: an optional installation without sudo apt install conda install google-sparsehash -c bioconda export C_INCLUDE_PATH=${CONDA_PREFIX}/include:$C_INCLUDE_PATH export CPLUS_INCLUDE_PATH=${CONDA_PREFIX}/include:CPLUS_INCLUDE_PATH pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git

Semantic-Kitti

• Download the Semantic-Kitti
• The dataset file structure is listed as follows:

./ ├── ├── ... └── path_to_semantic_kitti/ ├──sequences ├── 00/ │ ├── velodyne/ | |	├── 000000.bin | |	├── 000001.bin | |	└── ... │ └── labels/ | ├── 000000.label | ├── 000001.label | └── ... ├── 08/ # 08 is the validation set ├── 11/ # 11-21 is the test set └── ... └── 21/ 

• Symlink the paths to them as follows:

mkdir data ln -s /path_to_semantic_kitti data/semantic_kitti 

sh scripts/train.sh -p INTERPRETER_PATH -d DATASET -c CONFIG_NAME -n EXP_NAME 

for example:

sh scripts/train.sh -p python -d semantic_kitti -c spvcnn-mh -n mars3d-spvcnn 

For calling test script, exp folder generated in training process start by training script is needed.

Furthermore, currently, the weight path should be specified in test.py

sh scripts/test.sh -p INTERPRETER_PATH -d DATASET -c CONFIG_NAME -n EXP_NAME 

for example:

sh scripts/test.sh -p python -d semantic_kitti -c spvcnn-mh -n mars3d-spvcnn 

If you find this project useful in your research, please consider cite:

@inproceedings{liu2023mars3d, title={MarS3D: A Plug-and-Play Motion-Aware Model for Semantic Segmentation on Multi-Scan 3D Point Clouds}, author={Liu, Jiahui and Chang, Chirui and Liu, Jianhui and Wu, Xiaoyang and Ma, Lan and Qi, Xiaojuan}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={9372--9381}, year={2023} }