This repo is the PyTorch implementation of hand mesh reconstruction described in CMR and MobRecon.

• 2023-11-2. Add MobRecon (SpiralConv verison) model for more robust real-world applications. If you target on freely captured images, please try ./cmr/scripts/demo_mobrecon.sh.
• 2022-4-28. Wrap old-version code in cmr, including CMR demo/training/evaluation and Mobrecon demo/evaluation. Add mobrecon to release MobRecon training.
• 2021-12-7. Add MobRecon demo.
• 2021-6-10. Add Human3.6M dataset.
• 2021-5-20. Add CMR-G model.

• SpiralNet++
• Sub-pose aggregation
• Adaptive 2D-1D registration for mesh-image alignment
• DenseStack for 2D encoding
• Feature lifting with MapReg and PVL
• DSConv as an efficient mesh operator
• Complement data will be available here
• MobRecon training with consistency learning and complement data

• Environment

  conda create -n handmesh python=3.9 conda activate handmesh 

• Please follow official suggestions to install pytorch and torchvision. We use pytorch=1.11.0-cuda11.3, torchvision=0.12.0
• Requirements

  pip install -r requirements.txt 

  If you have difficulty in installing torch_sparse etc., please follow this link.
• Install MPI-IS Mesh from the source
• You should accept MANO LICENCE. Download MANO model from official website, then run

  ln -s /path/to/mano_v1_2/MANO_RIGHT.pkl template/MANO_RIGHT.pkl 

• Download the files you need from Google drive or Baidu cloud.

• Prepare pre-trained models as

  cmr/out/Human36M/cmr_g/checkpoints/cmr_pg_res18_human36m.pt cmr/out/FreiHAND/cmr_g/checkpoints/cmr_g_res18_moredata.pt cmr/out/FreiHAND/cmr_sg/checkpoints/cmr_sg_res18_freihand.pt cmr/out/FreiHAND/cmr_pg/checkpoints/cmr_pg_res18_freihand.pt cmr/out/FreiHAND/mobrecon_spconv/checkpoints/mobrecon_densestack.pt 

• Run

  ./cmr/scripts/demo_cmr.sh ./cmr/scripts/demo_mobrecon.sh 

  The prediction results will be saved in output directory, e.g., out/FreiHAND/mobrecon/demo.

• Explaination of the output

  

  • In an JPEG file (e.g., 000_plot.jpg), we show silhouette, 2D pose, projection of mesh, camera-space mesh and pose
  • As for camera-space information, we use a red rectangle to indicate the camera position, or the image plane. The unit is meter.
  • If you run the demo, you can also obtain a PLY file (e.g., 000_mesh.ply).
    • This file is a 3D model of the hand.
    • You can open it with corresponding software (e.g., Preview in Mac).
    • Here, you can get more 3D details through rotation and zoom in.

• Please download FreiHAND dataset from this link, and create a soft link in data, i.e., data/FreiHAND.
• Download mesh GT file freihand_train_mesh.zip, and unzip it under data/FreiHAND/training

• The official data is now not avaliable. Please follow I2L repo to download it.
• Download silhouette GT file h36m_mask.zip, and unzip it under data/Human36M.

• Please download the dataset from this link, and create a soft link in data, i.e., data/Ge.

• See this file for complement data. Then, create a soft link in data, i.e., data/CompHand.

${ROOT} |-- data | |-- FreiHAND | | |-- training | | | |-- rgb | | | |-- mask | | | |-- mesh | | |-- evaluation | | | |-- rgb | | |-- evaluation_K.json | | |-- evaluation_scals.json | | |-- training_K.json | | |-- training_mano.json | | |-- training_xyz.json | |-- Human3.6M | | |-- images | | |-- mask | | |-- annotations | | |-- J_regressor_h36m_correct.npy | |-- Ge | | |-- images | | |-- params.mat | | |-- pose_gt.mat | |-- Compdata | | |-- base_pose | | |-- trans_pose_batch1 | | |-- trans_pose_batch2 | | |-- trans_pose_batch3 

./cmr/scripts/eval_cmr_freihand.sh ./cmr/scripts/eval_mobrecon_freihand.sh 

• JSON file will be saved as out/FreiHAND/cmr_sg/cmr_sg.json. You can submmit this file to the official server for evaluation.

./cmr/scripts/eval_cmr_human36m.sh 

We re-produce the following results after code re-organization.

./cmr/scripts/train_cmr_freihand.sh ./cmr/scripts/train_cmr_human36m.sh ./mobrecon/scripts/train_mobrecon.sh 

A experiment log will be saved under cmr/out or mobrecon/out

@inproceedings{bib:CMR, title={Camera-Space Hand Mesh Recovery via Semantic Aggregationand Adaptive 2D-1D Registration}, author={Chen, Xingyu and Liu, Yufeng and Ma, Chongyang and Chang, Jianlong and Wang, Huayan and Chen, Tian and Guo, Xiaoyan and Wan, Pengfei and Zheng, Wen}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year={2021} } @inproceedings{bib:MobRecon, title={MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image}, author={Chen, Xingyu and Liu, Yufeng and Dong Yajiao and Zhang, Xiong and Ma, Chongyang and Xiong, Yanmin and Zhang, Yuan and Guo, Xiaoyan}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year={2022} }

Our implementation of SpiralConv is based on spiralnet_plus.

We also thank hand-graph-cnn, I2L-MeshNet_RELEASE, detectron2, smplpytorch(https://github.com/gulvarol/smplpytorch) for inspiring implementations.