This repository contains the code and pretrained models of

Mutual Learning to Adapt for Joint Human Parsing and Pose Estimation [PDF]
Xuecheng Nie, Jiashi Feng, and Shuicheng Yan
European Conference on Computer Vision (ECCV), 2018

• Python 3.5
• Pytorch 0.2.0
• OpenCV 3.0 or higher

1. Install Pytorch: Please follow the official instruction on installation of Pytorch.
2. Clone the repository

   git clone --recursive https://github.com/NieXC/pytorch-mula.git 

3. Download Look into Person (LIP) dataset and create symbolic links to the following directories

   ln -s PATH_TO_LIP_TRAIN_IMAGES_DIR dataset/lip/train_images ln -s PATH_TO_LIP_VAL_IMAGES_DIR dataset/lip/val_images ln -s PATH_TO_LIP_TEST_IMAGES_DIR dataset/lip/testing_images ln -s PATH_TO_LIP_TRAIN_SEGMENTATION_ANNO_DIR dataset/lip/train_segmentations ln -s PATH_TO_LIP_VAL_SEGMENTATION_ANNO_DIR dataset/lip/val_segmentations 

Run the following command to train the model from scratch (Default: 5-stage Hourglass based network):

sh run_train.sh 

or

CUDA_VISIBLE_DEVICES=0,1 python main.py -b 24 --lr 0.0015 

A simple way to record the training log by adding the following command:

2>&1 | tee exps/logs/mula_lip.log 

Some configurable parameters in training phase:

• --arch network architecture (HG (Hourglass) or VGG)
• -b mini-batch size
• --lr initial learning rate (0.0015 for HG based model and 0.0001 for VGG based model)
• --epochs total number of epochs for training
• --snapshot-fname-prefix prefix of file name for snapshot, e.g. if set '--snapshot-fname-prefix exps/snapshots/mula_lip', then 'mula_lip.pth.tar' (latest model), 'mula_lip_pose_best.pth.tar' (model with best validation PCK for human pose estimation) and 'mula_lip_parsing_best.pth.tar' (model with best validation mIoU for human parsing) will be generated in the folder 'exps/snapshots'
• --resume path to the model for recovering training
• -j number of workers for loading data
• --print-freq print frequency

Run the following command to evaluate the model on LIP validation set:

sh run_test.sh 

or

CUDA_VISIBLE_DEVICES=0 python main.py --evaluate True --calc-pck True --calc-miou True --resume exps/snapshots/mula_lip.pth.tar 

Run the following command to evaluate the model on LIP testing set:

CUDA_VISIBLE_DEVICES=0 python main.py --evaluate True --resume exps/snapshots/mula_lip.pth.tar --eval-data dataset/lip/testing_images --eval-anno dataset/lip/jsons/LIP_SP_TEST_annotations.json 

In particular, human pose estimation results will be saved as a .csv file followed the official evaluation format of LIP dataset for single-person human pose estimation. An example is provided in exps/preds/pose_results/pred_keypoints_lip.csv. Human parsing results will be saved as a set of .png images in the folder exps/preds/parsing_results, representing body part label maps for testing images.

Some configurable parameters in testing phase:

• --evaluate True for testing and false for training
• --resume path to the model for evaluation
• --calc-pck calculate PCK or not for validation of human pose estimation
• --calc-miou calculate mIoU or not for validation of human parsing
• --pose-pred-path path to the csv file for saving the human pose estimation results
• --parsing-pred-dir directory to the png images for saving the human parsing results
• --visualization visualize evaluation or not
• --vis-dir directory for saving the visualization results

The models generated with this code can be downloaded here: GoogleDrive. Training logs can also be found at the same folder for the reference.

If you use our code in your work or find it is helpful, please cite the paper:

@inproceedings{nie2018mula, title={Mutual Learning to Adapt for Joint Human Parsing and Pose Estimation}, author={Nie, Xuecheng and Feng, Jiashi and Yan, Shuicheng}, booktitle={ECCV}, year={2018} }