[New Algorithm Support] PifPaf (#322)

* pifpaf model: 1.pifpaf model added 2.pipaf training process adding * pifpaf training: preprocess finished: 1.pif_map generation 2.paf_map generation * pifpaf training: loss calculation added * pifpaf training: visualization processing added * pifpaf training: 1.pifpaf Config module integration added 3.pifpaf keypoint converter added, MPII left(1) 4.pifpaf parallel training added *pifpaf evaluation: 1pifpaf infer processing added * pifpaf debug: 1.train pre-processing debug finished 2.infer processing debug finished * pifpaf evaluating: 1.inference finished! 2.evaluation progressing 3.visualization progressing * pifpaf evaluation: 1.visualization finished! * pifpaf training debug: 1.debug progressing(1) * pifpaf training debug: 1.debug progressing(2) * pifpaf training debug: 1.debug processing(3) * pifpaf training debug: 1.debug progressing(4) * I.pipaf training debug: 1.debug progressing(5) II.code tidy up: 1.pose proposal * I.pifpaf training debug: 1.debug finished! II.docs tidy up 1.readme accuracy table added! * I.docs tidy-up 1.readme modified. * I.compatibility test 1.tensorflow 2.1.2 compatibity tested * I.pifpaf training 1.pifpaf training adjustment * I.pifpaf training 1.pifpaf training debuged II.resnet50 debug: 1.resnet50 backbone debug finished * I.Pifpaf evaluate: 1.pifpaf infer debug finished 2.pifpaf evaluate process debug finished

Author: Gyx-One

README.md

@@ -29,7 +29,11 @@ HyperPose is a library for building human pose estimation systems that can effic
 
 HyperPose has two key features, which are not available in existing libraries:
 
-- **Flexible training platform**: HyperPose provides flexible Python APIs to build many useful pose estimation models (e.g., OpenPose and PoseProposalNetwork). HyperPose users can, for example, customize data augmentation, use parallel GPUs for training, and replace deep neural networks (e.g., changing from ResNet to MobileNet), thus building models specific to their real-world scenarios.
+- **Flexible training platform**: HyperPose provides flexible Python APIs to provide a customise pipeline for developing various pose estimation models. HyperPose users can:
+* make use of uniform pipelines for train,evaluation,visualization,pre-processing and post-processing across various models (e.g., OpenPose,Pifpaf,PoseProposal Network)
+* customise model and dataset for their own use(e.g. user-defined model,user-defined dataset,mitiple dataset combination)
+* parallel training using multiple GPUs(using *Kungfu* adaptive distribute training library)
+thus building models specific to their real-world scenarios.
 - **High-performance pose estimation**: HyperPose achieves real-time pose estimation though a high-performance pose estimation engine. This engine implements numerous system optimizations: pipeline parallelism, model inference with TensorRT, CPU/GPU hybrid scheduling, and many others. This allows HyperPose to **run 4x FASTER than OpenPose and 10x FASTER than TF-Pose**.
 
 ## Documentation
@@ -81,6 +85,18 @@ We compare the prediction performance of HyperPose with [OpenPose 1.6](https://g
 | OpenPose (MobileNet) | 17.9 MB | 432 x 368 | **84.32 FPS** | 8.5 FPS (TF-Pose) |
 | OpenPose (ResNet18) | 45.0 MB | 432 x 368 | **62.52 FPS** | N/A |
 
+## Accuracy
+We evaluate accuracy of pose estimation models developed by hyperpose (mainly over Mscoco2017 dataset). the development environment is Ubuntu16.04, with 4 V100-DGXs and 24 Intel Xeon CPU. The training procedure takes 1~2 weeks using 1 V100-DGX for each model. (If you want to train from strach, loading the pretrained backbone weight is recommended.)
+
+| HyperPose Configuration | DNN Size | Input Size | Evaluate Dataset | Accuracy-hyperpose (Iou=0.50:0.95) | Accuracy-original (Iou=0.50:0.95) |
+| -------------------- | ---------- | ------------- | ---------------- | --------------------- | ----------------------- |
+| Openpose (vgg19) | 199 MB | 432 x 368 | Mscoco2014(random 1160 images) | 57.0 map | 58.4 map |
+| LightweightOpenpose (dailated mobilenet) | 17.7 MB | 432 x 368 | Mscoco2017(all 5000 images) | 46.1 map | 42.8 map |
+| LightweightOpenpose (mobilenet-thin) | 17.4 MB | 432 x 368 | Mscoco2017(all 5000 images) | 44.2 map | 28.06 map (Mscoco2014) |
+| LightweightOpenpose (tinyvgg) | 23.6 MB | 432 x 368 | Mscoco2017(all 5000 images) | 47.3 map | - |
+| LightweightOpenpose (resnet50) | 42.7 MB | 432 x 368 | Mscoco2017(all 5000 images) | 48.2 map | - |
+| PoseProposal (resnet18) | 45.2 MB | 384 x 384 | MPII(all 2729 images) | 54.9 map (Pckh) | 72.8 map (Pckh)|
+
 </a>
 <p align="center">
  <img src="./docs/markdown/images/demo-xbd.gif", width="600">

hyperpose/Config/__init__.py

@@ -65,6 +65,8 @@ def get_config():
  from .config_mbtopps import model,train,eval,data,log
  elif(update_model.model_type==MODEL.PoseProposal):
  from .config_ppn import model,train,eval,data,log
+ elif(update_model.model_type==MODEL.Pifpaf):
+ from .config_pifpaf import model,train,eval,data,log
  #merge settings with basic configurations
  model.update(update_model)
  train.update(update_train)

hyperpose/Config/config_pifpaf.py

@@ -0,0 +1,56 @@
+import os
+from .define import MODEL,DATA,TRAIN,BACKBONE
+from easydict import EasyDict as edict
+
+#model configuration
+model = edict()
+# number of keypoints + 1 for background
+model.n_pos = 17 
+model.num_channels=128
+# input size during training , 240
+model.hin = 368 
+model.win = 432
+# output size during training (default 46)
+model.hout = 46 
+model.wout = 54
+model.model_type = MODEL.Pifpaf
+model.model_name = "default_name"
+model.model_backbone=BACKBONE.Default
+model.data_format = "channels_first"
+# save directory
+model.model_dir = f"./save_dir/{model.model_name}/model_dir"
+
+#train configuration
+train=edict()
+train.batch_size = 4
+train.save_interval = 2000
+# total number of step
+train.n_step = 1000000
+# initial learning rate 
+train.lr_init = 1e-4
+# evey number of step to decay lr
+train.lr_decay_every_step = 136120 
+# decay lr factor
+train.lr_decay_factor = 0.2
+train.lr_decay_steps=[420000,630000]
+train.weight_decay_factor = 0.0
+train.train_type=TRAIN.Single_train
+train.vis_dir=f"./save_dir/{model.model_name}/train_vis_dir"
+
+#eval configuration
+eval =edict()
+eval.batch_size=8
+eval.vis_dir= f"./save_dir/{model.model_name}/eval_vis_dir"
+
+#data configuration
+data = edict()
+data.dataset_type = DATA.MSCOCO # coco, custom, coco_and_custom
+data.dataset_version = "2017" # MSCOCO version 2014 or 2017
+data.dataset_path = "./data"
+data.dataset_filter=None
+data.vis_dir=f"./save_dir/data_vis_dir"
+
+#log configuration
+log = edict()
+log.log_interval = 100
+log.log_path= f"./save_dir/{model.model_name}/log.txt"

hyperpose/Config/define.py

@@ -15,6 +15,7 @@ class MODEL(Enum):
  LightweightOpenpose=1
  PoseProposal=2
  MobilenetThinOpenpose=3
+ Pifpaf=4
 
 class DATA(Enum):
  MSCOCO=0
@@ -33,5 +34,5 @@ class KUNGFU(Enum):
 
 class OPTIM(Enum):
  Adam=0
- SGD=2
  RMSprop=1
+ SGD=2

hyperpose/Dataset/__init__.py

@@ -56,6 +56,9 @@ def get_dataset(config):
  elif(model_type==MODEL.PoseProposal):
  from .mscoco_dataset.define import ppn_input_converter as input_kpt_cvter
  from .mscoco_dataset.define import ppn_output_converter as output_kpt_cvter
+ elif(model_type==MODEL.Pifpaf):
+ from .mscoco_dataset.define import pifpaf_input_converter as input_kpt_cvter
+ from .mscoco_dataset.define import pifpaf_output_converter as output_kpt_cvter
  dataset=MSCOCO_dataset(config,input_kpt_cvter,output_kpt_cvter)
  dataset.prepare_dataset()
  elif(dataset_type==DATA.MPII):

hyperpose/Dataset/mscoco_dataset/dataset.py

@@ -58,7 +58,7 @@ def visualize(self,vis_num):
  '''
 
  train_dataset=self.get_train_dataset()
- visualize(self.vis_dir,vis_num,train_dataset,self.parts,self.colors,dataset_name="mpii")
+ visualize(self.vis_dir,vis_num,train_dataset,self.parts,self.colors,dataset_name="mscoco")
 
  def get_parts(self):
  return self.parts

hyperpose/Dataset/mscoco_dataset/define.py

@@ -86,4 +86,37 @@ def ppn_output_converter(kpt_list):
  kpts+=[0.0,0.0,0.0]
  else:
  kpts+=[x,y,1.0]
+ return kpts
+
+#convert kpts from pifpaf to mscoco
+from_pifpaf_converter={}
+for part_idx in range(0,len(CocoPart)):
+ from_pifpaf_converter[part_idx]=part_idx
+#convert kpts from mscoco to pifpaf
+to_pifpaf_converter={}
+for part_idx in range(0,len(CocoPart)):
+ to_pifpaf_converter[part_idx]=part_idx
+
+def pifpaf_input_converter(coco_kpts):
+ xs=coco_kpts[0::3]
+ ys=coco_kpts[1::3]
+ vs=coco_kpts[2::3]
+ lost_idx=np.where(vs<=0)[0]
+ xs[lost_idx]=-1000
+ ys[lost_idx]=-1000
+ cvt_kpts=np.array([xs,ys]).transpose()
+ return cvt_kpts
+
+def pifpaf_output_converter(kpt_list):
+ kpts=[]
+ for coco_idx in range(0,len(CocoPart)):
+ flag=False
+ if(coco_idx in to_pifpaf_converter):
+ model_idx=to_pifpaf_converter[coco_idx]
+ x,y=kpt_list[model_idx]
+ if(x>=0 and y>=0):
+ kpts==[x,y,1.0]
+ flag=True
+ if(not flag):
+ kpts+=[0.0,0.0,0.0] 
  return kpts

hyperpose/Model/__init__.py

@@ -77,6 +77,12 @@ def get_model(config):
  from .pose_proposal.utils import get_limbs
  model.parts=get_parts(dataset_type)
  model.limbs=get_limbs(dataset_type)
+ elif(model_type == MODEL.Pifpaf):
+ from .pifpaf.utils import get_parts
+ from .pifpaf.utils import get_limbs
+ model.parts=get_parts(dataset_type)
+ model.limbs=get_limbs(dataset_type)
+
  userdef_parts=config.model.userdef_parts
  userdef_limbs=config.model.userdef_limbs
  if(userdef_parts!=None):
@@ -93,18 +99,21 @@ def get_model(config):
  hin=model.hin,win=model.win,hout=model.hout,wout=model.wout,backbone=backbone,pretraining=pretraining,data_format=model.data_format)
  elif model_type == MODEL.LightweightOpenpose:
  from .openpose import LightWeightOpenPose as model_arch
- ret_model=model_arch(parts=model.parts,n_pos=len(model.parts),limbs=model.limbs,n_limbs=len(model.limbs),num_channels=model.num_channels,hin=model.hin,win=model.win,\
- hout=model.hout,wout=model.wout,backbone=backbone,pretraining=pretraining,data_format=model.data_format)
+ ret_model=model_arch(parts=model.parts,n_pos=len(model.parts),limbs=model.limbs,n_limbs=len(model.limbs),num_channels=model.num_channels,\
+ hin=model.hin,win=model.win,hout=model.hout,wout=model.wout,backbone=backbone,pretraining=pretraining,data_format=model.data_format)
  elif model_type == MODEL.MobilenetThinOpenpose:
  from .openpose import MobilenetThinOpenpose as model_arch
- ret_model=model_arch(parts=model.parts,n_pos=len(model.parts),limbs=model.limbs,n_limbs=len(model.limbs),num_channels=model.num_channels,hin=model.hin,win=model.win,\
- hout=model.hout,wout=model.wout,backbone=backbone,pretraining=pretraining,data_format=model.data_format)
+ ret_model=model_arch(parts=model.parts,n_pos=len(model.parts),limbs=model.limbs,n_limbs=len(model.limbs),num_channels=model.num_channels,\
+ hin=model.hin,win=model.win,hout=model.hout,wout=model.wout,backbone=backbone,pretraining=pretraining,data_format=model.data_format)
  elif model_type == MODEL.PoseProposal:
  from .pose_proposal import PoseProposal as model_arch
  ret_model=model_arch(parts=model.parts,K_size=len(model.parts),limbs=model.limbs,L_size=len(model.limbs),hnei=model.hnei,wnei=model.wnei,lmd_rsp=model.lmd_rsp,\
  lmd_iou=model.lmd_iou,lmd_coor=model.lmd_coor,lmd_size=model.lmd_size,lmd_limb=model.lmd_limb,backbone=backbone,\
  pretraining=pretraining,data_format=model.data_format)
- #print(f"\n!!!test in get_model: parts:{model.parts} limbs:{model.limbs}\n\n")
+ elif model_type == MODEL.Pifpaf:
+ from .pifpaf import Pifpaf as model_arch
+ ret_model=model_arch(parts=model.parts,n_pos=len(model.parts),limbs=model.limbs,n_limbs=len(model.limbs),hin=model.hin,win=model.win,\
+ scale_size=32,pretraining=pretraining,data_format=model.data_format)
  else:
  raise RuntimeError(f'unknown model type {model_type}')
  print(f"using {model_type.name} model arch!")
@@ -143,6 +152,8 @@ def get_train(config):
  from .openpose import single_train,parallel_train
  elif model_type == MODEL.PoseProposal:
  from .pose_proposal import single_train,parallel_train
+ elif model_type == MODEL.Pifpaf:
+ from .pifpaf import single_train,parallel_train
  else:
  raise RuntimeError(f'unknown model type {model_type}')
  print(f"training {model_type.name} model...")
@@ -193,6 +204,8 @@ def get_evaluate(config):
  from .openpose import evaluate
  elif model_type == MODEL.PoseProposal:
  from .pose_proposal import evaluate
+ elif model_type == MODEL.Pifpaf:
+ from .pifpaf import evaluate
  else:
  raise RuntimeError(f'unknown model type {model_type}')
  evaluate=partial(evaluate,config=config)