GPT2 using OpenVino speed up.
If you have python-3.9 installed you will have to install python-3.7.9 for this project to work. Follow the instructions below when building for first time (verified build on MacOS):
brew install pyenv # for syncing multitple versions on the machine pip3 install virtualenv # virtual-environment maker, can use any other package pyenv install 3.7.9 # install the specific version pyenv local 3.7.9 # set local (this folder) version to 3.7.9 export LOCAL_PY_VER_PATH=`pyenv which python3` # set path for convinience echo $LOCAL_PY_VER_PATH # [opt.] to check the path $LOCAL_PY_VER_PATH -m venv . # using the path above build a virtual environment in this folder source bin/activate # activate the local env pip3 install -r requirements.txt # install run dependencies When coming back to this project simply activate the virtualenv as and the rest will be ready for you:
source bin/activate To get the model in the ONNX format first run the file convert.py, this should dump gpt2.onnx file.
python3 convert.py For this you must first have openvino installed on your system. Download from here. Now I have added most of the requirements in my requirements.txt file, however you should also install those for OpenVino. After that run the following commands to setup environment variables:
export OPENVINO_FOLDER="path/to/openvino_2021" cd $OPENVINO_FOLDER/bin source setupvars.sh cd $OPENVINO_FOLDER/deployment_tools/model_optimizer pip3 install install -r requirements.txt pip3 install install -r requirements_onnx.txt If everything works correctly you will see an output like this:
[setupvars.sh] OpenVINO environment initialized Now come back to this repo, Openvino environment setup works correctly only if you are in the openvino_2021/bin folder. Now we run the script mo_onnx.py:
mo_onnx.py --help # to get meanings of arguments to be passed mkdir full_precision half_precision # full_precision is FP36 and other is FP16 mo_onnx.py --input_model gpt2.onnx \ --data_type=FP32/FP16 \ --output_dir=full_precision/half_precision If everything works correctly you should see 3 files in /fp32 folder:
gpt2.bin gpt2.mapping gpt2.xml To check if everything works fine run the script run.py. You should start seeing the outputs, the following is on the machine with following configuration:
MacBook Pro (13-inch, 2020, Four Thunderbolt 3 ports) Processor: 2 GHz Quad-Core Intel Core i5 Memory: 16 GB 3733 MHz LPDDR4X Graphics: Intel Iris Plus Graphics 1536 MB The performance results are as follows (2x boost):
---------------------------------------------------------------------- Loading Pytorch model :: Pytorch inference in 0.59065s ---------------------------------------------------------------------- Creating Inference Engine... Loading network Loading IR to the plugin... exec_net: <openvino.inference_engine.ie_api.ExecutableNetwork object at 0x12c531fb0> :: OpenVino inference in 0.26206s ---------------------------------------------------------------------- In order to test generation capabilities you can pass --g flag and get the following results:
---------------------------------------------------------------------- Loading Pytorch model Text shape: torch.Size([1, 127]) :: Pytorch inference in 0.46476s ---------------------------------------------------------------------- Testing generation :: Pytorch generation took (40 steps): 17.663s ---------------------------------------------------------------------- Creating Inference Engine... Loading network Loading IR to the plugin... exec_net: <openvino.inference_engine.ie_api.ExecutableNetwork object at 0x130aaffb0> :: OpenVino inference in 0.23262s ---------------------------------------------------------------------- Testing generation :: OpenVino generation took (40 steps): 6.220s ---------------------------------------------------------------------- When running on AWS c5.12xlarge and batching the data to 128 samples in a batch we see larger performance increase.
---------------------------------------------------------------------- Loading Pytorch model Pytorch inference in 3.55126s ---------------------------------------------------------------------- Creating Inference Engine... Loading network Loading IR to the plugin... exec_net: <openvino.inference_engine.ie_api.ExecutableNetwork object at 0x12c531fb0> ---------------------------------------------------------------------- OpenVino inference in 0.78668s ---------------------------------------------------------------------- Which is a 5x boost. Using OpenVino benchmarking tool we saw even more power throughput working at 134.29ms of first inference and 17ms as average processing time across 3522 runs. This is a massive 209x speed improvement.
This proves our hypothesis that larger CPU machines can take advantage of OpenVino's performance in a super-liear fashion.