The repo is the official implemenation of Cinematic Behavior Transfer via NeRF-based Differentiable Filming.
We have VirtualFilmStudio to showcase some of our explorations in filmmaking directions.
git clone https://github.com/VirtualFilmStudio/Cinetransfer.gitWe recommend you use anaconda to run our CinematicTransfer.
Please refer to SLAHMR work to install env. Notes:
- SLAHMR project have update, please use release branch.
- Use SLAHMR - Google Colab version to install is better. Conda env name will set to slahmr.
Prepare test data. Please place data according to the following framework.
|-Cinetransfer | |-camera_optim | |-data | |-torch_ngp | |-slahmr | | |-demo | | | |-videos | | | | |-arc2.mp4 |...Preprocess data by using SLAHMR.
cd {YOUR_ROOT}/Cinetransfer/slahmr/slahmr # delete slahmr default test dataset rm ../demo/videos/022691_mpii_test.mp4 # copy test data to slahmr demo cp ../../data/videos/arc2.mp4 ../demo/videos/ # preprocess data by using SLAHMR # If the server is not configured with virtual graphics, it is recommended to set 'run_vis=False' source activate slahmr && python run_opt.py data=video data.seq=arc2 data.root={YOUR_ROOT}/Cinetransfer/slahmr/demo run_opt=True run_vis=TrueAfter execution, an outputs folder will be generated. You can find a folder starting with arc2, which contains optimized smpl.
mkdir {YOUR_ROOT}/Cinetransfer/data/arc2 cd {YOUR_ROOT}/Cinetransfer/camera_optim # Copy the slahmr preprocessed data to the Cinetransfer/data folder and adjust the data structure source activate slahmr && python trans_slahmr_to_nerf.py --src_root {YOUR_ROOT}/Cinetransfer/slahmr/demo --dst_root {YOUR_ROOT}/Cinetransfer/data # Generate mask data source activate slahmr && python filmingnerf/preproc/gen_mask.py --data data --seq_name arc2 --data_path {YOUR_ROOT}/CinetransferCopy {YOUR_ROOT}/Cinetransfer/slahmr/outputs/.../arc2-xxxx/motion_chunks/arc2_{MAX_NUM}_world_results.npz to the {YOUR_ROOT}/Cinetransfer/data/arc2/ folder and rename it to arc2.npz.
conda deactivate cd {YOUR_ROOT}/Cinetransfer git clone --recursive https://github.com/ashawkey/torch-ngp.git mv torch-ngp torch_ngp cd torch_ngp conda create -n cinetrans python=3.9 -y source activate cinetrans && pip install -r requirements.txt source activate cinetrans && conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia # install the tcnn backbone source activate cinetrans && pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch # install all extension modules bash scripts/install_ext.sh cd raymarching source activate cinetrans && python setup.py build_ext --inplace # install to python path (you still need the raymarching/ folder, since this only install the built extension.) source activate cinetrans && pip install . Notes:
- If you have any problems installing
torch-ngpenvironment .You can refer to torch-ngp. - Rename
{YOUR_ROOT}/Cinetransfer/torch-ngpto{YOUR_ROOT}/Cinetransfer/torch_ngp.
cd {YOUR_ROOT}/Cinetransfer/camera_optim source activate cinetrans && sudo apt-get update source activate cinetrans && sudo apt install xvfb source activate cinetrans && pip install einops chardet source activate cinetrans && pip install pyrender source activate cinetrans && pip install smplx[all] source activate cinetrans && pip install imageio[ffmpeg] imageio[pyav]Open {YOUR_ROOT}/Cinetransfer/torch_ngp/dnerf/utils.py, in the file top add this
import os import sys sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))source activate cinetrans && xvfb-run --auto-servernum --server-num=1 python run_train_nerf.py --seq_name arc2By using camera_ui to optim first_frame.
cd {YOUR_ROOT}/Cinetransfer/camera_optim source activate cinetrans && pip install gradio source activate cinetrans && pip install plotly # temp visual result can be found in {out_dir}/{seq_name}/cache_first_pose.png source activate cinetrans && python run_camera_ui.py --seq_name arc2 --out_dir ../data/cam_optim_resWhen open the UI, please follow those steps to check first pose:
- Step1. Adjust the slide rail on the left to control the Camera pose.
- Step2. Click the 'render' button to view the rendering results.
- Step3. Finally click the 'optim' button to optimize a single frame.
Note: You can repeat the above process until you select the most suitable camera pose.
cd {YOUR_ROOT}/Cinetransfer/camera_optim source activate cinetrans && python run_camera_opt.py --seq_name arc2 --out_dir ../data/cam_optim_rescd {YOUR_ROOT}/Cinetransfer/camera_optim # smooth (optional) source activate cinetrans && xvfb-run --auto-servernum --server-num=1 python run_camera_smooth.py --seq_name arc2 --out_dir ../data/cam_optim_res source activate cinetrans && xvfb-run --auto-servernum --server-num=1 python run_camera_vis.py --seq_name arc2 --out_dir ../data/cam_optim_res
If you find this paper and repo useful for your research, please consider citing our paper.
@article{jiang2023cinematic, title={Cinematic Behavior Transfer via NeRF-based Differentiable Filming}, author={Jiang, Xuekun and Rao, Anyi and Wang, Jingbo and Lin, Dahua and Dai, Bo}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, year={2024} }The code is based on SLAHMR, torch-ngp, thanks to them!