This project has been my PhD project and is now migrating from a) being a PhD project to a longer term project that ultimately will need more general support and b) from PixelStreaming1 to a different system.

This project focuses on using Synthetic Data for Machine Learning frameworks and provides the infrastructure to make these processes work.

Introduction video: https://www.youtube.com/watch?v=H9cw_aE-l3A

Migration: There will be a living document of the up-to-date considerations, but for broader information, please check out my news posts and specifically the recent highlight about it.

This is a project that contains a framework to couple Simulation with Unreal Engine and Training Toolsets, entirely dependent on the standardized WebRTC itself. Standard compliant frameworks will be supported and we are going to present Demos soon.

You can find information on how this pipeline works here.

• Setup Unreal on a Compute Cluster
• Setup Pixelstreaming
• Setup OpenCV+Horovod+TensorFlow
• Import Data into Unreal
• SynavisUE Commands

An open port and the ability to listen on ports on a system is needed to run the Synavis framework. On most HPC systems, this will be permitted, and within high-throughput networks, port binding might not even be restricted.

In general, a discrete GPU is preferred, and if certain vendor-specific components are to be used, the corresponding GPU is necessary. This includes raytracing (which might require an RTX GPU) or less specific but instrumental chips such as encoding/decoding if H264 or H265 are to be used for the video stream.

This repository currently contains both the Synavis framework as well as the SynavisUE plugin. It aims to provide a central focus and discussion point for both software.

The Synavis framework is the piece of code that orchestrates the data streams between Python and the SynavisUE plugin. The SynavisUE plugin is the piece of code that takes control of the Unreal Engine application during runtime.

You do not need to do extra steps for the building of the SynavisUE plugin. Simply copy the folder into the Plugins/ folder of your project.

The Synavis framework requires CMake and a modern C++ compiler. It will try to seek out present installations of pybind11 and pybind-json, as well as the optional additional library FFmpeg when built for decoding. The easiest way of setting up the environment is either by using a venv or through the addition of a mamba environment.

Here, you can then create a build folder using cmake with cmake -H. -Bbuild and build with cmake --build build --config Release -j N with your number of concurrent build operations $N$.

CPlantBox needs to be built as well if the coupling workflows are to be replicated.

Synavis inaugural description --> in silico Plants: https://doi.org/10.1093/insilicoplants/diad022

Synavis HPC Use-Cases and infrastructural considerations --> Nordic e-Infrastructure Collaboration: https://link.springer.com/chapter/10.1007/978-3-031-86240-3_11

Newest article using Photosynthesis simulation in-the-loop: https://doi.org/10.1093/insilicoplants/diaf018

Encapsulated WebRTC