A software-in-the-loop simulator for lane keeping assist system (LKAS) using Vrep and C++. If you are using this framework please refer to the following paper:

[1] S. Mohamed, S. De, K. Bimpisidis, V, Nathan, D. Goswami, H. Corporaal, and T. Basten, “IMACS: A Framework for Performance Evaluation of Image Approximation in a Closed-loop System,” in: MECO, 2019.

The controller is designed in Matlab and the control matrices \Phi, \Gamma, and control gains K (and F) are copied to the src/LaneDetection_and_Control/lateralcontrol_multiple.cpp file. The LKAS controller is designed as explained in the following papers:

[2] S. Mohamed, A. U. Awan, D. Goswami, T. Basten, "Designing image-based control systems considering workload variations," in: CDC, 2019. [3] S. Mohamed, D. Zhu, D. Goswami, T. Basten, "Optimising quality-of-control for data-intensive multiprocessor image-based control systems considering workload variations," in: DSD, 2018. 

An initial framework using Vrep and Matlab has been presented in:

S. Mohamed, D. Zhu, D. Goswami, T. Basten, "DASA: an open-source design, analysis and simulation framework for automotive image-based control systems," in: MCAA Annual Conference, 2019. 

Tested using the following versions. If you would like to use latest versions, you might have to edit the syntax of some functions. Tested with the following versions:

• OS: Ubuntu 18.04
• g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0

1. Install dependent libraries
2. Install OpenCV
3. Install Eigen
4. Install VREP
5. Run IMACS_LKAS

It is suggested that you Fork this repository by clicking the "fork" button in the top right. This lets you have your own working copy.

Clone the forked (or directly this) repository:

git clone https://github.com/sajid-mohamed/cppVrepLKAS.git cd cppVrepLKAS pwd 

For brevity, $(root)=pwd, i.e. the path to cppVrepLKAS is called as $(root).

The following libraries might be needed for successful execution.

sudo apt-get install libtinfo-dev sudo apt-get install libjpeg-dev sudo apt-get install libtiff5-dev sudo apt-get install libpng-dev sudo apt-get install jasper sudo apt-get install libgtk-3-dev sudo apt-get install libopenblas-dev sudo apt-get install -y libavcodec-dev sudo apt-get install -y libavformat-dev sudo apt-get install libavutil-dev sudo apt-get install libswscale-dev sudo apt-get install valgrind # For openCV installation sudo apt-get install cmake ## qt4 sudo apt-get install qt4-qmake sudo apt-get install libqt4-dev 

If opencv is already installed, go to step 2.2.

For detailed OpenCV installation: follow steps from https://docs.opencv.org/trunk/d7/d9f/tutorial_linux_install.html.

cd $(root)/externalApps git clone https://github.com/opencv/opencv.git git clone https://github.com/opencv/opencv_contrib.git 

externalApps is a folder already available when you git clone cppVrepLKAS. It might be better to include all the external apps in this folder.

# Building OpenCV from Source Using CMake cd opencv mkdir build cd build cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local .. -DOPENCV_GENERATE_PKGCONFIG=ON make sudo make install 

Note the path of opencv*.pc file. This is the PKG_CONFIG_PATH. Typically this should be /usr/lib/pkgconfig. If you are using virtual machine this might be /usr/local/lib/pkgconfig.

In case you have errors with ccache (re-)install ccache as shown in FAQ 11.

If you have issues with java, see FAQ 12 and FAQ 13.

Make sure opencv*.pc file is in the /usr/lib/pkgconfig or change the PKG_CONFIG_PATH to point to this file. To change path follow the steps in FAQ 3 or FAQ 4 depending on whether you have sudo rights. Remember the path to your opencv installation as OPENCV_PATH.

To check if opencv*.pc is in the /usr/lib/pkgconfig:

cd /usr/lib/pkgconfig [ -f /usr/lib/pkgconfig/opencv*.pc ] && echo "File exist" 

You could also use dlocate to find the file.

If you cannot find it, you might not have done make install in step 2.1.

If the above does not work, try:

sudo apt-get install libopencv-dev 

cd $(root)/externalApps git clone https://gitlab.com/libeigen/eigen.git 

The path to eigen is needed later for EIGEN_PATH.

Download the linux version from https://coppeliarobotics.com/downloads.

The license to CoppeliaSim should be obtained from their website.

If you are in an academic institution, you could simply download the Edu version.

Make sure you copy the downloaded file (e.g. CoppeliaSim_Edu_V4_0_0_Ubuntu18_04.tar.xz) to the $(root)/externalApps folder.

cp CoppeliaSim_Edu_V4_0_0_Ubuntu18_04.tar.xz $(root)/externalApps cd $(root)/externalApps mkdir vrep tar -xvf CoppeliaSim_Edu_V4_0_0_Ubuntu18_04.tar.xz -C vrep --strip-components=1 

Generally for local tar file:

tar -xvf FILENAME -C FOLDER --strip-components=1 

The --strip-components flag is used when a tar file would naturally expand itself into a folder, let say, like github where it examples to repo-name-master folder. Of course you wouldn’t need the first level folder generated here so --strip-components set to 1 would automatically remove that first folder for you. The larger the number is set the deeper nested folders are removed.

The VREP_PATH is then $(root)/externalApps/vrep.

Please change the paths in the following files to the corresponding paths in your own system.

$(root)/cppVrepLKAS.pro $(root)/src/LaneDetection_and_Control/lane_detection.cpp 

1. If the external applications vrep, eigen or opencv are installed to another directory than the $(root)/externalApps, change the corresponding OPENCV_PATH, PKG_CONFIG_PATH, EIGEN_PATH, and VREP_PATH in cppVrepLKAS.pro to the corresponding PATH in your system.

To obtain the path, you could run pwd in a terminal opened from the corresponding folder.

2. Change line 121 of lane_detection.cpp to string out_string = "$(root)/out_imgs";.

cd $(root) bash run.sh 

The steps included in the bash script run.sh are the following:

Generate the Makefile using qmake.

cd $(root) qmake cppVrepLKAS.pro make 

When you want to run make in $(root) always run qmake cppVrepLKAS.pro first. make clean cleans the files generated by this make.

Open a vrep-scene. Vrep needs to be started before running IMACS framework.

cd $VREP_PATH ./coppeliaSim.sh ../../vrep-scenes/SmallBias_report.ttt 

If you are using an older version of vrep (coppeliaSim.sh is not found), please use the following command:

./vrep.sh ../../vrep-scenes/EnterCurveTest.ttt 

cd $(root) # Usage: ./imacsLKAS {version} {simstep} # Minimum one argument needed. (default values) ./imacs 1 0.005 ./imacsLKAS 1 

There might be some dialog windows opening in VREP if you are running it for the first time. This is normal.

You do not have to always close vrep to run imacs again. Just stop simulation in vrep and wait for sometime before executing ./imacsLKAS 1 0.005.

In case you encounter some other errors, you can debug using the following command:

valgrind ./imacsLKAS 1 

The main file: src/cpp_vrep_api/cpp_vrep_framework.cpp

• The simulation step time, the total simulation time, the initial wait time, the sampling period(s) and delay(s) are hardcoded in this file.
• If you want to add additional input arguments for imacsLKAS, you should do this here.

Controller Design: src/LaneDetection_and_Control/lateralcontrol_multiple.cpp

• The controller matrices are stored in this file. \Phi, \Gamma and K for multiple controllers are stored in the variables m_phi_aug, m_Gamma_aug and m_K2c.
• The controller currently used is an optimal LQR control for a regulation problem where the objective is for the output to reach zero. If you want to change the type of controller you can edit the function lateralController::compute_steering_angles().

Image Processing: src/LaneDetection_and_Control/lane_detection.cpp

• This file contains the image processing steps. Currently, we do the steps mentioned in [1]. You should be changing the functions in this file to include your own image processing algorithm.

matrix.hpp [-Wreorder] GivensQR.hpp [-Wreorder] extApiPlatform.c [-Wunused-result] 

$(ROOT)/src/cpp_vrep_api/my_vrep_api.cpp:59: vrepAPI::vrepAPI(): Assertion `m_clientID != -1 && "V-REP must be started: ./vrep.sh ../../vrep-scenes/EnterCurveTest.ttt"' failed. 

OpenCV Error: Assertion failed (mv && n > 0) in merge, file /home/om/hd/infra/build_opencv/opencv/modules/core/src/convert.cpp, line 327 terminate called after throwing an instance of 'cv::Exception' what(): /home/om/hd/infra/build_opencv/opencv/modules/core/src/convert.cpp:327: error: (-215) mv && n > 0 in function merge 

cd sudo gedit .bash_profile 

Add the export path commands in `.bash_profile' as shown below.

export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/lib/pkgconfig/ 

Save and close .bash_profile. Then, open .bashrc.

sudo gedit .bashrc 

Add the following line at the end of .bashrc.

source ~/.bash_profile 

Save and close the file. Now, the paths should be exported everytime you open a terminal.

If you want to export paths permanently, Open a terminal window using Ctrl+Alt+T Run the command

gedit ~/.profile 

Add the line(s) to the bottom of this file and save.

export PKG_CONFIG_PATH=/usr/lib/pkgconfig/ 

Log out and log in again. A safer way is to use quotes. Doing so is necessary if one or more directories in the original PATH contain spaces. E.g.:

export CLANG="$(root)/clang+llvm-9.0.0-x86_64-linux-gnu-ubuntu-18.04/bin/clang" 

Alternatively, you can set the paths in paths.sh and then source paths. Do not forget to include export command in the description here.

cd $(IMACSROOT) source paths.sh 

Change line 43 in src/cpp_vrep_api/cpp_vrep_framework.cpp if you want to simulate for a longer time, i.e. int simulation_time = 100; if you want to logically simulate for 100 sec.

To start a straight road scene (see vrep-scenes/SmallBias_report.ttt), replace line 7 in run.sh with the following

gnome-terminal -- sh -c 'cd externalApps/vrep; ./coppeliaSim.sh ../../vrep-scenes/SmallBias_report.ttt; exec bash' 

If you are making changes to the cppVrepLKAS.pro file, you would have to run the command make clean first in the terminal before running bash run.sh.

make clean bash run.sh 

The lateralcontrol_multiple.cpp has a sample controller implemented.

You are required to adapt the codes according to the control design you choose (especially the lateralController::compute_steering_angles() function). Ideally, you should rewrite the lateralController::compute_steering_angles() function.

The current code is for sequential implementation. To enforce pipelining, you have to change src/cpp_vrep_api/cpp_vrep_framework.cpp:123-134. Especially, have an array of control inputs (steering_angle values) and only actuate corresponding to the sampling period.

To implement the pipelining, you would also have to amend src/cpp_vrep_api/cpp_vrep_framework.cpp:129 to send the corresponding steering_angle instead of the latest one. You might also have to edit VrepAPI.sim_delay(time_step) accordingly.

sudo apt install -y ccache sudo /usr/sbin/update-ccache-symlinks echo 'export PATH="/usr/lib/ccache:$PATH"' | tee -a ~/.bashrc source ~/.bashrc && echo $PATH 

You can follow the tutorial here: https://www.digitalocean.com/community/tutorials/how-to-install-java-with-apt-on-ubuntu-18-04.

sudo apt install default-jre sudo apt install default-jdk sudo update-alternatives --config javac sudo update-alternatives --config java 

Choose the correct jdk version. jre does not have tools.jar Update the JAVA_HOME path

sudo gedit /etc/environment 

At the end of this file, add the following line, making sure to replace the path with your own copied path:

JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64/bin/" 

Modifying this file will set the JAVA_HOME path for all users on your system. Save the file and exit the editor.

Now reload this file to apply the changes to your current session:

source /etc/environment 

If you do not have sudo access, follow the steps in FAQ 4.