Skip to content

Commit fdfa66c

Browse files
author
Ashkan Vedadi Gargary
committed
new scripts for setup
1 parent 49e01be commit fdfa66c

File tree

5 files changed

+69
-0
lines changed

5 files changed

+69
-0
lines changed

paper_reproduced/scripts/README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,13 @@ Two sample datasets are included in the `dataset` folder. Large datasets (used i
1212
## Setup:
1313
**RapidJSON**, **simdjson**, **cuJSON**, and **pison** are already available in the related_works directory. No need for further installation. For **cudf** and **GPJSON**, it requires to install their library and create their enviroment to be used which are also explained in the `related_works/gpjson` and `related_works/cudf`'s `readme.md` section.
1414

15+
We provide a script to setup gpjson and cudf too. You can use:
16+
```
17+
./setup.sh
18+
```
19+
20+
It contains three scripts, `./setup/install_gpjson.sh` will download GraalVM (a dependency for gpjson) as well as clone and build gpjson. It's output also advises on what environment variables need to be set. `./setup/setup_cudf_env.sh` will setup cudf within a conda environment following steps from their documentation (https://github.com/rapidsai/cudf). `./setup/clean.sh` will just clean up the installation of gpjson, graalvm, and remove the cudf_env conda environment.
21+
1522
## 📊 Run all experiments
1623
We have added a new script called `run_experiments.sh`, which executes all experiments and generates the final figures and tables presented in the paper. For other methods, their libraries are already included in the `related_works` folder. If you need for further manually scritps checking and runnign per each figure you can follow the following subsections.
1724

paper_reproduced/scripts/setup.sh

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
#!/bin/bash
2+
3+
# Run all figure scripts sequentially
4+
./setup/clean.sh
5+
./setup/setup_cudf_env.sh
6+
./setip/install_gpjson.sh
7+
8+
9+
echo "✅ Setup Successfully!"
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#!/bin/bash -l
2+
3+
rm -rf graalvm-ce-java8-21.0.0.2/ gpjson/
4+
conda env remove -n cudf_env -y
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#!/bin/bash -l
2+
3+
# Notes:
4+
# Must have cuda installed (tested with toolkit 12.8)
5+
# Must have GPU enabled machine (tested on H100)
6+
7+
module purge
8+
module load cuda/12.8
9+
10+
# Download GraalVM
11+
if [ ! -d "graalvm-ce-java8-21.0.0.2" ]; then
12+
wget https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-21.0.0.2/graalvm-ce-java8-linux-amd64-21.0.0.2.tar.gz
13+
tar xvf graalvm-ce-java8-linux-amd64-21.0.0.2.tar.gz
14+
rm graalvm-ce-java8-linux-amd64-21.0.0.2.tar.gz
15+
fi
16+
17+
export JAVA_HOME=$PWD/graalvm-ce-java8-21.0.0.2
18+
export GRAALVM_HOME=$PWD/graalvm-ce-java8-21.0.0.2
19+
export PATH=$PWD/graalvm-ce-java8-21.0.0.2/bin:$PATH
20+
21+
which java
22+
java -version
23+
24+
if [ ! -d "gpjson" ]; then
25+
git clone https://github.com/koesie10/gpjson
26+
fi
27+
cd gpjson
28+
./gradlew copyToGraalVM -PgraalVMDirectory=$GRAALVM_HOME
29+
cd ..
30+
31+
echo -e "\n\n\n"
32+
echo "gpjson was installed"
33+
echo "Due to GraalVM being a dependency, modify the following environment variables:"
34+
echo "export JAVA_HOME=$PWD/graalvm-ce-java8-21.0.0.2"
35+
echo "export GRAALVM_HOME=$PWD/graalvm-ce-java8-21.0.0.2"
36+
echo "export PATH=$PWD/graalvm-ce-java8-21.0.0.2/bin:\$PATH"
37+
echo -e "\n\n"
38+
echo "This was tested on a machine using CUDA 12.8 and an H100 GPU"
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/bin/bash -l
2+
3+
module purge
4+
module load miniconda3
5+
module load cuda/12.8
6+
7+
conda create -n cudf_env -c rapidsai -c conda-forge cudf=25.08
8+
9+
echo "Conda environment for cudf created"
10+
echo "Run 'conda activate cudf_env' to activate"
11+

0 commit comments

Comments
 (0)