Benchmark Code

[szeyu]

Benchmark

Allow users to test on themselves to get the benchmark of model(s) on different backend. It will analyse the Token In / Out throughput for you in a statistical manner

Benchmark a Model

To benchmark a model, run this

• --backend cpu | ipex | openvino | directml
• --model_name Name of the Model
• --model_path Path to Model | Model Repo ID
• --token_in Number of Input Tokens (Max 2048)
• --token_out Number of Output Tokens

python ellm_benchmark.py --backend <cpu | ipex | openvino | directml> --model_name <Name of the Model> --model_path <Path to Model | Model Repo ID> --token_in <Number of Input Tokens (Max 2048)> --token_out <Number of Output Tokens>

Loop to benchmark the models

Customise your benchmarking config

# Define the models model_names = [ # model names ] # Define the model paths model_paths = [ # path to model in order to model names / model repo id ] # Define the token length token_in_out = [ (1024, 1024), (1024, 512), (1024, 256), (1024, 128), (512, 1024), (512, 512), (512, 256), (512, 128), (256, 1024), (256, 512), (256, 256), (256, 128), (128, 1024), (128, 512), (128, 256), (128, 128), ] # Choose backend backend = "cpu" backend = "directml" backend = "ipex" backend = "openvino" # Number of loops loop_count = 20

python loop_ellm_benchmark.py

Generate a Report (XLSX) of a Model's Benchmark

To Generate report for a model, run this

• --model_name Name of the Model

python analyse_detailed_benchmark.py --model_name <Name of the Model>

Generate Reports (XLSX) of Models' Benchmark

List out the models that you want to have report of benchmarking

model_names = [ # model names ]

python loop_analyse_detailed_benchmark.py