Community maintained hardware plugin for vLLM on Intel Gaudi

A safetensors extension to efficiently store sparse quantized tensors on disk

TPU inference for vLLM, with unified JAX and PyTorch support.

A high-throughput and memory-efficient inference and serving engine for LLMs

Community maintained hardware plugin for vLLM on Spyre

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

This repo hosts code for vLLM CI & Performance Benchmark infrastructure.

Intelligent Router for Mixture-of-Models

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

Community maintained hardware plugin for vLLM on Ascend