Installation

Lighteval can be installed from PyPI or from source. This guide covers all installation options and dependencies.

System Requirements

Python: 3.10 or higher
PyTorch: 2.0 or higher (but less than 3.0)
CUDA: Optional, for GPU acceleration

From PyPI

The simplest way to install Lighteval is from PyPI:

pip install lighteval

This installs the core package with all essential dependencies for basic evaluation tasks.

From Source

Source installation is recommended for developers who want to contribute to Lighteval or need the latest features:

git clone https://github.com/huggingface/lighteval.git
cd lighteval
pip install -e .

Optional Dependencies (Extras)

Lighteval provides several optional dependency groups that you can install based on your needs. Use the format pip install lighteval[<group>] or pip install -e .[<group>] for source installation.

Backend Extras

Extra	Description	Dependencies
`vllm`	Use VLLM as backend for high-performance inference	vllm>=0.10.0, ray, more_itertools
`tgi`	Use Text Generation Inference API	text-generation>=0.6.0
`litellm`	Use LiteLLM for unified API access	litellm, diskcache
`optimum`	Use Optimum for optimized models	optimum==1.12.0
`quantization`	Evaluate quantized models	bitsandbytes>=0.41.0, auto-gptq>=0.4.2
`adapters`	Evaluate adapter models (PEFT, Delta)	peft==0.3.0
`nanotron`	Evaluate Nanotron models	nanotron, tensorboardX

Task and Feature Extras

Extra	Description	Dependencies
`extended_tasks`	Extended evaluation tasks	langdetect, openai>1.87, tiktoken
`multilingual`	Multilingual evaluation support	stanza, spacy[ja,ko,th], jieba, pyvi
`math`	Mathematical reasoning tasks	latex2sympy2_extended==1.0.6

Storage and Logging Extras

Extra	Description	Dependencies
`s3`	Upload results to S3	s3fs
`tensorboardX`	Upload results to TensorBoard	tensorboardX
`wandb`	Log results to Weights & Biases	wandb
`trackio`	Log results to Trackio	trackio

Development Extras

Extra	Description	Dependencies
`quality`	Code quality tools	ruff>=v0.11.0, pre-commit
`tests`	Testing dependencies	pytest>=7.4.0, deepdiff
`docs`	Documentation building	hf-doc-builder, watchdog
`dev`	All development dependencies	Includes accelerate, quality, tests, multilingual, math, extended_tasks, vllm

< > Update on GitHub