Lighteval documentation
Lighteval
You are viewing main version, which requires installation from source. If you'd like
regular pip install, checkout the latest stable version (v0.9.2).
Lighteval
π€ Lighteval is your all-in-one toolkit for evaluating Large Language Models (LLMs) across multiple backends with ease. Dive deep into your modelβs performance by saving and exploring detailed, sample-by-sample results to debug and see how your models stack up.
Key Features
π Multi-Backend Support
Evaluate your models using the most popular and efficient inference backends:
transformers
: Evaluate models on CPU or one or more GPUs using π€ Acceleratenanotron
: Evaluate models in distributed settings using β‘οΈ Nanotronvllm
: Evaluate models on one or more GPUs using π VLLMcustom
: Evaluate custom models (can be anything)sglang
: Evaluate models using SGLang as backendinference-endpoint
: Evaluate models using Hugging Faceβs Inference Endpoints APItgi
: Evaluate models using π Text Generation Inference running locallylitellm
: Evaluate models on any compatible API using LiteLLMinference-providers
: Evaluate models using HuggingFaceβs inference providers as backend**: Distributed training and evaluation
π Comprehensive Evaluation
- Extensive Task Library: 1000s pre-built evaluation tasks
- Custom Task Creation: Build your own evaluation tasks
- Flexible Metrics: Support for custom metrics and scoring
- Detailed Analysis: Sample-by-sample results for deep insights
π§ Easy Customization
Customization at your fingertips: create new tasks, metrics or model tailored to your needs, or browse all our existing tasks and metrics.
βοΈ Seamless Integration
Seamlessly experiment, benchmark, and store your results on the Hugging Face Hub, S3, or locally.
Quick Start
Installation
pip install lighteval
Basic Usage
# Evaluate a model using Transformers backend
lighteval accelerate \
"model_name=openai-community/gpt2" \
"leaderboard|truthfulqa:mc|0"
Save Results
# Save locally
lighteval accelerate \
"model_name=openai-community/gpt2" \
"leaderboard|truthfulqa:mc|0" \
--output-dir ./results
# Push to Hugging Face Hub
lighteval accelerate \
"model_name=openai-community/gpt2" \
"leaderboard|truthfulqa:mc|0" \
--push-to-hub \
--results-org your-username