Entwickler-Tools
LM Evaluation Harness
LM Evaluation Harness — EleutherAI framework for few-shot evaluation of language models across many tasks, MIT
★ 14,4 Tsd. SterneMIT
Anleitung ansehen →LibreAI / Entwickler-Tools / evalscope
// devtools · modelscope/evalscope
evalscope
evalscope ist ein Open-Source-Projekt aus der Kategorie Entwickler-Tools von modelscope (modelscope/evalscope auf GitHub). Es hat 3.509 Sterne auf GitHub und 509 Forks, wird unter der Lizenz Apache-2.0 veröffentlicht und wurde zuletzt am 8. Oktober 2026 auf GitHub aktualisiert.
A streamlined framework for evaluating and benchmarking LLMs, VLMs and AIGC models.
★ 3,5 Tsd. Sterne⑂ 509 ForksApache-2.0aktualisiert Okt 2026
▦ Visual — das README dieses Projekts enthält keinen einbettbaren Screenshot,
darum hier ein Platzhalter. Die Anleitung unten ist trotzdem vollständig.
darum hier ein Platzhalter. Die Anleitung unten ist trotzdem vollständig.
// projektdaten
Projektdaten
- Repository
- modelscope/evalscope
- GitHub-Sterne
- 3.509
- Forks
- 509
- Lizenz
- Apache-2.0
- Zuletzt auf GitHub aktualisiert
- 8. Oktober 2026
- Kategorie
- Entwickler-Tools
- Eintragstyp
- Vollständige Installationsanleitung
- Offizielle Website
- Official site
// so funktioniert es
So funktioniert es
- Clarify your goal
Decide the first concrete outcome you want from it, so you can pick the right example, template or mode instead of exploring everything at once.
- Get the project
Clone the repository (or install the published package) and skim its README quick start for the requirements for your platform.
- Install and configure
Run the documented install step and set any required configuration — such as provider keys, data sources or local services — before the first run.
- Run your first task
Execute the smallest documented example end to end, confirm the output is what you expected, then adapt it to your real data or workflow.
// funktionen
Warum es so beliebt ist
Comprehensive benchmarks
Industry-recognised benchmarks including MMLU, C-Eval and GSM8K built in, with custom dataset support.
Multi-modal, multi-domain
Evaluate LLMs, VLMs, embedding, reranker and AIGC models in one framework.
Multiple backends
Native backend plus OpenCompass, VLMEvalKit and RAGEval integrations.
Agent evaluation mode
Drive benchmarks such as GSM8K, AIME and SWE-bench Agentic inside a controlled multi-turn AgentLoop with Docker sandbox and full traces.
Performance testing
Stress-test model services with TTFT, TPOT and throughput metrics, recordable to wandb/swanlab.
Reports and arena
Interactive web dashboard for comparison, plus pairwise Arena mode to rank models head-to-head.
Advertisement
// selbst ausführen
Führe es selbst aus
- Clone the repository
Get the source so you can inspect, configure and run it locally.
git clone https://github.com/modelscope/evalscope.git cd evalscope - Install EvalScope
Install the package from PyPI.
pip install evalscope - Evaluate a model API
Run a first benchmark against an OpenAI-compatible endpoint.
evalscope eval --model your-model-name --api-url $OPENAI_API_BASE_URL --api-key $OPENAI_API_KEY --eval-type openai_api --datasets gsm8k --limit 5
// anwendungsfälle
Was du damit bauen kannst
- Compare two candidate models before a production rollout
- Benchmark a fine-tuned model against its base on GSM8K and MMLU
- Stress-test a new inference deployment for TTFT and throughput
- Evaluate an agent on SWE-bench-style tasks with full traces
Advertisement
// häufige fragen
Häufige Fragen
Who maintains evalscope?
It is maintained by modelscope on GitHub (modelscope/evalscope), where the source, issues and releases live.
Is it free to use?
Yes — it is free and open source under Apache-2.0, which permits use, modification and commercial redistribution, with licence/notice retention.
Does it need a paid API key?
For API models you supply that provider's key; local models downloaded from ModelScope need no paid key, just compute.
How does it compare to OpenCompass or lm-eval used standalone?
Compared with OpenCompass or lm-eval used standalone, evalscope's distinguishing focus is: a streamlined framework for evaluating and benchmarking LLMs, VLMs and AIGC models. Pick it when that focus matches your need; the alternative may suit you better if its broader or different scope is the priority.
How do I get started?
Clone https://github.com/modelscope/evalscope (or install its published package), follow the README quick start to install and configure it, then run the smallest documented example — the Run section on this page mirrors those steps.
// weiter entdecken
Ähnliche Tools
Entwickler-Tools
ai-game-devtools
Resource hub for AI-powered game development — LLMs, world models, agents.
★ 1,4 Tsd. SterneMIT
Anleitung ansehen →Entwickler-Tools
LlamaFactory
Unified, efficient fine-tuning of 100+ LLMs and VLMs with a CLI and Web UI — ACL 2024
★ 75,4 Tsd. SterneApache-2.0
Anleitung ansehen →Entwickler-Tools
ms-swift
Fine-tune and deploy 600+ LLMs and 300+ multimodal models with PEFT or full-parameter training.
★ 15,8 Tsd. SterneApache-2.0
Anleitung ansehen →Entwickler-Tools
little-coder
A coding-agent harness optimized for smaller local LLMs — small models, serious coding work
★ 2,7 Tsd. SterneApache-2.0
Anleitung ansehen →Entwickler-Tools
backend.ai
Backend.AI is a streamlined, container-based computing cluster platform.
★ 674 SterneLGPL-3.0
Anleitung ansehen →// alternativen
Alternativen im Bereich Entwickler-Tools
Die anderen Projekte der Kategorie Entwickler-Tools mit den meisten Sternen im LibreAI-Katalog — die realistische Auswahl, falls evalscope nicht das Richtige ist.
- OpenCode★ 211,6 Tsd.
The open source coding agent
- Transformers★ 166,9 Tsd.
The model-definition framework behind modern AI — thousands of pretrained transformers for text, vision, and audio.
- claude-code★ 149,7 Tsd.
Anthropic's agentic coding tool that lives in your terminal and works across your whole codebase
- cc-switch★ 141,3 Tsd.
Cross-platform desktop all-in-one manager for Claude Code, Codex, OpenCode and other coding agents
- codex★ 128,2 Tsd.
OpenAI's lightweight coding agent that runs locally in your terminal
// mehr in devtools
Mehr in Entwickler-Tools
Verwandte Projekte der Kategorie Entwickler-Tools im Katalog, die evalscope bei den Sternen am nächsten stehen — zum Weiterstöbern auf beiden Seiten.
- twinny★ 3,7 Tsd.
Open-source AI coding assistant for VS Code. Code completion, chat, edits and reviews with local or hosted models. Your models, your infrastructure.
- AnyCrawl★ 3,5 Tsd.
A Node.js crawler that turns websites into LLM-ready data and extracts structured SERP results.
- mirage★ 3,7 Tsd.
A virtual terminal and filesystem that lets AI agents work across S3, Drive, Slack, Gmail and more.
- SocratiCode★ 3,3 Tsd.
Enterprise-grade codebase intelligence for AI agents — hybrid semantic search over 40M+ LOC, locally.
- abtop★ 3,7 Tsd.
Like htop, but for AI coding agents — watch Claude Code and Codex sessions in real time.
- sentrux★ 3,3 Tsd.
A real-time architectural sensor, in pure Rust, that helps AI agents keep code quality from slipping.
Hol dir evalscope
Kostenlos und offen — hol dir den Quellcode, gib dem Repository einen Stern und leg los.