// agents · idavidrein/gpqa

gpqa

gpqa is an open-source ai agents project by idavidrein (idavidrein/gpqa on GitHub). It has 540 GitHub stars and 58 forks, is released under the MIT license, and was last updated on GitHub on 30 September 2024.

A graduate-level benchmark even Google can't crack.

★ 540 stars⑂ 58 forksMITupdated 2024-09

View on GitHub ↗

▦ visual — this project's README ships no embeddable screenshot,
so here's a placeholder. The guide below is complete regardless.

// project facts

Project facts

Repository: idavidrein/gpqa
GitHub stars: 540
Forks: 58
License: MIT
Last updated on GitHub: 30 September 2024
Category: AI Agents
Listing: Full install guide

// how it works

How it works

Start with a use case
Pick one concrete task so you can judge whether the project fits.
Fetch the source
Clone the repo and check the prerequisites listed in the README.
Set up the environment
Install what it needs and configure credentials or model access.
Try the sample workflow
Run a minimal example end to end before wiring it into your stack.

// features

Why people love it

Graduate-level difficulty

Questions written by domain experts.

Google-proof

Resistant to simple search-based solving.

Reasoning-focused

Tests deep understanding, not trivia.

Influential

Widely cited in frontier-model evals.

Notebook format

Explore via Jupyter.

MIT licensed

Permissive licensing.

// run it

Run it yourself

Step 1 — Clone the repo
Download the source and move into the project folder.
```
git clone https://github.com/idavidrein/gpqa.git
cd gpqa
```
Step 2 — Install
Set up a Python environment with the notebook dependencies.
```
pip install -r requirements.txt
```
Step 3 — Explore the data
Open the notebooks and inspect the questions.
```
jupyter notebook
```

// use cases

What you can build

Evaluate a model on hard science questions.
Stress-test reasoning beyond memorization.
Compare frontier models on difficulty.
Study benchmark design for evals.

// faq

Questions, answered

Who maintains GPQA?

It is maintained by idavidrein at idavidrein/gpqa, with 58 forks. The last push was September 2024.

Is GPQA free to use?

Yes. It is released under the MIT license, which permits commercial use, modification, and redistribution. Read the LICENSE file for the exact terms.

Do I need a paid API to use it?

The dataset is free; you need model access to evaluate against it.

How does it compare to MMLU?

GPQA is deliberately harder and more reasoning-focused; MMLU is broader.

How do I get started?

The most-starred other AI Agents projects in the LibreAI catalogue — the realistic shortlist if gpqa is not the right fit.

openclaw★ 391.6k
A personal AI that really does things — on any OS, with your data staying yours.
superpowers★ 294.9k
An agentic skills framework & software development methodology that works.
ECC★ 275.2k
Agent harness optimization system with skills, memory and security for Claude Code, Codex and more
hermes-agent★ 252.0k
The agent that grows with you — a Python framework for adaptive AI agents.
deepseek-harness★ 245.5k
DeepSeek's plugin-based agent harness where everything is a plugin.

// more in agents

Get gpqa

Free and open — grab the source, star the repo, and start creating.

View on GitHub ↗← More AI Agents tools

gpqa

Project facts

How it works

Why people love it

Graduate-level difficulty

Google-proof

Reasoning-focused

Influential

Notebook format

MIT licensed

Run it yourself

What you can build

Questions, answered

Similar tools

SWE-bench_Pro-os

MMLU-Pro

SWE-bench

notfair-plugin

google_workspace_mcp

agentsight

Alternatives in AI Agents

More in AI Agents

Get gpqa