ResearchClawBench is a benchmark that measures whether AI coding agents can independently conduct scientific research — from reading raw data to producing publication-quality reports — and then rigorously evaluates the results against real human-authored papers.

Unlike benchmarks that test coding ability or factual recall, ResearchClawBench asks: given the same data and tools a human researcher had, can an AI agent arrive at the same (or better) scientific conclusions?

• 2026-03-20 🐈 Added Nanobot as a new agent — ultra-lightweight OpenClaw alternative with reliable multi-step tool execution. Agent config moved to agents.json for easy customization.
• 2026-03-19 🚀 Initial release with Claude Code, Codex CLI, and OpenClaw support. 40 tasks across 10 scientific domains.

Most AI benchmarks evaluate what models know. We evaluate what agents can do.

• Real science, not toy problems. 40 tasks sourced from published papers across 10 disciplines, each with complete experimental datasets.
• Two-stage pipeline. Autonomous research first, rigorous evaluation second — just like peer review.
• Fine-grained, multimodal scoring. A weighted checklist with text and image criteria, judged by an LLM acting as a strict peer reviewer.
• Agent-agnostic. Ships with first-class support for Claude Code, Codex CLI, and OpenClaw. Bring your own agent in one line.
• From Re-Discovery to New-Discovery. Scoring above 50 means matching the original paper; above 70 means surpassing it. The frontier is wide open.

Every task in ResearchClawBench is built through a rigorous, expert-driven pipeline to ensure scientific validity and reproducibility:

flowchart TD A["📄 High-Quality Paper Collection\n(Target Paper)"] --> B["🧑‍🔬 Human Expert Extraction\n(Core Task Instructions)"] B --> C["📋 Evaluation Checklist\n(Criteria + Keywords + Weights)"] B --> D["📂 Data & Related Work Collection\n(Datasets + Reference Papers)"] C --> E["✅ Human Reproduction & Validation\n(Verify checklist is reproducible)"] D --> E style A fill:#e0f2fe,stroke:#0284c7,stroke-width:2px style B fill:#fef3c7,stroke:#f59e0b,stroke-width:2px style C fill:#fce7f3,stroke:#ec4899,stroke-width:2px style D fill:#f0fdf4,stroke:#22c55e,stroke-width:2px style E fill:#f5f3ff,stroke:#8b5cf6,stroke-width:2px 

1. High-Quality Paper Collection — Domain experts select recent, high-impact publications with clear methodology and reproducible results across 10 scientific disciplines.

2. Expert Task Extraction — Human experts read each paper and distill the core research task into structured instructions, identifying the key scientific question, input data, and expected outputs.

3. Checklist Design — Experts create a fine-grained evaluation checklist with weighted criteria (text and image items), each with specific technical keywords that a judge must verify.

4. Data & Related Work Collection — The original datasets used in the paper are gathered, along with relevant reference materials, to form a self-contained research workspace.

5. Human Reproduction & Validation — Human researchers independently reproduce the paper's results using only the provided data and instructions, verifying that every checklist item is achievable. This ensures the benchmark is fair and the checklist is grounded in reality.

ResearchClawBench operates in two distinct stages:

flowchart LR subgraph Stage1["Stage 1 — Auto Research"] A["Raw Data\n+ Instructions"] --> B["AI Agent\n(autonomous)"] B --> C["Code\n+ Figures\n+ Report"] end subgraph Stage2["Stage 2 — Evaluation"] C --> D["LLM Judge"] E["Target Paper\n+ Checklist"] --> D D --> F["Per-Item Scores\n+ Reasoning"] end style Stage1 fill:#f0f4ff,stroke:#3b82f6,stroke-width:2px style Stage2 fill:#fff7ed,stroke:#f59e0b,stroke-width:2px 

Auto Research view — file explorer, live code output, and real-time agent conversation

The AI agent receives a workspace containing raw datasets, reference materials, and task instructions. It must independently:

1. Explore the data and understand the research question
2. Write code to analyze, model, and visualize the data
3. Produce a research report (report/report.md) with figures, methodology, results, and discussion

No hand-holding. No chain-of-thought hints. The agent works in its own sandboxed workspace with full tool access — just like a real researcher.

Evaluation view — target paper (left), AI report (center), scored checklist (right)

Once the agent finishes, its report is evaluated against the original published paper using a fine-grained checklist. The judge receives the task instructions, the AI report, and the checklist criteria — then scores each item using a dual-mode rubric:

flowchart TD subgraph Inputs I["INSTRUCTIONS.md\n(task background)"] R["Agent Report\n(text + figures)"] CL["Checklist\n(from target paper)"] end I & R & CL --> J["Multimodal LLM Judge"] J --> DET{"Determine\nEvaluation Mode"} DET -->|"Quantitative\nresults"| OBJ["Mode A: Objective\n(Metric Optimization)"] DET -->|"Qualitative\nreasoning"| SUB["Mode B: Subjective\n(Mechanism Analysis)"] OBJ --> SO["Score by metric\naccuracy vs paper"] SUB --> SS["Score by evidence\nstrength vs paper"] SO & SS --> T["Per-Item Scores\n+ Reasoning\n→ Weighted Total"] style Inputs fill:#f0f4ff,stroke:#3b82f6,stroke-width:2px style J fill:#fef3c7,stroke:#f59e0b,stroke-width:2px style OBJ fill:#dbeafe,stroke:#3b82f6,stroke-width:2px style SUB fill:#fce7f3,stroke:#ec4899,stroke-width:2px style T fill:#f0fdf4,stroke:#22c55e,stroke-width:2px 

Each checklist item includes:

• Specific criteria extracted from the paper's key contributions
• Technical keywords the judge must verify (e.g., "ROC-AUC improvement", "Monte Carlo integration")
• Weight reflecting the item's importance
• Type — text for methodology/findings, image for figure comparison (multimodal vision)

The judge automatically determines which evaluation mode applies to each item, then scores it with the corresponding rubric (see below).

For checklist items involving specific numerical results, metrics, or quantitative outcomes:

For checklist items involving theoretical explanations, mechanistic insights, or interpretive analysis:

Strict by design. The judge is highly skeptical of AI-generated content — plausible-sounding claims must be backed by concrete evidence. Longer reports do not score higher. Substance over style.

Each domain contains 4 carefully curated tasks with complete experimental data from real published research:

40 tasks total — each a self-contained research challenge selected from high-quality human-authored publications, spanning the full spectrum from data analysis to novel scientific insight.

git clone https://github.com/InternScience/ResearchClawBench.git cd ResearchClawBench pip install -r evaluation/requirements.txt

Create evaluation/.env with your scoring model credentials:

OPENAI_API_KEY=sk-xxx OPENAI_BASE_URL=https://api.openai.com/v1 SCORER_MODEL=gpt-5.1

python -m evaluation

Open http://localhost:5000 — browse tasks, pick an agent, hit Start Run, and watch the research happen live.

After a run completes, switch to the Evaluation tab and click Score. The multimodal LLM judge evaluates each checklist item and returns per-item scores with reasoning.

ResearchClawBench ships with built-in support for four frontier coding agents:

Agent	Command	Notes
Claude Code	claude -p ...	Anthropic, stream-JSON output
Codex CLI	codex exec --full-auto ...	OpenAI, full-auto mode
OpenClaw	openclaw agent ...	Self-hosted gateway, 3600s timeout
Nanobot	nanobot agent -m ...	Ultra-lightweight, reliable tool execution

Agent configuration is stored in evaluation/agents.json. To add a new agent, simply append an entry:

{ "my_agent": { "label": "My Agent", "icon": "M", "logo": "/static/logos/my_agent.svg", "cmd": "my-agent run -m <PROMPT> -w <WORKSPACE>" } }

The prompt injected into <PROMPT> is auto-generated from evaluation/instructions_tmpl.py, which combines a unified agent persona (autonomous execution guidelines, workspace sandbox rules) with task-specific instructions (description, data files, deliverables). All agents receive the exact same prompt — no code changes required, just edit the JSON file and restart the server.

You can view the leaderboard on our Website, which is updated in real time.

Leaderboard

The built-in dashboard aggregates the best score per (task, agent) pair and displays:

• Frontier chart — best score per task across all agents
• Leaderboard table — clickable cells linking to individual runs
• Per-task breakdown — view any agent's report, code, and score reasoning

The frontier represents the state of the art — every point above 50 is uncharted territory where AI surpasses human researchers on that specific task.

ResearchClawBench/ ├── evaluation/ # Core evaluation framework │ ├── server.py # Flask API + SSE streaming │ ├── run_task.py # Workspace setup + agent subprocess │ ├── score.py # Multimodal LLM scoring engine │ ├── config.py # Paths, constants, loads agents.json │ ├── agents.json # Agent presets (add your own here) │ ├── instructions_tmpl.py # Unified prompt template for all agents │ ├── utils.py # File tree, path safety, discovery │ ├── static/app.js # Single-file frontend │ └── templates/index.html # Entry point ├── tasks/ # 40 research tasks │ ├── Astronomy_000/ │ │ ├── task_info.json # Task description + data manifest │ │ ├── data/ # Raw experimental datasets │ │ ├── related_work/ # Reference papers │ │ └── target_study/ # Paper + checklist + images │ ├── Chemistry_000/ │ └── ... # 10 domains x 4 tasks └── workspaces/ # Generated at runtime (gitignored) 

We welcome contributions in several forms — see CONTRIBUTING.md for detailed guidelines.

• New tasks — Add research challenges in existing or new domains
• New agents — Add presets for emerging coding agents
• Bug reports — Open an issue

📧 Email: xu_wanghan@sjtu.edu.cn

📬 Community:

If you would like to cite our work, please use the following BibTeX.

@article{xu2025probing, title={Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows}, author={Xu, Wanghan and Zhou, Yuhao and Zhou, Yifan and Cao, Qinglong and Li, Shuo and Bu, Jia and Liu, Bo and Chen, Yixin and He, Xuming and Zhao, Xiangyu and others}, journal={arXiv preprint arXiv:2512.16969}, year={2025} }

🔝Back to top