Skip to main content

Code

The framework is about 48k lines of Python, 8.8k lines of Terraform and a 20k-line TypeScript dashboard, with 171 test files and a CI gate (ruff, black, mypy, pytest, gitleaks) — 809 commits between 2026-08-09 and 2026-09-09 in the development history, counted at the latter. The validated tree is published as a single-commit repository, psdhali/agent-evals, tagged v1.0, with the run artifacts attached as a release; every reference below is a link pinned to that tag.

Feedback​

Found a problem, or a number that does not add up? Open an issue. Questions and discussion: GitHub Discussions. No GitHub account: [email protected] or LinkedIn.

areapath
control plane: APIswebench_eval/orchestrator/api/main.py
control plane: dispatcher and plannerswebench_eval/orchestrator/control_plane/harness_dispatcher.py
control plane: run launch and keysswebench_eval/orchestrator/control_plane/run_launch.py
control plane: discovery probeswebench_eval/orchestrator/control_plane/ceiling_discovery.py
control plane: capacity observer and timelineswebench_eval/orchestrator/control_plane/capacity_observer.py
gateway: pacerswebench_eval/gateway/pacer.py
gateway: local proxy shimswebench_eval/gateway/local_proxy.py
harness adaptersswebench_eval/harnesses/
evaluation runnerswebench_eval/evaluation/swebench_runner.py
LLM judgeswebench_eval/analysis/judge.py, swebench_eval/analysis/judge_llm.py, config/judge_rubric.yaml
site exportswebench_eval/orchestrator/timeline_export.py, scripts/export_run_timeline.py
infrastructureinfra/terraform/ (three tiers: persistent, ui, eval; modules per service)
operator dashboardui/src/

License​

The framework is released under the MIT license. Patches and trajectories derived from Django, astropy, sympy, scikit-learn and the other SWE-bench repositories remain under those projects' own licenses; the SWE-bench dataset and the model providers' terms apply to the published outputs. See NOTICE in the site repository.