Multi-Agent Framework Benchmark

This benchmark compares four answer-generation architectures across two model backends on 20 freshness questions designed to defeat parametric knowledge.

Approaches

1. B0: Vanilla LLM — one direct model answer with no tools or search.
2. B1: LLM + Search — query generation, web search/fetch, and a single evidence-grounded answer.
3. B2: LLM + Search + Reflection — search-based draft, self-critique, and revised answer.
4. MA: Multi-Agent — Meta-Planner → Research → Synthesis → Validator → Skeptic → Final Answer with iterative convergence and deterministic grounding.

Question design

The 20-question freshness pack uses facts published within ~24-48 hours of the experiment (June 22-23, 2026), sourced from NASA releases, arXiv postings, a Google Developers post, and a GitHub release feed. Questions are designed so models cannot answer from training data — B0 scores ~2-3 across both models, confirming the freshness property.

Versions

Two versions of the pipeline were tested:

• v2.2 — Draft-then-validate orchestration: the MA loop changed from validating raw evidence before drafting to Research → Synthesis draft → Validator → Skeptic. This fixed the core failure mode where small models rejected semantically obvious evidence.
• v2.3 — Deterministic grounding: the validator no longer calculates a subjective “overall grounding” score. Instead, it outputs per-field verdicts for each required field, and the controller computes grounding as verified_fields / total_required_fields.

Results

Average scores (unified GPT-4o grader, 0-10 scale)

MA convergence

Headline finding

Multi-agent orchestration compresses the capability gap between small and large models.

MA on Ollama 7B (8.0-8.1) matches or beats MA on GPT-4o (7.8-7.9) and nearly matches B2 on GPT-4o (8.2-8.4). The overhead helps small models more than large models.

Implication: Proper orchestration + retrieval + deterministic verification can bring a small model up to frontier-model performance on evidence-grounded tasks — at ~100x lower cost.

Cost analysis

Trade-off: ~4-5x more wall-clock time and ~50% more tokens for ~100x lower cost and comparable accuracy. For batch/latency-tolerant workloads (research, evidence review, document analysis), this is a clear win.

Core learnings

1. The bottleneck was evidence consumption, not retrieval quality. Validating raw evidence before drafting caused small models to reject semantically obvious support. Draft-then-validate fixed it.
2. MA helps small models more than large models. GPT-4o already does implicit self-reflection; the 7B model needs explicit decomposition.
3. Deterministic grounding > LLM subjective grounding. Per-field verification eliminates validator judgment drift and improves convergence reliability.
4. The validator prompt is the critical control point. The entire v2.1→v2.2 improvement came from changing what the validator sees; v2.2→v2.3 came from changing how it reports.
5. Convergence is necessary but not sufficient. Grounding=100% means every claim is evidence-backed, but evidence quality depends on retrieval precision.

Next experiments

1. Multi-hop reasoning — questions requiring 2-3 evidence sources combined. Tests whether MA’s decomposition helps with reasoning, not just retrieval.
2. Code reasoning & traceability — questions about code behavior requiring retrieval of source code, docs, and tests. Directly aligned with the thesis on evidence-based verification.

Related research

Published research broadly supports these findings. See the related research companion document for a detailed mapping of each finding to verified papers from arXiv, ACL, NeurIPS, ICML, and EMNLP, including:

• Mixture-of-Agents (Wang et al., 2024) — open-source LLMs surpassing GPT-4o via layered aggregation
• Reflexion (Shinn et al., NeurIPS 2023) — verbal reinforcement learning validating draft-validate patterns
• SLMJury (Laddha et al., 2026) — shows debate degrades SLM accuracy, validating our skeptic backfire finding
• The Confident Liar (Keramati et al., ACL 2026) — explains role-asymmetric calibration in multi-agent debate
• Wu et al. (2024) — LLMs adopt incorrect retrieved content 60%+ of the time, explaining search-as-bottleneck

17 papers verified, covering support, contradiction, and nuance for each finding.

Data artifacts

• Aggregated results (v2.2 + v2.3)
• v2.2 full report
• v2.2 machine-readable summary
• v2.3 full report
• v2.3 machine-readable summary
• Freshness question pack
• Learnings and next steps
• Artifact directory README
• Dataset on GitHub

Claim boundary

These results document a controlled comparison on a fixed 20-question suite. The strongest supported claims are:

1. On freshness questions that defeat parametric knowledge, MA with proper grounding brings a 7B model within 0.2-0.4 points of the best baseline on a frontier model.
2. Draft-then-validate orchestration is necessary — validating raw evidence before drafting causes small models to reject valid support.
3. Deterministic grounding improves convergence reliability without hurting accuracy.
4. MA helps small models more than large models on this task type.

Claims do not generalize to all question types, all model sizes, real-time latency requirements, or domains outside evidence-grounded retrieval tasks. The 20-question sample supports directional claims but not statistical significance at conventional thresholds.