Benchmarks

Research Question Can a code knowledge graph (KG) with multi-agent navigation bring 7-8B parameter models closer to frontier-model performance on code generation, repair, and completion tasks? Sub-questions: Does KG navigation help small models write more correct code, or does it add noise they can’t filter? Does the multi-agent pipeline (draft → directional reasoning → test → refine) recover ground that single-agent KG loses? Which task types benefit most from graph evidence (generation vs. repair vs. completion)? Is there a model-capability floor below which KG navigation is ineffective — and does it differ from the legal KG floor? Can the pipeline enable small models to solve tasks that frontier B0 cannot? Experimental Setup Code Knowledge Graph A knowledge graph automatically extracted from a multi-module Python codebase (payments, user_service, notifications, email_service) using static analysis. ...

Research Question Can knowledge graph (KG) navigation bring a 7B parameter model to within 1 point of a frontier model on statutory reasoning tasks? Sub-questions: How much does KG navigation help small models vs. introduce noise? Which questions benefit most from graph evidence? What proportion of the gap is closeable via prompt engineering vs. graph quality vs. model capability? Is there a model-capability floor below which KG navigation is ineffective? Does natural language serialization of graph content help or hurt differently across model tiers? Experimental Setup Knowledge Graph A manually constructed knowledge graph covering six legal/technical domains: ...

Companion document to the 00n.ai Multi-Agent Benchmark This document maps our benchmark findings to the published research literature. Each finding is evaluated against verified papers from arXiv, ACL, NeurIPS, ICML, and EMNLP. Papers were confirmed to resolve and titles verified against abstracts. Finding 1: MA Compresses the Capability Gap Between Small and Large Models A 7B model with multi-agent orchestration (Ollama qwen2.5:7b, avg 8.0–8.1) matches frontier-model B2 performance (GPT-4o search+reflection, avg 8.2–8.4) on evidence-grounded tasks. ...

This benchmark compares four answer-generation architectures across two model backends on 20 freshness questions designed to defeat parametric knowledge. Approaches B0: Vanilla LLM — one direct model answer with no tools or search. B1: LLM + Search — query generation, web search/fetch, and a single evidence-grounded answer. B2: LLM + Search + Reflection — search-based draft, self-critique, and revised answer. MA: Multi-Agent — Meta-Planner → Research → Synthesis → Validator → Skeptic → Final Answer with iterative convergence and deterministic grounding. Question design The 20-question freshness pack uses facts published within ~24-48 hours of the experiment (June 22-23, 2026), sourced from NASA releases, arXiv postings, a Google Developers post, and a GitHub release feed. Questions are designed so models cannot answer from training data — B0 scores ~2-3 across both models, confirming the freshness property. ...

This is the canonical repoloc-bench entry for the iTrust requirement-to-code traceability benchmark package. It records the package and examples without copying the raw corpus into multiple places. Source Build brief: projects/state-estimation-traceability/research/dataset-benchmarking-buildout.md Package: projects/state-estimation-traceability/data/derived/paper1-benchmark/ GitHub repo: 00n-ai/repoloc-bench Reproduction bundle: benchmark_package_review_100 Snapshot commit: 64ecaab (Add benchmark reproduction bundle) What is included 34 requirements 4,907 methods 718 classes 307 gold traces masked gold-edge suites at 10/20/30% Claim boundary Primary branch is original requirement text. LLM-imputed requirements are auxiliary sensitivity data only. Gold links use traces with goldfinal == 'T'. This is a method-localization benchmark foundation, not yet a test-localization benchmark. Canonical package files manifest.json README.md README.json integrity.json EXAMPLES.md splits/ mask_suites/ systems/itrust/ Example outputs { "task_id": "orig:itrust:R1", "branch": "original", "system": "itrust", "requirement_id": 1, "gold_method_ids": [154, 2851, 2986, 2992, 3125, 3126, 4655], "positive_trace_count": 7 } { "node_id": "requirement:itrust:1", "original_text": "Add a New Patient ...", "imputed_text": "The system shall allow the addition of a new patient with basic information and generate a unique MID for them.", "gold_method_count": 7 } { "mask_percent": 10, "gold_edges": 307, "held_out_gold_edges": 39, "observed_gold_edges": 268 } Why this stays small The benchmark package itself remains the source of truth. This page is only the registry and summary layer, so we do not duplicate the underlying dataset here. ...