[{"content":"Agentic systems need an objective An agent is not just a tool that produces output. In the canonical definitions, an agent is a temporally persistent system that observes, retains state or memory, chooses actions, and acts in service of an objective.\nThat objective matters. Without it, the system is only reacting; with it, the system becomes agentic.\nThis is consistent with the core agent literature: Russell \u0026amp; Norvig treat agents as systems that perceive and act in an environment, Sutton \u0026amp; Barto define the RL agent as the decision-making side of the loop, and control theory treats the analogous object as a controller acting toward a desired outcome.\nWhat goes into an objective A useful objective should say:\nwhat success means what environment or domain it applies to what time horizon matters what constraints cannot be violated what tradeoffs matter when goals conflict what failure looks like So the objective is not the metric. It is the thing we care about.\nFor example:\n\u0026ldquo;Produce correct, useful, safe software changes over time\u0026rdquo; is an objective. Test pass rate, bug rate, latency, and review quality are possible measurements or proxies. Why measurement is needed If the objective is latent or only partially observable, we cannot usually optimize it directly. We need measurement signals that tell us whether the system is moving in the right direction.\nThat is why measurement exists:\nto observe outcomes to estimate hidden state to check whether the objective is improving to detect when the system is gaming the metric This is the same pattern seen in control theory, reinforcement learning, and measurement theory. In control, the true state is often not directly visible. In RL, reward is an operational signal, not always the full intended goal. In measurement theory, the construct and the measure are not the same thing.\nWhy proxies are used A proxy is a measurable stand-in for the objective.\nWe use proxies because:\nthe objective may be too broad to measure directly the objective may be long-horizon the objective may be hidden inside a larger process the objective may be expensive or slow to verify the objective may require multiple dimensions So instead of one score, we often need a proxy set.\nThe working framework The diagram makes the separation explicit:\nthe objective is what success means proxies are measurable stand-ins measurements produce signals evaluation combines the signals action changes the world, which feeds back into measurement and the objective How to write the objective Write the objective first in plain language. Then separate it from the metric. Then state the tradeoffs. Then state the failure modes.\nA good objective includes:\nTarget outcome Scope Time horizon Constraints Tradeoffs Failure modes How to get proxies from the objective Start by decomposing the objective into dimensions. Then ask what observable signals should move if the objective improves. Then choose multiple proxies that cover different dimensions. Then define how each proxy is measured. Then validate them against outcomes and failure cases. Then audit for Goodhart risk.\nSo the flow is:\nDefine the objective Decompose it Find observables Turn observables into proxies Measure those proxies Validate them Revise when they drift Where this leads next The next step is to discuss controls: how to add mechanisms that change state in ways we can measure through proxies. That is where policy, feedback, steering, and stability enter the picture.\nSources Russell \u0026amp; Norvig, Artificial Intelligence: A Modern Approach Sutton \u0026amp; Barto, Reinforcement Learning Åström \u0026amp; Murray, Feedback Systems Cronbach \u0026amp; Meehl, Construct Validity in Psychological Tests Manheim \u0026amp; Garrabrant, Categorizing Variants of Goodhart’s Law Internal project notes are used to shape this page but are not listed here because the public website does not have access to them.\n","permalink":"https://00n.ai/research/agents-objectives-proxies/","summary":"\u003ch2 id=\"agentic-systems-need-an-objective\"\u003eAgentic systems need an objective\u003c/h2\u003e\n\u003cp\u003eAn \u003ca href=\"/research/definitions/\"\u003eagent\u003c/a\u003e is not just a tool that produces output. In the \u003ca href=\"/research/definitions/\"\u003ecanonical definitions\u003c/a\u003e, an agent is a temporally persistent system that observes, retains state or memory, chooses actions, and acts in service of an \u003ca href=\"/research/definitions/\"\u003eobjective\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eThat objective matters. Without it, the system is only reacting; with it, the system becomes agentic.\u003c/p\u003e\n\u003cp\u003eThis is consistent with the core agent literature: Russell \u0026amp; Norvig treat agents as systems that perceive and act in an environment, Sutton \u0026amp; Barto define the RL agent as the decision-making side of the loop, and control theory treats the analogous object as a controller acting toward a desired outcome.\u003c/p\u003e","title":"Agents, Objectives, and Proxies"},{"content":"Canonical terms These are the project definitions used across the research site.\nAgent A temporally persistent system that maps observations and internal state to actions in an environment according to a policy, in service of an objective. Russell \u0026amp; Norvig ; Sutton \u0026amp; Barto ; Åström \u0026amp; Murray\nSingle agent One decision-making system acting on one environment. Sutton \u0026amp; Barto ; Åström \u0026amp; Murray\nPolicy The rule or function that maps decision state to action. Sutton \u0026amp; Barto ; Åström \u0026amp; Murray\nObjective The latent criterion of success that the system is trying to achieve. Russell \u0026amp; Norvig ; AERA/APA/NCME validity standards overview\nGoal What counts as success for the system. Russell \u0026amp; Norvig ; Sutton \u0026amp; Barto\nEnvironment The world or system the agent interacts with and can affect. Russell \u0026amp; Norvig ; Sutton \u0026amp; Barto\nObservation What the agent receives from the environment. Åström \u0026amp; Murray ; Sutton \u0026amp; Barto\nState The information the agent uses to decide. State may include current observations, internal variables, estimates, and memory. Sutton \u0026amp; Barto ; Åström \u0026amp; Murray\nMemory Retained state across steps. This includes short-term context, long-term traces, prior task progress, and learned heuristics. Sutton \u0026amp; Barto ; POMDP belief-state framing\nAction What the agent does to affect the environment. Russell \u0026amp; Norvig ; Sutton \u0026amp; Barto\nError The gap between current behavior or outcome and desired behavior or outcome. Åström \u0026amp; Murray ; optimal control framing\nSteering Changing behavior in response to error and feedback. Åström \u0026amp; Murray\nMeasurement The observable signal produced by tests, traces, audits, judges, sensors, or logs. Cronbach \u0026amp; Meehl ; AERA/APA/NCME validity standards\nProxy A measurable stand-in for the objective. Cronbach \u0026amp; Meehl ; Goodhart’s law overview\nEvaluation The function that aggregates measurements and proxies into a score or decision signal. AERA/APA/NCME validity standards ; balanced scorecard framing\nControl input A deliberate change applied to steer the system. Åström \u0026amp; Murray\nController The mechanism that selects control inputs or actions. Åström \u0026amp; Murray ; Sutton \u0026amp; Barto\nDynamics How actions and environment evolution change future state. Åström \u0026amp; Murray\nFeedback loop Observe → interpret → decide → act → update. Åström \u0026amp; Murray\nStability Whether the system converges, oscillates, or diverges under feedback. Åström \u0026amp; Murray\nRobustness Whether the agent still works under noise, delay, partial observability, or perturbation. Åström \u0026amp; Murray\nPartial observability When the agent cannot directly see the full state of the environment. POMDP framing ; Åström \u0026amp; Murray\nEstimation Inferring hidden or incomplete state from observations. Åström \u0026amp; Murray\nPolicy update A change to the policy based on experience, learning, or feedback. Sutton \u0026amp; Barto ; adaptive control framing\nModel A representation of how the environment or system behaves. Russell \u0026amp; Norvig ; Åström \u0026amp; Murray\nGoodhart effect When optimization pressure causes a proxy to stop tracking the true objective well. Goodhart’s law ; variants of Goodhart’s law\nUse this page as the link target When another page needs a definition, link here instead of restating the term.\nSource basis Russell \u0026amp; Norvig, Artificial Intelligence: A Modern Approach Sutton \u0026amp; Barto, Reinforcement Learning Åström \u0026amp; Murray, Feedback Systems Cronbach \u0026amp; Meehl, Construct Validity in Psychological Tests AERA/APA/NCME Standards for Educational and Psychological Testing Goodhart’s law ","permalink":"https://00n.ai/research/definitions/","summary":"\u003ch2 id=\"canonical-terms\"\u003eCanonical terms\u003c/h2\u003e\n\u003cp\u003eThese are the project definitions used across the research site.\u003c/p\u003e\n\u003ch3 id=\"agent\"\u003eAgent\u003c/h3\u003e\n\u003cp\u003eA temporally persistent system that maps observations and internal state to actions in an environment according to a policy, in service of an objective. \u003ca href=\"https://aima.cs.berkeley.edu/\"\u003eRussell \u0026amp; Norvig\u003c/a\u003e ; \u003ca href=\"http://incompleteideas.net/book/the-book.html\"\u003eSutton \u0026amp; Barto\u003c/a\u003e ; \u003ca href=\"https://introcontrol.mit.edu/_static/fall21/extras/Feedback%20Systems%20Murray.pdf\"\u003eÅström \u0026amp; Murray\u003c/a\u003e\u003c/p\u003e\n\u003ch3 id=\"single-agent\"\u003eSingle agent\u003c/h3\u003e\n\u003cp\u003eOne decision-making system acting on one environment. \u003ca href=\"http://incompleteideas.net/book/the-book.html\"\u003eSutton \u0026amp; Barto\u003c/a\u003e ; \u003ca href=\"https://introcontrol.mit.edu/_static/fall21/extras/Feedback%20Systems%20Murray.pdf\"\u003eÅström \u0026amp; Murray\u003c/a\u003e\u003c/p\u003e\n\u003ch3 id=\"policy\"\u003ePolicy\u003c/h3\u003e\n\u003cp\u003eThe rule or function that maps decision state to action. \u003ca href=\"http://incompleteideas.net/book/the-book.html\"\u003eSutton \u0026amp; Barto\u003c/a\u003e ; \u003ca href=\"https://introcontrol.mit.edu/_static/fall21/extras/Feedback%20Systems%20Murray.pdf\"\u003eÅström \u0026amp; Murray\u003c/a\u003e\u003c/p\u003e","title":"Definitions"},{"content":"Abstract This paper studies whether a hierarchical evidence tree, coupled with local support/residual measurements, can improve code localization and downstream coding-agent repair. The work is organized as a single experimental program with three phases: (i) construction of a grounded repository-state tree for traceability and localization, (ii) reformulation of traceability drift as evidence-weighted innovation over observable state rather than hyperbolic transport geometry, and (iii) a frozen-edit Codex augmentation replay to test whether routing and evidence packets reduce wasted code and wrong-region edits.\nThe broader literature framing is now narrower and stronger: attention and embedding retrieval are good at finding candidate regions, but the evidence needed to justify a requirement-satisfaction claim is relational, so the final support belongs in a graph or evidence subgraph.\nAcross Methods2Test routing, JHotDraw traversal, leaf localization, requirement-variant routing, Phase 2 calibration/drift experiments, and a six-task frozen Codex replay, the most consistent finding is that evidence-rich routing improves candidate narrowing, while the Codex augmentation packet remains mixed on patch churn and does not improve end-to-end repair success. Hyperbolic/Poincaré scoring does not outperform simpler baselines.\nTL;DR The tree is the win. It preserves structure, improves localization, and gives us a better way to route evidence toward likely edit regions. When we add that evidence to Codex, the agent’s search path changes, but the packet still does not reliably produce a final fix. So the current contribution is a stronger routing and error-measurement layer, not a full repair system.\nPurpose and scope This paper is an exploratory methodological study for our website and research program. Its purpose is to learn how to work effectively with AI research assistants, document open questions, and develop a repeatable research workflow. We used GPT-5.4 and GPT-5.5 to support literature review, synthesis, drafting, and analysis under author direction. The paper is intentionally framed as a first pass, not a final answer.\nWhat this paper is / is not Is: an initial methodological study of AI-assisted research; a record of open questions, working assumptions, and early experiments; a first attempt to learn repeatable ways of working with GPT-5.4 and GPT-5.5 as research assistants. Is not: a finished theory of AI-assisted research; a claim that the workflow is optimal or complete; a production guide or a substitute for human judgment and review. Manuscript Download the PDF manuscript Key results hierarchical evidence routing is stronger than flat similarity search for localization attention/embedding retrieval is a strong candidate-region finder, not the final proof layer local support/residual signals are useful for judging uncertainty hyperbolic/Poincaré geometry does not beat the simpler discrete routing baselines Codex augmentation changes routing and patch shape, but does not yet reliably improve repair success Reproducibility This paper is reproducible from the GitHub-backed research repo. The repo holds the code, data, experiment outputs, and paper artifacts.\nWhere to reproduce GitHub reproduction bundle: 00n-ai/repoloc-bench GitHub bundle snapshot: research/reproduction/benchmark_package_review_100 GitHub source of truth: state-estimation-traceability Key reproducibility note: research/reproducibility-and-release.md Benchmark summary: experiments/benchmark_package_review_100/README.md Benchmark metrics: experiments/benchmark_package_review_100/benchmark_package_summary.json Codex replay evidence: research/experiment-runs/2026-06-04T232257Z-h2_18-d1/metrics_summary.json Reproduction rule Re-run from the exact commit, with the documented command line and seed, and compare the new outputs to the saved summary files before publishing any claim here.\nRelated links Research index CV ","permalink":"https://00n.ai/research/evidence-weighted-routing-and-error-measurement/","summary":"\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003cp\u003eThis paper studies whether a hierarchical evidence tree, coupled with local support/residual measurements, can improve code localization and downstream coding-agent repair. The work is organized as a single experimental program with three phases: (i) construction of a grounded repository-state tree for traceability and localization, (ii) reformulation of traceability drift as evidence-weighted innovation over observable state rather than hyperbolic transport geometry, and (iii) a frozen-edit Codex augmentation replay to test whether routing and evidence packets reduce wasted code and wrong-region edits.\u003c/p\u003e","title":"Evidence-Weighted Routing and Error Measurement for Code Localization and Agentic Repair: A Three-Phase Study"},{"content":"Abstract This page collects notes on how to design multi-agent systems that are useful in practice: clear role boundaries, coordination patterns, shared state, task routing, evaluation, and failure handling. The focus is on systems that can explain what they are doing, route work intelligently, and remain measurable under real workloads.\nFocus areas role decomposition task routing and handoff shared memory and state coordination and control evaluation and failure modes guardrails and recovery separation of generation and evaluation Useful source Edward Y. Chang’s Multi-LLM Agent Collaborative Intelligence – The Path to Artificial General Intelligence is useful here because it treats multi-agent work as a pipeline where one set of agents generates or debates candidates and a separate set of judges evaluates them. That maps well to reliable multi-agent design: search first, judge later.\nIn particular, the book is helpful for:\nseparating candidate generation from evaluation using distinct judge agents aggregating supporting and opposing evidence recursive verification of claims This makes it a good supporting citation for agentic workflows that need judgment, not just generation.\nSource note This page uses Chang as a supporting source for multi-agent evaluation design, not as the main canonical definition of an agent.\nWorking note The goal is not to build the most complex agent swarm. It is to build the smallest agent system that can coordinate well, stay inspectable, and improve the quality of decisions.\n","permalink":"https://00n.ai/research/designing-multi-agent-systems/","summary":"\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003cp\u003eThis page collects notes on how to design multi-agent systems that are useful in practice: clear role boundaries, coordination patterns, shared state, task routing, evaluation, and failure handling. The focus is on systems that can explain what they are doing, route work intelligently, and remain measurable under real workloads.\u003c/p\u003e\n\u003ch2 id=\"focus-areas\"\u003eFocus areas\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003erole decomposition\u003c/li\u003e\n\u003cli\u003etask routing and handoff\u003c/li\u003e\n\u003cli\u003eshared memory and state\u003c/li\u003e\n\u003cli\u003ecoordination and control\u003c/li\u003e\n\u003cli\u003eevaluation and failure modes\u003c/li\u003e\n\u003cli\u003eguardrails and recovery\u003c/li\u003e\n\u003cli\u003eseparation of generation and evaluation\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"useful-source\"\u003eUseful source\u003c/h2\u003e\n\u003cp\u003eEdward Y. Chang’s \u003cem\u003eMulti-LLM Agent Collaborative Intelligence – The Path to Artificial General Intelligence\u003c/em\u003e is useful here because it treats multi-agent work as a pipeline where one set of agents generates or debates candidates and a separate set of judges evaluates them. That maps well to reliable multi-agent design: search first, judge later.\u003c/p\u003e","title":"Designing Multi-Agent Systems"},{"content":"Research Question Can a code knowledge graph (KG) with multi-agent navigation bring 7-8B parameter models closer to frontier-model performance on code generation, repair, and completion tasks?\nSub-questions:\nDoes KG navigation help small models write more correct code, or does it add noise they can\u0026rsquo;t filter? Does the multi-agent pipeline (draft → directional reasoning → test → refine) recover ground that single-agent KG loses? Which task types benefit most from graph evidence (generation vs. repair vs. completion)? Is there a model-capability floor below which KG navigation is ineffective — and does it differ from the legal KG floor? Can the pipeline enable small models to solve tasks that frontier B0 cannot? Experimental Setup Code Knowledge Graph A knowledge graph automatically extracted from a multi-module Python codebase (payments, user_service, notifications, email_service) using static analysis.\nMetric Count Nodes 178 Edges 348 Conventions 4 Modules 4 Node types: module, class, function, method, test, variable, enum_value, exception, convention\nEdge types: imports, calls, inherits, defines, tests, depends_on, raises, manages_state, has_enum_value, runtime_import, EXEMPLIFIES, FOLLOWS_CONVENTION\nConvention nodes capture behavioral contracts (e.g., \u0026ldquo;functions that validate user existence raise ValueError when not found\u0026rdquo;) and cross-module access patterns (e.g., \u0026ldquo;payments accesses user_service via _get_user_service() lazy import\u0026rdquo;).\nTasks 9 tasks across 3 difficulty types:\nTask Type Module Tests Description gen-cancel-subscription generation payments 4 Cancel subscription by refunding most recent completed payment gen-reset-password generation user_service 4 Reset password with old password verification gen-notify-user-deleted generation notifications 3 Send confirmation email + queue internal notification on user deletion repair-duplicate-email repair user_service 3 Fix duplicate email detection (case-insensitive) repair-revenue-refunded repair payments 3 Fix revenue calculation to exclude refunded payments repair-notification-queue repair notifications 3 Fix flush_notification_queue to send all before clearing comp-update-timestamp completion user_service 3 Add updated_at timestamp to User class and to_dict comp-payment-summary completion payments 5 Complete get_payment_summary with total, by_tier, and counts comp-notification-stats completion notifications 4 Complete get_notification_stats with total, by_type, unsent_count Models Role Model Provider Parameters Small model 1 qwen2.5:7b Ollama (local) 7B Small model 2 mistral:7b Ollama (local) 7B Small model 3 llama3.1:8b Ollama (local) 8B Frontier ceiling GLM-5.2 DeepInfra (frontier) Conditions B0 (Vanilla LLM): One direct model call with the task prompt. No tools, no graph. Temperature 0.3. The model sees only the task description.\nKG (Graph Navigation): Four-step single-agent pipeline:\nSeed — Deterministically retrieve all nodes from the target module (functions, classes, conventions) Decompose — LLM breaks the task into 2-4 sub-questions (knowledge needs) Navigate — For each sub-question, query the graph by keyword → retrieve nodes + 1-hop neighbors → LLM decides whether to explore further (8-step budget) Reason + Synthesize — LLM traces a reasoning path through visited nodes/edges, then synthesizes code following the reasoning path KG-NL (NL Serialization): Same as KG, but graph nodes are serialized as natural language descriptions (2-4 sentences per node) instead of teleographic JSON. Conventions include explicit validation vs. lookup pattern context.\nMA+KG (Multi-Agent + Directional Reasoning): Full pipeline with 4 components:\nKG-Agent (draft) — Same as KG above, produces draft code + visited evidence Directional Reasoning — LLM examines visited evidence + conventions + task and generates explicit \u0026ldquo;when X, do Y\u0026rdquo; implementation directives. Converts facts (\u0026ldquo;process_payment raises ValueError\u0026rdquo;) into direction (\u0026ldquo;when implementing user validation in payments, raise ValueError with \u0026rsquo;not found\u0026rsquo; message\u0026rdquo;) Test — Inject generated code into the module, run pytest, capture pass/fail Refine — If tests fail, LLM gets test output + directional directives and generates revised code. Up to 3 refinement rounds. Anti-regression guard: if refined code passes fewer tests than the previous round, keep the previous code. Directional Reasoning Fixes Four issues were identified and fixed before the full experiment:\nPre-filter conventions by task keywords (High). Only include a convention if the task description contains keywords related to it. Prevents irrelevant directives (e.g., ValueError validation convention applied to a data aggregation task).\nValidation vs. lookup pattern context (High). Convention nodes now distinguish validation functions (raise exceptions) from lookup functions (return None/False). Prevents the model from raising ValueError in data retrieval tasks where it should return None.\nPost-filter hallucinated function directives (Medium). Each directive\u0026rsquo;s function calls are checked against the graph\u0026rsquo;s known function inventory. Directives referencing nonexistent functions are stripped.\nSyntax-error refinement step (Medium). When a syntax error is detected, a focused LLM call attempts to fix it before running pytest. Prevents 0/0 scores from trivial syntax issues.\nEvaluation Pass@1: fraction of test cases passing after code injection. Each run injects the generated code into the original module, runs the task\u0026rsquo;s pytest suite, and counts passing tests. 5 runs per task/condition/model for variance estimation.\nResults Overall Pass@1 (avg across all tasks and runs) Condition qwen 7B mistral 7B llama 8B GLM-5.2 B0 28.1% 19.8% 45.9% — KG 16.9% 19.3% 24.5% — KG-NL 15.4% 20.2% 16.9% — MA+KG 23.9% 38.0% 48.3% — Frontier B0 — — — 83.9% MA+KG Lift over B0 (percentage points) Task qwen mistral llama gen-cancel-subscription −2 +0 −20 gen-reset-password +0 +8 +0 gen-notify-user-deleted −33 +33 +20 repair-duplicate-email +0 +40 −40 repair-revenue-refunded −40 +50 −33 repair-notification-queue +27 −2 +77 comp-update-timestamp +0 +20 +13 comp-payment-summary +4 +0 +52 comp-notification-stats +0 +12 −45 OVERALL −4.3 +18.1 +2.4 Gap to Frontier B0 Ceiling (83.9%) Condition qwen mistral llama B0 −55.8 −64.1 −38.0 MA+KG −60.0 −45.9 −35.6 Per-Task Detail by Model qwen2.5:7b Task B0 KG KG-NL MA+KG gen-cancel-subscription 40% 35% 20% 38% gen-reset-password 0% 0% 0% 0% gen-notify-user-deleted 73% 13% 13% 40% repair-duplicate-email 40% 20% 20% 40% repair-revenue-refunded 100% 67% 40% 60% repair-notification-queue 0% 13% 7% 27% comp-update-timestamp 0% 0% 0% 0% comp-payment-summary 0% 4% 4% 4% comp-notification-stats 0% 0% 35% 0% mistral:7b Task B0 KG KG-NL MA+KG gen-cancel-subscription 5% 0% 0% 5% gen-reset-password 0% 0% 0% 8% gen-notify-user-deleted 0% 0% 0% 33% repair-duplicate-email 60% 80% 100% 100% repair-revenue-refunded 33% 73% 27% 83% repair-notification-queue 80% 20% 40% 78% comp-update-timestamp 0% 0% 0% 20% comp-payment-summary 0% 0% 0% 0% comp-notification-stats 0% 0% 15% 12% llama3.1:8b Task B0 KG KG-NL MA+KG gen-cancel-subscription 90% 80% 40% 70% gen-reset-password 0% 0% 0% 0% gen-notify-user-deleted 40% 13% 0% 60% repair-duplicate-email 100% 20% 20% 60% repair-revenue-refunded 100% 47% 47% 67% repair-notification-queue 7% 27% 7% 83% comp-update-timestamp 0% 0% 0% 13% comp-payment-summary 16% 24% 4% 68% comp-notification-stats 60% 10% 35% 15% GLM-5.2 (Frontier B0 Ceiling) Task Score gen-cancel-subscription 100% gen-reset-password 75% gen-notify-user-deleted 100% repair-duplicate-email 80% repair-revenue-refunded 100% repair-notification-queue 0% comp-update-timestamp 100% comp-payment-summary 100% comp-notification-stats 100% Analysis Where MA+KG Helped Mistral: +18.1pp overall — the pipeline nearly doubles B0 performance. Standout tasks:\nrepair-duplicate-email: 60% → 100% (+40pp). The convention node for ValueError validation + the directional directive (\u0026ldquo;raise ValueError for duplicates\u0026rdquo;) gave mistral the exact pattern it needed. repair-revenue-refunded: 33% → 83% (+50pp). The graph\u0026rsquo;s Payment class nodes and refund logic edges provided the state machine knowledge mistral lacked. gen-notify-user-deleted: 0% → 33% (+33pp). The cross-module import convention (_get_user_service()) enabled mistral to correctly access user data from notifications. Llama: +2.4pp overall, with dramatic single-task transformations:\nrepair-notification-queue: 7% → 83% (+77pp). The clearest KG signature task — frontier B0 scores 0% on this task, but llama MA+KG scores 83%. The graph provides the cross-module send_email dependency that B0 can\u0026rsquo;t discover. comp-payment-summary: 16% → 68% (+52pp). The graph\u0026rsquo;s Payment class, payment_history function, and PaymentStatus enum nodes gave llama the full data model it needed. gen-notify-user-deleted: 40% → 60% (+20pp). The notification queue + email service cross-module edges enabled correct multi-module interaction. Where MA+KG Hurt Qwen: −4.3pp overall — the only model where B0 beats all KG conditions. The graph adds noise qwen can\u0026rsquo;t filter, and the refiner breaks working draft code. The anti-regression guard limits the damage but can\u0026rsquo;t fully prevent it.\nLlama on tasks where B0 was already strong:\ngen-cancel-subscription: 90% → 70% (−20pp). B0 was already excellent; the pipeline introduced irrelevant conventions and the refiner degraded the draft. repair-duplicate-email: 100% → 60% (−40pp). Same pattern — B0 nailed it, the pipeline overcomplicated. comp-notification-stats: 60% → 15% (−45pp). The KG provided irrelevant payment/user_service nodes that distracted from the simple notification iteration. The KG Signature Task: repair-notification-queue This task requires knowledge that the frontier model lacks — frontier B0 scores 0%. The graph provides the cross-module send_email dependency and the notification queue state machine that neither parametric knowledge nor task description reveals.\nModel B0 MA+KG Lift qwen 7B 0% 27% +27pp mistral 7B 80% 78% −2pp llama 8B 7% 83% +77pp GLM-5.2 0% — — Llama MA+KG (83%) vastly exceeds frontier B0 (0%) on this task. This is the clearest evidence that the code KG pipeline adds capabilities that even frontier parametric knowledge cannot provide — the graph encodes project-specific architectural knowledge (import conventions, cross-module dependencies) that no model knows without retrieval.\nKG Alone Hurts All Three Models Structured graph navigation (KG condition) hurts all three small models compared to B0:\nModel B0 KG Lift qwen 7B 28.1% 16.9% −11.2pp mistral 7B 19.8% 19.3% −0.5pp llama 8B 45.9% 24.5% −21.4pp The graph adds context the models can\u0026rsquo;t effectively use without the multi-agent guardrails (draft → directional reasoning → test → refine). MA+KG recovers and exceeds B0 for mistral (+18.1pp) and llama (+2.4pp) because the pipeline provides:\nTest feedback — the refiner knows which tests failed, not just which facts exist Directional reasoning — conventions are converted into actionable directives, not just presented as facts Anti-regression guard — prevents the refiner from breaking working draft code Model-Dependent Pattern Model B0 MA+KG MA+KG vs B0 Pattern qwen 7B 28.1% 23.9% −4.3pp Below capability floor — can\u0026rsquo;t synthesize graph evidence mistral 7B 19.8% 38.0% +18.1pp Sweet spot — low B0, high synthesis ability llama 8B 45.9% 48.3% +2.4pp Strong base — MA+KG helps on specific hard tasks Mistral is the ideal MA+KG candidate: low B0 (19.8%) means it lacks parametric knowledge, but it has enough synthesis capability to use graph evidence. The pipeline nearly doubles its performance.\nLlama is a strong base (B0=45.9%) that gets targeted help on hard tasks (repair-notification-queue +77pp, comp-payment-summary +52pp) but is hurt on easy tasks where B0 was already strong.\nQwen is below the capability floor for this task type. The graph adds noise it can\u0026rsquo;t filter, and the refiner degrades working code. This matches the legal KG finding where qwen had a +1.5 lift on legal reasoning (simpler synthesis) but the code domain is harder (multi-file, cross-module, state machines).\nComparison with Legal KG Benchmark This experiment replicates the legal KG benchmark pattern in the code domain:\nFinding Legal KG Coding KG Status KG alone hurts small models Mixed (helps qwen, hurts llama) Hurts all 3 Stronger in code MA+KG helps mid-tier models Yes (qwen +0.6) Yes (mistral +18.1pp) Confirmed Model-capability floor exists Yes (B0 \u0026lt; 5.0) Yes (qwen can\u0026rsquo;t use graph) Confirmed KG enables tasks frontier can\u0026rsquo;t do Not tested Yes (repair-notification-queue) New finding NL serialization helps weakest model Yes (llama: −2.7 → +1.2) No (doesn\u0026rsquo;t help any model) Domain difference Anti-regression guard needed N/A (no code refinement) Yes (7B refiner breaks draft) New finding The key new finding from the coding benchmark: the code KG enables small models to solve tasks that frontier B0 cannot solve (repair-notification-queue: 0% for frontier, 83% for llama MA+KG). This was not demonstrated in the legal domain because legal questions had no equivalent of project-specific architectural knowledge.\nLimitations Single codebase. The KG is extracted from one 4-module Python project. Generalization to larger codebases, other languages, or different architectural patterns is not demonstrated. 9 tasks only. The task set covers generation, repair, and completion, but is small. Statistical significance is not claimed. Automatic graph extraction. The KG is built by static analysis (AST walking). Dynamic behaviors, runtime configurations, and non-code conventions are not captured. Non-deterministic generation. Ollama models at temperature 0.3 produce variance across runs. 5 runs per condition provide variance estimation but not confidence intervals. Frontier B0 only. The frontier model was not run with MA+KG (no DeepInfra-compatible pipeline for the code KG). The ceiling comparison is B0-only. Convention extraction is heuristic. The 4 convention nodes are derived from AST patterns (raise statements, import patterns). Real-world conventions (naming, error messages, API contracts) may require manual annotation. Refinement is limited to 3 rounds. More rounds might help for complex tasks, but also increase the chance of the refiner breaking working code. No grader. Pass/fail is determined by pytest, not LLM grading. This is more objective than the legal benchmark but less nuanced (partial credit only from test counts). Reproduction Prerequisites # Software node v20+ (tested with v26.0.0) python3 with ast module pytest # Ollama (small models) ollama pull qwen2.5:7b ollama pull mistral:7b ollama pull llama3.1:8b # DeepInfra (frontier model) export DEEPINFRA_API_KEY=your_key Running the Experiment cd projects/coding-kg-experiment # Build the knowledge graph python3 src/extract_code_kg.py example-repo/ code-graph.json # Run full experiment (3 models × 4 conditions × 9 tasks × 5 runs) LLM_PROVIDER=ollama LLM_MODEL=qwen2.5:7b node src/run_full_experiment.mjs --runs 5 LLM_PROVIDER=ollama LLM_MODEL=mistral:7b node src/run_full_experiment.mjs --runs 5 LLM_PROVIDER=ollama LLM_MODEL=llama3.1:8b node src/run_full_experiment.mjs --runs 5 # Frontier B0 ceiling LLM_PROVIDER=deepinfra LLM_MODEL=deepinfra/zai-org/GLM-5.2 \\ node src/run_full_experiment.mjs --frontier --runs 5 Data Artifacts qwen2.5:7b results mistral:7b results llama3.1:8b results GLM-5.2 frontier B0 results Repository Structure projects/coding-kg-experiment/ ├── code-graph.json # KG: 178 nodes, 348 edges ├── example-repo/ # 4-module Python codebase │ ├── src/ │ │ ├── payments.py │ │ ├── user_service.py │ │ ├── notifications.py │ │ └── email_service.py │ └── tests/ ├── tasks/ │ ├── tasks.json # 9 task definitions │ └── tests/ # pytest suites per task ├── src/ │ ├── extract_code_kg.py # Graph extractor (AST-based) │ ├── code_kg_agent.js # Single-agent KG pipeline │ ├── ma_kg_agent.js # MA+KG pipeline with directional reasoning │ └── run_full_experiment.mjs # Experiment runner ├── results/ # JSON results per model └── docs/ ├── experiment-design.md # Full design document └── conventions-and-directional-reasoning.md # Convention extraction design Key Takeaways MA+KG beats B0 for 2 of 3 small models. Mistral +18.1pp, llama +2.4pp. The pipeline (draft → directional reasoning → test → refine) provides guardrails that single-agent KG navigation cannot. The code KG enables small models to solve tasks frontier B0 cannot. repair-notification-queue: 0% for frontier, 83% for llama MA+KG. The graph encodes project-specific architectural knowledge (import conventions, cross-module dependencies) that no model has parametrically. KG alone hurts all three small models. Structured graph navigation adds noise without the multi-agent guardrails. The value is in the pipeline, not the graph alone. Mistral is the ideal MA+KG candidate. Low B0 (19.8%) + sufficient synthesis capability = +18.1pp lift. The pipeline nearly doubles its performance. Qwen is below the capability floor for code synthesis. B0 beats all KG conditions. The graph adds noise qwen can\u0026rsquo;t filter, and the refiner degrades working code. Directional reasoning fixes worked. Convention pre-filtering prevented irrelevant directives, post-filtering caught hallucinated function names, and the anti-regression guard preserved draft wins. The model-capability floor differs by domain. qwen had +1.5 lift on legal KG (simpler synthesis) but −4.3pp on coding KG (harder: multi-file, cross-module, state machines). The floor is task-complexity-dependent, not just model-dependent. MA+KG narrows the gap to frontier. Llama MA+KG (48.3%) gets within 35.6pp of frontier B0 (83.9%) — the best of any small-model condition. Mistral MA+KG (38.0%) closes 18pp of gap that B0 alone couldn\u0026rsquo;t. Task-type dependency: MA+KG shines on hard tasks where B0 is weak. The biggest wins are on tasks where the model lacks parametric knowledge (repair-notification-queue, comp-payment-summary). On tasks where B0 is already strong, MA+KG can overcomplicate and hurt. Related Knowledge Graph Agent Benchmark (Legal) — Same experimental design on statutory reasoning tasks Multi-Agent Framework Benchmark — MA orchestration compresses the capability gap on evidence-grounded QA ","permalink":"https://00n.ai/research/coding-kg-agent-benchmark/","summary":"\u003ch2 id=\"research-question\"\u003eResearch Question\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eCan a code knowledge graph (KG) with multi-agent navigation bring 7-8B parameter models closer to frontier-model performance on code generation, repair, and completion tasks?\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eSub-questions:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eDoes KG navigation help small models write more correct code, or does it add noise they can\u0026rsquo;t filter?\u003c/li\u003e\n\u003cli\u003eDoes the multi-agent pipeline (draft → directional reasoning → test → refine) recover ground that single-agent KG loses?\u003c/li\u003e\n\u003cli\u003eWhich task types benefit most from graph evidence (generation vs. repair vs. completion)?\u003c/li\u003e\n\u003cli\u003eIs there a model-capability floor below which KG navigation is ineffective — and does it differ from the legal KG floor?\u003c/li\u003e\n\u003cli\u003eCan the pipeline enable small models to solve tasks that frontier B0 cannot?\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"experimental-setup\"\u003eExperimental Setup\u003c/h2\u003e\n\u003ch3 id=\"code-knowledge-graph\"\u003eCode Knowledge Graph\u003c/h3\u003e\n\u003cp\u003eA knowledge graph automatically extracted from a multi-module Python codebase (\u003ccode\u003epayments\u003c/code\u003e, \u003ccode\u003euser_service\u003c/code\u003e, \u003ccode\u003enotifications\u003c/code\u003e, \u003ccode\u003eemail_service\u003c/code\u003e) using static analysis.\u003c/p\u003e","title":"Coding Knowledge Graph Agent Benchmark"},{"content":"Research Question Can knowledge graph (KG) navigation bring a 7B parameter model to within 1 point of a frontier model on statutory reasoning tasks?\nSub-questions:\nHow much does KG navigation help small models vs. introduce noise? Which questions benefit most from graph evidence? What proportion of the gap is closeable via prompt engineering vs. graph quality vs. model capability? Is there a model-capability floor below which KG navigation is ineffective? Does natural language serialization of graph content help or hurt differently across model tiers? Experimental Setup Knowledge Graph A manually constructed knowledge graph covering six legal/technical domains:\nDomain Nodes Key Node Types EU GDPR 16 Statute, Article, Requirement, Penalty CCPA/CPRA 18 Statute, Article, Requirement, Exception HIPAA Breach Notification 14 Statute, Article, Requirement, Concept Section 230 10 Statute, Article, Concept DMCA Section 512 11 Statute, Article, Requirement RAG vs Fine-tuning 8 Paper, Finding, Concept Cross-references 10 Edges between statutes Graph stats: 91 nodes, 138 edges, 20 cross-domain bridge edges\nNode schema:\n{ \u0026#34;id\u0026#34;: \u0026#34;art-gdpr-6\u0026#34;, \u0026#34;label\u0026#34;: \u0026#34;GDPR Article 6 – Lawfulness of Processing\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;Article\u0026#34;, \u0026#34;domain\u0026#34;: \u0026#34;gdpr\u0026#34;, \u0026#34;summary\u0026#34;: \u0026#34;Processing of personal data is lawful only if at least one of six legal bases applies.\u0026#34;, \u0026#34;natural_language\u0026#34;: \u0026#34;Article 6 defines the six legal bases for lawful processing: consent, contract, legal obligation, vital interests, public task, and legitimate interests. At least one basis must apply before processing personal data. Without a valid legal basis, processing is unlawful by default.\u0026#34;, \u0026#34;key_points\u0026#34;: [\u0026#34;Consent (Art 6(1)(a))\u0026#34;, \u0026#34;Contract necessity\u0026#34;, \u0026#34;Legal obligation\u0026#34;, \u0026#34;Vital interests\u0026#34;, \u0026#34;Public task\u0026#34;, \u0026#34;Legitimate interests\u0026#34;], \u0026#34;evidence\u0026#34;: { \u0026#34;source\u0026#34;: \u0026#34;GDPR Regulation (EU) 2016/679\u0026#34;, \u0026#34;url\u0026#34;: \u0026#34;https://gdpr-info.eu/art-6-gdpr/\u0026#34;, \u0026#34;quote\u0026#34;: \u0026#34;Processing shall be lawful only if at least one of the following applies...\u0026#34; } } Edge schema:\n{ \u0026#34;source\u0026#34;: \u0026#34;art-230-c1\u0026#34;, \u0026#34;target\u0026#34;: \u0026#34;art-512c\u0026#34;, \u0026#34;relation\u0026#34;: \u0026#34;DIFFERS_FROM\u0026#34;, \u0026#34;assertion\u0026#34;: \u0026#34;230(c)(1): blanket publisher immunity. 512(c): immunity requires active compliance (agent, notice-and-takedown).\u0026#34;, \u0026#34;bridge\u0026#34;: \u0026#34;Section 230(c)(1) gives platforms blanket immunity for user content without any compliance steps, while DMCA 512(c) requires platforms to actively maintain a designated agent and respond to takedown notices. This is the key contrast: passive immunity vs. conditional safe harbor.\u0026#34; } Questions Five questions spanning all six domains, each requiring specific statutory knowledge:\n# Question Domain Key Facts in Answer Key Q1 Does EU GDPR require explicit consent for non-essential analytics cookies, and what is the maximum fine? GDPR Explicit opt-in required (Art 5/6/7); €20M or 4% turnover (Art 83 Tier 2) Q2 CCPA/CPRA notice requirements for data deletion and statutory deadline? CCPA 45-day response (extendable 45 more); verify identity; direct service providers; exceptions Q3 Difference between Section 230 immunity and DMCA safe harbor? §230/DMCA 230 = passive blanket immunity; 512 = conditional copyright safe harbor; 230(e)(2) preserves IP law Q4 HIPAA breach definition and notification timelines? HIPAA 4-factor risk assessment; 60-day individual notice; 500+ triggers media + HHS notice Q5 Does RAG always outperform fine-tuning for QA? ML Literature Complementary, not competing; hybrid outperforms either alone Models Role Model Provider Parameters Small model 1 qwen2.5:7b Ollama (local) 7B Small model 2 mistral:7b Ollama (local) 7B Small model 3 llama3.1:8b Ollama (local) 8B Frontier baseline GLM-5.2 DeepInfra (frontier) Unified grader GLM-5.2 DeepInfra (frontier) Note: The grader is the same model as the frontier baseline. The grader does not see which model produced which answer — answers are labeled B0/KG, not by model name.\nConditions B0 (Vanilla LLM): One direct model answer, no tools or retrieval. Temperature 0.3.\nKG-Agent (Graph Navigation): Four-step pipeline:\nDecompose — LLM breaks the question into 2–4 sub-questions (knowledge needs) Navigate — For each sub-question, query the graph by keyword → retrieve nodes + 1-hop neighbors → LLM decides whether to explore further Reason — LLM traces a reasoning path through visited nodes/edges, citing each fact and edge Synthesize — LLM writes final answer following the reasoning path Tier 1 Prompt Fixes (between initial and post-Tier 1 runs):\nSynthesis: \u0026ldquo;Confidence first\u0026rdquo; format, real legal citations (not node IDs), 300–500 word target, no verbatim repetition Reasoning: anti-repetition rule, keep path linear Navigation budget: 6 → 8 steps Tier 2 NL Serialization (between Tier 1 and Tier 2 runs):\nDomain-grouped serialization — nodes grouped by legal domain, noise deprioritized Natural language node descriptions — 2–4 sentence rich explanations replace telegraphic summaries Edge bridge sentences — cross-domain edges foregrounded with explanatory reasoning bridges LLM relevance filter — GLM-5.2 scores nodes 0–5 before presentation (threshold: keep 2+) Descriptive reference labels — navigation prompts use \u0026ldquo;Section 230(c)(1) — Publisher Immunity\u0026rdquo; instead of raw node ID \u0026ldquo;art-230-c1\u0026rdquo; Grading Unified GLM-5.2 grader scores each answer on 5 dimensions (0–10 scale): Accuracy, Citation Quality, Coverage, Calibration, Cognitive Load, and Overall. The grader sees the authoritative answer key and source. Answers are labeled B0/KG, not by model.\nResults Post-Tier 1 (prompt fixes: citation format, conciseness, 8-step budget) Model Cond Q1 Q2 Q3 Q4 Q5 Avg qwen2.5:7b B0 5.6 4.0 6.8 3.8 4.0 4.8 qwen2.5:7b KG 6.6 6.0 7.8 5.2 6.0 6.3 Lift +1.0 +2.0 +1.0 +1.4 +2.0 +1.5 mistral:7b B0 7.0 4.0 6.0 4.0 5.0 5.2 mistral:7b KG 7.0 5.0 7.0 4.0 8.0 6.2 Lift +0.0 +1.0 +1.0 +0.0 +3.0 +1.0 llama3.1:8b B0 7.6 4.0 4.0 6.0 6.0 5.5 llama3.1:8b KG 5.0 2.0 2.0 1.0 4.0 2.8 Lift −2.6 −2.0 −2.0 −5.0 −2.0 −2.7 Tier 2 (NL serialization + relevance filter + edge bridges) Model Cond Q1 Q2 Q3 Q4 Q5 Avg qwen2.5:7b B0 7.0 — 6.0 4.0 4.0 5.3 qwen2.5:7b KG 6.0 — 3.0 3.0 8.0 5.0 Lift −1.0 — −3.0 −1.0 +4.0 −0.3 mistral:7b B0 7.5 6.0 6.0 5.0 3.0 5.5 mistral:7b KG 6.0 5.0 6.0 3.0 0.0 4.0 Lift −1.5 −1.0 +0.0 −2.0 −3.0 −1.5 llama3.1:8b B0 5.0 5.0 4.0 4.0 6.0 4.8 llama3.1:8b KG 6.0 6.0 8.0 — 4.0 6.0† Lift +1.0 +1.0 +4.0 — −2.0 +1.2† †Q4 grading failed; average over 4 questions. \u0026ldquo;—\u0026rdquo; = grading failure.\nFrontier Baseline Model Q1 Q2 Q3 Q4 Q5 Avg GLM-5.2 B0 10 8 4 9.6 8 7.9 Cross-Tier Comparison Model Tier 1 B0 Tier 1 KG Tier 1 Lift Tier 2 B0 Tier 2 KG Tier 2 Lift qwen2.5:7b 4.8 6.3 +1.5 5.3 5.0 −0.3 mistral:7b 5.2 6.2 +1.0 5.5 4.0 −1.5 llama3.1:8b 5.5 2.8 −2.7 4.8 6.0† +1.2† Analysis Where KG Helped Q3 (§230 vs DMCA): Post-Tier 1 qwen scored 7.8 (frontier scored 4.0). The graph\u0026rsquo;s DIFFERS_FROM and EXCEPTS edges provided the 230(e)(2) IP preservation point and real statutory citations that the vanilla model couldn\u0026rsquo;t recall. Tier 2 llama scored 8.0 on the same question — the NL bridge sentences gave the weakest model the reasoning scaffolding it needed.\nQ2 (CCPA deletion): Post-Tier 1 qwen scored 6.0 (B0=4.0, +2.0 lift). The graph\u0026rsquo;s deletion-notice, procedure, verification, and exception nodes gave the small model the full statutory flow.\nQ5 (RAG vs FT): Tier 2 qwen scored 8.0 (B0=4.0, +4.0 lift). The RAG/FT literature nodes provided the comparative findings the model couldn\u0026rsquo;t recall.\nWhere KG Hurt llama3.1:8b Tier 1: −2.7 lift. The model is too weak to synthesize multi-source graph evidence. The 8-step budget exposed it to more noise without improving reasoning. This defines a model-capability floor.\nTier 2 mid-tier regression: qwen went from +1.5 to −0.3, mistral from +1.0 to −1.5. The NL serialization added token load and reasoning scaffolding that mid-tier models didn\u0026rsquo;t need — they could already build bridge sentences implicitly from the telegraphic format. The richer context introduced noise for models that didn\u0026rsquo;t need the help.\nCapability-Dependent Intervention The most striking finding: Tier 2 NL serialization helped the weakest model but hurt mid-tier models.\nModel Tier 1 → Tier 2 Lift Change Pattern qwen2.5:7b +1.5 → −0.3 (↓1.8) NL scaffolding adds noise for capable model mistral:7b +1.0 → −1.5 (↓2.5) Same — mid-tier doesn\u0026rsquo;t need scaffolding llama3.1:8b −2.7 → +1.2 (↑3.9) NL scaffolding provides bridge sentences the model couldn\u0026rsquo;t generate itself This suggests NL serialization is a capability-dependent intervention. It helps models below a capability floor by providing reasoning bridges they can\u0026rsquo;t construct themselves. It hurts models above the floor by adding token load and redundant context they could already generate internally.\nGap to Frontier Configuration Best Small KG Avg Frontier Avg Gap No KG (B0 only) 5.5 7.9 2.4 pts Tier 1 KG (best) 6.3 (qwen) 7.9 1.6 pts Tier 2 KG (best) 6.0† (llama) 7.9 1.9† pts KG navigation narrows the gap by 30–40% but does not close it. On individual knowledge-intensive questions (Q3), KG-augmented small models can exceed frontier performance — but this is not consistent across all questions.\nPer-Question Crossover Q3 is the one question where KG-augmented small models beat the frontier model:\nModel Q3 Score Frontier B0 (GLM-5.2) 4.0 Tier 1 qwen KG 7.8 Tier 2 llama KG 8.0 The frontier model\u0026rsquo;s parametric knowledge of §230/DMCA interaction was weak (it scored 4.0), while the graph provided exactly the structured knowledge the small models needed. On Q1 (GDPR), where the frontier model\u0026rsquo;s parametric knowledge is strong (scored 10), no small model configuration closed the gap.\nSaturation Experiment: Separating Graph Completeness from Model Capability The experiments above confound two factors: graph completeness (does the KG contain all needed knowledge?) and model capability (can the model synthesize the provided evidence into a high-quality answer?). To separate them, we constructed a saturating graph for Q1 — one where every fact, edge, and reasoning bridge needed to answer the question is explicitly present, with natural language bridge sentences connecting the full chain:\nArt 5 (lawfulness) → Art 6 (consent as legal basis) → Art 7 (consent conditions) → ePrivacy Art 5(3) (non-essential cookies require opt-in) → Art 83 Tier 2 (€20M or 4% turnover)\nGraph enrichment: Added 4 nodes (ePrivacy Directive, Art 5(3), non-essential cookies, essential cookies), 7 edges, and 15 bridge sentences. Graph grew from 87 to 91 nodes and 131 to 138 edges (20 with bridges).\nSaturation Results Model B0 KG (saturating) Lift Gap to Frontier (10.0) qwen2.5:7b 5.0 8.0 → 9.0 +3.0 → +4.0 1.0 mistral:7b 3.0 7.0 → 6.2 +4.0 → +1.6 3.8 llama3.1:8b 5.0 3.0 → 3.6 −2.0 → −1.4 6.4 Synthesis prompt fix: The initial saturating run produced step-by-step lab report format (\u0026ldquo;Step 1: FACT\u0026hellip;\u0026rdquo;). Fixed the synthesis prompt to require flowing legal analysis (\u0026ldquo;Write 2-4 flowing paragraphs. Do NOT use Step 1: FACT labels.\u0026rdquo;). This raised qwen from 8.0 to 9.0 — 1 point from frontier.\nQ1 Across All Runs Model Tier 1 KG Tier 2 KG Saturating KG qwen2.5:7b 6.6 6.0 8.0 mistral:7b 7.0 6.0 7.0 llama3.1:8b 5.0 6.0 3.0 Interpretation Graph completeness matters. qwen improved from 6.0 (Tier 2, incomplete graph) to 8.0 (saturating graph) — the ePrivacy Directive node and bridge sentences provided the missing knowledge. The gap to frontier narrowed from 4 points to 2 points.\nModel capability is still the ceiling. Even with a saturating graph (0 reasoning gaps for qwen), the score was 8.0, not 10.0. The remaining 2-point gap is synthesis quality: the answer was correct and well-cited but verbose and step-by-step rather than a polished legal analysis. The frontier model produced a concise, authoritative answer with correct citation hierarchy — the 7B model reproduced the facts but in a procedural format.\nCapability floor confirmed. llama3.1:8b scored 3.0 despite having the same saturating graph as qwen. Its answer focused on fine structure and missed the cookie consent analysis entirely — a synthesis failure, not a knowledge gap. Below B0 ≈ 5.0, graph completeness does not help because the model cannot synthesize multi-source evidence regardless of how complete it is.\nAnswer: Both are independent bottlenecks. A complete graph narrows the gap by 60–80% for capable models (qwen: 5 → 8, closing 3 of 5 points). The remaining 20–40% is synthesis capability that requires a larger model. Graph completeness and model capability are additive but independent — fixing one without the other yields suboptimal results.\nMulti-Agent KG: Draft-Validate-Refine on the Reasoning Graph The KG-agent is a single-agent pipeline — one model does everything. The MA+KG pipeline separates the workflow into 4 specialized roles:\nRetriever — Navigates the reasoning graph (decompose → keyword query → accumulate evidence) Drafter — Writes a draft answer using only graph evidence, with flowing legal analysis format Validator — Audits the draft against the graph as ground truth. For each claim: traces to a graph node? Node supports the claim? Edge supports the relationship? Flags unsupported claims, checks citations, identifies gaps Refiner — Revises the draft based on validation findings Key innovation: The validator uses the GRAPH as ground truth, not its own parametric knowledge. Every claim must trace to a node; every reasoning step must follow an edge.\nMA+KG Results Model Cond Q1 Q2 Q3 Q4 Q5 Avg Lift qwen2.5:7b B0 8.0 5.0 5.0 4.0 6.0 5.6 — qwen2.5:7b KG 4.0 6.0 7.0 4.0 8.0 5.8 +0.2 qwen2.5:7b MA+KG 7.0 6.0 6.0 5.0 7.0 6.2 +0.6 mistral:7b B0 5.0 6.0 7.6 6.0 5.0 5.9 — mistral:7b KG 4.0 3.0 8.0 2.0 6.0 4.6 −1.3 mistral:7b MA+KG 6.0 2.0 5.4 5.0 6.0 4.9 −1.0 llama3.1:8b B0 5.0 5.0 5.0 3.0 4.0 4.4 — llama3.1:8b KG 5.0 4.0 2.0 3.0 8.0 4.4 +0.0 llama3.1:8b MA+KG 6.0 2.0 4.0 7.0 3.0 4.4 +0.0 Gap to Frontier (7.9) Model Gap B0 Gap KG Gap MA+KG qwen2.5:7b 2.3 2.1 1.7 mistral:7b 2.0 3.3 3.0 llama3.1:8b 3.5 3.5 3.5 MA+KG Findings MA+KG improves consistency over single-agent KG (qwen). Single-agent KG was volatile: +2 on Q3, +2 on Q5, but −4 on Q1 — net lift +0.2. MA+KG was consistent: +1 on 4/5 questions — net lift +0.6. The validator-refiner cycle flattened the variance.\nMA+KG narrows the gap to frontier for qwen. Gap: B0=2.3 → KG=2.1 → MA+KG=1.7 — a 26% reduction from B0. This is the best non-saturating result measured.\nMA+KG does NOT help mistral or llama on average. The validator-refiner overhead hurts models that are either very capable (mistral) or very weak (llama).\nBiggest single-question lift in the entire series: llama Q4 B0=3 → MA+KG=7 (+4). The validator caught missing HIPAA 500+ thresholds and the refiner added them.\nMA+KG recovers from KG catastrophic failures. mistral Q4: KG=2 → MA+KG=5. llama Q3: KG=2 → MA+KG=4.\nMA+KG can also collapse. llama Q5: KG=8 → MA+KG=3 (−5). The refiner destroyed a strong draft — same small model that produced the draft can\u0026rsquo;t reliably improve it.\nMA+KG is a consistency improver, not a ceiling raiser. It trades peak performance for reduced variance.\nSynthesis Scaffolding: Can We Give Small Models Frontier Synthesis? The diagnosis from all previous experiments: the remaining gap is synthesis capability, not knowledge. Frontier models weave facts into flowing arguments, draw implications not in the graph, prioritize strongest points first, and use legal register.\nCan we scaffold this capability into small models through a multi-step pipeline?\nDesign Condition Retriever Synthesis Coach Implication Gen Drafter B0 — — — small (no graph) Synth-Self small small (self-coached) small small Synth-Coach small frontier (outline only, NOT answer) small small Frontier-Synth small — — frontier (from graph, upper bound) Key innovation in Synth-Coach: The frontier model sees the graph evidence and generates an argument OUTLINE (key points, logical flow, implication hints, register guidance) — NOT the answer. The small model uses the outline to construct the answer itself.\nSynthesis Scaffolding Results (qwen2.5:7b) Q B0 Synth-Self Synth-Coach Frontier-Synth Q1 (GDPR) 6.6 1.4 4.0 9.6 Q2 (CCPA) 4.0 6.0 7.0 9.0 Q3 (§230) 7.4 3.4 4.2 9.4 Q4 (HIPAA) 4.0 5.0 4.0 10.0 Q5 (RAG/FT) 6.0 3.0 5.0 10.0 Avg 5.6 3.8 4.8 9.6 Lift vs B0 — −1.8 −0.8 +4.0 Gap to Frontier B0 (7.9) Condition Avg Gap B0 5.6 2.3 KG (single) 5.8 2.1 MA+KG 6.2 1.7 (best small) Synth-Self 3.8 4.1 Synth-Coach 4.8 3.1 Frontier-Synth 9.6 −1.7 (beats frontier B0) Synthesis Findings Synthesis scaffolding HURTS small models on average. Synth-Self: −1.8 lift. Synth-Coach: −0.8 lift. Both are worse than doing nothing.\nThe multi-step pipeline is the problem, not the coaching. The outline adds 700–900 tokens of structure. The small model then has to integrate outline + implications + evidence + question into a single draft — that\u0026rsquo;s MORE synthesis work, not less. The intervention that was supposed to reduce synthesis difficulty actually increased it.\nSynth-Coach works when the model lacks knowledge (Q2 CCPA: 4→7) but hurts when it already knows the domain (Q3 §230: 7.4→4.2). The coaching pipeline overrides good parametric knowledge with worse graph-grounded synthesis.\nFrontier-Synth beats frontier B0 (9.6 vs 7.9). The graph HELPS the frontier model on precise citation questions — confirming that the graph is a knowledge asset, not a synthesis asset.\nThe uncomfortable truth: synthesis capability cannot be scaffolded through prompt engineering. The remaining ~1.7-point gap is model capability. You cannot scaffold your way to synthesis — the small model needs to actually do the synthesis, and it can\u0026rsquo;t.\nFull Ranking of All Interventions (qwen2.5:7b) Intervention Avg Gap to Frontier (7.9) Frontier-Synth (upper bound) 9.6 beats frontier B0 MA+KG (validator-refiner) 6.2 1.7 (best small) KG (single-agent) 5.8 2.1 B0 (vanilla) 5.6 2.3 Synth-Coach (frontier outline) 4.8 3.1 Synth-Self (self-coached) 3.8 4.1 The Three-Layer Gap Model The gap between small and frontier models splits into three parts:\nKnowledge gap → solvable (graph retrieval, saturation closes 60–90%) Error correction gap → solvable (validator-refiner adds +0.6 consistency) Synthesis capability gap → NOT solvable through prompt engineering (multi-step pipelines hurt small models more than they help) Interventions on the retrievable part work. Interventions on the synthesis part do not — at least not through prompt engineering alone.\nLimitations Grader = frontier model. The same GLM-5.2 serves as both frontier baseline and grader. While blinded to model identity, self-preference bias is possible. 5 questions only. The question set is small and domain-specific. Generalization is not demonstrated. Keyword-based retrieval (Tier 1). The queryGraph() function uses stopword-filtered keyword overlap, not semantic similarity. Returns noise nodes that can derail the agent. Tier 2\u0026rsquo;s LLM relevance filter partially addresses this. Non-deterministic generation. Ollama models are non-deterministic at temperature 0.3. B0 scores vary ±1.0 across runs. KG scores vary ±0.5. GLM-5.2 reasoning mode. The model defaults to thinking/reasoning that can consume all max_tokens and produce empty completions. Workaround: chat_template_kwargs: {enable_thinking: false}. Grading failures. Two grading failures in Tier 2 (Q2 qwen, Q4 llama) reduce the effective comparison set. Single grader. Inter-grader agreement was not measured. Reproduction Prerequisites # Software node v20+ (tested with v26.0.0) python3 (for graph analysis) # Ollama (small models) ollama pull qwen2.5:7b ollama pull mistral:7b ollama pull llama3.1:8b # DeepInfra (frontier model + grader) export DEEPINFRA_API_KEY=your_key_here Running the Experiment cd projects/multi-agent-framework # Tier 1: B0 vs KG-agent (all 3 models, 5 questions) DEEPINFRA_API_KEY=$KEY node src/kg-experiment-multi.js \\ --models qwen2.5:7b,mistral:7b,llama3.1:8b --questions 5 # Tier 2: B0 vs KG-agent with NL serialization DEEPINFRA_API_KEY=$KEY node src/kg-experiment-tier2.js \\ --models qwen2.5:7b,mistral:7b,llama3.1:8b # Frontier B0 baseline DEEPINFRA_API_KEY=$KEY node src/kg-experiment-frontier-b0.js # MA+KG: B0 vs KG vs MA+KG (draft-validate-refine) DEEPINFRA_API_KEY=$KEY node src/kg-experiment-ma.js \\ --models qwen2.5:7b,mistral:7b,llama3.1:8b --questions 5 # Synthesis scaffolding: B0 vs Synth-Self vs Synth-Coach vs Frontier-Synth DEEPINFRA_API_KEY=$KEY node scripts/run-synth-experiment.mjs \\ --models qwen2.5:7b --questions 5 # Capability experiments: frontier control, adversarial, abstention DEEPINFRA_API_KEY=$KEY node scripts/run-control-and-adversarial.mjs Repository Structure projects/multi-agent-framework/ ├── knowledge-graph/graph.json # KG: 91 nodes, 138 edges, 20 bridges ├── src/ │ ├── kg-agent.js # Tier 1 KG-agent │ ├── kg-agent-tier2.js # Tier 2 KG-agent (NL serialization) │ ├── kg-ma-agent.js # MA+KG agent (draft-validate-refine) │ ├── kg-synth-agent.js # Synthesis scaffolding agent │ ├── kg-experiment-multi.js # Tier 1 experiment runner │ ├── kg-experiment-tier2.js # Tier 2 experiment runner │ ├── kg-experiment-frontier-b0.js # Frontier baseline runner │ ├── kg-experiment-ma.js # MA+KG experiment runner │ ├── baselines.js # B0 + unified grader │ ├── agents.js # LLM call helper │ └── logger.js # Trace logging ├── logs/ │ ├── kg-multi-8408485f/ # Tier 1 qwen post-Tier 1 │ ├── kg-multi-70d770dc/ # Tier 1 mistral post-Tier 1 │ ├── kg-multi-cb7d5f56/ # Tier 1 llama post-Tier 1 │ ├── kg-tier2-f390efba/ # Tier 2 all models │ ├── kg-saturation-78fe42e1/ # Saturation experiment (Q1 only) │ ├── kg-capability-268cbc45/ # Capability experiments │ ├── kg-ma-7d3f0b30/ # MA+KG experiment │ ├── kg-synth-4321e4da/ # Synthesis scaffolding experiment │ └── kg-frontier-b0-f992ea91/ # Frontier B0 baseline ├── docs/ │ ├── kg-experiment-writeup.md # Full internal writeup │ ├── kg-agent-worked-example.md # End-to-end trace on Q3 │ ├── kg-gap-analysis.md # Gap analysis │ └── kg-learnings.md # Consolidated learnings (all experiments) └── scripts/ ├── enrich-graph-tier2.py # Graph enrichment (domain, NL, bridges) ├── saturate-gdpr-q1.py # Q1 saturating graph enrichment ├── run-saturation-experiment.mjs # Saturation experiment runner ├── run-control-and-adversarial.mjs # Capability experiments ├── run-synth-experiment.mjs # Synthesis scaffolding runner └── run-codex-design-first-replay.py # Codex replay harness Key Takeaways KG navigation provides consistent positive lift for mid-tier small models (+1.0 to +1.5 points with Tier 1 prompts), narrowing the gap to frontier by 30–40%. MA+KG (draft-validate-refine) is the best small-model intervention. MA+KG narrows the gap to frontier from 2.3 to 1.7 for qwen — a 26% reduction from B0 — without graph saturation. It improves consistency (no catastrophic collapses) but doesn\u0026rsquo;t raise the ceiling. Synthesis scaffolding does NOT work. Multi-step pipelines (outline → implications → draft) hurt small models on average (−0.8 to −1.8 lift). The pipeline adds cognitive load, not structure. Synthesis capability cannot be scaffolded through prompt engineering. The gap splits into three layers: knowledge (solvable via graph retrieval), error correction (solvable via validator-refiner), and synthesis capability (NOT solvable through prompt engineering). The remaining ~1.7-point gap is model capability. On knowledge-intensive questions where frontier parametric knowledge is weak, KG can exceed frontier performance (Q3: small models scored 7.8–8.0 vs frontier 4.0). NL serialization is capability-dependent. It helps the weakest model (llama: −2.7 → +1.2) but hurts mid-tier models (qwen, mistral) by adding redundant scaffolding. There is a model-capability floor. Below B0 ≈ 5.0, KG navigation can hurt. Above it, KG reliably helps. Graph completeness and model capability are independent bottlenecks. A saturating graph narrowed qwen\u0026rsquo;s gap from 5 to 1 point (80% closure). The remaining 1 point is synthesis quality. KG is a small-model intervention, not a universal enhancer. Frontier model + KG = 10.0 = frontier B0. The graph adds nothing when parametric knowledge is sufficient. KG improves false-premise rejection quality, not rejection rate. Rejections become better cited and more authoritative (qwen AdvQ3: 3.0 → 8.0) butnot more frequent. KG does not prevent hallucination. No model abstained when evidence was removed. But graph context improved parametric recall quality (qwen: 2.0 → 9.0 on abstention test via correct $750 recall triggered by surrounding nodes). Frontier-Synth (frontier model synthesizing from graph) beats frontier B0 (9.6 vs 7.9). The graph is a knowledge asset for frontier models on citation-heavy questions, even though it\u0026rsquo;s redundant for their parametric knowledge. Related Coding Knowledge Graph Agent Benchmark — Same experimental design applied to code generation, repair, and completion tasks Multi-Agent Framework Benchmark — Orchestration compresses the capability gap between small and frontier models Consolidated Learnings — Full learnings document with all experiments, open questions, and artifact index Worked Example: KG-Agent on Q3 — Full end-to-end trace ","permalink":"https://00n.ai/research/knowledge-graph-agent-benchmark/","summary":"\u003ch2 id=\"research-question\"\u003eResearch Question\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eCan knowledge graph (KG) navigation bring a 7B parameter model to within 1 point of a frontier model on statutory reasoning tasks?\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eSub-questions:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eHow much does KG navigation help small models vs. introduce noise?\u003c/li\u003e\n\u003cli\u003eWhich questions benefit most from graph evidence?\u003c/li\u003e\n\u003cli\u003eWhat proportion of the gap is closeable via prompt engineering vs. graph quality vs. model capability?\u003c/li\u003e\n\u003cli\u003eIs there a model-capability floor below which KG navigation is ineffective?\u003c/li\u003e\n\u003cli\u003eDoes natural language serialization of graph content help or hurt differently across model tiers?\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"experimental-setup\"\u003eExperimental Setup\u003c/h2\u003e\n\u003ch3 id=\"knowledge-graph\"\u003eKnowledge Graph\u003c/h3\u003e\n\u003cp\u003eA manually constructed knowledge graph covering six legal/technical domains:\u003c/p\u003e","title":"Knowledge Graph Agent Benchmark"},{"content":"Companion document to the 00n.ai Multi-Agent Benchmark\nThis document maps our benchmark findings to the published research literature. Each finding is evaluated against verified papers from arXiv, ACL, NeurIPS, ICML, and EMNLP. Papers were confirmed to resolve and titles verified against abstracts.\nFinding 1: MA Compresses the Capability Gap Between Small and Large Models A 7B model with multi-agent orchestration (Ollama qwen2.5:7b, avg 8.0–8.1) matches frontier-model B2 performance (GPT-4o search+reflection, avg 8.2–8.4) on evidence-grounded tasks.\nRelevant Papers 1. Mixture-of-Agents Enhances Large Language Model Capabilities\nAuthors: Jue Wang et al. Venue/Year: arXiv, June 2024 ID/URL: arXiv:2406.04692 Relation: Supports Key claim: MoA constructs a layered architecture where multiple LLM agents take outputs from the previous layer as auxiliary information. Using only open-source LLMs, MoA achieves 65.1% on AlpacaEval 2.0, surpassing GPT-4 Omni (57.5%). This directly demonstrates that orchestrating weaker open-source models can match or exceed a single strong frontier model. Synthesis: The MoA result is the clearest published precedent for our finding. Their layered aggregation of open-source models surpassing GPT-4o validates the core hypothesis that multi-agent orchestration can compress the capability gap. Our work extends this to evidence-grounded freshness questions with a draft-validate-ground pattern rather than layered aggregation. 2. More Agents Is All You Need\nAuthors: Deheng Ye et al. Venue/Year: TMLR (Transactions on Machine Learning Research), 2024 ID/URL: arXiv:2402.05120 Relation: Supports Key claim: Performance of LLMs scales with the number of agents instantiated via a simple sampling-and-voting method (Agent Forest). The enhancement is orthogonal to existing methods and correlates with task difficulty — harder tasks benefit more from additional agents. Synthesis: This paper provides a scaling law for agent数量: more agents → better performance, with diminishing returns on easy tasks. Our finding that a 7B model with MA matches GPT-4o is consistent with their observation that multi-agent scaling helps most where single-model performance is weakest. 3. Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms\nAuthors: Xinlin Wang, Mats Brorsson Venue/Year: arXiv, April 2026 ID/URL: arXiv:2604.19299 Relation: Partially relates / Nuanced Key claim: The first large-scale study of \u0026lt;10B models under three paradigms (base, single-agent + tools, multi-agent). Finds that single-agent systems achieve the best balance between performance and cost, while multi-agent setups add overhead with limited gains for small models. Synthesis: This paper partially contradicts our finding. It finds multi-agent overhead not worthwhile for \u0026lt;10B models in general tasks. However, our benchmark focuses on evidence-grounded freshness questions where the orchestration pattern (draft-validate-ground) is specifically designed for the task. The difference suggests that MA benefit is task-dependent: general reasoning may not benefit, but structured evidence-grounding tasks do. 4. Improving Factuality and Reasoning in Language Models through Multiagent Debate\nAuthors: Yilun Du et al. Venue/Year: arXiv, May 2023 ID/URL: arXiv:2305.14325 Relation: Supports Key claim: Multiple LLM instances proposing and debating over multiple rounds significantly enhances mathematical and strategic reasoning and improves factual validity, reducing hallucinations. The approach works with black-box models using identical procedures across tasks. Synthesis: This seminal paper established that multi-agent debate improves factuality — the same mechanism our benchmark exploits. Their finding that debate reduces hallucinations aligns with our observation that MA orchestration helps on evidence-grounded tasks where hallucination is the primary failure mode. Finding 2: MA Helps Small Models More Than Large Models On GPT-4o, MA slightly hurt vs B2 (7.8 vs 8.4). On Ollama 7B, MA slightly helped vs B2 (8.1 vs 8.0). The overhead benefits weaker models more.\nRelevant Papers 1. More Agents Is All You Need\nAuthors: Deheng Ye et al. Venue/Year: TMLR, 2024 ID/URL: arXiv:2402.05120 Relation: Supports Key claim: The degree of enhancement from multi-agent scaling is correlated with task difficulty. Harder tasks (where the base model is weaker) benefit more from additional agents. Synthesis: Our finding that MA helps the 7B model more than GPT-4o maps directly to their task-difficulty correlation. The 7B model starts from a weaker baseline (analogous to \u0026ldquo;harder\u0026rdquo; relative performance), so the multi-agent boost is more impactful. GPT-4o is already strong, leaving less room for improvement — and introducing overhead that can net hurt. 2. SLMJury: Can Small Language Models Judge as Well as Large Ones?\nAuthors: Anish Laddha, Nitesh Pradhan, Gaurav Srivastava Venue/Year: arXiv, June 2026 ID/URL: arXiv:2606.07810 Relation: Partially contradicts / Nuances Key claim: Benchmarks 16 SLM judges (0.6B–14B). Finds that multi-agent debate (Reflect-Critique-Refine) degrades accuracy across all tested configurations for small models, whereas top judges resist adversarial personas with ≤0.55% variance. No single SLM dominates across tasks. Synthesis: SLMJury\u0026rsquo;s finding that debate degrades SLM accuracy is a partial contradiction. However, their debate protocol (Reflect-Critique-Refine) differs from our draft-validate-ground pattern — ours uses deterministic grounding rather than peer critique. The difference suggests that the type of multi-agent interaction matters: structured grounding helps small models, while open-ended debate can hurt them. 3. Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms\nAuthors: Xinlin Wang, Mats Brorsson Venue/Year: arXiv, April 2026 ID/URL: arXiv:2604.19299 Relation: Partially contradicts Key claim: Multi-agent setups for \u0026lt;10B models add overhead with limited gains; single-agent + tools is the better paradigm for small models. Synthesis: This challenges our finding that MA helps the 7B model. The discrepancy may stem from task design: their evaluation covers general tasks, while ours targets evidence-grounded freshness where the draft-validate-ground pattern directly addresses the 7B\u0026rsquo;s main weakness (non-convergence/hallucination). The literature suggests MA for small models is task-dependent, not universally beneficial. 4. Self-Consistency Improves Chain of Thought Reasoning in Language Models\nAuthors: Xuezhi Wang et al. Venue/Year: ICLR 2023 ID/URL: arXiv:2203.11171 Relation: Partially relates Key claim: Sampling diverse reasoning paths and selecting the most consistent answer boosts performance by large margins (+17.9% GSM8K, +11.0% SVAMP). The method is more effective on harder tasks where single-path reasoning fails. Synthesis: Self-consistency is a single-model analogue to multi-agent voting. The finding that it helps more on harder tasks (where models are weaker) parallels our observation that MA helps the 7B more than GPT-4o. Both suggest that error-correction mechanisms have diminishing returns as base capability increases. Finding 3: Orchestration Matters More Than Retrieval Quality for Small Models The draft-then-validate pattern (v2.2) and deterministic grounding (v2.3) improved convergence from non-convergence to 85–90% on the 7B model.\nRelevant Papers 1. Reflexion: Language Agents with Verbal Reinforcement Learning\nAuthors: Noah Shinn et al. Venue/Year: NeurIPS 2023 ID/URL: arXiv:2303.11366 Relation: Supports Key claim: Agents verbally reflect on task feedback and maintain reflective text in episodic memory, achieving 91% pass@1 on HumanEval (surpassing GPT-4 at 80%). The framework is flexible across feedback types and sources. Synthesis: Reflexion validates the core idea behind our draft-validate pattern: structured verbal reflection on task outcomes improves subsequent decisions without weight updates. Our v2.2 draft-then-validate is analogous — the validator provides verbal feedback that grounds the drafter. The 91% → 80% gap over GPT-4 mirrors our 7B+MA matching GPT-4o. 2. Self-Refine: Iterative Refinement with Self-Feedback\nAuthors: Aman Madaan et al. Venue/Year: NeurIPS 2023 ID/URL: arXiv:2303.17651 Relation: Supports Key claim: A single LLM generates output, provides feedback, and refines iteratively — improving ~20% absolute on average across 7 tasks. Even GPT-4 benefits from self-refinement. Synthesis: Self-Refine demonstrates that the draft-feedback-refine loop is effective regardless of model size, but the gains are larger on initial lower-quality outputs. Our draft-then-validate pattern with separate roles (drafter + validator) is an inter-agent version of this same principle. 3. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection\nAuthors: Akari Asai et al. Venue/Year: arXiv, October 2023 ID/URL: arXiv:2310.11511 Relation: Supports Key claim: Self-RAG (7B and 13B) uses reflection tokens to adaptively retrieve passages, generate, and reflect on retrieved content — outperforming ChatGPT and retrieval-augmented Llama2-chat on QA, reasoning, and fact verification. The model learns to critique its own retrieval and generation. Synthesis: Self-RAG is the closest published precedent for our deterministic grounding pattern (v2.3). Both use a structured mechanism to ground generation in retrieved evidence. Self-RAG trains reflection tokens; our approach uses deterministic orchestration. The key insight is shared: for small models, explicit grounding signals matter more than for large models that can implicitly self-ground. 4. Large Language Model based Multi-Agents: A Survey of Progress and Challenges\nAuthors: Taicheng Guo et al. Venue/Year: arXiv, February 2024 (v2 April 2024) ID/URL: arXiv:2402.01680 Relation: Partially relates Key claim: Surveys multi-agent systems based on LLMs, identifying profiling, communication, and capacity-building mechanisms as essential dimensions. Highlights that agent orchestration patterns — role assignment, communication protocols, and memory — are the key design decisions. Synthesis: This survey provides the taxonomy context for our orchestration patterns. The finding that orchestration architecture (not just model capability) determines MA performance is consistent with our observation that v2.2/v2.3 patterns drove convergence improvements independent of retrieval quality. Finding 4: A Weak Skeptic on a Small Model Can Backfire In earlier experiments, a 7B skeptic agent overcorrected and dismissed valid evidence, scoring worse than vanilla.\nRelevant Papers 1. SLMJury: Can Small Language Models Judge as Well as Large Ones?\nAuthors: Anish Laddha, Nitesh Pradhan, Gaurav Srivastava Venue/Year: arXiv, June 2026 ID/URL: arXiv:2606.07810 Relation: Supports Key claim: Multi-agent debate (Reflect-Critique-Refine) degrades accuracy across all tested configurations for small models (0.6B–14B). The \u0026ldquo;overthinking effect\u0026rdquo; is domain-dependent: for most judges, quick 10-token verdicts match or beat extended reasoning on mathematical judging, while extended reasoning wins on general tasks by up to 23%. Synthesis: SLMJury directly validates our finding. A small model acting as critic/skeptic can hurt accuracy because it lacks the reasoning capacity to distinguish valid from invalid evidence. The \u0026ldquo;overthinking effect\u0026rdquo; they identify — where extended reasoning by a small model degrades performance — is the same mechanism as our 7B skeptic overcorrecting and dismissing valid evidence. 2. The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge\nAuthors: Ali Keramati, Justin Cheok, Jacob Horne, Mark Warschauer Venue/Year: ACL 2026 ID/URL: arXiv:2606.10296 Relation: Supports Key claim: In two-agent debate (Constructor + Auditor), confidence aligns with reasoned quality only for the Constructor, not the Auditor (AUROC 0.804 vs 0.634). The Auditor role is systematically less reliable at self-assessing reasoning quality — confidence-based failure detection fails for the Auditor. Synthesis: This paper explains why our 7B skeptic backfired. The Auditor/skeptic role has inherently worse calibration than the Constructor role. A small model in the Auditor position has both poor reasoning and poor self-assessment, leading to overcorrection. The role asymmetry they identify is the mechanism behind our finding. 3. The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents\nAuthors: Weihao Xuan et al. Venue/Year: arXiv, January 2026 ID/URL: arXiv:2601.07264 Relation: Supports Key claim: Evidence tools (e.g., web search) systematically induce severe overconfidence in agents due to noise in retrieved information, while verification tools (e.g., code interpreters) ground reasoning through deterministic feedback and mitigate miscalibration. Synthesis: This finding directly maps to our skeptic problem. A 7B skeptic processing retrieved evidence faces the overconfidence trap: noisy evidence + weak reasoning → overconfident dismissal. Our v2.3 deterministic grounding was designed precisely to replace the unreliable skeptic with a deterministic verification step, which this paper identifies as the correct approach. Finding 5: Search Infrastructure Is the Bottleneck for Single-Agent Approaches B1 (search) often scored lower than B0 (vanilla) because irrelevant retrieval degrades output.\nRelevant Papers 1. Quantifying the Tug-of-War Between an LLM\u0026rsquo;s Internal Prior and External Evidence\nAuthors: Eric Wu et al. Venue/Year: arXiv, April 2024 (v3 February 2025) ID/URL: arXiv:2404.10198 Relation: Strongly supports Key claim: LLMs are susceptible to adopting incorrect retrieved content, overriding their own correct prior knowledge over 60% of the time. The less confident a model is in its initial response, the more likely it is to adopt incorrect retrieved content. Even GPT-4o is affected. Synthesis: This paper provides the mechanistic explanation for our B1 \u0026lt; B0 finding. When search returns irrelevant or incorrect content, models adopt it over their own knowledge more than 60% of the time. This is not a minor effect — it\u0026rsquo;s a majority failure mode. Our finding that B1 (search) underperforms B0 (vanilla) is a direct consequence of this tug-of-war dynamic. 2. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval\nAuthors: Yufeng Du et al. Venue/Year: EMNLP 2025 Findings ID/URL: arXiv:2510.05381 Relation: Supports Key claim: Even when models can perfectly retrieve all relevant information, performance still degrades substantially (13.9%–85%) as input length increases. The sheer length of the input alone hurts performance, independent of retrieval quality and without any distraction. Synthesis: This extends our B1 \u0026lt; B0 finding: even correct search results hurt because they increase context length. The problem isn\u0026rsquo;t just irrelevant retrieval — it\u0026rsquo;s that any additional context, even relevant, degrades performance. This suggests B1\u0026rsquo;s failure is structural (context length) not just qualitative (relevance). 3. Retrieval-Augmented Generation for Large Language Models: A Survey\nAuthors: Yunfan Gao et al. Venue/Year: arXiv, December 2023 (v5 March 2024) ID/URL: arXiv:2312.10997 Relation: Partially relates Key claim: Comprehensive survey of RAG paradigms (Naive, Advanced, Modular). Identifies retrieval quality, generation grounding, and augmentation timing as the tripartite foundation. Notes that indiscriminate retrieval can diminish LM versatility or lead to unhelpful response generation. Synthesis: The survey acknowledges that indiscriminate retrieval is a known problem in RAG pipelines. Our B1 result is a specific instance of this general phenomenon — search without grounding mechanisms is counterproductive. 4. Knowledge Requirements Shape LLM Responses to Context-Memory Conflict\nAuthors: Kaiser Sun et al. Venue/Year: ACL 2026 ID/URL: arXiv:2506.06485 Relation: Supports Key claim: Performance degradation under context-memory conflict is driven by task-specific knowledge reliance and conflict plausibility. Strategies like rationales or context reiteration increase context reliance, helping context-only tasks but harming those requiring parametric knowledge. These effects bias model-based evaluation. Synthesis: This paper adds nuance to our B1 finding. The harm from search isn\u0026rsquo;t just about irrelevant content — it\u0026rsquo;s about the type of task. Evidence-grounded freshness questions require context reliance, but when search returns conflicting or irrelevant information, the model is pulled toward the wrong context. The task-dependent nature they identify explains why B1 hurts more on some questions than others. Synthesis Overall Picture Published research broadly supports our five findings, with important nuances:\nFinding Support Level Key Caveat F1: MA compresses gap Supported MoA (2406.04692) is the strongest precedent; task-dependence matters F2: MA helps small \u0026gt; large Partially supported \u0026ldquo;More Agents\u0026rdquo; scaling law supports; SLMJury and Rethinking Scale partially contradict for general tasks F3: Orchestration \u0026gt; retrieval Supported Reflexion, Self-Refine, Self-RAG all validate structured grounding patterns F4: Weak skeptic backfires Strongly supported SLMJury shows debate degrades SLM accuracy; Confident Liar shows role asymmetry F5: Search is bottleneck Strongly supported Wu et al. shows 60%+ adoption of incorrect retrieval; context length alone degrades performance Where Our Work Adds Novelty Evidence-grounded freshness as a benchmark domain. Most published MA benchmarks focus on math, coding, or general QA. Evidence-grounded freshness questions combine temporal reasoning with factual verification — a task structure where MA patterns specifically address the failure modes (hallucination, non-convergence) that dominate small-model performance.\nQuantification of the crossover effect. We show a specific crossover: MA helps 7B (+0.1) but hurts GPT-4o (−0.6). While \u0026ldquo;More Agents Is All You Need\u0026rdquo; shows the general principle, we quantify it for a specific orchestration pattern on a specific task type.\nDeterministic grounding vs. critic debate. Our v2.3 deterministic grounding pattern replaces the unreliable skeptic with a verification step. SLMJury\u0026rsquo;s finding that debate degrades SLM accuracy and The Confidence Dichotomy\u0026rsquo;s distinction between evidence tools (overconfidence-inducing) and verification tools (grounding) provide the theoretical basis for why deterministic grounding works where skeptic agents fail.\nB1 \u0026lt; B0 as a specific finding. While the RAG survey notes indiscriminate retrieval as a known problem, our benchmark quantifies it in the agent-search context: search infrastructure as a net negative for single-agent approaches on evidence-grounded tasks.\nCaveats from the Literature Task dependence. Rethinking Scale (2604.19299) finds MA overhead not worthwhile for \u0026lt;10B models on general tasks. Our positive MA result for the 7B may be specific to evidence-grounded freshness tasks where structured grounding directly addresses the main failure mode.\nDebate vs. structured grounding. SLMJury shows multi-agent debate (Reflect-Critique-Refine) degrades accuracy for small models, but our draft-validate-ground pattern is not open-ended debate — it\u0026rsquo;s structured with deterministic verification. The distinction matters: the type of multi-agent interaction determines whether small models benefit.\nGeneralization. Our benchmark uses specific models (qwen2.5:7b, GPT-4o) and a specific task type. While the literature supports the underlying mechanisms, the crossover point (where MA stops helping and starts hurting) will vary by model, task, and orchestration pattern.\nEvaluation reliability. The Confident Liar paper shows that confidence signals in multi-agent debate are role-asymmetric and unreliable for the Auditor. Our evaluation uses external scoring rather than agent self-assessment, which avoids this trap — but any LLM-as-judge evaluation of multi-agent output should account for this asymmetry.\nReferences Wang, J. et al. (2024). Mixture-of-Agents Enhances Large Language Model Capabilities. arXiv:2406.04692. https://arxiv.org/abs/2406.04692\nDu, Y. et al. (2023). Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv:2305.14325. https://arxiv.org/abs/2305.14325\nYe, D. et al. (2024). More Agents Is All You Need. TMLR. arXiv:2402.05120. https://arxiv.org/abs/2402.05120\nWang, X. \u0026amp; Brorsson, M. (2026). Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms. arXiv:2604.19299. https://arxiv.org/abs/2604.19299\nLaddha, A., Pradhan, N., \u0026amp; Srivastava, G. (2026). SLMJury: Can Small Language Models Judge as Well as Large Ones? arXiv:2606.07810. https://arxiv.org/abs/2606.07810\nWang, X. et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023. arXiv:2203.11171. https://arxiv.org/abs/2203.11171\nShinn, N. et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023. arXiv:2303.11366. https://arxiv.org/abs/2303.11366\nMadaan, A. et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023. arXiv:2303.17651. https://arxiv.org/abs/2303.17651\nAsai, A. et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. arXiv:2310.11511. https://arxiv.org/abs/2310.11511\nGuo, T. et al. (2024). Large Language Model based Multi-Agents: A Survey of Progress and Challenges. arXiv:2402.01680. https://arxiv.org/abs/2402.01680\nKeramati, A., Cheok, J., Horne, J., \u0026amp; Warschauer, M. (2026). The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge. ACL 2026. arXiv:2606.10296. https://arxiv.org/abs/2606.10296\nXuan, W. et al. (2026). The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents. arXiv:2601.07264. https://arxiv.org/abs/2601.07264\nWu, E. et al. (2024). Quantifying the Tug-of-War Between an LLM\u0026rsquo;s Internal Prior and External Evidence. arXiv:2404.10198. https://arxiv.org/abs/2404.10198\nDu, Y. et al. (2025). Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. EMNLP 2025 Findings. arXiv:2510.05381. https://arxiv.org/abs/2510.05381\nGao, Y. et al. (2024). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997. https://arxiv.org/abs/2312.10997\nSun, K. et al. (2026). Knowledge Requirements Shape LLM Responses to Context-Memory Conflict. ACL 2026. arXiv:2506.06485. https://arxiv.org/abs/2506.06485\nYadav, A. et al. (2026). More Capable, Less Cooperative? When LLMs Fail At Zero-Cost Collaboration. ICML 2026. arXiv:2604.07821. https://arxiv.org/abs/2604.07821\n","permalink":"https://00n.ai/research/related-research/","summary":"\u003cp\u003e\u003cstrong\u003eCompanion document to the 00n.ai Multi-Agent Benchmark\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis document maps our benchmark findings to the published research literature. Each finding is evaluated against verified papers from arXiv, ACL, NeurIPS, ICML, and EMNLP. Papers were confirmed to resolve and titles verified against abstracts.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"finding-1-ma-compresses-the-capability-gap-between-small-and-large-models\"\u003eFinding 1: MA Compresses the Capability Gap Between Small and Large Models\u003c/h2\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cem\u003eA 7B model with multi-agent orchestration (Ollama qwen2.5:7b, avg 8.0–8.1) matches frontier-model B2 performance (GPT-4o search+reflection, avg 8.2–8.4) on evidence-grounded tasks.\u003c/em\u003e\u003c/p\u003e","title":"Related Research: Multi-Agent Framework Benchmark Validation"},{"content":"This benchmark compares four answer-generation architectures across two model backends on 20 freshness questions designed to defeat parametric knowledge.\nApproaches B0: Vanilla LLM — one direct model answer with no tools or search. B1: LLM + Search — query generation, web search/fetch, and a single evidence-grounded answer. B2: LLM + Search + Reflection — search-based draft, self-critique, and revised answer. MA: Multi-Agent — Meta-Planner → Research → Synthesis → Validator → Skeptic → Final Answer with iterative convergence and deterministic grounding. Question design The 20-question freshness pack uses facts published within ~24-48 hours of the experiment (June 22-23, 2026), sourced from NASA releases, arXiv postings, a Google Developers post, and a GitHub release feed. Questions are designed so models cannot answer from training data — B0 scores ~2-3 across both models, confirming the freshness property.\nVersions Two versions of the pipeline were tested:\nv2.2 — Draft-then-validate orchestration: the MA loop changed from validating raw evidence before drafting to Research → Synthesis draft → Validator → Skeptic. This fixed the core failure mode where small models rejected semantically obvious evidence. v2.3 — Deterministic grounding: the validator no longer calculates a subjective \u0026ldquo;overall grounding\u0026rdquo; score. Instead, it outputs per-field verdicts for each required field, and the controller computes grounding as verified_fields / total_required_fields. Results Average scores (unified GPT-4o grader, 0-10 scale) Model B0 Vanilla B1 Search B2 Search + Reflection MA Multi-Agent v2.2 Ollama 7B 2.6 7.2 8.0 8.1 v2.2 GPT-4o 2.3 8.2 8.4 7.8 v2.3 Ollama 7B 2.5 7.5 8.1 8.0 v2.3 GPT-4o 2.3 7.8 8.2 7.9 MA convergence Version Ollama 7B GPT-4o v2.2 17/20 (85%) 19/20 (95%) v2.3 18/20 (90%) 19/20 (95%) Headline finding Multi-agent orchestration compresses the capability gap between small and large models.\nMA on Ollama 7B (8.0-8.1) matches or beats MA on GPT-4o (7.8-7.9) and nearly matches B2 on GPT-4o (8.2-8.4). The overhead helps small models more than large models.\nImplication: Proper orchestration + retrieval + deterministic verification can bring a small model up to frontier-model performance on evidence-grounded tasks — at ~100x lower cost.\nCost analysis Ollama MA GPT-4o B2 LLM calls/question ~6 ~4 Time/question 60-80s 15-20s Cost/question ~$0 (local) ~$0.05-0.10 API Convergence 90-95% N/A (single pass) Trade-off: ~4-5x more wall-clock time and ~50% more tokens for ~100x lower cost and comparable accuracy. For batch/latency-tolerant workloads (research, evidence review, document analysis), this is a clear win.\nCore learnings The bottleneck was evidence consumption, not retrieval quality. Validating raw evidence before drafting caused small models to reject semantically obvious support. Draft-then-validate fixed it. MA helps small models more than large models. GPT-4o already does implicit self-reflection; the 7B model needs explicit decomposition. Deterministic grounding \u0026gt; LLM subjective grounding. Per-field verification eliminates validator judgment drift and improves convergence reliability. The validator prompt is the critical control point. The entire v2.1→v2.2 improvement came from changing what the validator sees; v2.2→v2.3 came from changing how it reports. Convergence is necessary but not sufficient. Grounding=100% means every claim is evidence-backed, but evidence quality depends on retrieval precision. Next experiments Multi-hop reasoning — questions requiring 2-3 evidence sources combined. Tests whether MA\u0026rsquo;s decomposition helps with reasoning, not just retrieval. Code reasoning \u0026amp; traceability — questions about code behavior requiring retrieval of source code, docs, and tests. Directly aligned with the thesis on evidence-based verification. Related research Published research broadly supports these findings. See the related research companion document for a detailed mapping of each finding to verified papers from arXiv, ACL, NeurIPS, ICML, and EMNLP, including:\nMixture-of-Agents (Wang et al., 2024) — open-source LLMs surpassing GPT-4o via layered aggregation Reflexion (Shinn et al., NeurIPS 2023) — verbal reinforcement learning validating draft-validate patterns SLMJury (Laddha et al., 2026) — shows debate degrades SLM accuracy, validating our skeptic backfire finding The Confident Liar (Keramati et al., ACL 2026) — explains role-asymmetric calibration in multi-agent debate Wu et al. (2024) — LLMs adopt incorrect retrieved content 60%+ of the time, explaining search-as-bottleneck 17 papers verified, covering support, contradiction, and nuance for each finding.\nData artifacts Aggregated results (v2.2 + v2.3) v2.2 full report v2.2 machine-readable summary v2.3 full report v2.3 machine-readable summary Freshness question pack Learnings and next steps Artifact directory README Dataset on GitHub Claim boundary These results document a controlled comparison on a fixed 20-question suite. The strongest supported claims are:\nOn freshness questions that defeat parametric knowledge, MA with proper grounding brings a 7B model within 0.2-0.4 points of the best baseline on a frontier model. Draft-then-validate orchestration is necessary — validating raw evidence before drafting causes small models to reject valid support. Deterministic grounding improves convergence reliability without hurting accuracy. MA helps small models more than large models on this task type. Claims do not generalize to all question types, all model sizes, real-time latency requirements, or domains outside evidence-grounded retrieval tasks. The 20-question sample supports directional claims but not statistical significance at conventional thresholds.\n","permalink":"https://00n.ai/research/multi-agent-framework-benchmark/","summary":"\u003cp\u003eThis benchmark compares four answer-generation architectures across two model backends on 20 freshness questions designed to defeat parametric knowledge.\u003c/p\u003e\n\u003ch2 id=\"approaches\"\u003eApproaches\u003c/h2\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eB0: Vanilla LLM\u003c/strong\u003e — one direct model answer with no tools or search.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eB1: LLM + Search\u003c/strong\u003e — query generation, web search/fetch, and a single evidence-grounded answer.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eB2: LLM + Search + Reflection\u003c/strong\u003e — search-based draft, self-critique, and revised answer.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMA: Multi-Agent\u003c/strong\u003e — Meta-Planner → Research → Synthesis → Validator → Skeptic → Final Answer with iterative convergence and deterministic grounding.\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"question-design\"\u003eQuestion design\u003c/h2\u003e\n\u003cp\u003eThe 20-question freshness pack uses facts published within ~24-48 hours of the experiment (June 22-23, 2026), sourced from NASA releases, arXiv postings, a Google Developers post, and a GitHub release feed. Questions are designed so models cannot answer from training data — B0 scores ~2-3 across both models, confirming the freshness property.\u003c/p\u003e","title":"Multi-Agent Framework Benchmark"},{"content":"This is the canonical repoloc-bench entry for the iTrust requirement-to-code traceability benchmark package.\nIt records the package and examples without copying the raw corpus into multiple places.\nSource Build brief: projects/state-estimation-traceability/research/dataset-benchmarking-buildout.md Package: projects/state-estimation-traceability/data/derived/paper1-benchmark/ GitHub repo: 00n-ai/repoloc-bench Reproduction bundle: benchmark_package_review_100 Snapshot commit: 64ecaab (Add benchmark reproduction bundle) What is included 34 requirements 4,907 methods 718 classes 307 gold traces masked gold-edge suites at 10/20/30% Claim boundary Primary branch is original requirement text. LLM-imputed requirements are auxiliary sensitivity data only. Gold links use traces with goldfinal == 'T'. This is a method-localization benchmark foundation, not yet a test-localization benchmark. Canonical package files manifest.json README.md README.json integrity.json EXAMPLES.md splits/ mask_suites/ systems/itrust/ Example outputs { \u0026#34;task_id\u0026#34;: \u0026#34;orig:itrust:R1\u0026#34;, \u0026#34;branch\u0026#34;: \u0026#34;original\u0026#34;, \u0026#34;system\u0026#34;: \u0026#34;itrust\u0026#34;, \u0026#34;requirement_id\u0026#34;: 1, \u0026#34;gold_method_ids\u0026#34;: [154, 2851, 2986, 2992, 3125, 3126, 4655], \u0026#34;positive_trace_count\u0026#34;: 7 } { \u0026#34;node_id\u0026#34;: \u0026#34;requirement:itrust:1\u0026#34;, \u0026#34;original_text\u0026#34;: \u0026#34;Add a New Patient ...\u0026#34;, \u0026#34;imputed_text\u0026#34;: \u0026#34;The system shall allow the addition of a new patient with basic information and generate a unique MID for them.\u0026#34;, \u0026#34;gold_method_count\u0026#34;: 7 } { \u0026#34;mask_percent\u0026#34;: 10, \u0026#34;gold_edges\u0026#34;: 307, \u0026#34;held_out_gold_edges\u0026#34;: 39, \u0026#34;observed_gold_edges\u0026#34;: 268 } Why this stays small The benchmark package itself remains the source of truth. This page is only the registry and summary layer, so we do not duplicate the underlying dataset here.\n","permalink":"https://00n.ai/research/itrust-traceability-benchmark/","summary":"\u003cp\u003eThis is the canonical repoloc-bench entry for the iTrust requirement-to-code traceability benchmark package.\u003c/p\u003e\n\u003cp\u003eIt records the package and examples without copying the raw corpus into multiple places.\u003c/p\u003e\n\u003ch2 id=\"source\"\u003eSource\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eBuild brief: \u003ccode\u003eprojects/state-estimation-traceability/research/dataset-benchmarking-buildout.md\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003ePackage: \u003ccode\u003eprojects/state-estimation-traceability/data/derived/paper1-benchmark/\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eGitHub repo: \u003ca href=\"https://github.com/00n-ai/repoloc-bench\"\u003e00n-ai/repoloc-bench\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eReproduction bundle: \u003ca href=\"https://github.com/00n-ai/repoloc-bench/tree/main/research/reproduction/benchmark_package_review_100\"\u003ebenchmark_package_review_100\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eSnapshot commit: \u003ccode\u003e64ecaab\u003c/code\u003e (\u003ccode\u003eAdd benchmark reproduction bundle\u003c/code\u003e)\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"what-is-included\"\u003eWhat is included\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e34 requirements\u003c/li\u003e\n\u003cli\u003e4,907 methods\u003c/li\u003e\n\u003cli\u003e718 classes\u003c/li\u003e\n\u003cli\u003e307 gold traces\u003c/li\u003e\n\u003cli\u003emasked gold-edge suites at 10/20/30%\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"claim-boundary\"\u003eClaim boundary\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003ePrimary branch is original requirement text.\u003c/li\u003e\n\u003cli\u003eLLM-imputed requirements are auxiliary sensitivity data only.\u003c/li\u003e\n\u003cli\u003eGold links use traces with \u003ccode\u003egoldfinal == 'T'\u003c/code\u003e.\u003c/li\u003e\n\u003cli\u003eThis is a method-localization benchmark foundation, not yet a test-localization benchmark.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"canonical-package-files\"\u003eCanonical package files\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003emanifest.json\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eREADME.md\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eREADME.json\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eintegrity.json\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eEXAMPLES.md\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003esplits/\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003emask_suites/\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003esystems/itrust/\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"example-outputs\"\u003eExample outputs\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-json\" data-lang=\"json\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e{\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;task_id\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;orig:itrust:R1\u0026#34;\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;branch\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;original\u0026#34;\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;system\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;itrust\u0026#34;\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;requirement_id\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;gold_method_ids\u0026#34;\u003c/span\u003e: [\u003cspan style=\"color:#ae81ff\"\u003e154\u003c/span\u003e, \u003cspan style=\"color:#ae81ff\"\u003e2851\u003c/span\u003e, \u003cspan style=\"color:#ae81ff\"\u003e2986\u003c/span\u003e, \u003cspan style=\"color:#ae81ff\"\u003e2992\u003c/span\u003e, \u003cspan style=\"color:#ae81ff\"\u003e3125\u003c/span\u003e, \u003cspan style=\"color:#ae81ff\"\u003e3126\u003c/span\u003e, \u003cspan style=\"color:#ae81ff\"\u003e4655\u003c/span\u003e],\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;positive_trace_count\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#ae81ff\"\u003e7\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-json\" data-lang=\"json\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e{\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;node_id\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;requirement:itrust:1\u0026#34;\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;original_text\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;Add a New Patient ...\u0026#34;\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;imputed_text\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;The system shall allow the addition of a new patient with basic information and generate a unique MID for them.\u0026#34;\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;gold_method_count\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#ae81ff\"\u003e7\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-json\" data-lang=\"json\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e{\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;mask_percent\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#ae81ff\"\u003e10\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;gold_edges\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#ae81ff\"\u003e307\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;held_out_gold_edges\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#ae81ff\"\u003e39\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003e\u0026#34;observed_gold_edges\u0026#34;\u003c/span\u003e: \u003cspan style=\"color:#ae81ff\"\u003e268\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"why-this-stays-small\"\u003eWhy this stays small\u003c/h2\u003e\n\u003cp\u003eThe benchmark package itself remains the source of truth. This page is only the registry and summary layer, so we do not duplicate the underlying dataset here.\u003c/p\u003e","title":"iTrust Requirement-to-Code Traceability Benchmark"},{"content":"Sheraz Mahmood Solutions Architect\nLakeway, Texas • sheraz@00n.ai • LinkedIn • GitHub\nSummary AI-focused architect and researcher with two decades of experience in custom software development and enterprise systems architecture. Specializing in the design and deployment of governable, measurable, explainable, and production-grade AI systems. Proven track record of translating ambiguous business requirements into executable technical strategies, robust architectural decisions, and applied AI solutions.\nProfessional Experience Accenture\nAssociate Director \u0026amp; Solutions Architect\nLead architecture and solutioning across complex, enterprise-scale technical environments, drawing on 20 years of foundational software engineering experience. Drive the delivery of practical, production-ready AI systems, bridging the gap between research-only prototypes and scalable organizational solutions. Architect AI governance frameworks, focusing on risk measurement, bidirectional traceability, and evidence-based routing for organizational decision support. Education \u0026amp; Academic Focus University of Colorado Boulder\nMaster of Science in Artificial Intelligence | Expected 2026\nCurrent Research Focus: State Estimation for Bidirectional Traceability in Generative Software Architecture. Research Interests Neuro-symbolic AI \u0026amp; Knowledge Graphs: Integrating symbolic reasoning with neural networks for explainable AI. Control Theory for LLMs: Applying deterministic control mechanisms to large language models. Generative Software Architecture: Repository-state modeling, semantic localization, and deterministic AI systems. Selected Technical Work \u0026amp; Projects Systems Research: Authored paper-ready research focused on repository-state trees and semantic localization, complete with reproducible artifacts. Evaluation \u0026amp; Calibration: Designed and executed Phase 2 error/support measurement and calibration experiments for applied AI systems. AI Tooling Augmentation: Evaluated Codex augmentation, frozen replay scenarios, and modern AI coding assistants to optimize software engineering workflows. Technical Skills \u0026amp; Environment Core Competencies: AI Architecture \u0026amp; Delivery, Systems Design, Applied Research, Technical Writing, AI Governance. Languages \u0026amp; Automation: Python, Java, Repository Automation, Evaluation \u0026amp; Calibration scripting. ","permalink":"https://00n.ai/cv/","summary":"\u003ch1 id=\"sheraz-mahmood\"\u003eSheraz Mahmood\u003c/h1\u003e\n\u003cp\u003e\u003cstrong\u003eSolutions Architect\u003c/strong\u003e\u003cbr\u003e\nLakeway, Texas • \u003ca href=\"mailto:sheraz@00n.ai\"\u003esheraz@00n.ai\u003c/a\u003e • \u003ca href=\"https://www.linkedin.com/in/sherazmahmood\"\u003eLinkedIn\u003c/a\u003e • \u003ca href=\"https://github.com/00n-ai\"\u003eGitHub\u003c/a\u003e\u003c/p\u003e\n\u003ch3 id=\"summary\"\u003eSummary\u003c/h3\u003e\n\u003cp\u003eAI-focused architect and researcher with two decades of experience in custom software development and enterprise systems architecture. Specializing in the design and deployment of governable, measurable, explainable, and production-grade AI systems. Proven track record of translating ambiguous business requirements into executable technical strategies, robust architectural decisions, and applied AI solutions.\u003c/p\u003e\n\u003ch3 id=\"professional-experience\"\u003eProfessional Experience\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eAccenture\u003c/strong\u003e\u003cbr\u003e\n\u003cem\u003eAssociate Director \u0026amp; Solutions Architect\u003c/em\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eLead architecture and solutioning across complex, enterprise-scale technical environments, drawing on 20 years of foundational software engineering experience.\u003c/li\u003e\n\u003cli\u003eDrive the delivery of practical, production-ready AI systems, bridging the gap between research-only prototypes and scalable organizational solutions.\u003c/li\u003e\n\u003cli\u003eArchitect AI governance frameworks, focusing on risk measurement, bidirectional traceability, and evidence-based routing for organizational decision support.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"education--academic-focus\"\u003eEducation \u0026amp; Academic Focus\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eUniversity of Colorado Boulder\u003c/strong\u003e\u003cbr\u003e\n\u003cem\u003eMaster of Science in Artificial Intelligence\u003c/em\u003e | Expected 2026\u003c/p\u003e","title":"CV"}]