Knowledge Graph Agent Benchmark

Research Question

Can knowledge graph (KG) navigation bring a 7B parameter model to within 1 point of a frontier model on statutory reasoning tasks?

Sub-questions:

1. How much does KG navigation help small models vs. introduce noise?
2. Which questions benefit most from graph evidence?
3. What proportion of the gap is closeable via prompt engineering vs. graph quality vs. model capability?
4. Is there a model-capability floor below which KG navigation is ineffective?
5. Does natural language serialization of graph content help or hurt differently across model tiers?

Experimental Setup

Knowledge Graph

A manually constructed knowledge graph covering six legal/technical domains:

Graph stats: 91 nodes, 138 edges, 20 cross-domain bridge edges

Node schema:

{
  "id": "art-gdpr-6",
  "label": "GDPR Article 6 – Lawfulness of Processing",
  "type": "Article",
  "domain": "gdpr",
  "summary": "Processing of personal data is lawful only if at least one of six legal bases applies.",
  "natural_language": "Article 6 defines the six legal bases for lawful processing: consent, contract, legal obligation, vital interests, public task, and legitimate interests. At least one basis must apply before processing personal data. Without a valid legal basis, processing is unlawful by default.",
  "key_points": ["Consent (Art 6(1)(a))", "Contract necessity", "Legal obligation", "Vital interests", "Public task", "Legitimate interests"],
  "evidence": {
    "source": "GDPR Regulation (EU) 2016/679",
    "url": "https://gdpr-info.eu/art-6-gdpr/",
    "quote": "Processing shall be lawful only if at least one of the following applies..."
  }
}

Edge schema:

{
  "source": "art-230-c1",
  "target": "art-512c",
  "relation": "DIFFERS_FROM",
  "assertion": "230(c)(1): blanket publisher immunity. 512(c): immunity requires active compliance (agent, notice-and-takedown).",
  "bridge": "Section 230(c)(1) gives platforms blanket immunity for user content without any compliance steps, while DMCA 512(c) requires platforms to actively maintain a designated agent and respond to takedown notices. This is the key contrast: passive immunity vs. conditional safe harbor."
}

Questions

Five questions spanning all six domains, each requiring specific statutory knowledge:

Models

Note: The grader is the same model as the frontier baseline. The grader does not see which model produced which answer — answers are labeled B0/KG, not by model name.

Conditions

B0 (Vanilla LLM): One direct model answer, no tools or retrieval. Temperature 0.3.

KG-Agent (Graph Navigation): Four-step pipeline:

1. Decompose — LLM breaks the question into 2–4 sub-questions (knowledge needs)
2. Navigate — For each sub-question, query the graph by keyword → retrieve nodes + 1-hop neighbors → LLM decides whether to explore further
3. Reason — LLM traces a reasoning path through visited nodes/edges, citing each fact and edge
4. Synthesize — LLM writes final answer following the reasoning path

Tier 1 Prompt Fixes (between initial and post-Tier 1 runs):

• Synthesis: “Confidence first” format, real legal citations (not node IDs), 300–500 word target, no verbatim repetition
• Reasoning: anti-repetition rule, keep path linear
• Navigation budget: 6 → 8 steps

Tier 2 NL Serialization (between Tier 1 and Tier 2 runs):

• Domain-grouped serialization — nodes grouped by legal domain, noise deprioritized
• Natural language node descriptions — 2–4 sentence rich explanations replace telegraphic summaries
• Edge bridge sentences — cross-domain edges foregrounded with explanatory reasoning bridges
• LLM relevance filter — GLM-5.2 scores nodes 0–5 before presentation (threshold: keep 2+)
• Descriptive reference labels — navigation prompts use “Section 230(c)(1) — Publisher Immunity” instead of raw node ID “art-230-c1”

Grading

Unified GLM-5.2 grader scores each answer on 5 dimensions (0–10 scale): Accuracy, Citation Quality, Coverage, Calibration, Cognitive Load, and Overall. The grader sees the authoritative answer key and source. Answers are labeled B0/KG, not by model.

Results

Post-Tier 1 (prompt fixes: citation format, conciseness, 8-step budget)

Tier 2 (NL serialization + relevance filter + edge bridges)

†Q4 grading failed; average over 4 questions. “—” = grading failure.

Frontier Baseline

Cross-Tier Comparison

Analysis

Where KG Helped

Q3 (§230 vs DMCA): Post-Tier 1 qwen scored 7.8 (frontier scored 4.0). The graph’s DIFFERS_FROM and EXCEPTS edges provided the 230(e)(2) IP preservation point and real statutory citations that the vanilla model couldn’t recall. Tier 2 llama scored 8.0 on the same question — the NL bridge sentences gave the weakest model the reasoning scaffolding it needed.

Q2 (CCPA deletion): Post-Tier 1 qwen scored 6.0 (B0=4.0, +2.0 lift). The graph’s deletion-notice, procedure, verification, and exception nodes gave the small model the full statutory flow.

Q5 (RAG vs FT): Tier 2 qwen scored 8.0 (B0=4.0, +4.0 lift). The RAG/FT literature nodes provided the comparative findings the model couldn’t recall.

Where KG Hurt

llama3.1:8b Tier 1: −2.7 lift. The model is too weak to synthesize multi-source graph evidence. The 8-step budget exposed it to more noise without improving reasoning. This defines a model-capability floor.

Tier 2 mid-tier regression: qwen went from +1.5 to −0.3, mistral from +1.0 to −1.5. The NL serialization added token load and reasoning scaffolding that mid-tier models didn’t need — they could already build bridge sentences implicitly from the telegraphic format. The richer context introduced noise for models that didn’t need the help.

Capability-Dependent Intervention

The most striking finding: Tier 2 NL serialization helped the weakest model but hurt mid-tier models.

This suggests NL serialization is a capability-dependent intervention. It helps models below a capability floor by providing reasoning bridges they can’t construct themselves. It hurts models above the floor by adding token load and redundant context they could already generate internally.

Gap to Frontier

KG navigation narrows the gap by 30–40% but does not close it. On individual knowledge-intensive questions (Q3), KG-augmented small models can exceed frontier performance — but this is not consistent across all questions.

Per-Question Crossover

Q3 is the one question where KG-augmented small models beat the frontier model:

The frontier model’s parametric knowledge of §230/DMCA interaction was weak (it scored 4.0), while the graph provided exactly the structured knowledge the small models needed. On Q1 (GDPR), where the frontier model’s parametric knowledge is strong (scored 10), no small model configuration closed the gap.

Saturation Experiment: Separating Graph Completeness from Model Capability

The experiments above confound two factors: graph completeness (does the KG contain all needed knowledge?) and model capability (can the model synthesize the provided evidence into a high-quality answer?). To separate them, we constructed a saturating graph for Q1 — one where every fact, edge, and reasoning bridge needed to answer the question is explicitly present, with natural language bridge sentences connecting the full chain:

Art 5 (lawfulness) → Art 6 (consent as legal basis) → Art 7 (consent conditions) → ePrivacy Art 5(3) (non-essential cookies require opt-in) → Art 83 Tier 2 (€20M or 4% turnover)

Graph enrichment: Added 4 nodes (ePrivacy Directive, Art 5(3), non-essential cookies, essential cookies), 7 edges, and 15 bridge sentences. Graph grew from 87 to 91 nodes and 131 to 138 edges (20 with bridges).

Saturation Results

Synthesis prompt fix: The initial saturating run produced step-by-step lab report format (“Step 1: FACT…”). Fixed the synthesis prompt to require flowing legal analysis (“Write 2-4 flowing paragraphs. Do NOT use Step 1: FACT labels.”). This raised qwen from 8.0 to 9.0 — 1 point from frontier.

Q1 Across All Runs

Interpretation

Graph completeness matters. qwen improved from 6.0 (Tier 2, incomplete graph) to 8.0 (saturating graph) — the ePrivacy Directive node and bridge sentences provided the missing knowledge. The gap to frontier narrowed from 4 points to 2 points.

Model capability is still the ceiling. Even with a saturating graph (0 reasoning gaps for qwen), the score was 8.0, not 10.0. The remaining 2-point gap is synthesis quality: the answer was correct and well-cited but verbose and step-by-step rather than a polished legal analysis. The frontier model produced a concise, authoritative answer with correct citation hierarchy — the 7B model reproduced the facts but in a procedural format.

Capability floor confirmed. llama3.1:8b scored 3.0 despite having the same saturating graph as qwen. Its answer focused on fine structure and missed the cookie consent analysis entirely — a synthesis failure, not a knowledge gap. Below B0 ≈ 5.0, graph completeness does not help because the model cannot synthesize multi-source evidence regardless of how complete it is.

Answer: Both are independent bottlenecks. A complete graph narrows the gap by 60–80% for capable models (qwen: 5 → 8, closing 3 of 5 points). The remaining 20–40% is synthesis capability that requires a larger model. Graph completeness and model capability are additive but independent — fixing one without the other yields suboptimal results.

Multi-Agent KG: Draft-Validate-Refine on the Reasoning Graph

The KG-agent is a single-agent pipeline — one model does everything. The MA+KG pipeline separates the workflow into 4 specialized roles:

1. Retriever — Navigates the reasoning graph (decompose → keyword query → accumulate evidence)
2. Drafter — Writes a draft answer using only graph evidence, with flowing legal analysis format
3. Validator — Audits the draft against the graph as ground truth. For each claim: traces to a graph node? Node supports the claim? Edge supports the relationship? Flags unsupported claims, checks citations, identifies gaps
4. Refiner — Revises the draft based on validation findings

Key innovation: The validator uses the GRAPH as ground truth, not its own parametric knowledge. Every claim must trace to a node; every reasoning step must follow an edge.

MA+KG Results

Gap to Frontier (7.9)

MA+KG Findings

MA+KG improves consistency over single-agent KG (qwen). Single-agent KG was volatile: +2 on Q3, +2 on Q5, but −4 on Q1 — net lift +0.2. MA+KG was consistent: +1 on 4/5 questions — net lift +0.6. The validator-refiner cycle flattened the variance.

MA+KG narrows the gap to frontier for qwen. Gap: B0=2.3 → KG=2.1 → MA+KG=1.7 — a 26% reduction from B0. This is the best non-saturating result measured.

MA+KG does NOT help mistral or llama on average. The validator-refiner overhead hurts models that are either very capable (mistral) or very weak (llama).

Biggest single-question lift in the entire series: llama Q4 B0=3 → MA+KG=7 (+4). The validator caught missing HIPAA 500+ thresholds and the refiner added them.

MA+KG recovers from KG catastrophic failures. mistral Q4: KG=2 → MA+KG=5. llama Q3: KG=2 → MA+KG=4.

MA+KG can also collapse. llama Q5: KG=8 → MA+KG=3 (−5). The refiner destroyed a strong draft — same small model that produced the draft can’t reliably improve it.

MA+KG is a consistency improver, not a ceiling raiser. It trades peak performance for reduced variance.

Synthesis Scaffolding: Can We Give Small Models Frontier Synthesis?

The diagnosis from all previous experiments: the remaining gap is synthesis capability, not knowledge. Frontier models weave facts into flowing arguments, draw implications not in the graph, prioritize strongest points first, and use legal register.

Can we scaffold this capability into small models through a multi-step pipeline?

Design

Key innovation in Synth-Coach: The frontier model sees the graph evidence and generates an argument OUTLINE (key points, logical flow, implication hints, register guidance) — NOT the answer. The small model uses the outline to construct the answer itself.

Synthesis Scaffolding Results (qwen2.5:7b)

Gap to Frontier B0 (7.9)

Synthesis Findings

Synthesis scaffolding HURTS small models on average. Synth-Self: −1.8 lift. Synth-Coach: −0.8 lift. Both are worse than doing nothing.

The multi-step pipeline is the problem, not the coaching. The outline adds 700–900 tokens of structure. The small model then has to integrate outline + implications + evidence + question into a single draft — that’s MORE synthesis work, not less. The intervention that was supposed to reduce synthesis difficulty actually increased it.

Synth-Coach works when the model lacks knowledge (Q2 CCPA: 4→7) but hurts when it already knows the domain (Q3 §230: 7.4→4.2). The coaching pipeline overrides good parametric knowledge with worse graph-grounded synthesis.

Frontier-Synth beats frontier B0 (9.6 vs 7.9). The graph HELPS the frontier model on precise citation questions — confirming that the graph is a knowledge asset, not a synthesis asset.

The uncomfortable truth: synthesis capability cannot be scaffolded through prompt engineering. The remaining ~1.7-point gap is model capability. You cannot scaffold your way to synthesis — the small model needs to actually do the synthesis, and it can’t.

Full Ranking of All Interventions (qwen2.5:7b)

The Three-Layer Gap Model

The gap between small and frontier models splits into three parts:

1. Knowledge gap → solvable (graph retrieval, saturation closes 60–90%)
2. Error correction gap → solvable (validator-refiner adds +0.6 consistency)
3. Synthesis capability gap → NOT solvable through prompt engineering (multi-step pipelines hurt small models more than they help)

Interventions on the retrievable part work. Interventions on the synthesis part do not — at least not through prompt engineering alone.

Limitations

1. Grader = frontier model. The same GLM-5.2 serves as both frontier baseline and grader. While blinded to model identity, self-preference bias is possible.
2. 5 questions only. The question set is small and domain-specific. Generalization is not demonstrated.
3. Keyword-based retrieval (Tier 1). The queryGraph() function uses stopword-filtered keyword overlap, not semantic similarity. Returns noise nodes that can derail the agent. Tier 2’s LLM relevance filter partially addresses this.
4. Non-deterministic generation. Ollama models are non-deterministic at temperature 0.3. B0 scores vary ±1.0 across runs. KG scores vary ±0.5.
5. GLM-5.2 reasoning mode. The model defaults to thinking/reasoning that can consume all max_tokens and produce empty completions. Workaround: chat_template_kwargs: {enable_thinking: false}.
6. Grading failures. Two grading failures in Tier 2 (Q2 qwen, Q4 llama) reduce the effective comparison set.
7. Single grader. Inter-grader agreement was not measured.

Reproduction

Prerequisites

# Software
node v20+ (tested with v26.0.0)
python3 (for graph analysis)

# Ollama (small models)
ollama pull qwen2.5:7b
ollama pull mistral:7b
ollama pull llama3.1:8b

# DeepInfra (frontier model + grader)
export DEEPINFRA_API_KEY=your_key_here

Running the Experiment

cd projects/multi-agent-framework

# Tier 1: B0 vs KG-agent (all 3 models, 5 questions)
DEEPINFRA_API_KEY=$KEY node src/kg-experiment-multi.js \
  --models qwen2.5:7b,mistral:7b,llama3.1:8b --questions 5

# Tier 2: B0 vs KG-agent with NL serialization
DEEPINFRA_API_KEY=$KEY node src/kg-experiment-tier2.js \
  --models qwen2.5:7b,mistral:7b,llama3.1:8b

# Frontier B0 baseline
DEEPINFRA_API_KEY=$KEY node src/kg-experiment-frontier-b0.js

# MA+KG: B0 vs KG vs MA+KG (draft-validate-refine)
DEEPINFRA_API_KEY=$KEY node src/kg-experiment-ma.js \
  --models qwen2.5:7b,mistral:7b,llama3.1:8b --questions 5

# Synthesis scaffolding: B0 vs Synth-Self vs Synth-Coach vs Frontier-Synth
DEEPINFRA_API_KEY=$KEY node scripts/run-synth-experiment.mjs \
  --models qwen2.5:7b --questions 5

# Capability experiments: frontier control, adversarial, abstention
DEEPINFRA_API_KEY=$KEY node scripts/run-control-and-adversarial.mjs

Repository Structure

projects/multi-agent-framework/
├── knowledge-graph/graph.json           # KG: 91 nodes, 138 edges, 20 bridges
├── src/
│   ├── kg-agent.js                      # Tier 1 KG-agent
│   ├── kg-agent-tier2.js                # Tier 2 KG-agent (NL serialization)
│   ├── kg-ma-agent.js                   # MA+KG agent (draft-validate-refine)
│   ├── kg-synth-agent.js                # Synthesis scaffolding agent
│   ├── kg-experiment-multi.js            # Tier 1 experiment runner
│   ├── kg-experiment-tier2.js            # Tier 2 experiment runner
│   ├── kg-experiment-frontier-b0.js      # Frontier baseline runner
│   ├── kg-experiment-ma.js               # MA+KG experiment runner
│   ├── baselines.js                      # B0 + unified grader
│   ├── agents.js                         # LLM call helper
│   └── logger.js                         # Trace logging
├── logs/
│   ├── kg-multi-8408485f/               # Tier 1 qwen post-Tier 1
│   ├── kg-multi-70d770dc/               # Tier 1 mistral post-Tier 1
│   ├── kg-multi-cb7d5f56/               # Tier 1 llama post-Tier 1
│   ├── kg-tier2-f390efba/               # Tier 2 all models
│   ├── kg-saturation-78fe42e1/          # Saturation experiment (Q1 only)
│   ├── kg-capability-268cbc45/          # Capability experiments
│   ├── kg-ma-7d3f0b30/                  # MA+KG experiment
│   ├── kg-synth-4321e4da/               # Synthesis scaffolding experiment
│   └── kg-frontier-b0-f992ea91/         # Frontier B0 baseline
├── docs/
│   ├── kg-experiment-writeup.md          # Full internal writeup
│   ├── kg-agent-worked-example.md        # End-to-end trace on Q3
│   ├── kg-gap-analysis.md               # Gap analysis
│   └── kg-learnings.md                   # Consolidated learnings (all experiments)
└── scripts/
    ├── enrich-graph-tier2.py             # Graph enrichment (domain, NL, bridges)
    ├── saturate-gdpr-q1.py               # Q1 saturating graph enrichment
    ├── run-saturation-experiment.mjs     # Saturation experiment runner
    ├── run-control-and-adversarial.mjs   # Capability experiments
    ├── run-synth-experiment.mjs          # Synthesis scaffolding runner
    └── run-codex-design-first-replay.py  # Codex replay harness

Key Takeaways

1. KG navigation provides consistent positive lift for mid-tier small models (+1.0 to +1.5 points with Tier 1 prompts), narrowing the gap to frontier by 30–40%.
2. MA+KG (draft-validate-refine) is the best small-model intervention. MA+KG narrows the gap to frontier from 2.3 to 1.7 for qwen — a 26% reduction from B0 — without graph saturation. It improves consistency (no catastrophic collapses) but doesn’t raise the ceiling.
3. Synthesis scaffolding does NOT work. Multi-step pipelines (outline → implications → draft) hurt small models on average (−0.8 to −1.8 lift). The pipeline adds cognitive load, not structure. Synthesis capability cannot be scaffolded through prompt engineering.
4. The gap splits into three layers: knowledge (solvable via graph retrieval), error correction (solvable via validator-refiner), and synthesis capability (NOT solvable through prompt engineering). The remaining ~1.7-point gap is model capability.
5. On knowledge-intensive questions where frontier parametric knowledge is weak, KG can exceed frontier performance (Q3: small models scored 7.8–8.0 vs frontier 4.0).
6. NL serialization is capability-dependent. It helps the weakest model (llama: −2.7 → +1.2) but hurts mid-tier models (qwen, mistral) by adding redundant scaffolding.
7. There is a model-capability floor. Below B0 ≈ 5.0, KG navigation can hurt. Above it, KG reliably helps.
8. Graph completeness and model capability are independent bottlenecks. A saturating graph narrowed qwen’s gap from 5 to 1 point (80% closure). The remaining 1 point is synthesis quality.
9. KG is a small-model intervention, not a universal enhancer. Frontier model + KG = 10.0 = frontier B0. The graph adds nothing when parametric knowledge is sufficient.
10. KG improves false-premise rejection quality, not rejection rate. Rejections become better cited and more authoritative (qwen AdvQ3: 3.0 → 8.0) butnot more frequent.
11. KG does not prevent hallucination. No model abstained when evidence was removed. But graph context improved parametric recall quality (qwen: 2.0 → 9.0 on abstention test via correct $750 recall triggered by surrounding nodes).
12. Frontier-Synth (frontier model synthesizing from graph) beats frontier B0 (9.6 vs 7.9). The graph is a knowledge asset for frontier models on citation-heavy questions, even though it’s redundant for their parametric knowledge.

Related

• Coding Knowledge Graph Agent Benchmark — Same experimental design applied to code generation, repair, and completion tasks
• Multi-Agent Framework Benchmark — Orchestration compresses the capability gap between small and frontier models
• Consolidated Learnings — Full learnings document with all experiments, open questions, and artifact index
• Worked Example: KG-Agent on Q3 — Full end-to-end trace