Coding Knowledge Graph Agent Benchmark

Research Question

Can a code knowledge graph (KG) with multi-agent navigation bring 7-8B parameter models closer to frontier-model performance on code generation, repair, and completion tasks?

Sub-questions:

1. Does KG navigation help small models write more correct code, or does it add noise they can’t filter?
2. Does the multi-agent pipeline (draft → directional reasoning → test → refine) recover ground that single-agent KG loses?
3. Which task types benefit most from graph evidence (generation vs. repair vs. completion)?
4. Is there a model-capability floor below which KG navigation is ineffective — and does it differ from the legal KG floor?
5. Can the pipeline enable small models to solve tasks that frontier B0 cannot?

Experimental Setup

Code Knowledge Graph

A knowledge graph automatically extracted from a multi-module Python codebase (payments, user_service, notifications, email_service) using static analysis.

Node types: module, class, function, method, test, variable, enum_value, exception, convention

Edge types: imports, calls, inherits, defines, tests, depends_on, raises, manages_state, has_enum_value, runtime_import, EXEMPLIFIES, FOLLOWS_CONVENTION

Convention nodes capture behavioral contracts (e.g., “functions that validate user existence raise ValueError when not found”) and cross-module access patterns (e.g., “payments accesses user_service via _get_user_service() lazy import”).

Tasks

9 tasks across 3 difficulty types:

Models

Conditions

B0 (Vanilla LLM): One direct model call with the task prompt. No tools, no graph. Temperature 0.3. The model sees only the task description.

KG (Graph Navigation): Four-step single-agent pipeline:

1. Seed — Deterministically retrieve all nodes from the target module (functions, classes, conventions)
2. Decompose — LLM breaks the task into 2-4 sub-questions (knowledge needs)
3. Navigate — For each sub-question, query the graph by keyword → retrieve nodes + 1-hop neighbors → LLM decides whether to explore further (8-step budget)
4. Reason + Synthesize — LLM traces a reasoning path through visited nodes/edges, then synthesizes code following the reasoning path

KG-NL (NL Serialization): Same as KG, but graph nodes are serialized as natural language descriptions (2-4 sentences per node) instead of teleographic JSON. Conventions include explicit validation vs. lookup pattern context.

MA+KG (Multi-Agent + Directional Reasoning): Full pipeline with 4 components:

1. KG-Agent (draft) — Same as KG above, produces draft code + visited evidence
2. Directional Reasoning — LLM examines visited evidence + conventions + task and generates explicit “when X, do Y” implementation directives. Converts facts (“process_payment raises ValueError”) into direction (“when implementing user validation in payments, raise ValueError with ’not found’ message”)
3. Test — Inject generated code into the module, run pytest, capture pass/fail
4. Refine — If tests fail, LLM gets test output + directional directives and generates revised code. Up to 3 refinement rounds. Anti-regression guard: if refined code passes fewer tests than the previous round, keep the previous code.

Directional Reasoning Fixes

Four issues were identified and fixed before the full experiment:

1. Pre-filter conventions by task keywords (High). Only include a convention if the task description contains keywords related to it. Prevents irrelevant directives (e.g., ValueError validation convention applied to a data aggregation task).

2. Validation vs. lookup pattern context (High). Convention nodes now distinguish validation functions (raise exceptions) from lookup functions (return None/False). Prevents the model from raising ValueError in data retrieval tasks where it should return None.

3. Post-filter hallucinated function directives (Medium). Each directive’s function calls are checked against the graph’s known function inventory. Directives referencing nonexistent functions are stripped.

4. Syntax-error refinement step (Medium). When a syntax error is detected, a focused LLM call attempts to fix it before running pytest. Prevents 0/0 scores from trivial syntax issues.

Evaluation

Pass@1: fraction of test cases passing after code injection. Each run injects the generated code into the original module, runs the task’s pytest suite, and counts passing tests. 5 runs per task/condition/model for variance estimation.

Results

Overall Pass@1 (avg across all tasks and runs)

MA+KG Lift over B0 (percentage points)

Gap to Frontier B0 Ceiling (83.9%)

Per-Task Detail by Model

qwen2.5:7b

mistral:7b

llama3.1:8b

GLM-5.2 (Frontier B0 Ceiling)

Analysis

Where MA+KG Helped

Mistral: +18.1pp overall — the pipeline nearly doubles B0 performance. Standout tasks:

• repair-duplicate-email: 60% → 100% (+40pp). The convention node for ValueError validation + the directional directive (“raise ValueError for duplicates”) gave mistral the exact pattern it needed.
• repair-revenue-refunded: 33% → 83% (+50pp). The graph’s Payment class nodes and refund logic edges provided the state machine knowledge mistral lacked.
• gen-notify-user-deleted: 0% → 33% (+33pp). The cross-module import convention (_get_user_service()) enabled mistral to correctly access user data from notifications.

Llama: +2.4pp overall, with dramatic single-task transformations:

• repair-notification-queue: 7% → 83% (+77pp). The clearest KG signature task — frontier B0 scores 0% on this task, but llama MA+KG scores 83%. The graph provides the cross-module send_email dependency that B0 can’t discover.
• comp-payment-summary: 16% → 68% (+52pp). The graph’s Payment class, payment_history function, and PaymentStatus enum nodes gave llama the full data model it needed.
• gen-notify-user-deleted: 40% → 60% (+20pp). The notification queue + email service cross-module edges enabled correct multi-module interaction.

Where MA+KG Hurt

Qwen: −4.3pp overall — the only model where B0 beats all KG conditions. The graph adds noise qwen can’t filter, and the refiner breaks working draft code. The anti-regression guard limits the damage but can’t fully prevent it.

Llama on tasks where B0 was already strong:

• gen-cancel-subscription: 90% → 70% (−20pp). B0 was already excellent; the pipeline introduced irrelevant conventions and the refiner degraded the draft.
• repair-duplicate-email: 100% → 60% (−40pp). Same pattern — B0 nailed it, the pipeline overcomplicated.
• comp-notification-stats: 60% → 15% (−45pp). The KG provided irrelevant payment/user_service nodes that distracted from the simple notification iteration.

The KG Signature Task: repair-notification-queue

This task requires knowledge that the frontier model lacks — frontier B0 scores 0%. The graph provides the cross-module send_email dependency and the notification queue state machine that neither parametric knowledge nor task description reveals.

Llama MA+KG (83%) vastly exceeds frontier B0 (0%) on this task. This is the clearest evidence that the code KG pipeline adds capabilities that even frontier parametric knowledge cannot provide — the graph encodes project-specific architectural knowledge (import conventions, cross-module dependencies) that no model knows without retrieval.

KG Alone Hurts All Three Models

Structured graph navigation (KG condition) hurts all three small models compared to B0:

The graph adds context the models can’t effectively use without the multi-agent guardrails (draft → directional reasoning → test → refine). MA+KG recovers and exceeds B0 for mistral (+18.1pp) and llama (+2.4pp) because the pipeline provides:

1. Test feedback — the refiner knows which tests failed, not just which facts exist
2. Directional reasoning — conventions are converted into actionable directives, not just presented as facts
3. Anti-regression guard — prevents the refiner from breaking working draft code

Model-Dependent Pattern

Mistral is the ideal MA+KG candidate: low B0 (19.8%) means it lacks parametric knowledge, but it has enough synthesis capability to use graph evidence. The pipeline nearly doubles its performance.

Llama is a strong base (B0=45.9%) that gets targeted help on hard tasks (repair-notification-queue +77pp, comp-payment-summary +52pp) but is hurt on easy tasks where B0 was already strong.

Qwen is below the capability floor for this task type. The graph adds noise it can’t filter, and the refiner degrades working code. This matches the legal KG finding where qwen had a +1.5 lift on legal reasoning (simpler synthesis) but the code domain is harder (multi-file, cross-module, state machines).

Comparison with Legal KG Benchmark

This experiment replicates the legal KG benchmark pattern in the code domain:

The key new finding from the coding benchmark: the code KG enables small models to solve tasks that frontier B0 cannot solve (repair-notification-queue: 0% for frontier, 83% for llama MA+KG). This was not demonstrated in the legal domain because legal questions had no equivalent of project-specific architectural knowledge.

Limitations

1. Single codebase. The KG is extracted from one 4-module Python project. Generalization to larger codebases, other languages, or different architectural patterns is not demonstrated.
2. 9 tasks only. The task set covers generation, repair, and completion, but is small. Statistical significance is not claimed.
3. Automatic graph extraction. The KG is built by static analysis (AST walking). Dynamic behaviors, runtime configurations, and non-code conventions are not captured.
4. Non-deterministic generation. Ollama models at temperature 0.3 produce variance across runs. 5 runs per condition provide variance estimation but not confidence intervals.
5. Frontier B0 only. The frontier model was not run with MA+KG (no DeepInfra-compatible pipeline for the code KG). The ceiling comparison is B0-only.
6. Convention extraction is heuristic. The 4 convention nodes are derived from AST patterns (raise statements, import patterns). Real-world conventions (naming, error messages, API contracts) may require manual annotation.
7. Refinement is limited to 3 rounds. More rounds might help for complex tasks, but also increase the chance of the refiner breaking working code.
8. No grader. Pass/fail is determined by pytest, not LLM grading. This is more objective than the legal benchmark but less nuanced (partial credit only from test counts).

Reproduction

Prerequisites

# Software
node v20+ (tested with v26.0.0)
python3 with ast module
pytest

# Ollama (small models)
ollama pull qwen2.5:7b
ollama pull mistral:7b
ollama pull llama3.1:8b

# DeepInfra (frontier model)
export DEEPINFRA_API_KEY=your_key

Running the Experiment

cd projects/coding-kg-experiment

# Build the knowledge graph
python3 src/extract_code_kg.py example-repo/ code-graph.json

# Run full experiment (3 models × 4 conditions × 9 tasks × 5 runs)
LLM_PROVIDER=ollama LLM_MODEL=qwen2.5:7b node src/run_full_experiment.mjs --runs 5
LLM_PROVIDER=ollama LLM_MODEL=mistral:7b node src/run_full_experiment.mjs --runs 5
LLM_PROVIDER=ollama LLM_MODEL=llama3.1:8b node src/run_full_experiment.mjs --runs 5

# Frontier B0 ceiling
LLM_PROVIDER=deepinfra LLM_MODEL=deepinfra/zai-org/GLM-5.2 \
  node src/run_full_experiment.mjs --frontier --runs 5

Data Artifacts

• qwen2.5:7b results
• mistral:7b results
• llama3.1:8b results
• GLM-5.2 frontier B0 results

Repository Structure

projects/coding-kg-experiment/
├── code-graph.json                        # KG: 178 nodes, 348 edges
├── example-repo/                          # 4-module Python codebase
│   ├── src/
│   │   ├── payments.py
│   │   ├── user_service.py
│   │   ├── notifications.py
│   │   └── email_service.py
│   └── tests/
├── tasks/
│   ├── tasks.json                         # 9 task definitions
│   └── tests/                             # pytest suites per task
├── src/
│   ├── extract_code_kg.py                 # Graph extractor (AST-based)
│   ├── code_kg_agent.js                   # Single-agent KG pipeline
│   ├── ma_kg_agent.js                     # MA+KG pipeline with directional reasoning
│   └── run_full_experiment.mjs            # Experiment runner
├── results/                               # JSON results per model
└── docs/
    ├── experiment-design.md               # Full design document
    └── conventions-and-directional-reasoning.md  # Convention extraction design

Key Takeaways

1. MA+KG beats B0 for 2 of 3 small models. Mistral +18.1pp, llama +2.4pp. The pipeline (draft → directional reasoning → test → refine) provides guardrails that single-agent KG navigation cannot.
2. The code KG enables small models to solve tasks frontier B0 cannot. repair-notification-queue: 0% for frontier, 83% for llama MA+KG. The graph encodes project-specific architectural knowledge (import conventions, cross-module dependencies) that no model has parametrically.
3. KG alone hurts all three small models. Structured graph navigation adds noise without the multi-agent guardrails. The value is in the pipeline, not the graph alone.
4. Mistral is the ideal MA+KG candidate. Low B0 (19.8%) + sufficient synthesis capability = +18.1pp lift. The pipeline nearly doubles its performance.
5. Qwen is below the capability floor for code synthesis. B0 beats all KG conditions. The graph adds noise qwen can’t filter, and the refiner degrades working code.
6. Directional reasoning fixes worked. Convention pre-filtering prevented irrelevant directives, post-filtering caught hallucinated function names, and the anti-regression guard preserved draft wins.
7. The model-capability floor differs by domain. qwen had +1.5 lift on legal KG (simpler synthesis) but −4.3pp on coding KG (harder: multi-file, cross-module, state machines). The floor is task-complexity-dependent, not just model-dependent.
8. MA+KG narrows the gap to frontier. Llama MA+KG (48.3%) gets within 35.6pp of frontier B0 (83.9%) — the best of any small-model condition. Mistral MA+KG (38.0%) closes 18pp of gap that B0 alone couldn’t.
9. Task-type dependency: MA+KG shines on hard tasks where B0 is weak. The biggest wins are on tasks where the model lacks parametric knowledge (repair-notification-queue, comp-payment-summary). On tasks where B0 is already strong, MA+KG can overcomplicate and hurt.

Related

• Knowledge Graph Agent Benchmark (Legal) — Same experimental design on statutory reasoning tasks
• Multi-Agent Framework Benchmark — MA orchestration compresses the capability gap on evidence-grounded QA