Coding Knowledge Graph Agent Benchmark

Research Question Can a code knowledge graph (KG) with multi-agent navigation bring 7-8B parameter models closer to frontier-model performance on code generation, repair, and completion tasks? Sub-questions: Does KG navigation help small models write more correct code, or does it add noise they can’t filter? Does the multi-agent pipeline (draft → directional reasoning → test → refine) recover ground that single-agent KG loses? Which task types benefit most from graph evidence (generation vs. repair vs. completion)? Is there a model-capability floor below which KG navigation is ineffective — and does it differ from the legal KG floor? Can the pipeline enable small models to solve tasks that frontier B0 cannot? Experimental Setup Code Knowledge Graph A knowledge graph automatically extracted from a multi-module Python codebase (payments, user_service, notifications, email_service) using static analysis. ...

June 26, 2026 · 12 min · Sheraz Mahmood

Knowledge Graph Agent Benchmark

Research Question Can knowledge graph (KG) navigation bring a 7B parameter model to within 1 point of a frontier model on statutory reasoning tasks? Sub-questions: How much does KG navigation help small models vs. introduce noise? Which questions benefit most from graph evidence? What proportion of the gap is closeable via prompt engineering vs. graph quality vs. model capability? Is there a model-capability floor below which KG navigation is ineffective? Does natural language serialization of graph content help or hurt differently across model tiers? Experimental Setup Knowledge Graph A manually constructed knowledge graph covering six legal/technical domains: ...

June 25, 2026 · 18 min · Sheraz Mahmood