Structured Context Experiments: Graphs, Notes, and Scaffolding for Small Models

Overview

Six experiments, eleven measured conditions, one conclusion: structured context is the bottleneck — not raw model capability. When a small model gets the right structure at the right time, it approaches frontier performance. When the structure is irrelevant or conflicts with existing knowledge, it becomes noise or causes regression.

This page documents four experiments from the Boole Agent research line — one-shot structured note-taking with tool-based retrieval, synthetic novel-knowledge testing, and scaffolded step-by-step reasoning. The earlier multi-agent benchmark and code generation experiments are documented on their own pages.

Experiment 4: Boole Agent — One-Shot Structured Notes + Tool Retrieval

Research Question

Can a local agent that uses a Socratic reasoning graph as structured external memory — with 19 tools, ReAct agent loop, response validation, and automatic knowledge extraction — provide measurable lift over vanilla LLM? And does lift differ by model capability?

Mechanism

The Socratic reasoning graph is not a learning system. It is a persistent externalization of what the model understood during a teaching pass — structured notes the model generates for itself, then retrieves via tool calls. This is one-shot learning in the classic sense: a single exposure produces a reusable reference artifact. The graph is a retrieval interface to the model’s extracted understanding, not a knowledge acquisition mechanism.

Setup

• Tasks: 12 tasks across 4 categories (retrieval-grounded QA, tool-chain, reasoning+evidence, abstention-calibrated)
• Grader: GLM-5.2 (unified, same rubric as MA benchmark and code gen eval)
• Runs: 72 total (6 conditions × 12 tasks × 2 models)

Results

Key Findings

1. Agent lift is real for frontier models only. GLM-5.2 A1 (7.7) vs B0 (3.8) → +3.9 lift. Small model A1 (2.4) vs B0 (3.5) → -1.1 regression.
2. Graph adds marginal value beyond tools for frontier: A1 (7.7) vs A1ng (6.7) → +1.0 graph lift. For small: A1 (2.4) vs A1ng (2.4) → 0.0 graph lift.
3. Pattern is INVERTED from prior experiments. In the MA benchmark and Socratic code gen, structure helped small models MORE. Here, active tool orchestration requires base capability — passive briefing helps small models, but active ReAct loops hurt them.
4. Evidence quality is the strongest differentiator: frontier A1 = 8.1 (cites graph nodes, tool results), small A1 = 1.8 (doesn’t cite evidence even with tools).

Experiment 5: Synthetic RFG — Novel Knowledge Test

Research Question

Does one-shot structured note-taking with tool-based retrieval improve multi-step reasoning on completely novel (fictional) knowledge, across varying reasoning chain lengths? This eliminates the training-data recall confound.

Domain

Resonance Field Governance (RFG) — a 100% invented framework with 7 axioms across 8 concepts. No model has seen this in training.

Design

• Teaching phase: Model receives 4 lessons on RFG axioms → Socratic extractor decomposes understanding into graph nodes (one-shot note-taking)
• Transfer phase: 8 questions at 2/4/6/8-step chain lengths. Model retrieves relevant nodes via tool calls.

Frontier Results (GLM-5.2)

Chain Length Degradation (Overall):

Key Findings

1. One-shot structured notes improve reasoning on novel knowledge: +1.75 overall (seeded vs cold).
2. Biggest win is anti-hallucination: +3.24 (seeded 6.12 vs cold 2.88). Notes keep the model grounded — not smarter, more disciplined.
3. Structured notes extend effective chain length from ~2 steps to ~6 steps — 3× improvement in reasoning depth on novel knowledge. The seeded model holds perfect 8.0 quality through 6 steps; cold model peaks at 7.5 but hallucinates throughout.
4. Correctness is close (7.62 vs 7.25) — GLM-5.2 can guess many RFG values by analogy. The notes help with axiom identification and hallucination avoidance, not computation.
5. cold_ng is worst (3.75) — without graph or tools, the agent hallucinates or refuses.

Small Model Results (qwen2.5:7b)

The 7B model fails across all conditions (0.6–1.1 overall). Structured notes don’t help because the model can’t drive the tool retrieval loop. Consistent with the main Boole Agent eval: active tool orchestration requires base capability.

Experiment 6: Scaffolded Step-by-Step Chain Reasoning

Research Question

Does forcing one-step-at-a-time reasoning (external control structure) help small models that fail in freeform ReAct mode? Also tests a unified context_lookup tool that merges search + graph traversal into one call.

Design

• Domain: Same RFG fictional framework. Same 8 transfer questions at 2/4/6/8-step chain lengths.
• Stepwise loop: External control structure sends a targeted message per step: “Search your context graph for [concept]. Apply it and report only step N.” Each step’s output feeds into the next step’s prompt. Final consolidation call at end.

Models

Results — qwen2.5:7b

Stepwise hurts qwen2.5:7b. Overall: freeform 0.38 vs stepwise 0.12 (−0.26). Correctness is 0.0 across all conditions. The model hallucinates wrong formulas at each step regardless of scaffolding.

Results — gemma-3-4b-it

Stepwise helps gemma-3-4b. Overall: stepwise 1.12 vs freeform 0.25 (+0.87, 4.5× improvement). First small model where stepwise scaffolding shows a clear positive effect.

Grader: GLM-5.2 (unified rubric, same 5-dimension as synthetic RFG)

Cross-Model Comparison

*GLM-5.2 seeded from synthetic RFG (no stepwise condition run).

Key Findings

1. Stepwise scaffolding is model-dependent — not size-dependent. Gemma-3-4B (4B params) benefits; qwen2.5:7b (7B params) gets worse. The determining factor is whether the model can follow step instructions and apply retrieved axioms — not parameter count.

2. First non-zero correctness from a small model on RFG. Stepwise gemma-3-4b achieved correctness=2.5 at 2-step and 4-step chains. Freeform correctness is 0.0–0.5 across all small models. No prior intervention produced non-zero correctness from a small model on this domain.

3. Stepwise extends effective chain length for gemma-3-4b. Freeform collapses after 2 steps (1.0 → 0.0 → 0.0 → 0.0). Stepwise decays gracefully (2.0 → 1.5 → 0.5 → 0.5), still above zero at 8 steps. This parallels how structured notes extended GLM-5.2’s chain length from ~2 to ~6 steps in the synthetic RFG experiment.

4. Unified context_lookup tool didn’t solve the bottleneck. Merging search + graph traversal into one tool eliminated tool-discovery overhead, but both models still used the unified tool the same way they used separate tools. Tool discoverability ≠ reasoning capability.

5. Different small models have different bottlenecks. Qwen-2.5-7b: reasoning capability (can’t apply axioms even when retrieved). Gemma-3-4b: working memory / decomposition (can apply axioms when forced one at a time, but can’t decompose multi-step problems independently). Stepwise scaffolding addresses decomposition, not reasoning.

6. Qualitative behavior difference. In freeform, gemma-3-4b answers were almost all apologetic admissions of failure. In stepwise mode, it stopped apologizing and attempted actual computations. The stepwise structure appears to reduce confusion by focusing the model on one axiom at a time.

Cross-Experiment Evidence Matrix

Unified Conclusions

The Consistent Pattern

Structured context is the bottleneck — not raw model capability. When a small model gets the right structure at the right time, it approaches frontier performance. When the structure is irrelevant or conflicts with existing knowledge, it becomes noise or causes regression.

The Key Distinction: Passive vs Active Structure

Passive structure helps small models more than frontier models. Active structure helps frontier models only. The delivery mechanism of structure is a measurable system property that predicts lift direction.

The Mechanism: Briefing, Not Upgrading

The model isn’t getting smarter — it’s getting better context. The Socratic reasoning graph is a persistent externalization of the model’s own understanding, generated during a teaching pass, retrieved on demand. One-shot structured note-taking, not knowledge acquisition.

Model-Dependent Bottlenecks

Different small models fail for different reasons:

Structure interventions help when they address the model’s specific bottleneck. No single intervention helps all models.

Implications for Paper 1

1. Structure is the bottleneck — consistent across all six experiments and eleven conditions.
2. The interaction modality (passive vs active) is a measurable system property that predicts lift direction for small vs frontier models.
3. Anti-hallucination is a stronger signal than correctness — structured notes discipline the model more than they make it smarter.
4. Effective chain length is a measurable quantity — one-shot notes extend it 3× (from ~2 to ~6 steps) for frontier models; stepwise scaffolding extends it for gemma-3-4b (from 1 to 4+ steps).
5. Different small models have different bottlenecks — “small model” is not a monolithic category. Reasoning capability, working memory, and domain knowledge are independent bottlenecks.

Reproduction

Boole Agent Evaluation

cd projects/boole-agent

# Start backend
PYTHONPATH=backend python3 -m uvicorn backend.core.server:app --host 0.0.0.0 --port 8400

# Run all 72 evaluations (6 conditions × 12 tasks × 2 models)
python3 eval/run_eval.py --conditions all --models all

# Grade results (GLM-5.2 unified grader)
python3 eval/grade.py --raw eval/results/raw_<timestamp>.json --grader glm-5.2

Synthetic RFG Experiment

# Run synthetic eval (teaching → transfer → grade)
python3 eval/synthetic_eval.py --models frontier
python3 eval/synthetic_grader.py --raw eval/results/synthetic_raw_<timestamp>.json

Stepwise Experiment

# Run stepwise eval (freeform + stepwise conditions)
python3 eval/stepwise_eval.py --conditions freeform,stepwise --models gemma3
python3 eval/stepwise_grader.py --raw eval/results/stepwise_raw_<timestamp>.json

Key Files

Data Artifacts

• Full results: projects/boole-agent/eval/RESULTS.md
• Experiment reference (all 6 experiments): projects/multi-agent-framework/docs/experiment-reference.md
• Benchmark repo: 00n-ai/repoloc-bench

Claim Boundaries

1. Results are from controlled comparisons on fixed task sets (12 tasks for main eval, 8 for synthetic RFG, 8 for stepwise). Statistical significance is not claimed at conventional thresholds.
2. The RFG domain is small (7 axioms). Findings about chain-length degradation are directional, not precise measurements.
3. Only two small models were tested in the stepwise experiment (qwen2.5:7b and gemma-3-4b). The model-dependent finding is based on these two data points.
4. The frontier model (GLM-5.2) was not run in stepwise mode — the baseline is from the synthetic RFG experiment.
5. “One-shot structured note-taking” describes the mechanism, not a formal learning-theoretic claim. The model generates notes from a single exposure; no incremental learning is demonstrated.

Related

• Multi-Agent Framework Benchmark — MA orchestration compresses the capability gap on evidence-grounded QA
• Coding Knowledge Graph Agent Benchmark — Code KG with directional reasoning helps small models on code generation
• Evidence-Weighted Routing and Error Measurement — The Paper 1 thesis spine