When the answer is in the world,
does the model find it?

We give a model a complete, synthetic world of facts and relationships. Its task is simple to state: decide whether a conclusion follows. We vary the graph’s depth and breadth, then measure where reliable conclusions break down.

01 / WHAT WE TESTED

A world, a question, one model call.

Each “Every A is a B” rule creates a directed edge. A fact places an entity at a starting node. The answer is True only when the target can be reached by following the arrows.

Depth 4 · four implication hopsBreadth 2 · two children per internal predicate19 nodes · 17 edges including the starting fact

The world as a graph

The model received text, not this diagram. A and B labels below are presentation aliases only.

Show deterministic reference

Stated implicationGold proof pathReachable component in a False case

Select a node to see its exact symbol highlighted in the prompt.

Expected answer · independently verified

The exact prompt sent to Luna

System and user messages are shown verbatim, including the shuffled rules and original response instructions.

The selector changes only the entity’s starting fact. The rules and target remain the same.

Inspect the recorded model response

Unreachable does not mean absent. The target is in both worlds. In the False case it lies in the other disconnected component. The reference algorithm finds no path from the entity; the model must reach that conclusion from the supplied text.

Why use synthetic worlds, and what do depth and breadth mean?

Opaque, freshly generated symbols reduce direct factual-recall shortcuts. They do not rule out familiarity with the task type.
Depth counts implication hops along a spine. The entity’s starting fact adds one graph edge.
Breadth is the number of children per internal predicate, not the maximum width of a graph layer.
The world family contains two spines with distractor leaves. This is not a test of every possible graph topology.
The displayed pair is an actual recorded example, selected to illustrate a false positive, not a representative sample. Aggregate results below include every planned case.

02 / WHAT WE MEASURED

Where correct rejection falls below 90%.

Each setting contains 2,000 unreachable cases and 2,000 matched reachable controls. The primary measure is returning the correct False answer within the output budget, not just avoiding an incorrect True.

4models in the joint analysis

20measured model–complexity settings

80,000calls across two completed studies

95%simultaneous confidence across rejection settings

Confirmed above 90%Confirmed below 90%Unresolved relative to 90%

Dots show known correct rejections / all 2,000 planned cases. Whiskers are jointly corrected confidence intervals, widened conservatively for infrastructure unknowns. The dashed line is the 90% criterion. Separate rows are measured settings, not an interpolated frontier or a model leaderboard.

The correction matters. Luna at 32 × 2 scored 91.90%, but its joint interval crosses 90%, so it is unresolved. A supported Luna depth bracket remains: 16 × 4 above 90%, and 32 × 4 below 90%.

All results, including reachable controls

“Reachable accuracy” is secondary and descriptive; it is not covered by the rejection intervals.

Model

Model	Depth × breadth	Correct rejection	Joint interval	Decision	Unknown	Reachable accuracy

Unknowns stay in the planned denominator, not silently discarded. All 183 infrastructure unknowns are in Gemma’s unreachable cases; its above/below decisions survive assigning them either way. Token exhaustion and missing final answers are counted as failures.

03 / HOW TO INTERPRET IT

A measured reliability limit, not a universal capacity score.

What the evidence supports

Under the tested prompt, representation, and model-specific settings, all four models have measured settings above 90% successful correct rejection and settings below it.

Both unsupported conclusions and missed supported conclusions occur. Providing all required information is not sufficient to guarantee a correct answer.

What it does not establish

These are not production-safety limits, a ranking of intrinsic model intelligence, or a smooth boundary across untested graphs. Models used different complexity neighborhoods and provider settings.

Prompt effects remain unresolved: the True-only format example, asymmetric proof instructions, opaque labels, and rule ordering may contribute. They require separate controlled tests.

Statistical method, study provenance, and reproducibility

This is a retrospective joint analysis of completed confirmation cohorts, not a prospectively registered four-model study. All 20 correct-rejection settings are included. Two-sided exact Clopper–Pearson intervals use Bonferroni correction with tail α = 0.05 / 40. For C known successes and U unknowns, the lower limit uses C successes and the upper limit uses C + U successes out of 2,000.

The family covers successful correct rejection only. Explicit false-positive rates, reachable accuracy, proof validity, and model superiority are not additional jointly corrected claims. Shared worlds across models are paired. The analysis assumes independent sampled world units within each model-cell.

Luna used native OpenAI Batch with medium reasoning effort; Gemma, Hermes, and DeepSeek used DeepInfra Batch with their recorded settings. Each call had a 24,000-token output ceiling. Condition C and GLM are not included. The illustrated graph is a rendering of a recorded Condition A input, not an extra tool available to the model.

The page is an offline snapshot of completed results. No inference runs or external network requests are triggered by its controls.