Herman was retired on September 10, 2026. This site is preserved as a frozen archive and is no longer maintained or updated. Read the retrospective →

Experiment · active

Counting and spatial relations

Can the model obey exact object counts and left-right-inside-behind relationships?

Runs
117
Variable
spatial constraint
Prompt
spatial-counting-prompt v1.0

Protocol

Active round-one controlled comparison using one frozen prompt across five hosted model endpoints or quality tiers.

Frozen prompt template

CONTROL PROMPT AB-R1-SPATIAL-001. A wide 16:9 museum-catalog photograph of a matte charcoal tabletop containing exactly seven objects and nothing else: three red wooden cubes in one straight row on the left; two transparent glass spheres in the center; one small aged-brass pyramid directly behind the two spheres; and one folded cream paper crane on the far right. Camera elevated thirty degrees, all seven objects fully visible and separated, physically realistic materials, neutral softbox lighting, dark seamless background. No labels, no text, no watermark, no duplicate objects, no containers.

Review rubric

  • Prompt Adherence
  • Counting
  • Spatial Reasoning

Notes

One unseeded generation per model. Exact visible deviations are preserved in each run's review notes.

Evidence

Recorded runs

Generated specimen r1-spatial-grok-imagine r1-spatial-grok-imagine
successgrok-imagine-image

R1 Spatial Grok Imagine

Vision adherence score 10.0/10. Exact count and relations: 3 red cubes left, 2 glass spheres center, 1 brass pyramid behind them, 1 cream crane far right; 7 total.

Size
1280×720
Seed
not exposed
Method
first party tool
View experiment →
Generated specimen r1-spatial-minimax-image-01 r1-spatial-minimax-image-01
successimage-01

R1 Spatial Minimax Image 01

Vision adherence score 3.0/10. Count failure: 5 red cubes, 3 glass spheres, 1 brass pyramid, and 1 cream crane; 10 total. Left-to-right category order is otherwise preserved.

Size
1280×720
Seed
not exposed
Method
official api
View experiment →
Generated specimen r1-spatial-openai-gpt-image-2-high r1-spatial-openai-gpt-image-2-high
successgpt-image-2-high

R1 Spatial Openai Gpt Image 2 High

Vision adherence score 10.0/10. Exact count and relations: 3 red cubes left, 2 glass spheres center, 1 brass pyramid behind them, 1 cream crane far right; 7 total.

Size
1672×941
Seed
not exposed
Method
official api
View experiment →
Generated specimen r1-spatial-openai-gpt-image-2-low r1-spatial-openai-gpt-image-2-low
successgpt-image-2-low

R1 Spatial Openai Gpt Image 2 Low

Vision adherence score 10.0/10. Exact count and relations: 3 red cubes left, 2 glass spheres center, 1 brass pyramid behind them, 1 cream crane far right; 7 total.

Size
1672×941
Seed
not exposed
Method
official api
View experiment →
Generated specimen r1-spatial-openai-gpt-image-2-medium r1-spatial-openai-gpt-image-2-medium
successgpt-image-2-medium

R1 Spatial Openai Gpt Image 2 Medium

Vision adherence score 10.0/10. Exact count and relations: 3 red cubes left, 2 glass spheres center, 1 brass pyramid behind them, 1 cream crane far right; 7 total.

Size
1672×941
Seed
not exposed
Method
official api
View experiment →
Generated specimen r2-spatial-minimax-image-01-trial-02 r2-spatial-minimax-image-01-trial-02
successimage-01

R2 Spatial Minimax Image 01 Trial 02

Exact-count FAIL. Visible: 3 red wooden cubes in one straight left row; 1 small aged-brass pyramid mid-composition (between cubes and spheres, not behind the spheres); 3 spherical objects (1 dark sphere on a small square base, 2 transparent/frosted glass spheres); 1 cream paper crane far right. Counting the square base …

Size
1280×720
Seed
not exposed
Method
official api
View experiment →
Generated specimen r2-spatial-minimax-image-01-trial-03 r2-spatial-minimax-image-01-trial-03
successimage-01

R2 Spatial Minimax Image 01 Trial 03

Exact-count FAIL. Visible L→R: clustered red wooden cubes (~5, not 3 in one straight row); 2 transparent glass spheres; 1 aged-brass pyramid to the right of the spheres (not behind them); 1 crumpled cream paper form; 1 larger cream origami crane/boat on the far right. Total ~10 discrete objects vs required 7. Category …

Size
1280×720
Seed
not exposed
Method
official api
View experiment →
Generated specimen r2-spatial-minimax-image-01-trial-04 r2-spatial-minimax-image-01-trial-04
successimage-01

R2 Spatial Minimax Image 01 Trial 04

Exact object count PASS (7). L→R: 3 red wooden cubes in one straight left row; small transparent glass sphere; small aged-brass pyramid; large transparent glass sphere; cream paper crane far right. Spatial FAIL: pyramid sits between the two spheres in a single frontal row rather than directly behind the sphere pair. …

Size
1280×720
Seed
not exposed
Method
official api
View experiment →
Generated specimen r2-spatial-minimax-image-01-trial-05 r2-spatial-minimax-image-01-trial-05
successimage-01

R2 Spatial Minimax Image 01 Trial 05

Composition FAIL despite total discrete count of 7. L→R: ~4 red wooden cubes clustered/left (not 3 in one straight row); 1 transparent glass sphere (not 2); 1 aged-brass pyramid to the right of the single sphere (not behind a sphere pair); 1 cream paper crane far right. Missing second sphere; pyramid behind-relation …

Size
1280×720
Seed
not exposed
Method
official api
View experiment →