The unit of evidence
A run is the smallest publishable unit. It contains the exact prompt, declared model identity, dimensions, exposed generation settings, timestamps, access method, source provenance, rights note, review outcome, and a content-addressed artifact.
Unknown provider values are stored as null. They are never guessed.
Controlled experiments
Each experiment freezes a versioned prompt template and declares one controlled variable. Repeated unseeded trials are labeled as such. Cross-model comparisons become valid only when the prompt and exposed settings match closely enough for the stated research question.
Review rubric
Reviews separate prompt adherence from aesthetic preference. The active rubric includes identity, composition, typography, material realism, anatomy, mechanical coherence, artifact severity, and production readiness. Review text must identify visible deviations rather than collapse everything into one score.
Failure taxonomy
blocked, moderated, error, timeout, and discarded are first-class outcomes. A failure can still be aesthetically useful, but the failed constraint remains explicit. Empty failure counts are reported as dataset state, not evidence of model quality.
Access-policy boundary
Automated provider adapters require an official API, a first-party permitted tool, or local inference. Manual-only providers stay manual-only in code and data. No CAPTCHA bypass, App Check bypass, restriction evasion, or undocumented browser automation is part of this project.
Verification gates
- JSON Schema validation
- ID uniqueness and referential integrity
- secret-like URL query rejection
- asset path, byte count, MIME, and SHA-256 verification
- orphan and duplicate-asset detection
- deterministic catalog compilation
- Python tests and lint
- Hugo build and route verification
- desktop and mobile browser QA before release