Reproducibility: Making Your AI-Assisted Work Re-Runnable

Traditional reproducibility says: publish your code and data, and anyone can re-run. AI adds three new failure modes: the model you used may be deprecated; the same prompt may produce different outputs next week; and stochastic sampling means even identical prompts produce different outputs across runs.

The lock-down checklist

1. 1Record the exact model ID (e.g., 'gpt-4o-2024-08-06', 'claude-sonnet-4-5-20250929')
2. 2Record the temperature, top-p, and max-tokens settings
3. 3Set a random seed where the API supports it
4. 4Version-control every prompt as a text file, not buried in a notebook
5. 5Cache the raw model outputs alongside the derived results
6. 6Document the date of every API call — model behavior changes silently

What to publish

• The full prompts (as .txt or .md files in your repo)
• The model IDs and settings (as a config file)
• The raw responses you actually used (as JSON)
• Any post-processing code
• A README that walks through re-running the whole pipeline

The big idea: AI-assisted work is only reproducible if you treat prompts as code and cache outputs as data. Everything else is a promise you can't keep.