By Editorial — IT Researches
Reproducibility is one of those topics that everyone agrees is important and almost no one prioritises until a specific incident makes the cost of its absence unavoidable. We have been on the receiving end of enough of those incidents — ours and our clients' — to have settled on a small set of practices that we now apply by default. They are unglamorous, they are not novel, and they are the difference between a research practice that scales and one that does not.
The first practice is deterministic data snapshots. Every experiment is bound to a named snapshot of the input data, identified by a content hash, with the schema written down separately and versioned independently. "The latest data" is not allowed as an input to a comparable experiment, and we have stopped accepting any experiment writeup that does not name its snapshot.
The second is environment pinning at the level of the lockfile. Reproducibility at the level of "this requirements.txt" is not reproducibility; reproducibility at the level of a fully resolved lockfile, with hashes, in a container image whose digest is recorded next to the experiment, is. The cost of doing this once is small. The cost of not having done it, when the experiment in question turns out to be the basis of a regulatory submission three years later, is enormous.
The third is structured experiment metadata. Every experiment writes a single JSON document at the moment it is launched: the snapshot id, the image digest, the seed, the model configuration, the metric definitions and the operator. The document is the canonical record; every chart and every table downstream is reconstructible from it. We index the documents in a small internal store and search across them; the search is the most-used internal tool we have built.
The fourth is metric definitions as code. We have learned the hard way that two teams looking at "the same metric" are usually looking at two slightly different definitions of it. We define metrics as small, named, versioned functions; the definition is referenced by id wherever the metric is used; a metric whose definition has changed gets a new id. The discipline is mildly irritating until the day it saves a release.
The fifth is the periodic replay. Every quarter we pick a small number of historical experiments — usually three or four — and re-run them from scratch, using the snapshots, images and configurations of record. The intent is not to validate the original result; the intent is to validate the reproducibility infrastructure itself. Almost every replay surfaces something — a deprecated dependency, a snapshot that has been pruned, a container image that no longer pulls — and surfacing it in a planned exercise is much cheaper than surfacing it in the middle of an incident.
Together, these practices add about ten per cent to the elapsed time of a typical experiment in the first month of their introduction, and somewhere closer to zero by the end of the second. They subtract a great deal more than that from the cost of the next year of operation. Reproducibility is not a virtue; it is a piece of cheap insurance that the discipline of doing applied AI for the long run has taught us not to skip.