By Editorial — IT Researches
Since 2011 we have shipped a great many machine-learning systems into production. The technology has changed beyond recognition; the operational mistakes, in our experience, have not. The five we list here are the ones we still see in 2025, in organisations that ought to know better, and in organisations whose teams are quietly aware they are making them but have not yet found the political space to fix them.
The first mistake is treating the feature store as an architecture rather than a discipline. Most enterprise feature stores end up as a directory of slightly different versions of the same features, owned by nobody, refreshed on schedules nobody can quite remember, and serving a long tail of models that do not really need them. The fix is unglamorous: name an owner, write the contracts down, retire features aggressively, and resist the urge to publish a feature unless at least two consumers can be named at the time it is published.
The second mistake is operating dashboards no one looks at. We have lost count of the model-monitoring dashboards we have inherited that nobody opens between incidents. The trick is not to build prettier dashboards but to make the alerts the primary product and the dashboards the secondary one. An alert that arrives in the on-call channel with a one-line description, a link to the suspected feature drift, and a runbook entry is worth more than a wall of charts that quietly confirms the system is working most of the time.
The third mistake is conflating reproducibility with replayability. Reproducibility — being able to retrain the same model on the same data and get a comparably good result — is a research virtue. Replayability — being able to ask, at 3 a.m. on a Tuesday in eighteen months' time, exactly which model produced a specific decision, on which features, with which feature values — is an operational obligation. Most teams under-invest in the latter because the former is more interesting. Regulators do not share that preference.
The fourth mistake is shipping without a rollback. The phrase "we'll roll back if it goes wrong" usually means "we have not actually rehearsed a rollback for this model in this environment under load". A rollback that has been rehearsed once a quarter, with the on-call team that will need to execute it, is worth ten that exist only on a wiki page. The corollary is that every model deployment plan should include the rollback as a first-class artefact, not as an appendix.
The fifth mistake is treating evaluation as a one-time gate rather than a continuous practice. Pre-deployment evaluation tells you whether the model is fit to ship; continuous evaluation tells you whether it is fit to continue. The good teams have written down, for each production model, the precise definition of "still healthy" — the metrics, their thresholds, the segments those thresholds apply to, and the cadence on which they are checked — and the action that is automatically taken when health degrades.
None of the above is novel. All of it has been written about, often by people closer to the front line than we are. The reason it bears repeating in 2025 is that the arrival of foundation models has, at the margins, made teams less careful about the operational basics rather than more. The new capability is exciting; the underlying discipline is the same as it has always been; the projects that will still be running in five years are the ones whose teams understood the difference.