Machine Learning on Big Data

We design ML platforms around a small set of opinions: features should be defined once and reused; experiments should be reproducible by default; models should be versioned and registered before they reach production; and inference paths should be observable in the same way the underlying services are. The opinions are not novel; the discipline of applying them consistently is what separates platforms that compound from platforms that calcify.

Our preferred toolkit varies by environment. We have built around managed offerings such as Databricks, SageMaker, Vertex AI and Azure ML, around open-source stacks (MLflow, Feast, Ray, Kubeflow, Argo) on Kubernetes, and around hybrids of the two. The choice is dictated by your existing investments, your team's operational comfort and the latency profile of the workloads in question.

Across all of these we insist on a few non-negotiables: every artefact (data set, feature set, model) is identified by a content-addressable hash; every promotion between environments is logged with the lineage of inputs and the test results that justified it; and every model in production is paired with a monitor that knows what good looks like.

• Reproducible training pipelines
• Feature stores with online and offline parity
• Vector stores for retrieval and similarity
• Model registry, staging and rollback
• Drift, performance and cost monitoring
• Cost-aware training and inference

Distributed training is justified less often than it is attempted. Many of our clients arrive convinced they need a multi-node cluster, only to find that a well-tuned gradient-boosted model on a single large machine outperforms it for less money and less operational pain. We start by establishing a strong single-machine baseline, and we scale out only when the data, the model architecture or the latency targets clearly require it.

When distribution genuinely is required — for foundation-model fine-tuning, for distributed deep recommenders, or for very large feature spaces — we use Ray, PyTorch DDP, DeepSpeed, Spark MLlib or Horovod as appropriate, and we pay close attention to the data path. Distributed training that is bottlenecked on storage or shuffle is a depressingly common failure mode, and one whose cure rarely involves more GPUs.

Reproducibility is treated as a property of the training pipeline, not as a hopeful aspiration. Random seeds, library versions, environment definitions, data snapshots and hyperparameters are all captured automatically and tied to the resulting model artefact. Three months later, when a regulator or an analyst asks why the model behaves the way it does, the trail is intact.

“More GPUs is rarely the answer; better data plumbing usually is.”

Inference architecture is dictated by the consumer, not by the model. A risk score that informs a once-a-day batch decision wants a different platform from a real-time recommendation served at the ninety-ninth-percentile under tens of milliseconds. We design serving paths around the consumer's latency, throughput and freshness requirements, and we are unsentimental about reusing infrastructure across use cases when it makes sense.

For real-time inference we typically use a combination of a feature store with online serving, a model server (Triton, BentoML, KServe or a managed equivalent) and a request-side cache where appropriate. For batch inference, we lean on the same training cluster, with deterministic windowing and the same lineage controls as upstream pipelines. For streaming inference we integrate the model into the stream-processing topology directly, scoring events in flight.

We pay close attention to cost at this layer because it is where ML projects most reliably grow expensive. Quantisation, distillation, batching, request collapsing, autoscaling on real demand signals and a willingness to use smaller models when they suffice routinely cut inference cost by an order of magnitude with no measurable loss of accuracy.

A model in production is a living artefact. Data shifts, concepts shift, upstream services change, and downstream consumers find new ways to use the output. We design ML systems with the expectation that they will be retrained dozens of times across their useful life, with each retraining following the same controlled, auditable pathway as the first deployment.

Drift detection is wired in by default — both feature drift and label drift, where labels are available — and is paired with explicit thresholds for action. Some models retrain on a schedule; others retrain on signal; the choice depends on how labels arrive and how costly a stale model is. Either way, retraining produces a new candidate that competes with the incumbent on the same evaluation harness, and the winner is promoted under the same release controls used originally.

Where models are exposed to adversarial conditions — fraud, abuse, gaming — we add red-teaming and shadow deployment as standard practice. Production traffic flows to both the live model and a candidate, and the candidate's behaviour is compared in detail before promotion. The discipline is mundane and the safety it provides is substantial.

Across our engagements, the most consistent outcome is not a single model — it is a measurable acceleration of the ML lifecycle. Clients who needed months to take a model from notebook to production reliably do so in weeks. Models that previously required a dedicated team to operate quietly run themselves between routine reviews. The capacity that used to be consumed maintaining what already exists is freed up to build what is next.

The second consistent outcome is a defensible cost curve. Because feature work is reused, because inference is right-sized, because compute is monitored as a first-class concern, the marginal cost of the next ML use case falls. This is the property that distinguishes a platform from a portfolio of pilots.