Your data pipeline is the product

The demo trap

A model demo is a controlled experiment on a clean CSV. Production is a continuous stream of partial updates, schema drift, missing values, and upstream systems that change without telling anyone. The gap between those two worlds is where most AI projects die, not because the model is bad, but because the data it ingests is no longer the data it was trained on. We have watched teams spend six months tuning a gradient boosted model to squeeze out an extra two points of AUC, then ship it against a pipeline that has never seen a null value it didn't choke on.

The pattern repeats across industries: retail demand forecasting, fraud scoring, churn prediction. The proof of concept works because someone hand curated the input file. Nobody hand curates production. Once the model is live, it inherits every quirk of every upstream system: a warehouse job that used to run at 2am now runs at 6am, a third party API that quietly renamed a field, a currency column that switched from cents to whole units after a migration nobody flagged to the data team.

What a production pipeline actually needs

• Schema contracts between producers and consumers, enforced in CI.
• Freshness checks with alerts that reach a human within minutes, not dashboards.
• Lineage: every prediction must be traceable to the inputs that produced it.
• A rollback path that can restore yesterday's pipeline without redeploying the model.
• Automated backfill so a fixed upstream bug does not leave a permanent scar in the training data.

These are not MLOps luxuries. They are the minimum viable product for any predictive system that touches money, customers, or operations. The organisations that treat them as first-class engineering get models that age gracefully. The others get a series of 'incidents' that are really design failures wearing an incident report as a disguise.

The uncomfortable part is that none of this is glamorous work. Nobody presents a schema contract at a board meeting. But every serious outage we have been called in to diagnose traces back to one of these five items missing, not to the model architecture. Fix the boring layer first.

Who owns what, in practice

Ownership ambiguity is the single biggest predictor of pipeline rot. When a data source has no named owner, nobody notices when it breaks until a customer complains. We push every client toward a simple rule: every table, every feed, every API has one name attached to it, and that name is on call for it.

• The data engineering team owns extraction, transformation, and freshness SLAs.
• The domain team that generates the data owns its meaning and any schema changes.
• The ML team owns feature definitions and consumes the contract, it does not author the source.
• A rotating on call owns the pager for freshness and drift alerts, seven days a week.

The numbers we watch

The commodity move is to stop treating the pipeline as the unglamorous part of the project. It is the product. A well engineered pipeline makes models interchangeable: swap the algorithm, keep the data contract. That is what lets you improve the model without betting the business on every release. It also makes procurement simpler: you can buy a better model from a vendor without renegotiating your entire data estate.

We track one more number internally that rarely makes it into a client dashboard: mean time to detect silent failure. Anecdotally it runs from a few minutes at the disciplined shops to three to four weeks at the shops that discover the problem when quarterly numbers look strange. Three to four weeks of a fraud model scoring on stale features is not a bug, it is a business event.

What to change on Tuesday