Model drift is why a document-extraction model that passed every test at launch quietly gets worse in production. The model did not break. The documents changed. A new broker template, a scanned page where there used to be native text, a clause convention that shifts across a market cycle: each moves the input away from the data the model was measured against, and extraction quality decays field by field. The dangerous part is that the decay is invisible without measurement. Accuracy falls a point a quarter while the interface looks identical, and the first signal an operator gets is a wrong number that reaches an investment committee. This is a production ai problem, not a laboratory one.
Key Takeaways
Model drift is the gap that opens between the data a model was trained and validated on and the data it sees in production; it degrades extraction quality without changing a line of code.
NIST defines drift as the tendency for AI system performance to shift over time once deployed, and names monitoring as the control that detects it.
Data drift and concept drift are different failures: data drift is a change in the input distribution, concept drift is a change in the relationship between input and answer, per the Gama concept-drift survey.
Drift is silent by default. Without a labeled data audit against ground truth, a model can lose accuracy quarter over quarter before anyone notices.
A model does not decay because it was built wrong. It decays because the world it was measured against has moved.
What is model drift in a CRE extraction model?
Model drift is the decline in a deployed model's accuracy over time as production data diverges from the data the model was trained and validated on. The model is static; the world is not. As broker formats, document quality, and market conventions change, the inputs shift away from the training distribution, and extraction quality falls even though nothing about the model changed.
The National Institute of Standards and Technology (NIST) frames this directly in its AI Risk Management Framework. In the Manage function, NIST states that system performance and trustworthiness may shift over time once a system is deployed, a phenomenon it calls drift, and that regular monitoring is what lets an organization detect and respond to it. Its Manage 2.4 guidance goes further, requiring mechanisms to supersede, disengage, or deactivate systems whose performance becomes inconsistent with intended use. The framework treats drift not as an edge case but as the expected condition of any model left running.
For a CRE extraction pipeline, that means the launch benchmark has a shelf life. A model validated at 96% field accuracy on last year's offering memoranda is not a model that reads this quarter's memoranda at 96%. The number was true on the day it was measured and decays from there. Extraction accuracy is a workflow property that has to be re-measured, not a specification stamped once at delivery. The same discipline that keeps AI hallucination in CRE in check, tying outputs to sources, is what surfaces drift before it reaches a committee.
Data drift or concept drift: which one degrades extraction quality?
Both degrade extraction quality, but through different mechanisms, and the distinction determines the fix. Data drift is a change in the input distribution: the model sees documents that look statistically unlike its training set. Concept drift is a change in the relationship between the input and the correct answer: the same input now maps to a different label. One is a shift in what arrives, the other a shift in what is right.
The academic formalization comes from the concept-drift survey by Gama and colleagues, published in ACM Computing Surveys in 2014. In that framing, data drift is a change in the input distribution P(X) while the mapping from input to answer holds, and concept drift is a change in the conditional relationship P(y given X), where the target the model must predict has itself moved. In extraction, data drift looks like a new rent-roll layout the model has never parsed. Concept drift looks like a field definition changing under you: what your team once abstracted as "base rent" now excludes a component that a new lease convention breaks out separately, so the label the model was trained to produce is no longer the label you want.
Drift type | What changes | Typical CRE cause | How you detect it |
|---|---|---|---|
Data drift (covariate shift) | Input distribution P(X) | New broker template, scanned instead of native PDF, unfamiliar asset class | Input monitoring, distribution and file-type checks, confidence-score decline |
Concept drift | Relationship P(y given X) | Field definitions revised, clause conventions shift across a market cycle | Labeled audit against a refreshed ground-truth set; falling accuracy on stable inputs |
Label or prior drift | Class balance P(y) | Deal mix shifts toward a document type the model rarely saw | Output-rate monitoring; extraction volumes per field diverge from history |
Upstream drift | The pipeline feeding the model | OCR engine update, preprocessing change, source-system export change | Version diffing, a fixed canary document set run on every release |
Data drift is usually the earlier and more common signal, because formats change constantly. Concept drift is rarer and more dangerous, because the model can look confident while producing the wrong target. A model evaluation that only checks whether the model is sure of itself misses concept drift entirely; only comparison against fresh ground truth catches it.
How much does model drift cost over four quarters?
Drift is expensive because the cost is a review tax that compounds quarter over quarter, and most of it is invisible. A worked example makes the size concrete. All figures below are derived from stated inputs, not measured from any specific system.
Assume a pipeline processes 500 offering memoranda per quarter, each yielding 40 extracted fields, for 20,000 fields per quarter. At launch the model runs at 96% field accuracy, a 4% error rate. Now assume drift erodes accuracy by 1.5 percentage points per quarter as templates and conventions move, a representative decay rate for an unmaintained model, not a measured constant.
Quarter | Field accuracy | Error rate | Erroneous fields (of 20,000) | Extra errors vs launch |
|---|---|---|---|---|
Launch (Q0) | 96.0% | 4.0% | 800 | 0 |
Q1 | 94.5% | 5.5% | 1,100 | 300 |
Q2 | 93.0% | 7.0% | 1,400 | 600 |
Q3 | 91.5% | 8.5% | 1,700 | 900 |
By Q3 the model produces 1,700 wrong fields a quarter, 900 more than at launch. Put a review cost on the caught errors: assume each error a human catches and corrects takes 6 minutes. The 900 additional errors cost 5,400 minutes, or 90 analyst-hours, in Q3 alone. At a loaded analyst rate in the range of 75 dollars per hour, that is roughly 6,750 dollars of added review in a single quarter, on top of the launch-level review the process already carried. Annualize the widening gap and the drift tax runs into the low tens of thousands of dollars per year for one pipeline.
That figure only counts errors a reviewer catches. The costlier half is the errors that survive review, the missed fields that never surface. As with the tradeoff covered in extraction accuracy metrics, a wrong field caught in review is the cheap error; a wrong field that reaches a valuation is the expensive one. Drift raises both, and it raises the silent one without ever showing up on a dashboard that is not measuring for it.
How do you detect and correct model drift in production?
You detect drift by measuring the live model against ground truth on a schedule, and you correct it by retraining on data that reflects the new distribution. Detection is a monitoring problem, correction is a data problem, and neither happens on its own. NIST names continuous monitoring as the mechanism that lets an organization detect and respond to drift once a system is deployed.
Detection has two layers. The first is a data audit of inputs: track file types, template signatures, page counts, and per-field confidence scores, and alert when the production distribution moves away from the training distribution. This catches data drift early, often before accuracy visibly falls, because the inputs change before the outputs are scored. The second layer is a periodic labeled audit: sample production documents, have a human establish the correct answers, and score the model against them. This is the only layer that catches concept drift, which can leave the inputs looking normal while the correct answer moves underneath.
Correction follows detection. When a labeled audit shows accuracy has decayed past a set threshold, retrain on data drawn from the current distribution, including the new templates and revised conventions that caused the drift, then re-validate and reset the benchmark. The principle is that a model is not a delivered artifact but a maintained one. Firms that run a standing data audit and a retraining trigger keep extraction quality flat. Firms that treat the launch number as permanent accumulate a widening error gap they cannot see. What to detect is captured in the working notion of model drift itself: the deployed model and the world it serves have quietly stopped agreeing.
Frequently Asked Questions
What is the difference between data drift and concept drift? Data drift is a change in the input distribution: the model sees documents statistically unlike its training data, such as a new broker template. Concept drift is a change in the relationship between input and correct answer, such as a field definition being revised, so the same input now maps to a different label. Data drift is more common; concept drift is harder to detect and more dangerous.
How do you detect model drift in a document-extraction pipeline? Run two monitors. First, a data audit of inputs that tracks file types, template signatures, and per-field confidence scores and alerts when the production distribution moves. Second, a periodic labeled audit that samples production documents, establishes correct answers by hand, and scores the model against them. Input monitoring catches data drift early; only the labeled audit catches concept drift.
How often should a CRE extraction model be retrained? There is no fixed interval; retrain on a trigger, not a calendar. Set an accuracy threshold from a recurring labeled audit and retrain when the live model falls below it, or when input monitoring flags a sustained distribution shift such as a new dominant template. NIST frames drift as the expected condition of any deployed system, so monitoring should run continuously and retraining should follow the measurement.
Conclusion
Model drift is the default trajectory of any extraction model left running against a changing document supply. The model does not degrade because it was built poorly. It degrades because the offering memoranda, leases, and rent rolls it reads this quarter are not the ones it was measured against, and no benchmark stamped at launch survives that movement. Data drift shifts what arrives, concept drift shifts what is correct, and both erode extraction quality a point at a time while the interface looks unchanged. The cost is a review tax that compounds and a set of silent misses that never reach a dashboard.
For the operator, the takeaway is to stop treating an accuracy figure as a specification and start treating it as a reading that expires. A production ai system needs a standing data audit against ground truth and a retraining trigger tied to it, because the only way to know a model has drifted is to keep measuring it. The firms that hold extraction quality flat assume decay and measure for it. The firms that assume the launch number holds find out otherwise the quarter a wrong field clears review and reaches the committee.