Model drift is the gradual decay of an AI model's predictive accuracy after deployment, caused by the live data or the input-to-output relationship shifting away from what the model was trained on. In commercial real estate document extraction, it means a model that read leases well last year quietly reads them worse as document formats, clause language, and market conventions change.
What Causes Model Drift?
Model drift is caused by the world moving while the model stays fixed. A model learns static patterns from historical documents, but the documents it sees in production keep changing: new lease templates, unfamiliar clause phrasing, revised rent roll layouts, and shifting market terms. The trained patterns slowly stop matching the live inputs, and accuracy falls without any code changing.
Drift splits into two forms that fail differently. Data drift is a change in the input distribution, such as a new property management system exporting rent rolls in an unfamiliar format. Concept drift is a change in the relationship between input and correct answer, such as "base rent" starting to exclude a component it once included after a market convention shifts. Data drift is easier to catch because you can compare input distributions directly; concept drift hides until outputs are checked against ground truth.
Cause | Type | How it shows up in extraction |
New lease template | Data drift | Model sees layouts absent from training |
Unfamiliar clause phrasing | Data drift | Wording patterns shift under the model |
Redefined field convention | Concept drift | Correct answer for a field changes meaning |
New document source | Data drift | Export format and structure differ |
Why Model Drift Matters
Model drift matters because a model's accuracy at launch is not a promise about next quarter. Research summarized by NannyML, drawing on a study across institutions including MIT and Harvard, found that 91% of machine learning models degrade over time once deployed. In extraction, undetected drift means a field that was reliable starts feeding wrong rents and dates into a pro forma, and no one notices until a deal turns on the number.
The danger is that drift is silent. Unlike a crash, a drifting model keeps returning confident, well-formatted answers that are gradually less correct. This is why post-deployment monitoring, not one-time validation, is the operating assumption. As a design principle, a model that is not measured against fresh ground truth should be assumed to be drifting, because most of them are.
Example
Model drift is easiest to see when accuracy is measured over time on the same field. An extraction model launches reading base rent at 97% accuracy on a monthly sample of 400 leases. Over three quarters a large landlord migrates to a new lease template the model never trained on. The table tracks measured accuracy on that field.
Quarter | Leases sampled | Base rent accuracy | Errors |
Q1 (launch) | 400 | 97.0% | 12 |
Q2 | 400 | 94.5% | 22 |
Q3 | 400 | 90.0% | 40 |
Q4 (after retrain) | 400 | 96.5% | 14 |
Accuracy falls from 97.0% to 90.0% across three quarters, a 7 percentage point drop that more than triples the error count from 12 to 40. Nothing in the model changed; the incoming documents did. Because each output still looked clean, the decay was invisible until the field was measured against verified values. After the model is fine-tuned on the new template in Q4, accuracy recovers to 96.5%. The lesson is that drift is a monitoring problem before it is a modeling one.
Variations and Edge Cases
Model drift comes in distinct patterns, and naming them helps a team pick the right monitor. The variants below differ in what shifts and how fast, which changes both how you detect them and how urgently you respond.
Variant | Definition |
Data drift | The input distribution changes; the target relationship holds |
Concept drift | The input-to-output relationship changes; inputs may look the same |
Sudden drift | A change happens abruptly, such as a new template rolled out overnight |
Gradual drift | Accuracy erodes slowly as conventions shift over months |
Seasonal drift | A recurring pattern, such as year-end lease amendments, that returns on a cycle |
Model Drift vs AI Hallucination
Model drift is often conflated with AI hallucination, but they operate on different timescales. Model drift is a population-level decay of accuracy over time as live data diverges from training data, measurable only across many documents. AI hallucination is a single confident output invented for one document in one moment, with no basis in the source in front of it.
The distinction shapes the defense. Drift is caught by monitoring accuracy against ground truth over time and retraining when it falls. Hallucination is caught per field by grounding outputs in the source and flagging low-confidence values for review. A drifting model can be accurate on the next document; a hallucinating one is wrong on this one.
Frequently Asked Questions
What is model drift in AI?Model drift is the gradual decline of an AI model's accuracy after deployment, caused by live data or the input-to-output relationship shifting away from the training data. A model that performed well at launch reads new documents progressively worse until it is monitored and retrained.
What is the difference between data drift and concept drift?Data drift is a change in the input distribution, such as a new document format, while the correct answer for each input stays the same. Concept drift is a change in the relationship between input and correct answer, such as a field convention being redefined. Data drift is easier to detect than concept drift.
How is model drift detected?Model drift is detected by monitoring model outputs against fresh ground truth and comparing live input distributions to training distributions using statistical tests such as Population Stability Index or the Kolmogorov-Smirnov test. Falling accuracy on a tracked field signals drift and triggers retraining.
Related Terms
Extraction Accuracy
Model Fine-Tuning
Confidence Score
Human-in-the-Loop
AI Hallucination