Active learning is a machine learning training method in which the model selects the most informative unlabeled examples and asks a human to label only those. In commercial real estate document extraction, it improves a model on leases and rent rolls by having reviewers label the documents it is least certain about, reaching target accuracy with far fewer annotations.
How Does Active Learning Work?
Active learning works as a loop. The model scores a large pool of unlabeled documents, selects the ones it is least certain about, and sends only those to a human for labeling. The new labels are added to the training set, the model retrains, and the cycle repeats. Each round spends labeling effort where it teaches the model the most.
The selection step is what makes active learning efficient. The most common strategy is uncertainty sampling, where the model queries the examples on which its predictions are least confident, since those carry the most new information. A lease with a clean, standard rent table teaches the model little; an amended lease with a buried renewal clause the model reads poorly teaches it a lot. By prioritizing the hard cases, the model reaches a given accuracy with a fraction of the labels random sampling would require.
Step | What happens |
Score | Model runs over a pool of unlabeled documents |
Select | Least-certain examples are chosen (uncertainty sampling) |
Label | A human annotates only the selected examples |
Retrain | New labels are added and the model updates |
Repeat | The loop continues until accuracy targets are met |
Why Active Learning Matters
Active learning matters because labeling is the expensive, slow bottleneck in building an extraction model, and most labels teach the model almost nothing. Hand-labeling every clause of every lease is costly and largely wasted on documents the model already reads correctly. Active learning concentrates human effort on the examples that move accuracy, so a small annotation budget goes further.
The efficiency gains are documented. Research on active learning for named entity recognition, the same field-labeling task that underlies lease extraction, reports reaching target accuracy with 30 to 70 percent fewer labels when the selection function is good, and one deep active learning study for NER found roughly 20 to 25 percent of the data sufficient to match full-data performance (Shen et al., "Deep Active Learning for Named Entity Recognition," arXiv 1707.05928). For a CRE team building a custom extractor, that is the difference between labeling ten thousand documents and labeling two thousand.
Example
Active learning is easiest to see against the labeling budget it saves. A team needs a lease extractor to hit 95% field accuracy and estimates it would take 10,000 labeled leases with random sampling. They run active learning instead, retraining after each batch of 500 labeled documents.
Round | Documents labeled (cumulative) | Selection | Field accuracy |
Start | 500 | Random seed set | 78% |
3 | 1,500 | Uncertainty sampling | 89% |
5 | 2,500 | Uncertainty sampling | 94% |
6 | 3,000 | Uncertainty sampling | 95% |
The model reaches the 95% target at 3,000 labeled documents instead of 10,000, a 70% reduction, consistent with the 30 to 70 percent range reported for NER-style tasks. The saved labels are the ones active learning never asked for: the thousands of clean, standard leases the model already read correctly and would have learned nothing from.
Variations and Edge Cases
Active learning behaves differently depending on how examples are selected and where the labels come from. The variants below change which documents the loop asks about.
Variant | Behavior |
Uncertainty sampling | Selects the least-confident predictions; simple and widely used |
Diversity-aware | Balances uncertainty with coverage so queries are not all near-duplicates |
Query-by-committee | Multiple models vote; examples they disagree on are selected |
Cold start | Early rounds lack signal; a random seed set is labeled first |
Sampling bias | Chasing only hard cases can skew the training set away from common documents |
Active Learning vs Human-in-the-Loop
Active learning is often confused with human-in-the-loop, but they solve different problems. Active learning is a training-time method: the model chooses which examples a human should label to build a better model faster. Human-in-the-loop is a run-time method: a human reviews and corrects the model's outputs on real documents as they are processed.
Put simply, active learning spends human effort to teach the model, and human-in-the-loop spends human effort to catch the model's mistakes in production. The same reviewers may do both, and the loops look similar, but one improves the model before deployment while the other guards its output after.
Frequently Asked Questions
How much labeling does active learning save?It varies by task and selection strategy, but research on named entity recognition, the labeling task behind field extraction, reports reaching target accuracy with 30 to 70 percent fewer labels, and some studies match full-data performance with only about 20 to 25 percent of the data.
What is uncertainty sampling in active learning?Uncertainty sampling is the most common selection strategy, in which the model queries the examples it is least confident about. Those least-certain predictions carry the most new information, so labeling them improves the model faster than labeling random or already-easy examples.
Is active learning the same as human-in-the-loop?No. Active learning is a training method where the model picks which examples a human should label to improve the model. Human-in-the-loop is a production method where a human reviews and corrects the model's live outputs. One builds the model; the other guards its results.
Related Terms
Training Data
Ground Truth
Human-in-the-Loop
Confidence Score
Extraction Accuracy