Why Extraction Accuracy Needs Precision and Recall, Not a Single Number

Extraction accuracy metrics fail the operator when they collapse into one number, because a single accuracy figure hides the two errors that behave nothing alike: the field the model got wrong and the field the model never found. In CRE document extraction, those two failures carry different costs. A wrong number gets caught in review. A missing clause never enters the model at all. Precision and recall separate them; a lone accuracy score fuses them into a figure that looks reassuring and tells you almost nothing about what you missed.

Key Takeaways

• Extraction accuracy metrics need precision and recall because the two errors they measure, wrong extractions and missed extractions, carry different costs in CRE underwriting.

• Precision is TP / (TP + FP): of what the model extracted, how much was correct. Recall is TP / (TP + FN): of what existed, how much the model found.

• A worked example below shows a model at 92% overall accuracy hiding a 75% recall, meaning one in four co-tenancy clauses went undetected.

• Accuracy is misleading when one outcome dominates. A naive classifier on 90% spam data scores 90% accuracy while catching zero legitimate mail, per DataCamp's F1 guide.

• For extraction that feeds a valuation, recall is usually the metric to protect, because a missed clause is a silent error and a wrong one is a catchable one.

What Do Precision and Recall Actually Measure?

Precision and recall measure two different failures. Precision asks: of the fields the model extracted, what share were correct. Recall asks: of the fields that existed, what share the model found. Precision is TP / (TP + FP) and recall is TP / (TP + FN), where FP is a wrong extraction and FN is a missed one. They move in opposite directions.

The distinction is not academic. A model tuned to extract only what it is sure of has high precision-recall on the precision side and low on recall: it is rarely wrong but misses a lot. A model tuned to catch everything has high recall and low precision: it finds every clause but flags many that are not there. There is a genuine tradeoff. As the scikit-learn documentation puts it, increasing precision typically reduces recall and the reverse, which is exactly why a single number cannot describe both.

Why Does a Single Accuracy Number Mislead?

A single accuracy number misleads because it rewards the model for being right about the common case while hiding the rare case that matters. Accuracy is (TP + TN) / total. When most documents do not contain a given clause, a model that simply predicts "no clause" scores high accuracy while catching almost none of the clauses that exist.

DataCamp's F1 guide gives the canonical illustration: on a dataset that is 90% spam, a classifier that labels everything spam reaches 90% accuracy and never once identifies a legitimate email. The number looks strong; the model is useless. CRE extraction has the same shape. Co-tenancy clauses, purchase options, and unusual escalation terms are rare across a portfolio. A model that under-predicts them still posts high accuracy, because the denominator is dominated by the leases that lack them. The extraction accuracy figure a vendor quotes is almost always this blended number, and blended is where the risk hides.

How Can 92% Accuracy Hide a 75% Recall? A Worked Example

A worked example makes the gap concrete. Suppose you run 100 leases through a model to detect a co-tenancy clause. In reality, 20 leases contain one. The model flags 18 leases as having the clause. Of those 18, it is right on 15 and wrong on 3. So it correctly found 15, wrongly flagged 3, and missed 5 real clauses entirely.

Now compute each metric from those counts. True positives (TP) = 15. False positives (FP) = 3. False negatives (FN) = 5. True negatives (TN) = the 80 leases without the clause, minus the 3 wrongly flagged, so 77.

The accuracy figure is 92%. It is technically true and operationally dangerous. Recall is 75%, which means one in four co-tenancy clauses went undetected. In a shopping-center acquisition, an undetected co-tenancy trigger is exactly the kind of miss that resets a valuation after close. The 92% describes the leases without the clause. The 75% describes the risk. As the working principle goes: report the number that describes your worst case, not the one that describes your easiest. The two here differ by 17 points, and only one of them is about the deal.

Which Metric Should an Operator Optimize For?

An operator should usually protect recall in extraction that feeds a valuation, because a missed field is a silent error and a wrong field is a catchable one. A false positive surfaces in review; a human sees a flagged clause and confirms or rejects it. A false negative surfaces nowhere. The clause simply never appears, so no reviewer knows to look.

This is why the confidence score and the review workflow matter more than the headline metric. A pipeline can run at high recall and accept lower precision, then route every flagged field to a human-in-the-loop reviewer who clears the false positives quickly. That design turns the cheap error (a wrong flag, caught in seconds) into the one you tolerate, and drives the expensive error (a missing clause, caught never) toward zero. The metric to optimize is not F1 in the abstract. It is the recall on the specific high-stakes fields, options, co-tenancy, escalations, whose absence would move the number.

Frequently Asked Questions

What is the difference between precision and recall in extraction? Precision is the share of extracted fields that are correct: TP / (TP + FP). Recall is the share of existing fields the model found: TP / (TP + FN). Precision exposes wrong extractions; recall exposes missed ones. They typically move in opposite directions, so no single number captures both.

Why is a single accuracy number misleading for document extraction? Because accuracy rewards being right about the common case and hides the rare one. When most documents lack a given clause, a model that under-predicts it still scores high accuracy. DataCamp's F1 guide shows a classifier on 90% spam scoring 90% accuracy while catching zero legitimate mail.

Should extraction optimize for precision or recall? Usually recall, for fields that feed a valuation. A false positive is caught by a reviewer in seconds; a false negative is a missing clause no one knows to look for. A well-designed pipeline runs at high recall and routes flagged fields to a human who clears the false positives.

Conclusion

Extraction accuracy is not one number, and treating it as one is how a missed clause reaches a model unnoticed. Precision and recall exist because the two failures they name behave differently and cost differently. A wrong field is loud and cheap to fix. A missing field is silent and expensive, because it changes the valuation without ever appearing on screen. A blended accuracy figure fuses them and reports the easy one.

The operator takeaway is to stop accepting a single accuracy score and start asking for recall on the fields that move the deal. Firms that measure recall on options and co-tenancy triggers know what they might be missing. Firms that accept 92% learn about the missing quarter after close, when the clause that was never extracted turns out to have been in the lease all along. The number to trust is not the one that looks best. It is the one that describes what you failed to find.

Related

• Precision and Recall

• Extraction Accuracy

• Human-in-the-Loop

• Why AI Lease Abstraction Accuracy Is Hard: The Last Twenty Percent

Related Reading

• Data Provenance in CRE: Why Source Citations Matter

• Data Validation Rules: The Guardrails That Keep AI Extraction From Shipping Garbage

• Design the Extraction Workflow Around the Exceptions, Not the Straight-Through Path

• OCR Does Not Read a Document. It Guesses Pixels, and on Scanned OMs It Guesses Wrong

• Structured Data Is the Real Product of AI Extraction, Not the Summary