Most CRE AI vendor accuracy claims cannot be falsified, and that is the problem. A single percentage with no denominator, no test set, and no metric definition is not a measurement. It is a marketing figure wearing a decimal point. "99% accurate" tells you nothing about whether the system reads the fields that carry your risk, because accuracy is defined by whoever computed it, on data you never see, using a formula they never state. To evaluate a vendor, stop reading the headline number and force it to answer four questions: which metric, on what test set, against whose ground truth, and measured per field or per document.
Key Takeaways
A CRE AI vendor accuracy figure is meaningless without a stated denominator, a described test set, and a named metric. Accuracy alone can be true and useless at the same time.
Accuracy, precision, and recall measure different failures. A model can post 95% accuracy and still miss half of the rare, high-value fields you actually care about.
The National Institute of Standards and Technology defines precision as the fraction of retrieved items that are relevant and recall as the fraction of relevant items retrieved. One number cannot substitute for both.
Document-level accuracy and field-level accuracy are different measurements. A vendor quoting the higher one and letting you assume the lower one is not lying, but is not informing you either.
The only accuracy claim worth trusting is one you can reproduce on your own documents against your own ground truth.
Why Are Most CRE AI Vendor Accuracy Claims Unfalsifiable?
Most CRE AI vendor accuracy claims are unfalsifiable because they omit the three things that would let you check them: the sample the number was computed on, the definition of a correct answer, and the metric formula. A claim you cannot reproduce is not evidence. It is an assertion, and assertions do not survive diligence.
Consider what "95% accurate" leaves out. Accurate on which documents, retail leases or industrial, clean PDFs or scanned amendments? Correct by whose judgment, and at what tolerance, is a date off by one day wrong? Averaged over how many fields, and weighted how? Each omission moves the number. A vendor can report a real figure that is honestly computed and still tells you nothing about your portfolio, because it was computed on documents that look nothing like yours.
The academic evaluation literature has treated this as settled for decades. Manning, Raghavan, and Schütze, in the Stanford text "Introduction to Information Retrieval," build the entire field of system evaluation on a test collection: a fixed set of documents, a fixed set of information needs, and a set of relevance judgments treated as ground truth. Remove any one of the three and the metric loses meaning. A vendor number that arrives without its test collection is a metric stripped of the thing that made it a measurement.
What Is the Difference Between Precision, Recall, and a Single Accuracy Number?
Precision and recall measure two different failures, and a single accuracy number hides both. The National Institute of Standards and Technology, through its TREC evaluation program, defines precision as the proportion of retrieved items that are relevant and recall as the proportion of relevant items that are retrieved. Precision punishes false positives. Recall punishes false negatives. Accuracy, the raw share of correct calls, can stay high while recall collapses.
The gap matters most when the thing you are looking for is rare. If a co-tenancy clause appears in only 15% of your leases, a model that never flags one is right 85% of the time and useful zero percent of the time. Accuracy rewards it for the common case. Recall exposes it. This is why practitioners report both, and why the F1 score exists at all: it is the harmonic mean of precision and recall, designed to stay low if either one is low. As David Powers argues in "What the F-measure doesn't measure," even F1 carries hidden assumptions, which is the point. No single number is neutral, and every one you accept without its definition is a number you have chosen to trust blind.
Metric | What it answers | The failure it catches |
|---|---|---|
Accuracy | What share of all calls were correct | None specifically. It averages everything, including the easy majority |
Precision | Of the fields the model extracted, how many were right | False positives: confident, wrong values entering your model |
Recall | Of the fields that existed, how many did the model find | False negatives: real clauses the model never surfaced |
F1 | The balance of precision and recall | Either one being low. It cannot be gamed by one alone |
How Does a 95% Accuracy Claim Hide Field-Level Failure?
A 95% accuracy claim hides field-level failure by averaging the easy majority of fields with the hard, high-value minority. Standard fields drive the average up. The clauses that carry underwriting risk drag it down, and the average buries them. The way to see it is to stop accepting the aggregate and compute precision and recall on one critical field.
Work the numbers. A vendor tests extraction across 1,000 field instances and reports 950 correct, a true 95%. Break the 1,000 apart. Suppose 900 are standard fields, base rent, tenant name, commencement date, extracted at 99%, which is 891 correct. The remaining 100 are hard fields, escalation formulas, co-tenancy triggers, percentage-rent breakpoints, extracted at 59%, which is 59 correct. Add them: 891 plus 59 is 950. The headline 95% is arithmetically honest and operationally misleading, because the fields you underwrite against run at 59%.
Now zoom to a single high-value field with a confusion matrix. Across 200 leases, 30 actually contain a co-tenancy clause. The model flags 20 leases as having one. Of those 20, 15 are correct and 5 are wrong.
Outcome | Count |
|---|---|
True positives (flagged, real) | 15 |
False positives (flagged, not real) | 5 |
False negatives (real, missed) | 15 |
True negatives (correctly cleared) | 165 |
From that matrix, accuracy is 15 plus 165 over 200, which is 90%. Precision is 15 over 20, which is 75%. Recall is 15 over 30, which is 50%. The model missed half of every co-tenancy clause in the portfolio, and the accuracy figure still reads 90%, because 165 correct negatives swamp the 15 misses. If your diligence depends on catching co-tenancy exposure, 90% accuracy and 50% recall are not the same finding, and only one of them is disqualifying. For more on why one number cannot carry this weight, see why extraction accuracy is precision and recall, not a single number and the definition of precision and recall.
What Questions Force a Vendor to Prove an Accuracy Claim?
Four questions force a vendor to either produce evidence or reveal there is none: which metric, on what test set, against whose ground truth, and per field or per document. A vendor that can answer all four with specifics has done the measurement work. One that deflects to the headline number has given you a slogan. The table below is the script.
Question to ask | What a strong answer looks like | What a weak answer looks like |
|---|---|---|
Which metric produced this number? | Names precision, recall, and F1 per field, and defines what counts as correct | "Overall accuracy," with no formula and no per-field breakdown |
What documents was it measured on? | A described test set: property types, document conditions, sample size | "Across thousands of documents," with no composition stated |
Whose ground truth did you score against? | Human-labeled gold set, labeling protocol, adjudication of disagreements | The model's own output, or no stated source of truth |
Field-level or document-level? | Field-level, reported for the high-value fields specifically | A document-level or token-level figure quoted as if it were field-level |
Can I reproduce it on my documents? | Yes, run a blind test on your leases with your labels | "Our benchmarks already prove it," declines a live test |
The single most revealing question is the last one. An accuracy claim you can reproduce on your own documents against your own ground truth is the only one that survives contact with your portfolio. This is also why grounding every extracted value to its source matters as much as the accuracy number itself: a value you can trace to a page is a value you can audit, and an audited field is the only one whose accuracy you ever truly know. See extraction accuracy for how field-level measurement is defined.
Frequently Asked Questions
Is a higher accuracy percentage always better when comparing CRE AI vendors? No. A higher number computed on easier documents, a looser definition of correct, or a document-level average is worse evidence than a lower number computed on your document types with field-level precision and recall. Compare methods before you compare percentages.
What is the difference between field-level and document-level accuracy? Document-level accuracy asks whether a document was processed correctly overall. Field-level accuracy asks whether each extracted value is correct. A document can be 90% field-accurate and still carry a wrong escalation, so field-level, reported per critical field, is the measurement that matters for underwriting.
Why does ground truth matter so much in an accuracy claim? Ground truth is the set of known-correct answers the model is scored against. Without a stated, human-verified ground truth, an accuracy number is scored against nothing, or against the model's own output, which measures consistency rather than correctness.
Can I trust a vendor benchmark instead of running my own test? Only as a starting point. A vendor benchmark is measured on the vendor's chosen data. The only claim that binds is one reproduced on a blind sample of your own documents, labeled by your team, scored with a metric you defined in advance.
Conclusion
A CRE AI vendor accuracy claim is only as good as the test set, ground truth, and metric behind it, and most claims arrive with none of the three. The number itself is not the deliverable. The reproducibility is. A vendor that names its metric, describes its sample, states its source of truth, and agrees to a blind test on your documents has given you something you can verify. A vendor that repeats a headline percentage has given you a figure you cannot check and therefore cannot use.
The operator move is to invert the burden of proof. Do not ask a vendor how accurate the system is. Ask it to prove a number on your leases, per field, against labels you control. Firms that run that test learn what their tooling actually reads before it reaches the model. Firms that accept the headline inherit the errors the headline was built to hide.