A benchmark dataset is a fixed, labeled collection of documents with known correct answers, used to measure and compare how accurately AI systems extract information. In commercial real estate document work, a benchmark dataset provides the ground-truth leases, rent rolls, and forms against which an extraction model's precision and recall are scored on the same yardstick.
What Is a Benchmark Dataset?
A benchmark dataset is a shared test set of documents paired with verified correct answers, so competing models are graded on identical inputs by the same metric. It separates the test data from the training data, holds the labels fixed, and defines a scoring rule such as F1 or accuracy, which makes one model's result directly comparable to another's.
Public document-AI benchmarks anchor most reported numbers. DocVQA, introduced by Mathew et al. in 2021, contains about 50,000 questions over 12,767 document images and is scored with ANLS, an edit-distance metric. FUNSD, from Jaume et al. in 2019, contains 199 scanned forms with 9,707 semantic entities and is scored with entity-level F1.
Benchmark | Content | Metric |
DocVQA (2021) | ~50,000 questions, 12,767 images | ANLS |
FUNSD (2019) | 199 forms, 9,707 entities | Entity-level F1 |
CORD | ~11,000 receipts, structured fields | F1 |
Why Benchmark Datasets Matter
Benchmark datasets matter because an accuracy claim means nothing without a fixed test set behind it. A vendor reporting "95 percent accuracy" on its own hand-picked pages is not comparable to a competitor's number. A shared benchmark forces both to answer the same questions on the same documents, which turns a marketing figure into a measurable one.
Benchmarks also expose the gap to human performance. On DocVQA, the original authors reported human accuracy of 94.36 percent, a ceiling that shows how much headroom automated systems still have. For CRE specifically, public benchmarks like FUNSD use generic forms, not leases or rent rolls, so a strong FUNSD score does not guarantee accuracy on a co-tenancy clause. The document types differ enough that a domain benchmark, built from real CRE documents with verified answers, is the only honest measure of CRE extraction quality.
Example
A benchmark score is clearest with a worked count. Suppose a benchmark of 50 leases contains 500 labeled rent-escalation entities as ground truth. An extraction model returns 460 entities, of which 437 are correct.
Metric | Formula | Value |
Precision | 437 / 460 | 0.950 |
Recall | 437 / 500 | 0.874 |
F1 | 2 (P R) / (P + R) | 0.910 |
Precision is 437 divided by 460, which is 0.950. Recall is 437 divided by 500, which is 0.874. F1 is 2 times (0.950 times 0.874) divided by (0.950 plus 0.874), which is 1.6606 divided by 1.824, or about 0.910. Reported on the same 50-lease benchmark, that 0.910 F1 is directly comparable to any other model's F1 on the identical set, which a self-selected demo can never be.
Variations and Edge Cases
Benchmark datasets vary in construction, and the design choices decide what a score actually proves. The variants below matter when reading a vendor's reported number.
Variant | Behavior |
Public benchmark | Open, widely cited; risks models training on the test data |
Held-out test split | Reserved portion never seen in training; guards against memorization |
Domain benchmark | Built from in-domain documents like real leases; most relevant to CRE |
Golden set | Small, expert-verified sample used for ongoing quality checks |
Contaminated benchmark | Test data leaked into training; inflates the score |
Benchmark Dataset vs Ground Truth
A benchmark dataset is often confused with ground truth, but one contains the other. Ground truth is the set of verified correct answers for a document. A benchmark dataset is the assembled, fixed collection of documents plus their ground truth, packaged with a scoring rule for comparison.
Ground truth is the label on a single field, such as the true base rent on one lease. A benchmark dataset is the whole standardized test: many documents, all their ground-truth labels, a train-test split, and a defined metric. Ground truth answers what is correct; a benchmark dataset answers how well a system finds what is correct, across a fixed and shareable set.
Frequently Asked Questions
What is a benchmark dataset in document AI?A benchmark dataset is a fixed, labeled collection of documents with known correct answers, used to measure and compare how accurately AI systems extract information. In commercial real estate it provides the ground-truth leases, rent rolls, and forms against which an extraction model's precision and recall are scored on the same yardstick.
What are common document extraction benchmarks?Common benchmarks include DocVQA, with about 50,000 questions over 12,767 document images scored by ANLS, and FUNSD, with 199 forms and 9,707 entities scored by entity-level F1. CORD covers roughly 11,000 receipts. These use generic forms and receipts, not CRE leases, so a domain benchmark is needed for real estate.
Why is a benchmark dataset better than a vendor demo?A benchmark dataset grades every model on identical documents by the same metric, so the number is comparable across systems. A self-selected demo shows hand-picked pages with no fixed test set, so its accuracy figure cannot be verified or compared to anyone else's.
Related Terms
Ground Truth
Extraction Accuracy
Precision and Recall
Named Entity Recognition
Document Extraction