Model evaluation is the process of measuring how well an AI system performs by comparing its outputs against a set of known correct answers and scoring the result with defined metrics. In document processing, it answers a direct question: on a labeled sample of real documents, how often does the system extract the right value, and how often does it miss or invent one.
What Is Model Evaluation?
Model evaluation is the measurement of a model's performance against ground truth, a set of examples with known correct answers held separate from the data the model was built on. Per the Wikipedia entry on statistical model validation, model evaluation, also called model criticism, tests whether a model holds up on data it did not train on. The output is a set of metrics that quantify quality rather than an impression that the system "seems to work."
For an extraction system, ground truth is a batch of documents where a human has recorded the correct value for each field. The system runs on those documents, and its answers are compared field by field to the human labels. The comparison produces numbers an operator can act on.
Term | Meaning in evaluation |
|---|---|
Ground truth | Human-verified correct answers held aside for testing |
Metric | A defined score such as precision, recall, or F1 |
Test set | Documents used only to measure, never to build the model |
Error | A field the system extracted wrong, missed, or invented |
Which Metrics Measure Extraction Quality?
The core metrics for extraction quality are precision, recall, and their combination, the F1 score. Per standard machine learning references, precision is the share of the system's answers that are correct, recall is the share of the correct answers the system found, and F1 is the harmonic mean that balances the two. The pair matters because a system can look good on one and fail on the other.
The trade-off is the point. A system tuned to answer only when certain has high precision but may skip hard fields, lowering recall. A system that answers everything has high recall but returns more wrong values, lowering precision. Which to favor depends on the field.
Metric | Question it answers | Formula |
|---|---|---|
Precision | Of the values it extracted, how many were correct? | True positives / (true positives + false positives) |
Recall | Of the correct values, how many did it find? | True positives / (true positives + false negatives) |
F1 score | What is the balance of the two? | Harmonic mean of precision and recall |
What Does Model Evaluation Mean for CRE?
Model evaluation matters for CRE because an extraction system that is right most of the time is not the same as one that is right on the fields that move a deal, and only measurement tells them apart. A system might extract tenant names perfectly and still miss lease expiration dates, or read clean rent rolls well and fail on scanned ones. Evaluation on a labeled sample of the operator's own document types surfaces exactly where the system is reliable and where a human still needs to check.
For a document processing pipeline, this reframes the buying and trust decision. Instead of accepting a headline accuracy claim, an operator can ask for precision and recall by field and by document type, measured on documents like their own. As a labeled representative point, a single blended accuracy number can hide large gaps between easy fields and hard ones, so field-level metrics on a representative test set give a truer read than one aggregate figure. Where recall is lower, the workflow routes those fields to human review; where precision is high, they pass with less checking.
The quotable point for an operator: a model is only as trustworthy as its evaluation, so ask for precision and recall by field on documents like yours, not a single accuracy headline.
Example
An operator tests an extraction system on 100 rent rolls where every field has a human-verified answer. For the lease-expiration field, the system returns a value on 90 documents, of which 81 are correct, and it leaves 10 blank. The counts feed precision and recall directly.
Quantity | Value | Meaning |
|---|---|---|
Values returned | 90 | Documents where the system gave an answer |
Correct values | 81 | Returned answers that matched ground truth |
Precision | 0.90 | 81 correct out of 90 returned |
Recall | 0.81 | 81 correct out of 100 documents |
Precision of 0.90 says that when the system reports an expiration date, it is right nine times in ten. Recall of 0.81 says it captured 81 of the 100 true dates. The gap tells the operator to send the missing and low-confidence expirations to human review while trusting the returned values more freely. Those two numbers, computed from a labeled sample, are worth more than any single accuracy claim.
Related Terms
Precision and Recall
Data Extraction