A confidence score AI extraction returns is a prediction of how often a field is correct, not a grade on how well the model tried. A well-calibrated model that outputs 0.90 is telling you that, across many fields scored 0.90, about 90% are right and 10% are wrong. That is the whole meaning of the number. Operators who treat a confidence score as a quality badge, rather than as a probability with a knowable error rate, either trust fields they should check or check fields they never needed to. The value of the score is that it lets you route work: high scores flow through, low scores go to a person, and the threshold that divides them is a business decision, not a technical one.
Key Takeaways
A confidence score is a calibrated probability of correctness. A field scored 0.90 in a well-calibrated system is wrong roughly one time in ten, per the standard definition of calibration in machine learning.
The score tells you likelihood of error, not cost of error. A misread parking-ratio field and a misread base-rent field can share a confidence score and carry wildly different consequences.
The threshold that separates auto-accept from human review is a business decision. Parseur recommends routing fields below 90% confidence to human review for critical data.
Human-in-the-loop review raises document extraction accuracy from roughly 80% to 95% and higher, per practitioner benchmarks, by spending review time only where the score is low.
The dangerous field is the confident wrong one. Calibration and source citations, not the raw score alone, are what make a confidence score safe to act on.
What Is a Confidence Score in AI Extraction?
A confidence score in AI extraction is a number, usually between 0 and 1, that estimates the probability a given extracted value is correct. In a well-calibrated model, the score matches the empirical hit rate: predictions made at 0.80 confidence are correct about 80% of the time, and predictions at 0.90 are correct about 90% of the time, per the standard machine-learning definition of calibration.
That definition carries a consequence most operators skip past. The score is a claim about a population of similar predictions, not a verdict on the single field in front of you. When a rent-roll extraction returns a lease expiration at 0.72, the system is saying that dates scored 0.72 are wrong about 28% of the time. You still do not know whether this particular date is one of the wrong ones. The score sizes the risk. It does not resolve it.
What Does a 0.90 Confidence Score Actually Mean?
A 0.90 confidence score in a well-calibrated system means that, across all fields the model scores at 0.90, about 90% will be correct and about 10% will be wrong. It is an expected error rate for that bucket, not a promise about any single field. The critical word is calibrated. An uncalibrated model can output 0.90 and be right only 70% of the time.
Calibration is the property that makes the number usable. As one machine-learning reference on calibration puts it, "a model is well-calibrated if its prediction's confidence scores accurately reflect the probability of those predictions being correct." When that holds, the score becomes an instrument you can set thresholds against. When it does not, the score is decoration. Before an operator trusts any extraction platform's confidence scores, the question to ask is not "how high are they" but "are they calibrated," which you verify by checking whether fields scored 0.90 are in fact right about nine times in ten on your own documents.
Confidence bucket | Meaning in a calibrated model | Expected errors per 100 fields |
0.99 | Right about 99 times in 100 | ~1 |
0.95 | Right about 95 times in 100 | ~5 |
0.90 | Right about 90 times in 100 | ~10 |
0.80 | Right about 80 times in 100 | ~20 |
0.70 | Right about 70 times in 100 | ~30 |
How Do You Set a Confidence Threshold for Human Review?
You set a confidence threshold by deciding, per field, how much error you can tolerate before a human must check the value. Fields above the threshold auto-accept; fields below route to review. Parseur recommends human review when confidence drops below 90% for critical data. Many production systems run two thresholds: a high auto-accept line near 0.95 and a lower review line near 0.65.
The threshold is not one number for the whole document. It is a per-field policy that trades two costs against each other: the cost of shipping a wrong value and the cost of a person reviewing a right one. A high threshold catches more errors and sends more work to review. A low threshold automates more and lets more errors through. The right setting depends on what the field feeds. A tenant contact field feeding a mailing list can sit at a permissive threshold. A base-rent field feeding a valuation and a debt-service-coverage calculation should sit at a strict one, because the downstream cost of a wrong number is measured in dollars, not in a re-sent letter.
This is why a single global accuracy figure misleads. A system can post 95% overall accuracy and still be unsafe if the 5% of errors concentrate in the fields that drive the model. The threshold policy exists to put review attention exactly where error is expensive.
Why Is a Confident Wrong Answer the Real Danger?
The real danger is a field the model scores high and gets wrong, because a high score is the signal that tells the workflow to skip review. A low-confidence error gets caught by design; the score routes it to a person. A high-confidence error sails through untouched. This is where miscalibration and hallucination do their damage.
Two forces make confident errors more than a rounding problem. First, miscalibration: an uncalibrated model reports 0.95 on fields it gets right only 80% of the time, so the auto-accept lane fills with errors the threshold was supposed to stop. Second, in language-model extraction, hallucination produces fluent, plausible values that carry high confidence and no basis in the document. One clinical extraction study found hallucinations in 33% of cases overall, and notably higher in incorrect answers, which is exactly the pattern that defeats naive trust in the number.
The defense is not a higher threshold. It is grounding. Every extracted field should carry a citation back to the source page and coordinates, so a disputed value is settled by opening the document rather than by trusting the score. As a rule an operator can quote: a confidence score without a source citation is an opinion, and a confidence score with one is a checkable claim. The citation is what turns the model from an oracle into a witness you can cross-examine.
How Much Review Should a Confidence-Scored Workflow Actually Require?
A confidence-scored workflow should send only the low-confidence minority to human review, not the whole document. The point of the score is to concentrate scarce review time. Practitioner benchmarks report that human-in-the-loop review lifts document extraction accuracy from about 80% to 95% and higher, precisely because reviewers spend their time on flagged fields rather than re-reading everything.
The metric that governs this is straight-through processing rate, the share of documents or fields that clear automatically with no human touch. Industry STP benchmarks run from about 26% globally for hard international transactions to roughly 67% for best-in-class implementations, per KlearStack and Hyperscience practitioner data. Whatever your STP rate, the remaining share is exception handling: the fields the confidence threshold flagged for a person.
Workflow | What it optimizes | Where it fails |
Auto-accept everything | Speed and cost | Ships every high-confidence error straight into the model |
Review everything | Perceived safety | Burns reviewer time on fields the model reads perfectly, leaving no time for the hard ones |
Confidence-routed review | Reviewer attention on the fields most likely to be wrong | Requires calibrated scores and source citations to be trustworthy |
The middle row is the only defensible configuration at volume. It works only when two conditions hold: the scores are calibrated, so the threshold means what it says, and every field is grounded to a source, so a flagged field resolves fast. Miss either condition and the workflow degrades into false confidence.
Frequently Asked Questions
What is a good confidence score for AI extraction? There is no single good score, because the number is a probability, not a grade. A 0.95 confidence field is right about 95% of the time in a calibrated model, which is strong for a low-stakes field and still too loose for a value feeding a valuation. Set the acceptable score per field based on the cost of an error, not on a universal cutoff.
When should an AI extraction be reviewed by a human? An extraction should be reviewed when its confidence score falls below the threshold you set for that field's risk, or when the field is high-stakes regardless of score. Parseur recommends human review below 90% confidence for critical data. The safest configuration routes low-confidence and high-consequence fields to a person and auto-accepts the rest.
Can you trust a high confidence score? Only if the model is calibrated and the field is grounded to a source. A high score from an uncalibrated model, or a fluent hallucination from a language model, can be confidently wrong. A source citation that links the value back to its page turns the score from an opinion into a claim you can verify.
Conclusion
A confidence score is a prediction about a population of similar fields, calibrated to a real error rate, and nothing more. Read it as a grade and you will trust confident errors and re-check obvious wins. Read it as a probability and it becomes the instrument that decides where a human belongs.
The operator takeaway is to stop asking whether an extraction platform's scores are high and start asking whether they are calibrated and grounded. A calibrated score tells you how often a bucket of fields is wrong. A source citation tells you which one. Together they let you set thresholds that put review attention where error is expensive and let everything else flow through. Firms that read the score this way spend their scarce judgment on the fields that carry risk. Firms that treat the number as a badge inherit the errors it was quietly reporting all along.
Related
AI Gets 80% of a Lease Right. The Other 20% Is Where the Money Is