A multimodal model is an AI system that takes in and reasons over more than one type of data at once, such as text, images, tables, and layout, rather than a single format. It encodes each input type and fuses them into a shared representation, so it can read a scanned page as both pixels and words and answer questions about it.
What Is a Multimodal Model?
A multimodal model is a machine learning model trained to process and relate multiple data modalities, most often text plus images, in one system. Per Stanford HAI and IBM, a multimodal model typically encodes each modality separately and then fuses the encodings, letting the model connect what it sees to what it reads. This is the difference between a text-only model that needs a page transcribed first and a model that looks at the page directly.
For commercial real estate documents, the relevant modalities are the visual layout of a page and the text on it. A rent roll, a survey, or an offering memorandum carries meaning in its structure, not just its words: a number's position in a column, a label above a table, a stamp on a page. A multimodal model can use that visual context directly.
Modality | Example in a CRE document |
|---|---|
Text | Lease clauses, dates, tenant names, dollar amounts |
Image and layout | Table grids, column alignment, signatures, stamps, floor plans |
Structured tables | Rent rolls, operating statements, T-12 grids |
Handwriting and marks | Initials, margin notes, checkboxes on a form |
How Does a Multimodal Model Differ From OCR Plus a Text Model?
A multimodal model differs from an OCR-then-text pipeline by reading the page as an image and text together rather than in two disconnected steps. A traditional pipeline runs optical character recognition to flatten a page into a string, then feeds that string to a text-only model, discarding the layout. A multimodal model keeps the layout, so it can tell that a value sits inside a specific cell or under a specific header.
The practical effect shows up on messy documents. When a scan is skewed, a table spans columns, or a figure sits beside a caption, the OCR string loses the spatial relationships that told a human what the number meant. A multimodal model that sees position can hold those relationships together.
Approach | How it reads a page | Weakness |
|---|---|---|
OCR plus text model | Convert page to a flat string, then process text only | Loses layout, columns, and spatial context |
Multimodal model | Process pixels, layout, and text together | Higher compute cost per page |
What Do Multimodal Models Mean for CRE?
Multimodal models matter for CRE because the documents that drive deals are visual, not clean text, and extraction quality depends on reading layout. Rent rolls, T-12s, offering memoranda, surveys, and leases arrive as scans, photos, and PDFs where the structure carries the meaning. A model that reads layout directly can pull a tenant's expiration date from the right row, tie a footnote to the line it modifies, and separate two adjacent tables that a flat transcript would merge.
For a document processing system, this changes what is extractable and how reliable it is. A field that lives in a specific cell of a rent roll, or a total at the bottom of an operating statement, is easier to locate correctly when the model sees the grid. As a labeled representative point, extraction accuracy on structured tables generally improves when a system uses layout-aware reading rather than flat text, though the exact gain depends on document quality and the fields involved.
The quotable point for an operator: CRE documents hide meaning in their layout, so a model that reads the page as an image can extract fields a text-only pipeline would misplace.
Example
An underwriter receives a scanned rent roll where two properties are stacked in one PDF, each with its own table, and a footnote flags one tenant as month-to-month. A text-only pipeline transcribes the page into a single string, and the two tables and the footnote blur together, so the month-to-month flag can attach to the wrong tenant.
Field | Text-only pipeline risk | Multimodal model behavior |
|---|---|---|
Tenant name and row | May shift if columns misalign | Reads the cell in its row and column |
Lease expiration | Can bind to the wrong tenant | Stays tied to the correct row |
Month-to-month footnote | May attach to the nearest text | Links the mark to the flagged row |
Property separation | Two tables can merge | Keeps each table distinct by layout |
The multimodal model reads the same page as an image, keeps each table separate, and ties the footnote to the row it points at. The underwriter gets a rent roll where the flag lands on the right tenant, which is the difference between an accurate lease-expiration schedule and a corrupted one.
Related Terms
Data Extraction
Model Evaluation