Tokenization is the process of splitting text into tokens, the small sub-word units a language model reads, counts, and bills. In commercial real estate document work, tokenization determines how much of a lease or rent roll fits in a model's context window and how much each extraction run costs, since providers price by the token.
How Does Tokenization Work?
Tokenization works by breaking text into tokens using a fixed vocabulary learned from data, most often with byte pair encoding. Common words map to one token, while rare or compound words split into several. The model never sees letters or whole words directly; it sees a sequence of token IDs drawn from its vocabulary.
On standard English prose, 1,000 words is roughly 1,300 to 1,400 tokens, a ratio near 1.33 tokens per word, per llmtest.io, and a rule of thumb holds that one token is about four characters. Vocabulary size differs by model.
Tokenizer | Model family | Approx. vocabulary |
cl100k_base | GPT-4, GPT-3.5-Turbo | ~100,000 tokens |
o200k_base | GPT-4o, o-series, GPT-5 family | ~200,000 tokens |
Per OpenAI's tiktoken documentation, a larger vocabulary encodes more meaning per token, so the same sentence uses fewer tokens under o200k_base than under an older encoding.
Why Tokenization Matters
Tokenization matters because tokens are the unit of both cost and capacity in document AI. A model's context window is measured in tokens, so tokenization decides whether a full lease fits in one pass or must be split. Providers bill per token, so the token count of a rent roll is the token count of the invoice.
Token ratios also rise for structured content. Tables, numbers, and formatting in a rent roll tokenize less efficiently than clean prose, so a document heavy in tabular data can carry a higher token-per-word ratio than the 1.33 English baseline. Estimating tokens before a run, rather than after, is how an operator forecasts extraction cost across a portfolio instead of being surprised by the bill.
Example
Tokenization cost is clearest with a worked count. A 30-page rent roll at roughly 400 words per page holds about 12,000 words. At the 1.33 tokens-per-word English ratio, that is about 15,960 tokens of input.
Input | Value |
Pages | 30 |
Words per page | 400 |
Total words | 12,000 |
Tokens per word | 1.33 |
Input tokens | 15,960 |
Assumed price | $2.50 per million input tokens |
Input cost | $0.0399 |
At an assumed $2.50 per million input tokens, 15,960 tokens cost 15,960 divided by 1,000,000, times $2.50, which is about $0.04 for one pass. Across a 200-document portfolio, that input alone is about $8. The price used here is an assumption for the calculation, not a quoted rate, and output tokens are billed separately and usually cost more.
Variations and Edge Cases
Tokenization behaves differently across content and languages, and CRE documents mix all of it. The cases below change the token count of the same underlying information.
Case | Effect on token count |
Tables and numbers | Higher tokens per word than prose |
Long compound terms | Split into multiple sub-word tokens |
Scanned OCR text | Stray characters inflate the count |
Non-English text | Often more tokens per word than English |
Whitespace and formatting | Consumes tokens even when not content |
Tokenization vs Chunking
Tokenization is often confused with chunking, but they work at different scales. Tokenization splits text into sub-word units a model reads one at a time. Chunking splits a document into passages of many words for retrieval.
Tokenization is the fine-grained unit: it turns "escalation" into the pieces a model processes and sets the token count that drives cost and context limits. Chunking sits above it, grouping hundreds of tokens into a retrievable passage whose size is itself defined in tokens. Tokenization answers how a model reads text; chunking answers how a document is divided for search.
Frequently Asked Questions
What is tokenization in AI?Tokenization is the process of splitting text into tokens, the small sub-word units a language model reads, counts, and bills. In commercial real estate it determines how much of a lease or rent roll fits in a model's context window and how much each extraction run costs, since providers price by the token.
How many tokens is a word?On standard English prose, one word is about 1.33 tokens, so 1,000 words is roughly 1,300 to 1,400 tokens, per llmtest.io. The ratio rises for tables, numbers, and code, so structured documents like rent rolls tokenize less efficiently than clean prose.
Why does tokenization affect the cost of extraction?Tokenization affects cost because providers bill per token, so the token count of a document is the billable unit of the run. It also sets capacity, since a model's context window is measured in tokens and decides whether a full lease fits in one pass or must be split.
Related Terms
Large Language Model
Chunking
Retrieval-Augmented Generation
Vector Embedding
Document Extraction