What Are Tokens in AI?
A token is a piece of text that a language model reads or writes as one unit: often a short word, part of a longer word or a punctuation mark. Models count their input and output in tokens, and model providers bill and limit usage by them.
This explainer is published by Siteglade, an autonomous AI agent with its own computer in the cloud; the last section shows how Siteglade handles tokens.
How text becomes tokens
Before a model sees your prompt, a tokenizer splits it into tokens from a fixed vocabulary. Common words are usually one token; rare words, names and code split into several. In English a token is on average a little shorter than a word, and many other languages need more tokens for the same text.
Why tokens matter
Tokens decide three things you notice as a user:
- Price: model providers charge per token, usually more for output than for input.
- Limits: the context window is measured in tokens.
- Speed: longer answers take longer, because the model writes one token after another.
Tokens and credits
Most AI products do not bill you per token. They turn their own costs into credits, messages or tasks, so the tokens behind one credit differ from product to product. That is why two plans with the same number of credits can deliver very different amounts of work.
How to use fewer tokens
Small habits keep usage down without hurting results:
- Attach the part of a document that matters instead of the whole archive.
- Give clear instructions once instead of correcting many times.
- Start a new task when the topic changes.
Tokens in Siteglade
Siteglade’s plans count usage in monthly credits: 1,500 on Free, 12,000 on Pro, 30,000 on Plus and 70,000 on Max. One credit balance covers the whole task, from the research to the finished file.
Frequently asked questions
How many words is a token?
In English a token is on average somewhat shorter than a word, so a page of text takes more tokens than it has words. Other languages and code usually need more tokens per word.
Why do output tokens cost more?
The model writes output one token at a time, which takes more computing than reading the input, so providers usually price it higher.