Tokens, Cost & Latency, Explained From Zero

Cost and speed decide whether an AI feature can survive in production, and both come down to one unit almost no one mentions at first: the token. This guide starts from nothing and builds up what a token is, why it governs your bill and your wait time, and the handful of habits — guarded by your eval — that keep an AI feature affordable and responsive.

1 · The hidden unit under everything

Underneath every AI interaction is a unit that governs what you pay and how long you wait: the token. Models don't process text as whole words or as letters — they break it into tokens, which are chunks of text roughly the size of a short word or a piece of one. A rough rule of thumb: a token is about three-quarters of an English word, so a hundred words is around 130 tokens. Every piece of text has a token count — your prompt, the documents you attach, the model's reply, all of it. This matters because tokens are the currency and the clock of AI: you're billed per token, and you wait per token. So understanding tokens is the key to understanding both your bill and your speed.

2 · You pay per token, both ways

Here's how the money works, with one twist people miss: you pay for tokens in both directions. The input tokens — everything you send, including your prompt, instructions, any attached documents, and the whole conversation history — are billed. And the output tokens — everything the model writes back — are billed too, usually at a higher rate than input. So a request's cost is roughly input tokens plus output tokens, each at their price. That twist explains a lot. It means a giant prompt stuffed with documents costs you on every request, even before the model says a word — which is why the context-engineering lesson “don't cram the window” is also a cost lesson. And it means asking for a shorter answer literally saves money. Once you see that every token in and out has a price tag, the cost levers become obvious: send fewer tokens, receive fewer tokens.

3 · You also wait per token

Tokens don't just cost money — they cost time, and that's latency: the delay between asking and getting an answer. Models generate output one token at a time, in sequence, so the single biggest driver of how long a user waits is how many tokens the model has to produce. A one-line answer comes back quickly; a five-paragraph essay makes the user wait noticeably longer, because the model is literally writing it out token by token. This reframes response time as something you control: if a feature feels sluggish, a huge cause is often simply that you're asking for — or allowing — a longer answer than needed. Input length matters somewhat too, and the model's size matters (bigger models generate each token slower, tying back to model selection), but output length is usually the dominant, most controllable factor. So “make it faster” very often just means “make the answer shorter” — the same lever that also makes it cheaper.

4 · Trim the input

The first practical lever is trimming the input, and this is where the whole course connects. Every token you send costs money and adds a little time, so a bloated prompt is a standing tax on every request. This is exactly the context-engineering discipline — but now with a dollar value. Don't attach ten retrieved documents when three answer the question; those extra seven are billed forever. Don't send the entire conversation history when a short summary of the older part would do — we saw summarising helps the model focus, and now we see it also cuts cost. Don't pad instructions with redundant boilerplate. The habit is to ask, of everything in your prompt: does this token earn its place by helping the answer? If not, cut it — you'll save on every call and shave a little latency. Lean prompts aren't just cleaner for the model; they're directly cheaper and faster. Trimming the input is the cost lever you pull constantly.

5 · Trim the output

The second lever is trimming the output, and since output tokens are typically the priciest and slowest, this one hits hardest. Models, left alone, often over-explain — padding answers with preamble, caveats, and restatements the user never asked for. Every extra word costs you at the higher output rate and makes the user wait longer as it's generated one token at a time. So instruct the model toward the length you actually need: “answer in one sentence,” “give just the key points,” “be concise.” Structured output helps too — asking for a short, defined shape rather than free-flowing prose naturally caps the length. This isn't about crippling the model; it's about not paying for verbosity nobody wanted. A crisp two-line answer is frequently better for the user as well as cheaper and faster — a rare case where the economical and the quality choice point the same way. Ask for exactly as much answer as the job needs, and no more.

6 · Small savings times huge volume

Why sweat a few tokens? Because of scale, and this is the mindset shift that makes it click. On one request, none of this matters — a few extra tokens is a fraction of a cent and a few milliseconds. But a real application doesn't make one request. It makes thousands, then millions, over weeks and months. At that volume, tiny per-request waste multiplies into an enormous, ongoing bill and a persistently sluggish experience. Trim a couple hundred needless tokens from a prompt that runs a million times, and you've saved a fortune and sped up every user. This is why token-thinking is a core operational skill, not a micro-optimisation: the savings live in the multiplication. It also means the biggest wins come from your highest-volume paths — find the requests you make most often and make those lean first. Small savings times huge volume equals real money and real speed — that's the whole economic logic of running AI in production.

7 · Tune with your eval as the guardrail

One last, crucial safeguard ties this whole series together. When you trim tokens to save cost and time, you're changing the prompt — shorter context, tighter output — and any change can affect quality. Cut too much and the model loses information it needed, and your answers quietly degrade. So don't optimise cost blindly: use your evaluation as the guardrail. Make your trimming change, then re-run the eval and check the quality score held. If it did, you've won on both fronts — cheaper, faster, and just as good. If it dropped, you cut something that mattered, so dial it back. This is the disciplined loop the whole series has been building toward: measure quality, make a change — grounding, choosing an approach, picking a model, or trimming tokens — and re-measure to confirm it helped without breaking anything. Cost, speed, and quality stop being a guessing game and become something you steer deliberately, with the eval keeping you honest.

8 · A simple test you can run this week

9 · Glossary — every term, spelled out

Token

A chunk of text the model processes — roughly three-quarters of an English word.

Input tokens

Everything you send: prompt, instructions, attached documents, conversation history — all billed.

Output tokens

Everything the model writes back — billed too, usually at a higher rate, and generated one at a time.

Latency

The delay before an answer, driven mostly by how many output tokens the model must produce.

Trimming

Cutting needless input and capping output length to reduce cost and latency.

Eval guardrail

Re-running your evaluation after a cost cut to confirm quality didn't quietly drop.