Context Windows and Attention | Beyond the Prompt

Lesson 2: Context Windows and Attention

Blackboard

The limits of what the model can see

10 min

Concept 1

Context windows are finite

A context window is the amount of text the model can process at once. Even the best models max out around a few hundred thousand tokens — roughly two books. Anything beyond that gets cut off or must be handled differently.

Concept 2

Tokens are not words

LLMs process tokens, not words. A token is roughly three-quarters of an English word. When a model advertises a 200,000-token context window, that translates to roughly 150,000 words.

Concept 3

Attention degrades over distance

The attention mechanism is powerful but imperfect. It struggles to find specific facts buried in very long contexts. The 'needle in a haystack' benchmark tests this by hiding one sentence in a massive document and asking the model to find it.

Concept 4

Bigger context does not solve everything

Even with infinite context, reading your entire drive every time you ask a question would be slow and expensive. Techniques like RAG exist because targeted retrieval is faster and more reliable than brute-force reading.

Pause and Think

If you uploaded a 200-page report to a chatbot, what would you expect it to miss or get wrong?

Flashcards

Key Terms

Click or press Enter to flip a card. Review these before the practice question.

Additional Resources

Article — Context Rot (Chroma Research)
Chroma's own research showing LLM accuracy drops as you stuff more into the context window, even on tasks that should be easy. The numbers behind this lesson's warning about bigger-is-not-better.
Google Research — LLM Scientific Problem Solving
Google Research testing LLMs on scientific reasoning with long inputs. Interesting because it shows cases where more context actually helps — and cases where it still doesn't matter.
Article — Context Windows (Epoch AI)
Epoch AI charts tracking how context windows have grown over time. The gap between what vendors advertise and what models actually use well is striking.
Paper — Context Window Limits
NoLiMa paper that measured effective context length on real tasks and found it can be 99% smaller than the advertised number. That stat alone makes it worth reading.
Glossary — Context window
A short definition.
Context rot: why AI agents decay the longer they run
What happens as context fills up, in practice.

Practice Question

Why might RAG still be useful even if context windows become much larger?

Does not count for certificate

Choose one answer, use the hint if needed, then check your answer.

Use the hint before checking if you need a nudge.

Choose one answer, then click Check Answer.