Improving RAG | Beyond the Prompt

Lesson 10: Improving RAG

Blackboard

From basic retrieval to precision

10 min

Concept 1

Chunking

Instead of embedding whole documents, you split them into chapters, sections, or paragraphs. The vector database stores both document-level and chunk-level embeddings. This allows retrieval to be more precise — finding the right section, not just the right document.

Concept 2

HyDE: Hypothetical Document Embeddings

A clever technique: use the LLM to generate a fake answer to the user's question, embed that fake answer, and use it to search the vector database. The fake answer's embedding is often closer to the real document than the original short query would be.

Concept 3

The breadth of RAG research

Between 2020 and 2025, dozens of RAG variations were published: re-ranking retrieved documents, speculative RAG, hybrid search combining keyword and semantic retrieval, and more. The field is moving fast.

Concept 4

Is RAG future-proof?

If context windows grow to infinity, does RAG become irrelevant? Probably not. The search engine analogy holds: even if you could read the entire web, you would still want a search engine to narrow down where to look. Targeted retrieval is faster, cheaper, and provides sources.

Pause and Think

If you embed an entire 200-page document as one vector, what information do you lose?

Flashcards

Key Terms

Click or press Enter to flip a card. Review these before the practice question.

Additional Resources

Paper — HiChunk: Hierarchical Chunking for RAG
HiChunk paper proposing a benchmark for hierarchical chunking in RAG. Useful if you want a framework for actually testing which chunking strategy works best on your own documents.
Google Research — Speculative RAG
Google Research on Speculative RAG — a smaller model drafts answers from retrieved docs, then a bigger model checks them. One of the more interesting optimisation tricks touched on in this lesson.
Article — Chunking Strategies for RAG
Practitioner walkthrough of chunking strategies: fixed-size, recursive, semantic, hierarchical. Lists the trade-offs for each, which helps when you're deciding what to try first.
Paper — Agentic AI Systems Survey
Broad survey of agentic AI — architectures, tool use, memory, orchestration. The sections on retrieval-augmented agents tie directly into the advanced RAG ideas in this lesson.
Glossary — Reranking
A short definition.
Glossary — Chunking
A short definition.

Practice Question

A user's short query retrieves the wrong documents because the query embedding is too far from the relevant document embedding. Which technique specifically addresses this problem?

Does not count for certificate

Choose one answer, use the hint if needed, then check your answer.

Use the hint before checking if you need a nudge.

Choose one answer, then click Check Answer.