Testing and Evaluating Prompts

Lesson 6: Testing and Evaluating Prompts

Blackboard

Measure before you ship

10 min

Concept 1

Start with manual error analysis

Read 20 outputs yourself. You will spot patterns — the model is too verbose, it misses key points, the tone is wrong. This hands-on review builds intuition that automated metrics alone cannot give you.

Concept 2

LLM judges scale evaluation

An LLM judge is another model tasked with grading the output. It can do pairwise comparison (which summary is better?), single-answer grading (rate this from 1 to 5), or rubric-guided grading with detailed criteria for each score level.

Concept 3

Rubrics make judging consistent

A rubric defines what a 5/5 looks like versus a 1/5. For example: 5 = under 100 words, mentions three distinct key points, opens with an overview. You can even give the judge few-shot examples of each score to improve consistency.

Concept 4

Test across models

Platforms like Promptfoo let you run the same prompt against five different LLMs simultaneously and compare results in a table. This helps you choose the right model for your task and spots when one model consistently outperforms others.

Pause and Think

If you had to choose between two versions of a summarisation prompt, how would you decide which is better?

Flashcards

Key Terms

Click or press Enter to flip a card. Review these before the practice question.

Additional Resources

Guide — LLM Evaluation (Langfuse)
Langfuse guide on building LLM evaluation pipelines — offline metrics, regression testing, LLM-as-judge. If you want to actually build what this lesson describes, start here.
Paper — LLM Evaluation Methods
Survey covering human, automatic, and hybrid ways to evaluate LLMs. Good for seeing the trade-offs between scoring strategies when you're picking one for your own project.
Glossary — Benchmark
What benchmarks measure, and what they miss.
AI tools directory — AI benchmark tools
Leaderboards and evaluation tools.

Practice Question

You want an LLM judge to rate summaries consistently. What should you provide alongside the scoring scale?

Does not count for certificate

Choose one answer, use the hint if needed, then check your answer.

Use the hint before checking if you need a nudge.

Choose one answer, then click Check Answer.