Lesson 6: Crawlability, robots.txt, and the Agent-File Debate

Last updated: 2026-05-17

Before an agent can read a page, it has to be allowed to fetch it, find it, and parse it efficiently. Two tiny files at the root of a site — robots.txt and sitemap.xml — control most of that. A handful of newer files (llms.txt, AGENTS.md, markdown mirrors) have been proposed; this lesson covers what each one actually does, which assistants honour it, and where Google's May 2026 guidance comes down.

Lesson 6: Crawlability, robots.txt, and the Agent-File Debate

Blackboard

What helps every system, and what is still vendor-specific

13 min

Pause and Think

Practice Question

Question

Does not count for certificate

Choose one answer, use the hint if needed, then check your answer.

Use the hint before checking if you need a nudge.

Choose one answer, then click Check Answer.