Encyclopedia / SEO & GEO

robots.txt

A file at the site root that tells crawlers where not to go.

It's a request, not a lock: polite bots obey it and anyone else can ignore it. It's also where you allow or block AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot) one by one, and where you point to your sitemap.

User-agent: *
Disallow: /drafts

User-agent: GPTBot
Allow: /

Sitemap: https://ada.example/sitemap.xml

Related terms

Sitemap Robots meta / noindex GEO (generative engine optimization)

Where it shows up in a page load

  1. Step 11: Meanwhile: the crawlers. Googlebot and AI crawlers make the same trip. They read robots.txt and the sitemap, fetch each page and read the head (title, description, canonical, structured data) and the content. That is everything search results and AI answers have to work with, so SEO and GEO start with the website itself.

Watch the full request journey →

More in SEO & GEO

How search engines and AI systems find, read and describe the site.

Title tag Meta description Canonical URL Robots meta / noindex Sitemap Open Graph & social cards Structured data (JSON-LD) Core Web Vitals Internal links Backlinks Search Console GEO (generative engine optimization) llms.txt

Open in the interactive encyclopedia →