GEO Improvement: Add robots.txt to docs site
Source audit: GEO Audit Report — 2026-07-23
Audit date: 2026-07-23
Run: §30026886630
Finding
Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot) — all 20 audited pages on github.github.com/gh-aw score 0/18 on the robots.txt check because no robots.txt file exists at that origin.
Why this matters
Without a robots.txt, AI crawlers (GPTBot, ClaudeBot, PerplexityBot) cannot confirm they have explicit permission to index the documentation site. This is the single largest score gap across the entire site: fixing it would add +18 points per page site-wide, raising the sitemap average from 38/100 to approximately 56/100. The README already allows 27 AI bots explicitly and scores 15/18 on this check — the docs site is the missing half.
Suggested fix
- Add a
robots.txt file to the static assets of the Astro docs site (served at https://github.github.com/gh-aw/robots.txt) with contents similar to the repo's own robots.txt:
User-agent: *
Allow: /
# AI crawlers — explicit allow
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Bingbot
Allow: /
User-agent: cohere-ai
Allow: /
User-agent: YouBot
Allow: /
Sitemap: https://github.github.com/gh-aw/sitemap.xml
- Place the file at
docs/public/robots.txt (or wherever the Astro project's static public directory is located).
- Verify it is served correctly:
curl https://github.github.com/gh-aw/robots.txt
- Re-run
gh aw run geo-optimizer-skill to confirm the score improves.
Generated by 🌍 GEO Optimizer Daily Audit · sonnet46 · 35.4 AIC · ⌖ 12.5 AIC · ⊞ 5K · ◷
GEO Improvement: Add robots.txt to docs site
Source audit: GEO Audit Report — 2026-07-23
Audit date: 2026-07-23
Run: §30026886630
Finding
Why this matters
Without a
robots.txt, AI crawlers (GPTBot, ClaudeBot, PerplexityBot) cannot confirm they have explicit permission to index the documentation site. This is the single largest score gap across the entire site: fixing it would add +18 points per page site-wide, raising the sitemap average from 38/100 to approximately 56/100. The README already allows 27 AI bots explicitly and scores 15/18 on this check — the docs site is the missing half.Suggested fix
robots.txtfile to the static assets of the Astro docs site (served athttps://github.github.com/gh-aw/robots.txt) with contents similar to the repo's own robots.txt:docs/public/robots.txt(or wherever the Astro project's static public directory is located).curl https://github.github.com/gh-aw/robots.txtgh aw run geo-optimizer-skillto confirm the score improves.