robots.txt and Sitemap Paths Attackers Still Miss
robots.txt is not access control. It is a polite suggestion to crawlers that often doubles as a map of sensitive directories. Sitemaps are the opposite signal: paths marketing wants indexed. Together they outline what the site thinks is important—and what it hopes bots will ignore.
Read both on every host, not only the apex.
robots.txt as a treasure map
Fetch /robots.txt and note Disallow and Allow paths. Classic hits:
/admin,/backup,/api/internal/search?with parameter hints- Staging comments left by humans ("# temporary lock on /migrate")
Comments are fair game. So are sitemap directives pointing at non-obvious XML indexes.
Do not file "robots discloses /admin" as a high severities by itself. Visit the path. If it 401s with a normal login, you found recon. If it serves a directory listing, an unprotected panel, or an old installer, you found a bug.
I've watched teams block /api/v2/secret-feature in robots while the route stayed wide open without auth. The robots line was the clue; missing authorization was the finding.
Sitemaps and nested indexes
Parse sitemap.xml, sitemap index files, and any gzipped variants. Marketing sitemaps mostly list blogs. Product sitemaps sometimes list tenant-specific URLs, reset flows, or deprecated locales.
Diff sitemap URLs against what a shallow crawl of the UI produces. Anything in the sitemap but not linked from the homepage nav deserves a manual visit.
Watch for alternate hosts in sitemap <loc> entries—CDN, localization, or a forgotten shop-old. domain that is still in scope.
Multi-host and multi-env habits
www, bare apex, and api. often ship different robots files. So do country TLDs. Check mobile subdomains. Check the host behind a ticket portal separate from the marketing brand.
Some frameworks generate robots dynamically. Compare logged-out versus logged-in fetches if the app customizes crawler rules per session—rare, but when it happens the authenticated copy is chatty.
Also fetch /humans.txt and /security.txt while you are there. They are not robots, but the same low-effort pass often surfaces contact paths, old PGP keys, or a staging URL someone documented for researchers.
Compare Crawl-delay and nonstandard extensions only as flavor. The money is still the path list and whether those paths enforce auth today.
What good reports sound like
"robots.txt lists /debug/metrics; unauthenticated GET returns Prometheus data for the payment service" is a report. "Here is their robots.txt" is not.
If a Disallow path redirects to SSO and stops, say that. Inflating it into "admin panel exposed" gets you bounced. If the same path returns an API JSON error with stack traces, that is a different finding—attach the body.
Recommend treating robots as documentation for defenders too: do not Disallow paths you are unwilling to harden, and keep sitemaps free of non-public URLs. For hunters, five minutes on robots and sitemaps still beats another unfocused content-discovery run across the whole dictionary.