What robots.txt controls
Robots.txt controls crawler requests for URL paths on one site host. A cooperative crawler checks the file before crawling and follows the group that matches its crawler token.
Treat robots.txt like a sign beside an open corridor. The sign can ask polite visitors to skip one room. A locked door still protects the room.
The distinction matters during SEO audits. Robots.txt can reduce unwanted crawler visits. It cannot secure private files, block malicious requests, or promise removal from search results.
Which parts form a robots.txt file?
A robots.txt file contains groups and rules. A group starts with one or more User-agent lines. The crawler then reads Allow and Disallow lines beneath that group.
The User-agent value names a crawler product token. Googlebot is one token. GPTBot and OAI-SearchBot are separate OpenAI tokens.
Example:
In this example, Googlebot cannot crawl paths under /nogooglebot/. The star group applies to crawlers without a stronger matching group. The Sitemap line names the XML sitemap location.
How robots.txt rule matching works
A crawler first chooses the strongest user-agent group. The product token match is case-insensitive, so Googlebot and googlebot match the same crawler token.
Path rules then compete by specificity. The rule with the longest matching path wins. If matching Allow and Disallow rules have equal strength, Allow wins.
Example: /shop/private-product.html matches Disallow: /shop/ and Allow: /shop/private-product.html. The longer Allow rule wins because it matches more of the URL path.
Where robots.txt must sit
Robots.txt must sit at the root of the site host that it controls. For https://www.example.com/, the file belongs at https://www.example.com/robots.txt.
The file does not control every version of a domain. A rule on https://example.com/ does not control https://www.example.com/. A rule on HTTPS does not control HTTP.
Subdomains need their own files when crawlers reach them separately. A staging host needs login protection when pages should remain private.
What fetch errors change for crawlers
Fetch results can change crawler behavior. A missing robots.txt file can leave paths open for crawling. A server failure can stop crawling because the crawler cannot read the rules.
Googlebot treats most 4xx robots.txt responses as no crawl restrictions. For 5xx errors, Googlebot stops crawling for the first 12 hours and keeps retrying.
If Googlebot still cannot fetch a new file, it can rely on the last good file for up to 30 days. Without a cached file, Google can treat the site as having no crawl restrictions after that process.
How robots.txt differs from noindex and sitemaps
Robots.txt controls crawling. Noindex controls whether a fetched page can appear in Google Search. An XML sitemap lists URLs for crawler discovery.
| Control | Job | When to choose |
|---|---|---|
| robots.txt | Limits crawler requests | Crawl paths need access rules |
| noindex | Removes fetched pages from Google Search | Search appearance needs removal |
| XML sitemap | Lists URLs for discovery | Crawlers need a URL list |
Noindex fails when robots.txt blocks the same page from Googlebot. The crawler must fetch the page before it can read the noindex rule.
A Sitemap line can appear inside robots.txt. The Sitemap directive stands apart from user-agent groups, so placement inside the file does not change crawler matching.
How AI crawler tokens should be listed
AI crawler rules need exact names from official crawler documentation. OpenAI separates OAI-SearchBot for search from GPTBot for generative model training.
Those two OpenAI settings work independently. A site can allow OAI-SearchBot for ChatGPT Search while disallowing GPTBot for training crawl collection.
Do not group AI crawlers under one label unless the access choice is the same. Different crawler names can serve different product jobs.
How to audit robots.txt before publishing
A safe robots.txt check starts with public file access. Open /robots.txt in a browser first. Then test chosen URL paths against each crawler token.
Record five fields for each test: crawler token, tested URL, matched rule, expected access, and review date. This record prevents memory-based edits.
Check one failure case before publishing. User-agent: * with Disallow: / blocks every path for crawlers that follow the star group.
Which limits should stay in the audit record?
A robots.txt note should name its limits beside the rules. The file guides cooperative crawlers, but it does not force every crawler to obey.
The audit record should separate crawl access from search indexing. It should also separate public crawler rules from private access controls.
Robots.txt cannot promise rankings, mentions, AI citations, or search traffic. Evidence for those outcomes must come from crawl tests, server logs, Search Console records, or platform reports.
Manish Singh is the Team Lead at IMMWIT, where he brings over 14 years of experience in SEO, UX, and digital marketing. Known for helping businesses rank, scale, and grow smarter online, he blends strategic thinking with AI and NLP-backed insights. His hands-on approach to semantic SEO and UX design turns ideas into real results clients can see and trust.