Direct answer: A robots.txt file is a UTF-8 text file placed at the root of your domain (https://example.com/robots.txt) that tells compliant crawlers which URLs they may request. It supports the User-agent, Allow, Disallow, and Sitemap directives. Disallow does not prevent indexing — if a URL is blocked but linked from another site, Google can still index the address.
How Robots.txt Works
Every request from Googlebot, Bingbot, or any compliant crawler begins with one question: “May I crawl this site and which URLs are off limits?” The answer lives in a single text file at the root of your domain.
A compliant crawler fetches https://yourdomain.com/robots.txt, parses the directives, and applies the rules that match its User-agent token. If the file is missing, malformed, or returns a 5xx error, the crawler typically behaves as if everything is allowed and proceeds.
Three rules govern the file:
- One file per host. Subdomains and protocols (HTTP vs HTTPS) need separate files.
- UTF-8 encoded text. A single UTF-8 BOM or wrong MIME type blocks parsing.
- Case-sensitive paths, case-insensitive field names. User-agent: and user-agent: are equivalent; /Admin/ and /admin/ are two different rules.
Robots.txt Directives — Cheat-Sheet
| Directive | Purpose | Google Support |
|---|---|---|
| User-agent | Names the crawler the rules apply to | Required |
| Allow | Permits crawling of a path or pattern | Supported |
| Disallow | Blocks crawling of a path or pattern | Supported |
| Sitemap | Absolute URL of an XML sitemap | Supported |
| Crawl-delay | Politeness delay between requests | Ignored by Googlebot |
| Host | Preferred domain | Yandex only |
Source: Google Search Central’s robots.txt specification.
Copy-Paste Robots.txt Examples
Example 1 — Allow Everything (Recommended Default)
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlThis is the safest starting point for a new site. Every compliant crawler may request every URL, and the sitemap gives Googlebot the discovery cue it needs.
Example 2 — WordPress with Admin Blocked
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /search/
Disallow: /?s=
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/uploads/
Sitemap: https://example.com/sitemap.xmlNotice the Allow for admin-ajax.php: blocking it can break AJAX-powered features in plugins like WooCommerce.
Example 3 — Block AI and Unwanted Bots
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /private/
Disallow: /tmp/Always keep the Googlebot block open when chasing organic traffic. Surgical blocking beats blanket Disallow: /.
Example 4 — Staging Environment
User-agent: *
Disallow: /Use this only on staging.example.com. Placing it on the live domain will de-list every URL you care about — a mistake we see almost weekly during a technical SEO audit cleanup.
Example 5 — Wildcards and $ End Marker
User-agent: *
Disallow: /*.pdf$
Disallow: /print/
Disallow: /*?*sort=- * matches any sequence of characters.
- $ anchors the rule to the end of the URL path.
- Disallow: /*.pdf$ blocks /whitepaper.pdf but not /whitepaper/page.
Robots.txt Generator — Faster, Safer Output
Hand-coding is fine, but typos cost crawl budget. A validated robots.txt generator can produce a correct file in under a minute, and a search-engine spider simulator lets you confirm which URLs a real crawler can fetch before you publish.
Sitemap Directive and Cross-Topic Connections
Every robots.txt file should end with a Sitemap: line pointing to your XML sitemap. The directive can technically sit anywhere in the file, but placing it at the bottom keeps the rules easy to read. For the full discovery story, pair this with our technical SEO audit guide.
Common Robots.txt Mistakes
1. Believing Disallow Removes URLs From Google
This is the #1 misconception. Google has repeated this point: a page disallowed in robots.txt can still be indexed if other sites link to it. Use noindex meta tags or password protection to actually keep a page out of the index.
2. Using Crawl-delay to Slow Googlebot
Add Crawl-delay: 10 and Google will silently ignore it. The directive is honored by Bing and Yandex, not by Google. To slow Googlebot, adjust the crawl rate settings in Google Search Console.
3. Blocking /js or /css
Modern search engines render JavaScript. If you block your theme’s /js/ or /css/ directories, you are telling Google not to fetch the assets it needs to render the page, which often tanks rankings.
4. Mixed Case Paths
Disallow: /Admin/ does nothing if your CMS serves /admin/. Either lowercase everything on the server or write both rules.
5. Wrong File Location, Wrong MIME Type
The file must live at https://example.com/robots.txt (root). A redirect to /pages/robots.txt is treated by Google as a missing file. It must also be served as text/plain (or a compatible MIME), not text/html.
6. Placing the File in a Robots Subfolder
HTML alternate-link references do not work for robots.txt. There is one canonical location — the domain root.
7. Noindex Inside Robots.txt
Adding Noindex: /page/ to robots.txt is not supported by Google. To de-index, use an X-Robots-Tag: noindex HTTP header, a noindex meta tag, or 404/410 responses.
Test Before You Publish
- Open Google Search Console’s robots.txt report for the live domain.
- Paste the candidate rules into the editor.
- Enter a sample URL and confirm “Allowed” or “Blocked” matches intent.
- Run the same URLs through a search-engine spider simulator to mimic Googlebot’s fetch behavior.
- Save. Changes apply on the next crawl.
Real-World Scenario — Single-Location Service Business
A Dallas-based HVAC company accidentally published a staging Disallow: / on the live site. Organic traffic dropped 71% within 48 hours. Removing the rule and validating through Google Search Console restored indexing within a week. Pair robots.txt hygiene with strong local SEO and a complete technical SEO audit to prevent repeated incidents.
Comparison Table — robots.txt vs noindex vs Password Protection
| Mechanism | Effect on Crawling | Effect on Indexing | When to Use |
|---|---|---|---|
| robots.txt Disallow | Stops crawl | Does not stop indexing | Low-value URLs (faceted nav, internal search) |
| noindex meta tag | Allows crawl (if not blocked) | Removes from index | Thin pages, staging pages, gated content you still want crawled |
| Password protection (.htpasswd) | Blocks crawl | Prevents indexing | True confidential content |
| 404 / 410 status | Returns error | Removes from index | Permanently removed URLs |
Action Checklist
- Place the file at the exact domain root.
- Serve as UTF-8 / text/plain.
- Use User-agent: * for default rules.
- Add Sitemap: pointing to your XML sitemap.
- Avoid blocking /js, /css, or asset directories.
- Remove any Crawl-delay rules if Google is your primary crawler.
- Never add Noindex: inside robots.txt.
- Validate with the Google Search Console tester.
- Validate with a spider simulator.
- Monitor Coverage and Crawl Stats reports weekly.
Conclusion
A clean robots.txt file is the gateway between your site and every compliant crawler. With the examples and checklist above, you can publish a syntactically correct ruleset within minutes. For folder-level crawl issues, log audit findings, and rendering penalties, graduate to the complete technical SEO audit guide.
FAQs
What is robots.txt used for?
It tells compliant crawlers which URLs they may request. It does not guarantee de-indexing — use noindex or password protection for that.
Where should I place my robots.txt file?
At the exact root of your domain — https://example.com/robots.txt — served as UTF-8 text.
Does Google obey the Crawl-delay directive?
No. Google ignores it. Change the crawl rate in Google Search Console.
Will Disallow keep a page out of Google?
No. Disallow blocks crawling only. Google can index a blocked URL if it is linked from elsewhere.
Is robots.txt case sensitive?
Field names are case-insensitive. URL paths are case-sensitive.
Can I use wildcards in robots.txt?
Yes. Google supports * (any sequence) and $ (end of path) anchor.
How do I point Google to my XML sitemap?
Add Sitemap: https://example.com/sitemap.xml anywhere in the file.
What is the safest robots.txt for a new site?
User-agent: * / Allow: / plus a Sitemap: directive allowing every crawler.
Why is my robots.txt not working?
Common causes: wrong location (not at root), wrong MIME type, blocking /js or /css assets, case mismatches, or 5xx errors on the file.
How do I test robots.txt before going live?
Use Google Search Console’s robots.txt tester plus a search-engine spider simulator.
