How To Write Robots Txt
📖 Bu rehber ToolPazar ekibi tarafından hazırlanmıştır. Tüm araçlarımız ücretsiz ve reklamsızdır.
What robots.txt actually controls
Robots.txt tells well-behaved crawlers which URLs they may request. It does not enforce anything — malicious scrapers ignore it — and it does not directly stop a page from appearing in search results. A URL blocked in robots.txt can still be indexed if Google discovers it via external links; the search result simply has no snippet because the crawler never read the page body.
File location and format
Maximum file size honored by Google is 500 KiB. Anything past that is truncated, and the truncation can land mid-rule. Keep real-world files well under 50 KiB.
User-agent: targeting specific crawlers
Google, Bing, and most modern crawlers support two pattern characters beyond plain prefix matching:
Disallow and Allow
You can list multiple sitemaps or point to a sitemap index. This is the only directive that actively tells crawlers where to find content — the rest only tell them where not to go.
Wildcards and end-of-URL anchor
Combine with HTTP Basic Auth or IP allowlisting — robots.txt alone is a courtesy, not a gate. If a competitor or scraper finds the staging URL, they will ignore your robots.txt and crawl it anyway.
Sitemap directive
For a production site with a private admin area and a search results page you do not want indexed:
Crawl-delay — limited support
Google Search Console has a robots.txt report under Settings that shows the last-fetched copy, any parse errors, and the timestamp. For live testing against specific URLs, use the URL Inspection tool — it reports whether a given URL is blocked and which rule blocked it.
Noindex is no longer supported here
Before deploying, run the full file through a syntax checker that understands Google’s precedence rules. A misplaced Allow/Disallow can look harmless but silently block a whole section.