What is the purpose of robots.txt?

A robots.txt file instructs search engine crawlers (Googlebot, Bingbot) which URLs and directories they can or cannot access on your website.

Where should the robots.txt file be placed?

The robots.txt file must always be placed at the root level of your website domain (e.g. https://example.com/robots.txt).

Fast, Private & Client-Side Developer Suite

100% Client-Side • Private

Developer Utilities

Create SEO-optimized robots.txt files with custom user-agents, disallow rules, and sitemaps.

Loading tool...

Generate valid sitemap.xml files with lastmod, changefreq, and priority tags from URL lists

Generate SEO meta tags

Generate SPF, DMARC, and DKIM DNS TXT records for domain email security

Configure crawler directives for all bots (*) and specific engines (Googlebot, Bingbot).
Add directory paths you want to disallow (e.g. /admin/, /private/, /api/).
Specify your XML sitemap URL.
Copy the generated text or download it directly as robots.txt for your web server root.

What is the purpose of robots.txt?: A robots.txt file instructs search engine crawlers (Googlebot, Bingbot) which URLs and directories they can or cannot access on your website.
Where should the robots.txt file be placed?: The robots.txt file must always be placed at the root level of your website domain (e.g. https://example.com/robots.txt).

Review policy: references and behavior notes are checked whenever the tool implementation changes.