Robots.txt

A simple text file that controls which pages search engine crawlers can access on your website

What Is Robots.txt?

Robots.txt is a plain text file placed in your website's root directory that tells search engine crawlers which pages or sections they can and cannot access. This file acts as the first point of contact between your site and search engine bots, providing instructions about crawler behaviour before they begin indexing your content.

Google's specifications define how this file communicates with crawlers through simple text commands. Whilst the file cannot legally prevent access—anyone can view it by visiting yourdomain.com/robots.txt—reputable search engines respect the directives you specify. This makes it an essential tool for managing crawler traffic and preventing indexing issues.

Simple explanation: This file is like a set of instructions at your website's front door telling search engine bots which rooms they're allowed to enter. It helps you control what gets indexed and manage server resources.

Why Robots.txt Matters

Understanding the importance of proper implementation:

• Crawler management: Controls how search engines access your site
• Resource conservation: Prevents bots from wasting server resources on unimportant pages
• Privacy protection: Keeps private or sensitive pages out of search results
• Duplicate prevention: Blocks crawlers from indexing duplicate content versions
• Technical organisation: Prevents search engines from accessing administrative areas
• Crawl budget optimisation: Focuses bot attention on your most important content

Key Takeaway

Every website should have a properly configured file to guide search engine crawlers effectively. Use this tool to block access to admin areas, search results pages, duplicate content, and low-value pages that waste crawl budget. Never use it to hide sensitive information—blocked pages can still appear in search results without descriptions, and malicious actors ignore directives entirely. Instead, use proper authentication and noindex tags for genuinely private content. Regularly audit your configuration to ensure you're not accidentally blocking important pages. Test changes before implementing them to avoid catastrophic mistakes that could deindex your entire site. Remember that different crawlers may interpret directives slightly differently, so verify your file works correctly across major search engines.

How Robots.txt Works

The mechanics behind crawler control:

When a search engine bot visits your website, it first checks for this file at your root domain. The file must be located at yourdomain.com/robots.txt—no other location works. If found, the bot reads the directives before crawling any other pages. It uses simple syntax with User-agent lines specifying which bots the rules apply to, followed by Allow or Disallow directives indicating permitted or forbidden paths.

Wildcards enable pattern matching for flexible control. For example, "Disallow: /*.pdf$" blocks all PDF files regardless of location. Comments preceded by hash symbols help document your choices for future reference. The file affects crawler behaviour but doesn't guarantee compliance—whilst major search engines respect these directives, the file holds no legal authority.

Basic Robots.txt Syntax

Understanding the file structure:

• User-agent: Specifies which crawler the rules apply to
• Disallow: Tells crawlers not to access specified paths
• Allow: Overrides disallow rules for specific paths
• Sitemap: Points crawlers to your XML sitemap location
• Crawl-delay: Requests time between requests (not supported by Google)

A basic configuration might contain: "User-agent: * / Disallow: /admin/ / Sitemap: https://yourdomain.com/sitemap.xml" This tells all crawlers to avoid the admin directory whilst providing the sitemap location.

Common Use Cases

When to implement blocking directives:

Administrative Areas

Block crawlers from accessing WordPress admin pages, login areas, and backend systems that offer no value in search results and waste crawl budget.

Duplicate Content

Prevent indexing of print versions, session ID URLs, or paginated archives that create duplicate content issues.

Search Results Pages

Stop internal search results from being indexed, as these typically provide poor user experience and thin content.

Shopping cart pages, checkout processes, and private user areas should also be blocked to keep them out of search indices.

Critical Mistakes to Avoid

How improper configuration causes problems:

• Blocking important content: Accidentally disallowing pages you want indexed
• Blocking CSS/JavaScript: Prevents Google from rendering pages properly
• Security misuse: Treating this file as a security measure for sensitive data
• Syntax errors: Incorrect formatting causing unintended blocking
• Conflicting directives: Multiple rules that contradict each other

The most catastrophic mistake involves accidentally blocking your entire website with "Disallow: /" under a wildcard user-agent. This single line can deindex your entire site within days. Always test changes carefully before deployment.

Testing Your Robots.txt

Verifying proper configuration:

Google Search Console provides a tester that shows exactly how Googlebot interprets your file. This tool highlights syntax errors, shows which pages are blocked, and allows you to test changes before pushing them live. Use this tester whenever modifying your configuration to catch mistakes before they harm your search visibility.

Manually verify your file by visiting yourdomain.com/robots.txt in a browser. The file should be publicly accessible and display plain text. Check that your directives match your intentions and that no typos or formatting issues exist.

Robots.txt vs Meta Robots Tags

Understanding the difference:

These tools serve different purposes. This file prevents crawlers from accessing pages entirely—the bot never sees the content. Meta robots tags get placed on individual pages and tell crawlers how to handle content they've already accessed. Use this method for pages you don't want crawled at all. Use meta robots tags or X-Robots-Tag HTTP headers for pages you want crawled but not indexed.

Importantly, pages blocked by this file can still appear in search results if other sites link to them. The search engine knows the URL exists but cannot access content or descriptions. This creates awkward search listings with just a URL and no snippet.

Sitemap Declaration

Including your XML sitemap:

Most robots.txt files include a sitemap directive pointing crawlers to the XML sitemap location. This helps search engines discover all your important pages efficiently. The directive simply states: "Sitemap: https://yourdomain.com/sitemap.xml" placed at the bottom of the file.

You can declare multiple sitemaps if needed. This proves particularly useful for large sites with separate sitemaps for different content sections or for sites managing multiple sitemap files due to size limitations.

Advanced Directives

Beyond basic blocking:

Wildcard characters enable sophisticated pattern matching. The asterisk (*) matches any sequence of characters, useful for blocking entire file types or URL patterns. The dollar sign ($) indicates the end of a URL, ensuring exact matching. These wildcards let you create precise rules without listing every individual URL.

Different user-agents allow bot-specific rules. You might block aggressive crawlers whilst allowing Google and Bing full access. However, bots can claim any user-agent they choose, so this doesn't guarantee compliance from malicious scrapers.

Monitoring Crawler Access

Tracking how bots interact:

Server logs reveal which crawlers visit your site and which pages they access. Review these logs periodically to identify aggressive bots consuming excessive resources or crawlers ignoring your directives. Google Search Console also reports crawl statistics, showing how Googlebot interacts with your site over time.

If you notice problematic crawler behaviour, you can add specific user-agent blocks to your configuration or implement server-level blocks for persistent offenders.

Related SEO Terms

• Crawling — How search engines discover pages
• Indexing — Adding pages to search databases
• Meta Robots — Page-level crawler directives
• XML Sitemap — List of your site's pages
• Crawl Budget — How much search engines crawl

Need Help With Technical SEO?

Our SEO experts can audit your robots.txt configuration and ensure search engines crawl your site efficiently.