Robots.txt is a simple text file placed at the root of a website (e.g., yoursite.com/robots.txt) that gives instructions to search engine crawlers about which pages or sections of the site they should or shouldn’t crawl.
It’s commonly used to prevent search engines from wasting crawl budget on low-value pages, such as admin login pages, internal search result pages, or duplicate content, while still allowing full access to the pages that matter for ranking.
A misconfigured robots.txt file is one of the most damaging technical SEO mistakes possible — accidentally blocking an entire site or key section from being crawled can cause a dramatic, sudden traffic loss. It’s one of the first files we check in any technical audit.
Robots.txt configuration is reviewed as part of our Technical SEO process.