Trusted WordPress tutorials, when you need them most.
Beginner’s Guide to WordPress
25 Million+
Websites using our plugins
Years of WordPress experience
WordPress tutorials
by experts

What is: Robots.txt

Editorial Note: We earn a commission from partner links on WPBeginner. Commissions do not affect our editors' opinions or evaluations. Learn more about Editorial Process.

Robots.txt is a text file that allows a website to provide instructions to web-crawling bots.

It tells search engines like Google which parts of your website they can and cannot access when indexing your site.

That makes robots.txt a powerful tool for SEO and can also be used to ensure that certain pages do not appear in Google search results.

Glossary: Robots.txt

How Does Robots.txt Work?

Robots.txt is a text file that you can create to tell search engine bots which pages to crawl and index on your website. It is normally stored in the root directory of your website.

Search engines like Google use web crawlers, sometimes called web robots, to archive and categorize websites. Most bots are configured to search for a robots.txt file on the server before they read any other file from the website. A bot does this to see if a website’s owner has special instructions on crawling and indexing their site.

The robots.txt file contains a set of instructions that request the bot to ignore specific files or directories. This may be for privacy or because the website owner believes that the contents of those files and directories are irrelevant to the website’s categorization in search engines.

Here is an example of a robots.txt file:

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/

In this example, the asterisk ‘*’ after ‘User-Agent’ specifies that the instructions are for all search engines.

Next, we allow search engines to crawl and index files in our WordPress uploads folder. Then, we disallow them from crawling and indexing plugins and WordPress admin folders.

Note that if you don’t disallow a URL, then search engine bots will assume they can index it.

Finally, we have provided the URL of our XML sitemap.

How to Create a Robots.txt File in WordPress

The easiest way to create a robots.txt file is using All in One SEO. It’s the best WordPress SEO plugin on the market and comes with an easy-to-use robots.txt file generator.

Robots.txt file editor in All in One SEO

Another tool you can use is WPCode, a powerful code snippets plugin that lets you add custom code to your website easily and safely. The Pro version includes a handy feature that lets you quickly edit the robots.txt file.

However, if you are familiar with code, then you can create the robots.txt file manually. You will need to use an FTP client to edit the robots.txt file. Alternatively, you can use the file manager provided by your WordPress hosting.

FTP connection robots.txt

For more details on creating a robots.txt file, see our guide on how to optimize your WordPress robots.txt for SEO.

How to Use Robots.txt to Stop Search Engines Crawling a Site

Search engines are the biggest source of traffic for most websites. However, there are a few reasons why you might want to discourage search engines from indexing your site.

For example, if you are still building your website, then you won’t want it to appear in search results. The same is true of private blogs and business intranets.

You can use disallow rules in your robots.txt file to ask search engines not to index your entire website or just certain pages. You’ll find detailed instructions in our guide on how to stop search engines from crawling a WordPress site.

You can use tools like All in One SEO to automatically add these rules to your robots.txt file.

Asking Search Engines Not to Index a Single Page

It is important to note that not all bots will honor a robots.txt file. Some malicious bots will even read the robots.txt file to find which files and directories they should target first.

Also, even if a robots.txt file instructs bots to ignore specific pages on the site, those pages may still appear in search results if they are linked to other crawled pages.

We hope this article helped you learn more about robots.txt in WordPress. You may also want to see our Additional Reading list below for related articles on useful WordPress tips, tricks, and ideas.

If you liked this article, then please subscribe to our YouTube Channel for WordPress video tutorials. You can also find us on Twitter and Facebook.

Additional Reading

Editorial Staff

Editorial Staff at WPBeginner is a team of WordPress experts led by Syed Balkhi with over 16 years of experience in WordPress, Web Hosting, eCommerce, SEO, and Marketing. Started in 2009, WPBeginner is now the largest free WordPress resource site in the industry and is often referred to as the Wikipedia for WordPress.

The Ultimate WordPress Toolkit

Get FREE access to our toolkit - a collection of WordPress related products and resources that every professional should have!