WPBeginner

Beginner's Guide for WordPress

  • Blog
    • Beginners Guide
    • News
    • Opinion
    • Showcase
    • Themes
    • Tutorials
    • WordPress Plugins
  • Start Here
    • How to Start a Blog
    • Create a Website
    • Start an Online Store
    • Best Website Builder
    • Email Marketing
    • WordPress Hosting
    • Business Name Ideas
  • Deals
    • Bluehost Coupon
    • SiteGround Coupon
    • WP Engine Coupon
    • HostGator Coupon
    • Domain.com Coupon
    • Constant Contact
    • View All Deals »
  • Glossary
  • Videos
  • Products
X
☰
Beginner's Guide for WordPress / Start your WordPress Blog in minutes
Choosing the Best
WordPress Hosting
How to Easily
Install WordPress
Recommended
WordPress Plugins
View all Guides

WPBeginner» Blog» Tutorials» How to Optimize Your WordPress Robots.txt for SEO

How to Optimize Your WordPress Robots.txt for SEO

Last updated on February 27th, 2021 by Editorial Staff
808 Shares
Share
Tweet
Share
Pin
Free WordPress Video Tutorials on YouTube by WPBeginner
How to Optimize Your WordPress Robots.txt for SEO

Recently one of our readers asked us for tips on how to optimize the robots.txt file to improve SEO.

Robots.txt file tells search engines how to crawl your website which makes it an incredibly powerful SEO tool.

In this article, we will show you how to create a perfect robots.txt file for SEO.

Using WordPress robots.txt file to improve SEO

What is robots.txt file?

Robots.txt is a text file that website owners can create to tell search engine bots how to crawl and index pages on their site.

It is typically stored in the root directory, also known as the main folder, of your website. The basic format for a robots.txt file looks like this:

User-agent: [user-agent name]
Disallow: [URL string not to be crawled]

User-agent: [user-agent name]
Allow: [URL string to be crawled]


Sitemap: [URL of your XML Sitemap]

You can have multiple lines of instructions to allow or disallow specific URLs and add multiple sitemaps. If you do not disallow a URL, then search engine bots assume that they are allowed to crawl it.

Here is what a robots.txt example file can look like:

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/

Sitemap: https://example.com/sitemap_index.xml

In the above robots.txt example, we have allowed search engines to crawl and index files in our WordPress uploads folder.

After that, we have disallowed search bots from crawling and indexing plugins and WordPress admin folders.

Lastly, we have provided the URL of our XML sitemap.

Do You Need a Robots.txt File for Your WordPress Site?

If you don’t have a robots.txt file, then search engines will still crawl and index your website. However, you will not be able to tell search engines which pages or folders they should not crawl.

This will not have much of an impact when you’re first starting a blog and do not have a lot of content.

However as your website grows and you have a lot of content, then you would likely want to have better control over how your website is crawled and indexed.

Here is why.

Search bots have a crawl quota for each website.

This means that they crawl a certain number of pages during a crawl session. If they don’t finish crawling all pages on your site, then they will come back and resume crawl in the next session.

This can slow down your website indexing rate.

You can fix this by disallowing search bots from attempting to crawl unnecessary pages like your WordPress admin pages, plugin files, and themes folder.

By disallowing unnecessary pages, you save your crawl quota. This helps search engines crawl even more pages on your site and index them as quickly as possible.

Another good reason to use robots.txt file is when you want to stop search engines from indexing a post or page on your website.

It is not the safest way to hide content from the general public, but it will help you prevent them from appearing in search results.

What Does an Ideal Robots.txt File Look Like?

Many popular blogs use a very simple robots.txt file. Their content may vary, depending on the needs of the specific site:

User-agent: *
Disallow:
 
Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml

This robots.txt file allows all bots to index all content and provides them a link to the website’s XML sitemaps.

For WordPress sites, we recommend the following rules in the robots.txt file:

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/

Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml

This tell search bots to index all WordPress images and files. It disallows search bots from indexing WordPress plugin files, WordPress admin area, the WordPress readme file, and affiliate links.

By adding sitemaps to robots.txt file, you make it easy for Google bots to find all the pages on your site.

Now that you know what an ideal robots.txt file look like, let’s take a look at how you can create a robots.txt file in WordPress.

How to Create a Robots.txt File in WordPress?

There are two ways to create a robots.txt file in WordPress. You can choose the method that works best for you.

Method 1: Editing Robots.txt File Using All in One SEO

All in One SEO also known as AIOSEO is the best WordPress SEO plugin in the market used by over 2 million websites.

It’s easy to use and comes with a robots.txt file generator.

If you don’t have already have the AIOSEO plugin installed, you can see our step by step guide on how to install a WordPress plugin.

Note: Free version of AIOSEO is also available and has this feature.

Once the plugin is installed and activated, you can use it to create and edit your robots.txt file directly from your WordPress admin area.

Simply go to All in One SEO » Tools to edit your robots.txt file.

AIOSEO robots.txt editor

First, you’ll need to turn on the editing option, by clicking the ‘Enable Custom Robots.txt’ toggle to blue.

With this toggle on, you can create a custom robots.txt file in WordPress.

AIOSEO enable custom robots.txt

All in One SEO will show your existing robots.txt file in the ‘Robots.txt Preview’ section at the bottom of your screen.

This version will show the default rules that were added by WordPress.

Robots.txt default rules

These default rules tell the search engines not to crawl your core WordPress files, allows the bots to index all content, and provides them a link to your site’s XML sitemaps.

Now, you can add your own custom rules to improve your robots.txt for SEO.

To add a rule, enter a user agent in the ‘User Agent’ field. Using a * will apply the rule to all user agents.

Then, select whether you want to ‘Allow’ or ‘Disallow’ the search engines to crawl.

Next, enter filename or directory path in the ‘Directory Path’ field.

Add rule in robots.txt

The rule will automatically be applied to your robots.txt. To add another rule clicks the ‘Add Rule’ button.

We recommend adding rules until you create the ideal robots.txt format we shared above.

Your custom rules will look like this.

Robots.txt custom rules file

Once you’re done, don’t forget to click on the ‘Save Changes’ button to store your changes.

Method 2. Edit Robots.txt file Manually Using FTP

For this method, you will need to use an FTP client to edit robots.txt file.

Simply connect to your WordPress hosting account using an FTP client.

Once inside, you will be able to see the robots.txt file in your website’s root folder.

FTP connection robots.txt

If you don’t see one, then you likely don’t have a robots.txt file.

In that case, you can just go ahead and create one.

FTP connection create robots.txt

Robots.txt is a plain text file, which means you can download it to your computer and edit it using any plain text editor like Notepad or TextEdit.

After saving your changes, you can upload it back to your website’s root folder.

How to Test Your Robots.txt File?

Once you have created your robots.txt file, it’s always a good idea to test it using a robots.txt tester tool.

There are many robots.txt tester tools out there, but we recommend using the one inside Google Search Console.

First, you’ll need to have your website linked with Google Search Console. If you haven’t done this yet, see our guide on how to add your WordPress site to Google Search Console.

Then, you can use the Google Search Console Robots Testing Tool.

Select website property robots.txt tester

Simply select your property from the dropdown list.

The tool will automatically fetch your website’s robots.txt file and highlight the errors and warnings if it found any.

Robots.txt tester results

Final Thoughts

The goal of optimizing your robots.txt file is to prevent search engines from crawling pages that are not publicly available. For example, pages in your wp-plugins folder or pages in your WordPress admin folder.

A common myth among SEO experts is that blocking WordPress category, tags, and archive pages will improve crawl rate and result in faster indexing and higher rankings.

This is not true. It’s also against Google’s webmaster guidelines.

We recommend that you follow the above robots.txt format to create a robots.txt file for your website.

We hope this article helped you learn how to optimize your WordPress robots.txt file for SEO. You may also want to see our ultimate WordPress SEO guide and the best WordPress SEO tools to grow your website.

If you liked this article, then please subscribe to our YouTube Channel for WordPress video tutorials. You can also find us on Twitter and Facebook.

808 Shares
Share
Tweet
Share
Pin
Popular on WPBeginner Right Now!
  • How to Start Your Own Podcast (Step by Step)

    How to Start Your Own Podcast (Step by Step)

  • Checklist

    Checklist: 15 Things You MUST DO Before Changing WordPress Themes

  • Revealed: Why Building an Email List is so Important Today (6 Reasons)

    Revealed: Why Building an Email List is so Important Today (6 Reasons)

  • Google Analytics in WordPress

    How to Install Google Analytics in WordPress for Beginners

About the Editorial Staff

Editorial Staff at WPBeginner is a team of WordPress experts led by Syed Balkhi. Trusted by over 1.3 million readers worldwide.

The Ultimate WordPress Toolkit

112 Comments

Leave a Reply
  1. sean says:
    Aug 3, 2020 at 10:31 pm

    Hi, what are the pros and cons of blocking wp-content/uploads
    Thank you

    Reply
    • WPBeginner Support says:
      Aug 4, 2020 at 11:01 am

      If you block your uploads folder then search engines would not normally crawl your uploaded content like images.

      Reply
  2. Piyush says:
    Jul 1, 2020 at 8:07 am

    thanks for solve my problem

    Reply
    • WPBeginner Support says:
      Jul 1, 2020 at 8:42 am

      You’re welcome :)

      Reply
  3. Ravi kumar says:
    Jun 26, 2020 at 2:11 pm

    Sir i m very confused about robot.txt many time i submitted site map in blogger but the after 3,4 days coming the same issue what is the exactly robot.txt.. & how submit that please guide me

    Reply
    • WPBeginner Support says:
      Jun 30, 2020 at 10:19 am

      It would depend on your specific issue, you may want to take a look at our page below:
      https://www.wpbeginner.com/glossary/robots-txt/

      Reply
  4. Prem says:
    Jun 26, 2020 at 2:10 pm

    If I no index a url or page using robots.txt file, does google shows any error in search console?

    Reply
    • WPBeginner Support says:
      Jun 30, 2020 at 10:15 am

      No, Google will not list the page but if the page is listed it will not show an error.

      Reply
  5. Bharat says:
    Jun 24, 2020 at 2:35 pm

    Hi
    I have a question
    i receive google search console coverage issue warning for blocked by robots.txt
    /wp-admin/widgets.php
    My question is, can i allow for wp-admin/widgets.php to robots.txt and this is safe?

    Reply
    • WPBeginner Support says:
      Jun 25, 2020 at 8:51 am

      IF you wanted to you can but that is not a file that Google needs to crawl.

      Reply
  6. Anthony says:
    Jun 6, 2020 at 5:55 pm

    Hi there, I’m wondering if you should allow: /wp-admin/admin-ajax.php?

    Reply
    • WPBeginner Support says:
      Jun 8, 2020 at 3:52 pm

      Normally, yes you should.

      Reply
      • Jaira says:
        Jul 22, 2020 at 10:20 am

        May I know why you should allow /wp-admin/admin-ajax.php?

        Reply
        • WPBeginner Support says:
          Jul 23, 2020 at 10:15 am

          It is used by different themes and plugins to appear correctly for search engines.

  7. Amila says:
    May 13, 2020 at 10:23 am

    Hello! I really like this article and as I’m a beginner with all this crawling stuff I would like to ask something in this regard. Recently, Google has crawled and indexed one of my websites on a really terrible way, showing the pages in search results which are deleted from the website. The website didn’t have discouraged search engine from indexing in the settings of WordPress at the beginning, but it did later after Google showed even 3 more pages in the search results (those pages also doesn’t exist) and I really don’t understand how it could happen with “discourage search engine from indexing” option on. So, can the Yoast method be helpful and make a solution for my website to Google index my website on the appropriate way this time? Thanks in advance!

    Reply
    • WPBeginner Support says:
      May 14, 2020 at 8:54 am

      The Yoast plugin should be able to assist in ensuring the pages you have are indexed properly, there is a chance before you discouraged search engines from crawling your site your page was cached.

      Reply
      • Amila says:
        May 15, 2020 at 2:49 am

        Well yes and from all pages, it cached the once who doesn’t exist anymore. Anyway, as the current page is on “discourage” setting on, is it better to keep it like that for now or to uncheck the box and leave the Google to crawl and index it again with Yoast help? Thanks! With your articles, everything became easier!

        Reply
        • WPBeginner Support says:
          May 15, 2020 at 9:02 am

          You would want to have Google recrawl your site once it is set up how you want.

  8. Pradhuman Kumar says:
    May 12, 2020 at 2:10 pm

    Hi I loved the article, very precise and perfect.
    Just a small suggestion kindly update the image ROBOTS.txt tester, as Google Console is changed and it would be awesome if you add the link to check the robots.txt from Google.

    Reply
    • WPBeginner Support says:
      May 13, 2020 at 8:52 am

      Thank you for the feedback, we’ll be sure to look into updating the article as soon as we are able.

      Reply
  9. Kamaljeet Singh says:
    Mar 18, 2020 at 3:36 pm

    My blog’s robots.txt file was:
    User-Agent: *
    crawl-delay: 10

    After reading this post, I have changed it into your recommended robots.txt file. Is that okay that I removed crawl-delay

    Reply
    • WPBeginner Support says:
      Mar 19, 2020 at 8:50 am

      It should be fine, crawl-delay tells search engines to slow down how quickly to crawl your site.

      Reply
  10. reena says:
    Dec 9, 2019 at 1:10 am

    Very nicely described about robot.text, i am very happy
    u r very good writer

    Reply
    • WPBeginner Support says:
      Dec 9, 2019 at 12:00 pm

      Thank you, glad you liked our article :)

      Reply
  11. JJ says:
    Nov 26, 2019 at 7:02 am

    What is Disallow: /refer/ page ? I get a 404, is this a hidden wp file?

    Reply
    • Editorial Staff says:
      Nov 26, 2019 at 8:18 am

      We use /refer/ to redirect to various affiliate links on our website. We don’t want those to be indexed since they’re just redirects and not actual content.

      Reply
  12. Sagar Arakh says:
    Oct 22, 2019 at 1:56 am

    Thank you for sharing. This was really helpful for me to understand robots.txt
    I have updated my robots.txt to the ideal one you suggested. i will wait for the results now

    Reply
    • WPBeginner Support says:
      Oct 22, 2019 at 10:16 am

      You’re welcome, glad you’re willing to use our recommendations :)

      Reply
  13. Akash Gogoi says:
    Sep 3, 2019 at 5:14 am

    Very helpful article. Thank you very much.

    Reply
    • WPBeginner Support says:
      Sep 4, 2019 at 9:59 am

      Glad our article was helpful :)

      Reply
  14. Zingylancer says:
    Aug 23, 2019 at 1:38 am

    Thanks for share this useful information about us.

    Reply
    • WPBeginner Support says:
      Aug 23, 2019 at 9:44 am

      Glad we could share this information about the robots.txt file :)

      Reply
  15. Jasper says:
    Jul 26, 2019 at 2:53 am

    thanks for update information for me. Your article was good for Robot txt. file. It gave me a piece of new information. thanks and keep me updating with new ideas.

    Reply
    • WPBeginner Support says:
      Jul 26, 2019 at 9:50 am

      Glad our guide was helpful :)

      Reply
  16. Imran says:
    Jul 23, 2019 at 3:15 pm

    Thanks , I added robots.txt in WordPress .Very good article

    Reply
    • WPBeginner Support says:
      Jul 24, 2019 at 9:52 am

      Thank you, glad our article was helpful :)

      Reply
  17. Steve says:
    Jun 10, 2019 at 7:30 pm

    Thanks for this – how does it work on a WP Multisite thou?

    Reply
    • WPBeginner Support says:
      Jun 11, 2019 at 10:16 am

      For a multisite, you would need to have a robots.txt file in the root folder of each site.

      Reply
  18. Pacifique Ndanyuzwe says:
    Apr 30, 2019 at 10:19 am

    My wordpress site is new and my robot.txt by default is
    user-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    I want google to crawl and index my content. Is that robot.txt okay?

    Reply
    • WPBeginner Support says:
      May 1, 2019 at 10:52 am

      You can certainly use that if you wanted

      Reply
  19. Ritesh Seth says:
    Mar 11, 2019 at 10:36 pm

    Great Airticle…

    I was confused from so many days about Robots.txt file and Disallow links. Have copied the tags for robots file. Hope this will solve the issue of my Site

    Reply
    • WPBeginner Support says:
      Mar 12, 2019 at 10:45 am

      We hope our article will help as well :)

      Reply
  20. Kurt says:
    Feb 20, 2019 at 7:11 pm

    The files in the screenshots of your home folder are actually located under the public_html folder under my home folder.

    I did not have a /refer folder under my public_html folder.

    I did not have post or page xml files anywhere on my WP account.

    I did include an entry in the robots.txt file I created to disallow crawling my sandbox site. I’m not sure that’s necessary since I’ve already selected the option in WP telling crawlers not to crawl my sandbox site, but I don’t think it will hurt to have the entry.

    Reply
    • WPBeginner Support says:
      Feb 21, 2019 at 10:24 am

      Some hosts do rename public_html to home which is why you see it there. You would want to ensure Yoast is active for the XML files to be available. The method in this article is an additional precaution to help with preventing crawling your site :)

      Reply
  21. Ahmed says:
    Jan 24, 2019 at 4:58 pm

    Great article

    Reply
    • WPBeginner Support says:
      Jan 25, 2019 at 10:20 am

      Thank you :)

      Reply
  22. ASHOK KUMAR JADON says:
    Jan 18, 2019 at 1:50 am

    Hello, such a nice article you solve my problem. So Thank You so much

    Reply
    • WPBeginner Support says:
      Jan 18, 2019 at 10:38 am

      Glad our article could help :)

      Reply
  23. Elyn Ashton says:
    Jan 15, 2019 at 10:40 pm

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php <– This is my robot.txt code but im confuse why my /wp-admin is index? How to no index it?

    Reply
    • WPBeginner Support says:
      Jan 16, 2019 at 11:09 am

      If it was indexed previously you may need to give time for the search engine’s cache to clear

      Reply
  24. Ashish kumar says:
    Jan 8, 2019 at 12:41 am

    This website really inspire me to start a blog .Thank you lost of tema.this website each and every article have rich of information and explanation.when i have some problem at first i visit this blog . Thank You

    Reply
    • WPBeginner Support says:
      Jan 8, 2019 at 10:48 am

      Glad our articles can be helpful :)

      Reply
  25. Anna says:
    Jan 3, 2019 at 7:00 am

    I am trying to optimise robots for my website using Yoast. However Tools in Yoast does not have the option for ‘File Editor’.
    There are just two options
    (i) Import and Export
    (ii) Bulk editor
    May you please advise how this can be addressed. Could it be that I am on a free edition of Yoast?

    Reply
    • WPBeginner Support says:
      Jan 3, 2019 at 1:30 pm

      The free version of Yoast still has the option, your installation may be disallowing file editing in which case you would likely need to use the FTP method.

      Reply
  26. Emmanuel Husseni says:
    Dec 18, 2018 at 3:04 pm

    I really find this article helpful because I really don’t know much on how robot.txt works but now I do.

    pls what I don’t understand is how do I find the best format of robot.txt to use on my site (I mean one that works generally)?

    I noticed lots of big blogs I check ranking high on search engine uses different robot.txt format..

    I’ll be clad to see a reply from you or just anyone that can help

    Reply
    • Editorial Staff says:
      Dec 21, 2018 at 6:46 am

      Having a sitemap and allowing the areas that need to be allowed is the most important part. The disallow part will vary based on each site. We shared a sample in our blog post, and that should be good for most WordPress sites.

      Reply
    • WPBeginner Support says:
      Dec 21, 2018 at 9:12 am

      Hey Emmanuel,

      Please see the section regarding the ideal robots.txt file. It depends on your own requirements. Most bloggers exclude WordPress admin and plugin folders from the crawl.

      Reply
      • Emmanuel Husseni says:
        Dec 26, 2018 at 2:51 am

        Thank you so much.

        now I understand. I guess I’ll start with the general format for now.

        Reply
  27. jack says:
    Jul 31, 2018 at 3:45 am

    Well written article, I recommend the users to do sitemap before creating and enabling their ROBOTS text it will help your site to crawl faster and indexed easily.

    Jack

    Reply
  28. Connie S Owens says:
    Apr 17, 2018 at 9:48 am

    I would like to stop the search engines from indexing my archives during their crawl.

    Reply
  29. Emmanuel Nonye says:
    Jan 8, 2018 at 6:26 pm

    Thanks alot this article it was really helpful

    Reply
  30. Cherisa says:
    Nov 13, 2017 at 4:28 pm

    I keep getting the error message below on google webmaster. I am basically stuck. A few things that were not clear to me on this tutorial is where do I find my site’s root files, how do you determine if you already have a “robots.txt” and how do you edit it?

    Reply
    • WPBeginner Support says:
      Nov 14, 2017 at 4:30 am

      Hi Cherisa,

      Your site’s root folder is the one that contains folders like wp-admin, wp-includes, wp-content, etc. It also contains files like wp-config.php, wp-cron.php, wp-blogheader.php, etc.

      If you cannot see a robots.txt file in this folder, then you don’t have one. You can go a head and create a new one.

      Reply
      • Cherisa says:
        Nov 14, 2017 at 3:47 pm

        Thank you for your response. I have looked everywhere and can’t seem to locate these root files as you describe. Is there a path directory I can take that leads to this folder. Like it is under Settings, etc?

        Reply
  31. Devender says:
    Oct 25, 2017 at 3:33 am

    I had a decent web traffic to my website. Suddenly dropped to zero in the month of May. Till now I have been facing the issue. Please help me to recover my website.

    Reply
  32. Haris Aslam says:
    Oct 17, 2017 at 7:32 am

    Hello There Thank you For This Information, But I Have A Question
    That I Just Create The Sitemap.xml and Robots.txt File, & Its Crawling well. But How Can I Create “Product-Sitemap.xml”
    There is all list of product in sitemap.xml file. Do I Have To Create Product-sitemap.xml separately?

    and submit to google or bing again ?
    Can You please Help me out…
    Thank You

    Reply
  33. Mahadi Hassan says:
    Sep 5, 2017 at 9:27 pm

    I have a problem on robots.txt file setting. Only one robots.txt is showing for all websites. Please help me to show separate robots.txt file of all websites. I have all separate robots.txt file of all individual website. But only one robots.txt file is showing in browser for all websites.

    Reply
  34. Debu Majumdar says:
    Jul 31, 2017 at 6:26 pm

    Please explain why did you include
    Disallow: /refer/
    in the beginner Robots.txt example? I do not understand the implications of this line. Is this important for the beginner? You have explained the other two Disallowed ones.

    Thanks.

    Reply
    • WPBeginner Support says:
      Jul 31, 2017 at 10:36 pm

      Hi Debu,

      This example was from WPBeginner’s robots.txt file. At WPBeginner we use ThirstyAffiliates to manage affiliate links and cloak URLs. Those URLs have /refer/ in them, that’s why we block them in our robots.txt file.

      Reply
  35. Evaristo says:
    Apr 28, 2017 at 8:09 am

    How can I put all tags/mydomain.Com in nofollow? In robots.txt to concentrate the link Juice? Thanks.

    Reply
  36. harsh kumar says:
    Apr 15, 2017 at 12:13 pm

    hey,,i am getting error in yoast seo regarding site map..once i click on fix it ,,,it’s coming again..my site html is not loading properly

    Reply
  37. Tom says:
    Apr 9, 2017 at 5:39 am

    I’ve just been reviewing my Google Webmaster Tools account and using the Search Console, I’ve found the following:
    Page partially loaded
    Not all page resources could be loaded. This can affect how Google sees and understands your page. Fix availability problems for any resources that can affect how Google understands your page.

    This is because all CSS stylesheets associated with Plugins are disallowed by the default robots.txt.

    I understand good reasons why I shouldn’t just make this allowable, but what would be an alternative as I would suspect the Google algorithms are marking down the site for not seeing these.

    Reply
  38. Suren says:
    Mar 8, 2017 at 5:34 pm

    Hi,

    Whenever, I search my site on the google this text appears below the link: “A description for this result is not available because of this site’s robots.txt”

    How, can i solve this issue?

    Regards

    Reply
    • WPBeginner Support says:
      Mar 8, 2017 at 5:40 pm

      Hi Suren,

      Seems like someone accidentally changed your site’s privacy settings. Go to Settings » Reading page and scroll down to ‘Search engine visibility’ section. Make sure that the box next to is unchecked.

      Reply
  39. Divyesh says:
    Feb 19, 2017 at 11:47 am

    Hello

    As i seen in webmaster tool, i got robot.txt file like below :

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    let me know is that okey ? or should i use any other ?

    Reply
  40. John Cester says:
    Dec 24, 2016 at 12:36 am

    I want to know, does it a good idea to block (disallow) “/wp-content/plugins/” in robots.tx? Every time i remove a plugin it shows 404 error in some pages of that plugin.

    Reply
  41. Himanshu singh says:
    Dec 23, 2016 at 12:40 pm

    I loved this explanation. As a beginner I was very confused about robot.txt file and its uses. But now I know what is its purpose.

    Reply
  42. rahul says:
    Nov 28, 2016 at 10:10 am

    in some robot.txt file index.php has been disallowed. Can you explain why ? is it a good practice.

    Reply
  43. Waleed Barakat says:
    Nov 16, 2016 at 8:19 am

    Thanks for passing by this precious info.

    Reply
  44. Awais Ahmed says:
    Nov 10, 2016 at 2:32 am

    Can you please tell me why this happening on webmaster tool:

    Network unreachable: robots.txt unreachableWe were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.

    robots.txt file exist but still

    Reply
  45. Dozza says:
    Nov 9, 2016 at 5:24 am

    Interesting update from the Yoast team on this at

    Quote: “The old best practices of having a robots.txt that blocks access to your wp-includes directory and your plugins directory are no longer valid.”

    Reply
  46. natveimaging says:
    Oct 2, 2016 at 11:29 am

    Allow: /wp-content/uploads/

    Shouldn’t this be?
    Disallow: /wp-content/uploads/

    Because you are aware that google will index all your uploads pages as public URLs right? And then you will get slapped with errors for the page itself. Is there something I am missing here?

    Reply
    • nativeimaging says:
      Oct 2, 2016 at 11:31 am

      Overall, its the actual pages that google crawls to generate image maps, NOT the uploads folders. Then you would have a problem of all the smaller image sizes, and other images that are for UI will also get indexed.

      This seems to be the best option:
      Disallow: /wp-content/uploads/

      If i’m incorrect, please explain this to me so I can understand your angle here.

      Reply
  47. Jason says:
    Sep 25, 2016 at 10:05 pm

    My blog url not indexing do i need to change my robots.txt?
    Im using this robots.txt

    Reply
  48. iyan says:
    Aug 4, 2016 at 12:00 pm

    how to create robot txt which is ONLY allow index for page and Post.. thanks

    Reply
  49. Simaran Singh says:
    Jul 11, 2016 at 11:41 am

    I am not sure what’s the problem but my robots.txt has two versions.
    One at http://www.example.com/robots.txt and second at example.com/robots.txt

    Anybody, please help! Let me know what can be the possible cause and how to correct it?

    Reply
    • WPBeginner Support says:
      Jul 11, 2016 at 10:03 pm

      Most likely, your web host allows your site to be accessed with both www and non-www urls. Try changing robots.txt using an FTP client. Then examine it from both URLs if you can see your changes on both URLs then this means its the same file.

      Reply
      • Simaran Singh says:
        Jul 11, 2016 at 11:44 pm

        Thanks for the quick reply. I have already done that, but I am not able to see any change. Is there any other way to resolve it?

        Reply
  50. Martin conde says:
    Apr 22, 2016 at 11:16 am

    Yoasts blogpost about this topic was right above yours in my search so of course I checked them both. They are contradicting each other a little bit.. For example yoast said that disallowing plugin directories and others, might hinder the Google crawlers when fetching your site since plugins may output css or js. Also mentioned (and from my own experience), yoast doesn’t add anything sitemap related to the robots.txt, rather generates it so that you can add it to your search console. Here is the link to his post, maybe you can re-check because it is very hard to choose whose word to take for it ;)

    Reply
« 1 2

Leave a Reply Cancel reply

Thanks for choosing to leave a comment. Please keep in mind that all comments are moderated according to our comment policy, and your email address will NOT be published. Please Do NOT use keywords in the name field. Let's have a personal and meaningful conversation.

Over 1,320,000+ Readers

Get fresh content from WPBeginner

Featured WordPress Plugin
TrustPulse
TrustPulse
Instantly get 15% more conversions with social proof. Learn More »
How to Start a Blog How to Start a Blog
I need help with ...
Starting a
Blog
WordPress
Performance
WordPress
Security
WordPress
SEO
WordPress
Errors
Building an
Online Store
Useful WordPress Guides
    • 7 Best WordPress Backup Plugins Compared (Pros and Cons)
    • How to Fix the Error Establishing a Database Connection in WordPress
    • Why You Need a CDN for your WordPress Blog? [Infographic]
    • 30 Legit Ways to Make Money Online Blogging with WordPress
    • Self Hosted WordPress.org vs. Free WordPress.com [Infograph]
    • Free Recording: WordPress Workshop for Beginners
    • 24 Must Have WordPress Plugins for Business Websites
    • How to Properly Move Your Blog from WordPress.com to WordPress.org
    • 5 Best Contact Form Plugins for WordPress Compared
    • Which is the Best WordPress Popup Plugin? (Comparison)
    • Best WooCommerce Hosting in 2021 (Comparison)
    • How to Fix the Internal Server Error in WordPress
    • How to Install WordPress - Complete WordPress Installation Tutorial
    • Why You Should Start Building an Email List Right Away
    • How to Properly Move WordPress to a New Domain Without Losing SEO
    • How to Choose the Best WordPress Hosting for Your Website
    • How to Choose the Best Blogging Platform (Comparison)
    • WordPress Tutorials - 200+ Step by Step WordPress Tutorials
    • 5 Best WordPress Ecommerce Plugins Compared
    • 5 Best WordPress Membership Plugins (Compared)
    • 7 Best Email Marketing Services for Small Business (2021)
    • How to Choose the Best Domain Registrar (Compared)
    • The Truth About Shared WordPress Web Hosting
    • When Do You Really Need Managed WordPress Hosting?
    • 5 Best Drag and Drop WordPress Page Builders Compared
    • How to Switch from Blogger to WordPress without Losing Google Rankings
    • How to Properly Switch From Wix to WordPress (Step by Step)
    • How to Properly Move from Weebly to WordPress (Step by Step)
    • Do You Really Need a VPS? Best WordPress VPS Hosting Compared
    • How to Properly Move from Squarespace to WordPress
    • How to Register a Domain Name (+ tip to get it for FREE)
    • HostGator Review - An Honest Look at Speed & Uptime (2021)
    • SiteGround Reviews from 4464 Users & Our Experts (2021)
    • Bluehost Review from Real Users + Performance Stats (2021)
    • How Much Does It Really Cost to Build a WordPress Website?
    • How to Create an Email Newsletter the RIGHT WAY (Step by Step)
    • Free Business Name Generator (A.I Powered)
    • How to Create a Free Business Email Address in 5 Minutes (Step by Step)
    • How to Install Google Analytics in WordPress for Beginners
    • How to Move WordPress to a New Host or Server With No Downtime
    • Why is WordPress Free? What are the Costs? What is the Catch?
    • How to Make a Website in 2021 – Step by Step Guide
Deals & Coupons (view all)
LiveChat logo
LiveChat Inc Coupon
Get a 30 day free trial and 30% OFF LiveChat, one of the best live chat service providers for WordPress users.
LearnDash
LearnDash Coupon
Get the lowest price on the best learning management system (LMS) plugin for WordPress.
Featured In
About WPBeginner®

WPBeginner is a free WordPress resource site for Beginners. WPBeginner was founded in July 2009 by Syed Balkhi. The main goal of this site is to provide quality tips, tricks, hacks, and other WordPress resources that allows WordPress beginners to improve their site(s).

Join our team: We are Hiring!

Site Links
  • About Us
  • Contact Us
  • FTC Disclosure
  • Privacy Policy
  • Terms of Service
  • Free Blog Setup
  • Free Business Tools
  • Growth Fund
Our Sites
  • OptinMonster
  • MonsterInsights
  • WPForms
  • SeedProd
  • Nameboy
  • RafflePress
  • Smash Balloon
  • AIOSEO

Copyright © 2009 - 2021 WPBeginner LLC. All Rights Reserved. WPBeginner® is a registered trademark.

Managed by Awesome Motive | WordPress hosting by SiteGround | WordPress Security by Sucuri.