Do you want to optimize your WordPress robots.txt file? Not sure why and how robots.txt file is important for your SEO? We have got you covered. In this article, we will show you how to optimize your WordPress robots.txt for SEO and help you understand the importance of robots.txt file.
Recently, a user asked us if they need a robots.txt file and what is the importance of it? Your site’s robots.txt file play an important role in your site’s overall SEO performance. It basically allows you to communicate with search engines and let them know which parts of your site they should index.
Do I Really Need a Robots.txt File?
Absence of a robots.txt file will not stop search engines from crawling and indexing your website. However, it is highly recommended that you create one. If you want to submit your site’s XML sitemap to search engines, then this is where search engines will look for your XML sitemap unless you have specified it in Google Webmaster Tools.
We highly recommend that if you do not have a robots.txt file on your site, then you immediately create one.
Where is the Robots.txt file? How to Create a Robots.txt file?
Robots.txt file usually resides in your site’s root folder. You will need to connect to your site using an FTP client or by using cPanel file manager to view it.
It is just like any ordinary text file, and you can open it with a plain text editor like Notepad.
If you do not have a robots.txt file in your site’s root directory, then you can always create one. All you need to do is create a new text file on your computer and save it as robots.txt. Next, simply upload it to your site’s root folder.
Video Tutorial
If you don’t like the video or need more instructions, then continue reading.
How to Use Robots.txt file?
The format for robots.txt file is actually quite simple. The first line usually names a user agent. The user agent is actually the name of the search bot you are trying to communicate with. For example, Googlebot or Bingbot. You can use asterisk * to instruct all bots.
The next line follows with Allow or Disallow instructions for search engines, so they know which parts you want them to index, and which ones you don’t want indexed.
See a sample robots.txt file:
User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /readme.html
In this sample robots.txt file for WordPress, we have instructed all bots to index our image upload directory.
In the next two lines we have disallowed them to index our WordPress plugins directory and the readme.html file.
Optimizing Your Robots.txt File for SEO
In the guidelines for webmasters, Google advises webmasters to not use robots.txt file to hide low quality content. If you were thinking about using robots.txt file to stop Google from indexing your category, date, and other archive pages, then that may not be a wise choice.
Remember, the purpose of robots.txt is to instruct bots what to do with the content they crawl on your site. It does not stop bots from crawling your website.
There are other WordPress plugins which allow you to add meta tags like nofollow and noindex in your archive pages. WordPress SEO plugin also allows you to do this. We are not saying that you should have your archive pages deindexed, but if you wanted to do it, then that’s the proper way of doing it.
You do not need to add your WordPress login page, admin directory, or registration page to robots.txt because login and registration pages have noindex tag added as meta tag by WordPress.
It is recommend that you disallow readme.html file in your robots.txt file. This readme file can be used by someone who is trying to figure out which version of WordPress you are using. If this was an individual, then they can easily access the file by simply browsing to it.
On the other hand if someone is running a malicious query to locate WordPress sites using a specific version, then this disallow tag can protect you from those mass attacks.
You can also disallow your WordPress plugin directory. This will strengthen your site’s security if someone is looking for a specific vulnerable plugin to exploit for a mass attack.
Adding Your XML Sitemap to Robots.txt File
If you are using Yoast’s WordPress SEO plugin or some other plugin to generate your XML sitemap, then your plugin will try to automatically add your sitemap related lines into robots.txt file.
However if it fails, then your plugin will show you the link to your XML Sitemaps which you can add to your robots.txt file manually like this:
Sitemap: http://www.example.com/post-sitemap.xml Sitemap: http://www.example.com/page-sitemap.xml
What Does an Ideal Robots.txt File Should Look Like?
Honestly, many popular blogs use very simple robots.txt files. Their contents vary, depending on the needs of the specific site:
User-agent: * Disallow: Sitemap: http://www.example.com/post-sitemap.xml Sitemap: http://www.example.com/page-sitemap.xml
This robots.txt file simply tells all bots to index all content and provides the links to site’s XML sitemaps.
Here is another example of a robots.txt file, this time it is the one we use here on WPBeginner:
User-Agent: * Allow: /?display=wide Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /readme.html Disallow: /refer/ Sitemap: https://www.wpbeginner.com/post-sitemap.xml Sitemap: https://www.wpbeginner.com/page-sitemap.xml Sitemap: https://www.wpbeginner.com/deals-sitemap.xml Sitemap: https://www.wpbeginner.com/hosting-sitemap.xml
That’s all. We hope this article helped you learn how to optimize your WordPress robots.txt file for SEO. You may also want to see our guide on 9 Best WordPress SEO plugins and tools that you should use.
If you liked this article, then please subscribe to our YouTube Channel for WordPress video tutorials. You can also find us on Twitter and Facebook.
I would like to stop the search engines from indexing my archives during their crawl.
Thanks alot this article it was really helpful
I keep getting the error message below on google webmaster. I am basically stuck. A few things that were not clear to me on this tutorial is where do I find my site’s root files, how do you determine if you already have a “robots.txt” and how do you edit it?
Hi Cherisa,
Your site’s root folder is the one that contains folders like wp-admin, wp-includes, wp-content, etc. It also contains files like wp-config.php, wp-cron.php, wp-blogheader.php, etc.
If you cannot see a robots.txt file in this folder, then you don’t have one. You can go a head and create a new one.
Thank you for your response. I have looked everywhere and can’t seem to locate these root files as you describe. Is there a path directory I can take that leads to this folder. Like it is under Settings, etc?
I had a decent web traffic to my website. Suddenly dropped to zero in the month of May. Till now I have been facing the issue. Please help me to recover my website.
Hello There Thank you For This Information, But I Have A Question
That I Just Create The Sitemap.xml and Robots.txt File, & Its Crawling well. But How Can I Create “Product-Sitemap.xml”
There is all list of product in sitemap.xml file. Do I Have To Create Product-sitemap.xml separately?
and submit to google or bing again ?
Can You please Help me out…
Thank You
I have a problem on robots.txt file setting. Only one robots.txt is showing for all websites. Please help me to show separate robots.txt file of all websites. I have all separate robots.txt file of all individual website. But only one robots.txt file is showing in browser for all websites.
Please explain why did you include
Disallow: /refer/
in the beginner Robots.txt example? I do not understand the implications of this line. Is this important for the beginner? You have explained the other two Disallowed ones.
Thanks.
Hi Debu,
This example was from WPBeginner’s robots.txt file. At WPBeginner we use ThirstyAffiliates to manage affiliate links and cloak URLs. Those URLs have /refer/ in them, that’s why we block them in our robots.txt file.
How can I put all tags/mydomain.Com in nofollow? In robots.txt to concentrate the link Juice? Thanks.
hey,,i am getting error in yoast seo regarding site map..once i click on fix it ,,,it’s coming again..my site html is not loading properly
I’ve just been reviewing my Google Webmaster Tools account and using the Search Console, I’ve found the following:
Page partially loaded
Not all page resources could be loaded. This can affect how Google sees and understands your page. Fix availability problems for any resources that can affect how Google understands your page.
This is because all CSS stylesheets associated with Plugins are disallowed by the default robots.txt.
I understand good reasons why I shouldn’t just make this allowable, but what would be an alternative as I would suspect the Google algorithms are marking down the site for not seeing these.
Hi,
Whenever, I search my site on the google this text appears below the link: “A description for this result is not available because of this site’s robots.txt”
How, can i solve this issue?
Regards
Hi Suren,
Seems like someone accidentally changed your site’s privacy settings. Go to Settings » Reading page and scroll down to ‘Search engine visibility’ section. Make sure that the box next to is unchecked.
Hello
As i seen in webmaster tool, i got robot.txt file like below :
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
let me know is that okey ? or should i use any other ?
I want to know, does it a good idea to block (disallow) “/wp-content/plugins/” in robots.tx? Every time i remove a plugin it shows 404 error in some pages of that plugin.
I loved this explanation. As a beginner I was very confused about robot.txt file and its uses. But now I know what is its purpose.
in some robot.txt file index.php has been disallowed. Can you explain why ? is it a good practice.
Thanks for passing by this precious info.
Can you please tell me why this happening on webmaster tool:
Network unreachable: robots.txt unreachableWe were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.
robots.txt file exist but still
Interesting update from the Yoast team on this at
Quote: “The old best practices of having a robots.txt that blocks access to your wp-includes directory and your plugins directory are no longer valid.”
Allow: /wp-content/uploads/
Shouldn’t this be?
Disallow: /wp-content/uploads/
Because you are aware that google will index all your uploads pages as public URLs right? And then you will get slapped with errors for the page itself. Is there something I am missing here?
Overall, its the actual pages that google crawls to generate image maps, NOT the uploads folders. Then you would have a problem of all the smaller image sizes, and other images that are for UI will also get indexed.
This seems to be the best option:
Disallow: /wp-content/uploads/
If i’m incorrect, please explain this to me so I can understand your angle here.
My blog url not indexing do i need to change my robots.txt?
Im using this robots.txt
how to create robot txt which is ONLY allow index for page and Post.. thanks
I am not sure what’s the problem but my robots.txt has two versions.
One at http://www.example.com/robots.txt and second at example.com/robots.txt
Anybody, please help! Let me know what can be the possible cause and how to correct it?
Most likely, your web host allows your site to be accessed with both www and non-www urls. Try changing robots.txt using an FTP client. Then examine it from both URLs if you can see your changes on both URLs then this means its the same file.
Thanks for the quick reply. I have already done that, but I am not able to see any change. Is there any other way to resolve it?
Yoasts blogpost about this topic was right above yours in my search so of course I checked them both. They are contradicting each other a little bit.. For example yoast said that disallowing plugin directories and others, might hinder the Google crawlers when fetching your site since plugins may output css or js. Also mentioned (and from my own experience), yoast doesn’t add anything sitemap related to the robots.txt, rather generates it so that you can add it to your search console. Here is the link to his post, maybe you can re-check because it is very hard to choose whose word to take for it
As I’m Not Good in Creating this Robotstxt file So Can I use your Robots.txt file by changing the parameters like url and sitemap of my site is it good? or should I create a different one
Hi,
Today i got this mail from Google “Googlebot cannot access CSS and JS files”…what can be the solution?
Thanks
Let me guess… You are using CDN services to import CSS and JS files.
or
It may be possible that you have written wrong syntax in these file.
I have a question about adding Sitemaps. How can I add Yahoo and Bing Sitemap to Robots file and WordPress Directory?
Thanks for the elaborate outline of using the robots file. Does anyone know if Yahoo is using this robots.txt too, and does it obey the rules mentioned in the file? I ask this since I have a “Disallow” for a certain page in my file, but I do receive traffic coming from Yahoo on that page. Nothing from Google, as it should be. Thanks in advance.
correction…
“If you are using Yoast’s WordPress SEO plugin or some other plugin to generate your XML sitemap, then your plugin will try to automatically add your sitemap related lines into robots.txt file.”
Not true. WordPress SEO doesn’t add the sitemap to the robots.txt
“I’ve always felt linking to your XML sitemap from your robots.txt is a bit nonsense. You should be adding them manually to your Google and Bing Webmaster Tools and make sure you look at their feedback about your XML sitemap. This is the reason our WordPress SEO plugin doesn’t add it to your robots.txt.”
https://yoast.com/wordpress-robots-txt-example/
Also more recommended is not to disallow the wp-plugins directory (reasons see Yoast’s post)
And personally I like to simply remover the readme.txt file…
I understood it robots.txt file and use of robots file. What is the site map how do I create sitemap for my site.
After reading Google’s documentation I’m under the impression that the directive to use in the robots.txt file is disallow which only tells the bots what they can and cannot crawl. It does not tell them what can and cannot be indexed. You need to use the noindex robots meta tag to have a page noindexed.
really good article for seo optimized robots.txt file. But I need you to give a tutorial on how to upload robots.txt file to server. As, being a beginner it seems to be a drastic problem to upload that file.
By the way thanks to share such beneficial information.
-Nitin
Upload it to your server/public_hmtl/(Your-site-name) … in this folder
What is the best way to add code to HTTacess to block multiple spam bot refers for their url and Ip address if no URL is given
I know if you get wrong syntax when doing httacess it can take your site off line I am a newbie and need to block these annoying multiple urls from Russia, China, Ukraine etc.
Many thanks
Thanks for those tips….robot txt files now amended! yay!
why have to add Allow: !!!
if I mention only which have to Disallow that enough. Don’t have to write code for Allow because Googlebot or Bingbot will crawl all other thing automatically.
So why should I use Allow again?
Since lacking the Robots.txt file doesn’t stop the site from being crawled, I find myself curious: Is there any sort of hard data on exactly how much having the file will improve SEO performance?
I did a quick Google search, and didn’t see any sort of quantitative data on it. It’s about half a million articles saying, “Hey, this makes SEO better!” but I’d really like to know what we’re talking about here, even generally.
Is this a 5% boost? 50? 500?
Search engines don’t share such data. While not having a robots.txt file does not stop search engines from crawling or indexing a website. However, it is a recommended best practice.
Thanks for taking the time to get back to me, I appreciate it!
Good timing on this. I was trying to find this information just yesterday.
The reason I was searching for it is that Google Webmaster tools was telling me that they could not properly crawl my site because I was blocking various resources needed for the proper rendering of the page.
These resources included those found in /wp-content/plugins/
After I allowed this folder, the warning disappeared.
Any thoughts on this?
You can safely ignore those warnings. It is only a warning if you actually had content there that you would want to get indexed. Sometimes users have restricted search bots and have forgotten about it. These warnings come in handy in those situations.
I have a question that many people out there use static sitemap, and you know what they have wordpress site. Should we us static or sitemap genrated by worpress.
NIce, i like the way Wpbeginner have, and with that rank in alexa, look to me is one of the best way to sue robots.txt..
what does my robots.txt file mean if it looks like this?
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
It’s just saying that search engines should not index your wp-admin folder, wp-includes folder, and the xml-rpc file. Sometimes disallowing /wp-includes/ can block certain scripts for search engines specially if your site is using those scripts. This can hurt your SEO.
The best thing to do is go to Google Webmaster Tools and fetch your website as a bot there. If everything loads fine, then you have nothing to worry about. If it says that scripts are blocked, then you may want to take out the wp-includes line.
I see no logic in your idea about having a script within the includes directory that can be used by a crawn / robot .. and another, if there is an isolated case, it is better after specify the default wordpress allow this file you imagine that search engines use his scripts! after all, wordpress has certainly not by default an robots.txt anti serch engines!!!!