Recently, one of our users asked us how they can stop search engines from crawling and indexing their WordPress site? There are many scenarios when you would want to stop search engines from crawling your website or listing it in search results. In this article, we will show you how to stop search engines from crawling a WordPress site.
Why and Who Would Want to Stop Search Engines
For most websites, search engines are the major source of traffic. You may ask, why would anyone want to block search engines?
Believe it or not, there are many users who work on their sites by putting them on a live publicly accessible domain instead of creating a local development environment or creating a development site.
Some people create project management sites using WordPress. There are many people who use WordPress to create private blogs. In all these situations, you probably don’t want to be indexed by search engines and be found when you are not ready for that.
A common misconception is that if I do not have links pointing to my domain, then search engines will probably never find my website. This is not completely true.
There are many ways search engines can find a website linked elsewhere. For example:
- Your domain name could have been previously owned by someone else and they still have some links pointing to your website now.
- Some domain search site’s results could get indexed with your link on them.
- There are literally thousands of pages with just list of domain names, your site can appear on one of those.
There are many things happening on the web and none of them are under your control. However, your website is still under your control, and you can instruct search engines to not follow or index your website.
Blocking Search Engines from Crawling and Indexing Your WordPress Site
WordPress comes with a built-in feature that allows you to instruct search engines not to index your site. All you need to do is visit Settings » Reading and check the box next to Search Engine Visibility option.
When this box is checked, WordPress adds this line to your website’s header:
<meta name='robots' content='noindex,follow' />
WordPress also modifies your site’s robots.txt file and add these lines to it:
User-agent: * Disallow: /
These lines ask robots (web crawlers) not to index your pages. However, it is totally up to search engines to accept this request or ignore it. Even though most search engines respect this, some page or random image from your site may get indexed.
How to Make Sure Your Site Doesn’t Appear in Search Results?
The most effective way to block search engines from a live website is by password protecting your entire WordPress site on the server level. This means when anyone accesses your website they are asked to provide a username and password even before they reach WordPress. This includes search engines as well. Upon login failure, they are shown 401 error and the bots turn away. Here is how to password protect an entire WordPress site.
Password Protecting an Entire Site using cPanel
If your WordPress hosting provider offers cPanel access to manage your hosting account, then you can protect your entire site using cPanel. Simply login to cPanel dashboard and then click on password protect directories.
This will bring up a popup where you need to choose the document root.
On the next screen, select the folder where your WordPress site is installed. It is usually public_html or www directory. After that, check the box next to ‘Password protect this directory’ option. Next, provide a name to the protected directory and hit the save button.
Later, you will see a success message and link to go back. Clicking on the go back link will bring you to the password protection screen. Now, you need to add a username and password which will be required to view your website.
That’s all your website is now password protected, and no one including the search engines can access your website.
Here is another tutorial on how to password protect your WordPress site. However, in this tutorial, you will be using a plugin. If for some reason you deactivate the plugin, then your site will be accessible to search engines again.
We hope this article helped you stop search engines from crawling or indexing your WordPress site.
If you liked this article, then please subscribe to our YouTube Channel for WordPress video tutorials. You can also find us on Twitter and Google+.
if we disable indexing the wordpress site to prevent it from hackers, then will it affect google indexing or not, and if then how to allow google to index my pages
Doesn’t seem to work. Followed the directions with confidence. My subdomain is still publicly viewable.
Hi Andy,
If you have password protected your subdomain directory, then please review its settings again in cPanel. If everything looks alright, then you may want to contact your WordPress hosting provider.
Yeah, whats written under “Discourage search engines from indexing site” is very true. Google doesnt seems to honor this
Hi, I have a doubt that if website have 1 million subpages, search engine block all subpage when we add
User-agent: *
Disallow: /
in robots file?
Yes. Adding Disallow:/ in robot.txt will tell all search engine bots, do not crawl this site or web page.
i want memebership pages member profile page to be not indes by any search engines how to do ????
I tried this but it messed up my .htccess for my WordPress install.
So proceed with caution.
Hello,
I’m currently having a WordPress.com Blog (Free Plan) & desire to sell my Digital Stuff online without setting-up a website and relying wholly on the Blog, at first (have been short of funds). Thus, going through with this thought I have made changes to my Blog, under this after making the payment, the customer is redirected to my blog page haivng a link to download the digital content. So, I just want the payment gateway to be able to redirect the traffic to the Download page on my blog and not have it searchable via search engines, etc. & neither be it listed under the WordPress.com Posts list/Pages, etc. because if the download page is searchable by the Search Engines & listed under my Blog’s post, I wouldn’t earn anything because then the customer would directly download the content without making the payment.
Regards,
Joojo.
I’m learning WP now and designing my site. For now, I don’t want the site crawled, so I’ve checked the Search Engine Visibility Box and I’ve add the password protection. However, in a few months, I’ll want the site to go live and I’ll want visitors to my site. If I uncheck the Search Engine Visibility box and take off the password protection, will the robots nofollow header automatically be updated? Or will I have to manually update this in the header section?
Yes, it will be. However, to be extra sure you need to submit your site’s xml sitemap using Google webmaster tools. This will allow you to see if any crawl errors are found on your site.
Is it possible to block only specific pages on your site, rather than blocking the entire site?
WordPress offers a Password Protect Option on any of your pages and posts. Simply change “visibility” in your page or post edit to “password protected”.
Please take a look at this thread.
How can I get Google and bing to crawl my site while stopping SeMalt from crawling my site. SeMalt sends traffic to my site making it look like I have more hits, however thesclicks leave my site straight away and by so doing so increase the bounce rate to my wordpress site.
Any help on this matter would be great.
Patrick
Thanks for this tip. I didn’t know this was possible