WPBeginner

Beginner's Guide for WordPress

  • Blog
    • Beginners Guide
    • News
    • Opinion
    • Showcase
    • Themes
    • Tutorials
    • WordPress Plugins
  • Start Here
    • How to Start a Blog
    • Create a Website
    • Start an Online Store
    • Best Website Builder
    • Email Marketing
    • WordPress Hosting
    • Business Name Ideas
  • Deals
    • Bluehost Coupon
    • SiteGround Coupon
    • WP Engine Coupon
    • HostGator Coupon
    • Domain.com Coupon
    • Constant Contact
    • View All Deals »
  • Glossary
  • Videos
  • Products
X
☰
Beginner's Guide for WordPress / Start your WordPress Blog in minutes
Choosing the Best
WordPress Hosting
How to Easily
Install WordPress
Recommended
WordPress Plugins
View all Guides

WPBeginner» Blog» Beginners Guide» How to Stop Search Engines from Crawling a WordPress Site

How to Stop Search Engines from Crawling a WordPress Site

Last updated on November 28th, 2018 by Editorial Staff
166 Shares
Share
Tweet
Share
Pin
Free WordPress Video Tutorials on YouTube by WPBeginner
How to Stop Search Engines from Crawling a WordPress Site

Recently, one of our users asked us how they can stop search engines from crawling and indexing their WordPress site? There are many scenarios when you would want to stop search engines from crawling your website or listing it in search results. In this article, we will show you how to stop search engines from crawling a WordPress site.

Stop search engines from crawling your WordPress site

Why and Who Would Want to Stop Search Engines

For most websites, search engines are the biggest source of traffic. You may ask, why would anyone want to block search engines?

When starting out, a lot of people don’t know how to create a local development environment or a staging site. If you’re developing your website live on a publicly accessible domain name, then you likely don’t want Google to index your under construction or maintenance mode page.

There are also many people who use WordPress to create private blogs, and they don’t want those indexed in search results because they’re private.

Also some people use WordPress for project management or intranet, and you wouldn’t want your internal documents being publicly accessible.

In all the above situations, you probably don’t want search engines to index your website.

A common misconception is that if I do not have links pointing to my domain, then search engines will probably never find my website. This is not completely true.

There are many ways search engines can find a website linked elsewhere. For example:

  1. Your domain name could have been previously owned by someone else, and they still have some links pointing to your website now.
  2. Some domain search site’s results could get indexed with your link on them.
  3. There are literally thousands of pages with just list of domain names, your site can appear on one of those.

There are a lot of things happening on the web and most of them are not under your control. However, your website is still under your control, and you can instruct search engines to not index or follow your website.

Video Tutorial

Subscribe to WPBeginner

If you don’t like the video or need more instructions, then continue reading.

Blocking Search Engines from Crawling and Indexing Your WordPress Site

WordPress comes with a built-in feature that allows you to instruct search engines not to index your site. All you need to do is visit Settings » Reading and check the box next to Search Engine Visibility option.

Search engine visibility setting in WordPress

When this box is checked, WordPress adds this line to your website’s header:

<meta name='robots' content='noindex,follow' />

WordPress also modifies your site’s robots.txt file and add these lines to it:

User-agent: *
Disallow: /

These lines ask robots (web crawlers) not to index your pages. However, it is totally up to search engines to accept this request or ignore it. Even though most search engines respect this, some page or random image from your site may get indexed.

How to Make Sure Your Site Doesn’t Appear in Search Results?

The default WordPress visibility settings instruct search engines not to index your website. However, search engines may still end up crawling and indexing a page, file, or image from your website.

The only way to to make sure that search engines don’t index or crawl your website at all is by password protecting your entire WordPress site on the server level.

This means when anyone tries to access your website, they are asked to provide a username and password even before they reach WordPress. This includes search engines as well. Upon login failure, they are shown 401 error and the bots turn away. Here is how to password protect an entire WordPress site.

Method 1: Password Protecting an Entire Site using cPanel

If your WordPress hosting provider offers cPanel access to manage your hosting account, then you can protect your entire site using cPanel. All you have to do is log in to your cPanel dashboard and then click on ‘Password Protect Directories’ icon under the ‘Security’ section.

Password protect directories

Next, you will need to choose the folder where you have installed WordPress. Usually, it is the public_html folder.

Select the root folder

If you have multiple WordPress sites installed under public_html directory, then you need to click on the folder icon to browse and select the folder for the website you want to password protect.

On the next screen, you need to enter a name for the protected directory and click on the ‘Save’ button.

Enter a name for protected directory

Your information will be saved and cPanel will reload the page. After that, you need to add an authorized user by entering a username and password.

Add authorized user

That’s all, you have successfully added password protection to your WordPress site.

Now, whenever a user or search engine visits your website they will be prompted to enter the username and password you created earlier to view the site.

Password protected website

Method 2: Password Protecting WordPress with a Plugin

If you are using a managed WordPress hosting solution, sometimes you don’t have access to cPanel.

In that case, you have the option to use different WordPress plugins to password protect your website. Below are the two most popular solutions:

  1. SeedProd – it’s the #1 coming soon and maintenance mode plugin for WordPress, used on over 800,000+ websites. It comes with complete access control and permissions features that you can use to hide your website from everyone including search engines. We have a step by step how-to guide for SeedProd.
  2. Password Protected – it’s a very simple way to password protect your WordPress site with a single password (no user creation needed). See our step by step guide on how to password protect a WordPress site.

We hope this article helped you stop search engines from crawling or indexing your WordPress site. You may also want to see our ultimate step by step WordPress security guide for beginners.

If you liked this article, then please subscribe to our YouTube Channel for WordPress video tutorials. You can also find us on Twitter and Facebook.

166 Shares
Share
Tweet
Share
Pin
Popular on WPBeginner Right Now!
  • How to Properly Move Your Blog from WordPress.com to WordPress.org

  • How to Start Your Own Podcast (Step by Step)

    How to Start Your Own Podcast (Step by Step)

  • How to Fix the Error Establishing a Database Connection in WordPress

    How to Fix the Error Establishing a Database Connection in WordPress

  • Revealed: Why Building an Email List is so Important Today (6 Reasons)

    Revealed: Why Building an Email List is so Important Today (6 Reasons)

About the Editorial Staff

Editorial Staff at WPBeginner is a team of WordPress experts led by Syed Balkhi. Trusted by over 1.3 million readers worldwide.

The Ultimate WordPress Toolkit

20 Comments

Leave a Reply
  1. Charlotte says:
    Oct 5, 2020 at 3:11 pm

    This is was just what I was looking for. After I moved my dev site to the client’s live site, I didn’t want any traffic coming back to the old one. Thank you!

    Reply
    • WPBeginner Support says:
      Oct 6, 2020 at 10:04 am

      Glad our guide was helpful :)

      Reply
  2. Ryan says:
    Mar 14, 2019 at 2:49 pm

    I deleted my site permanently from wordpress and when I google my name, the old wordpress site still appears, despite the fact that it’s gone. Can someone from the support team help me to delete it? I don’t want my clients getting confused with my actual, live site.

    Reply
    • WPBeginner Support says:
      Mar 18, 2019 at 3:28 pm

      For your old site in the results on Google if you crawl the site as google it can help update Google’s cache. For more ways to completely remove the site you would want to take a look at our post here: https://www.wpbeginner.com/wp-tutorials/how-to-permanently-delete-a-wordpress-site-from-internet/

      Reply
  3. Divyanshu says:
    Feb 4, 2018 at 11:48 am

    if we disable indexing the wordpress site to prevent it from hackers, then will it affect google indexing or not, and if then how to allow google to index my pages

    Reply
  4. Andy says:
    Dec 17, 2017 at 9:32 am

    Doesn’t seem to work. Followed the directions with confidence. My subdomain is still publicly viewable.

    Reply
    • WPBeginner Support says:
      Dec 17, 2017 at 8:19 pm

      Hi Andy,

      If you have password protected your subdomain directory, then please review its settings again in cPanel. If everything looks alright, then you may want to contact your WordPress hosting provider.

      Reply
    • Jay says:
      Feb 20, 2018 at 5:36 am

      Yeah, whats written under “Discourage search engines from indexing site” is very true. Google doesnt seems to honor this

      Reply
  5. chinju says:
    Dec 15, 2016 at 6:36 am

    Hi, I have a doubt that if website have 1 million subpages, search engine block all subpage when we add
    User-agent: *
    Disallow: /
    in robots file?

    Reply
    • Shekhar Sardar says:
      Sep 24, 2017 at 5:52 am

      Yes. Adding Disallow:/ in robot.txt will tell all search engine bots, do not crawl this site or web page.

      Reply
  6. anand says:
    Apr 7, 2016 at 2:58 am

    i want memebership pages member profile page to be not indes by any search engines how to do ????

    Reply
  7. Kenneth says:
    Jan 12, 2016 at 3:55 pm

    I tried this but it messed up my .htccess for my WordPress install.

    So proceed with caution.

    Reply
  8. Joojo says:
    Dec 24, 2015 at 2:10 am

    Hello,

    I’m currently having a WordPress.com Blog (Free Plan) & desire to sell my Digital Stuff online without setting-up a website and relying wholly on the Blog, at first (have been short of funds). Thus, going through with this thought I have made changes to my Blog, under this after making the payment, the customer is redirected to my blog page haivng a link to download the digital content. So, I just want the payment gateway to be able to redirect the traffic to the Download page on my blog and not have it searchable via search engines, etc. & neither be it listed under the WordPress.com Posts list/Pages, etc. because if the download page is searchable by the Search Engines & listed under my Blog’s post, I wouldn’t earn anything because then the customer would directly download the content without making the payment.

    Regards,
    Joojo.

    Reply
  9. Jennifer says:
    Mar 30, 2015 at 4:33 pm

    I’m learning WP now and designing my site. For now, I don’t want the site crawled, so I’ve checked the Search Engine Visibility Box and I’ve add the password protection. However, in a few months, I’ll want the site to go live and I’ll want visitors to my site. If I uncheck the Search Engine Visibility box and take off the password protection, will the robots nofollow header automatically be updated? Or will I have to manually update this in the header section?

    Reply
    • WPBeginner Support says:
      Mar 31, 2015 at 9:07 pm

      Yes, it will be. However, to be extra sure you need to submit your site’s xml sitemap using Google webmaster tools. This will allow you to see if any crawl errors are found on your site.

      Reply
  10. Ryan says:
    Jan 19, 2015 at 5:56 pm

    Is it possible to block only specific pages on your site, rather than blocking the entire site?

    Reply
    • Chad says:
      Feb 13, 2015 at 5:01 pm

      WordPress offers a Password Protect Option on any of your pages and posts. Simply change “visibility” in your page or post edit to “password protected”.

      Reply
  11. WPBeginner Staff says:
    Nov 16, 2014 at 4:16 pm

    Please take a look at this thread.

    Reply
  12. Patrick says:
    Nov 14, 2014 at 5:47 am

    How can I get Google and bing to crawl my site while stopping SeMalt from crawling my site. SeMalt sends traffic to my site making it look like I have more hits, however thesclicks leave my site straight away and by so doing so increase the bounce rate to my wordpress site.
    Any help on this matter would be great.
    Patrick

    Reply
  13. Pawan Reddy says:
    Nov 14, 2014 at 2:08 am

    Thanks for this tip. I didn’t know this was possible

    Reply

Leave a Reply Cancel reply

Thanks for choosing to leave a comment. Please keep in mind that all comments are moderated according to our comment policy, and your email address will NOT be published. Please Do NOT use keywords in the name field. Let's have a personal and meaningful conversation.

Over 1,320,000+ Readers

Get fresh content from WPBeginner

Featured WordPress Plugin
TrustPulse
TrustPulse
Instantly get 15% more conversions with social proof. Learn More »
How to Start a Blog How to Start a Blog
I need help with ...
Starting a
Blog
WordPress
Performance
WordPress
Security
WordPress
SEO
WordPress
Errors
Building an
Online Store
Useful WordPress Guides
    • 7 Best WordPress Backup Plugins Compared (Pros and Cons)
    • How to Fix the Error Establishing a Database Connection in WordPress
    • Why You Need a CDN for your WordPress Blog? [Infographic]
    • 30 Legit Ways to Make Money Online Blogging with WordPress
    • Self Hosted WordPress.org vs. Free WordPress.com [Infograph]
    • Free Recording: WordPress Workshop for Beginners
    • 24 Must Have WordPress Plugins for Business Websites
    • How to Properly Move Your Blog from WordPress.com to WordPress.org
    • 5 Best Contact Form Plugins for WordPress Compared
    • Which is the Best WordPress Popup Plugin? (Comparison)
    • Best WooCommerce Hosting in 2020 (Comparison)
    • How to Fix the Internal Server Error in WordPress
    • How to Install WordPress - Complete WordPress Installation Tutorial
    • Why You Should Start Building an Email List Right Away
    • How to Properly Move WordPress to a New Domain Without Losing SEO
    • How to Choose the Best WordPress Hosting for Your Website
    • How to Choose the Best Blogging Platform (Comparison)
    • WordPress Tutorials - 200+ Step by Step WordPress Tutorials
    • 5 Best WordPress Ecommerce Plugins Compared
    • 5 Best WordPress Membership Plugins (Compared)
    • 7 Best Email Marketing Services for Small Business (2020)
    • How to Choose the Best Domain Registrar (Compared)
    • The Truth About Shared WordPress Web Hosting
    • When Do You Really Need Managed WordPress Hosting?
    • 5 Best Drag and Drop WordPress Page Builders Compared
    • How to Switch from Blogger to WordPress without Losing Google Rankings
    • How to Properly Switch From Wix to WordPress (Step by Step)
    • How to Properly Move from Weebly to WordPress (Step by Step)
    • Do You Really Need a VPS? Best WordPress VPS Hosting Compared
    • How to Properly Move from Squarespace to WordPress
    • How to Register a Domain Name (+ tip to get it for FREE)
    • HostGator Review - An Honest Look at Speed & Uptime (2020)
    • SiteGround Reviews from 4196 Users & Our Experts (2020)
    • Bluehost Review from Real Users + Performance Stats (2020)
    • How Much Does It Really Cost to Build a WordPress Website?
    • How to Create an Email Newsletter the RIGHT WAY (Step by Step)
    • Free Business Name Generator (A.I Powered)
    • How to Create a Free Business Email Address in 5 Minutes (Step by Step)
    • How to Install Google Analytics in WordPress for Beginners
    • How to Move WordPress to a New Host or Server With No Downtime
    • Why is WordPress Free? What are the Costs? What is the Catch?
    • How to Make a Website in 2020 – Step by Step Guide
Deals & Coupons (view all)
Advanced Coupons
Advanced WooCommerce Coupons
Get 50% off the Advanced Coupons smart coupons plugin for WooCommerce.
ProfilePress
ProfilePress Coupon
Get 20% OFF on ProfilePress WordPress ultimate user and profile plugin.
Featured In
About WPBeginner®

WPBeginner is a free WordPress resource site for Beginners. WPBeginner was founded in July 2009 by Syed Balkhi. The main goal of this site is to provide quality tips, tricks, hacks, and other WordPress resources that allows WordPress beginners to improve their site(s).
Join our team: We are Hiring!

Site Links
  • About Us
  • Contact Us
  • FTC Disclosure
  • Privacy Policy
  • Terms of Service
  • Free Blog Setup
  • Free Business Tools
Our Sites
  • OptinMonster
  • MonsterInsights
  • WPForms
  • SeedProd
  • Nameboy
  • RafflePress
  • Smash Balloon

Copyright © 2009 - 2021 WPBeginner LLC. All Rights Reserved. WPBeginner® is a registered trademark.

Managed by Awesome Motive | WordPress hosting by SiteGround | WordPress CDN by MaxCDN | WordPress Security by Sucuri.