WPBeginner

Beginner's Guide for WordPress

  • Blog
    • Beginners Guide
    • News
    • Opinion
    • Showcase
    • Themes
    • Tutorials
    • WordPress Plugins
  • Start Here
  • Deals
  • Glossary
  • Videos
  • Products
X
☰
Beginner's Guide for WordPress / Start your WordPress Blog in minutes
Choosing the Best
WordPress Hosting
How to Easily
Install WordPress
Recommended
WordPress Plugins
View all Guides

WPBeginner» Glossary» R» Robots.txt

What is: Robots.txt

31 Shares
Share
Tweet
Share

Robots.txt is a text file which allows a website to provide instructions to web crawling bots.

Search engines like Google use these web crawlers, sometimes called web robots, to archive and categorize websites. Mosts bots are configured to search for a robots.txt file on the server before it reads any other file from the website. It does this to see if a website’s owner has some special instructions on how to crawl and index their site.

The robots.txt file contains a set of instructions that request the bot to ignore specific files or directories. This may be for the purpose of privacy or because the website owner believes that the contents of those files and directories is irrelevant to the categorization of the website in search engines.

If a website has more than one subdomain, each subdomain must have its own robots.txt file. It is important to note that not all bots will honor a robots.txt file. Some malicious bots will even read the robots.txt file to find which files and directories they should target first. Also, even if a robots.txt file instructs bots to ignore a specific pages on the site, those pages may still appear in search results of they are linked to by other pages that are crawled.

Additional Reading

  • SEO
  • How to Add Your WordPress Site to Google Webmaster Tools
31 Shares
Share
Tweet
Share
Popular on WPBeginner Right Now!
  • Error Establishing a Database Connection in WordPress

    How to Fix the Error Establishing a Database Connection in WordPress

  • Checklist

    Checklist: 15 Things You MUST DO Before Changing WordPress Themes

  • How to Properly Move Your Blog from WordPress.com to WordPress.org

  • Why Build Your Email List Today

    Revealed: Why Building Your Email List is so Important Today!

About the Editorial Staff

Editorial Staff at WPBeginner is a team of WordPress experts led by Syed Balkhi. Page maintained by Syed Balkhi.

Over 600,000+ Readers

Get fresh content from WPBeginner

Featured WordPress Plugin
OptinMonster
OptinMonster
Convert website visitors into email subscribers. Learn More »
How to Start a Blog How to Start a Blog
I need help with ...
Starting a
Blog
WordPress
Performance
WordPress
Security
WordPress
SEO
WordPress
Errors
Building an
Online Store
Useful WordPress Guides
    • 7 Best WordPress Backup Plugins Compared (Pros and Cons)
    • How to Fix the Error Establishing a Database Connection in WordPress
    • Why You Need a CDN for your WordPress Blog? [Infographic]
    • 25 Legit Ways to Make Money Online Blogging with WordPress
    • Self Hosted WordPress.org vs. Free WordPress.com [Infograph]
    • Free Recording: WordPress Workshop for Beginners
    • 24 Must Have WordPress Plugins for Business Websites
    • How to Properly Move Your Blog from WordPress.com to WordPress.org
    • 5 Best Contact Form Plugins for WordPress Compared
    • Which is the Best WordPress Popup Plugin? (Comparison)
    • Best WooCommerce Hosting in 2018 (Comparison)
    • How to Fix the Internal Server Error in WordPress
    • How to Install WordPress - Complete WordPress Installation Tutorial
    • Why You Should Start Building an Email List Right Away
    • How to Properly Move WordPress to a New Domain Without Losing SEO
    • How to Choose the Best WordPress Hosting for Your Website
    • How to Choose the Best Blogging Platform (Comparison)
    • WordPress Tutorials - 200+ Step by Step WordPress Tutorials
    • 5 Best WordPress Ecommerce Plugins Compared
    • 5 Best WordPress Membership Plugins (Compared)
    • 7 Best Email Marketing Services for Small Business (2018)
    • Which is the Best WordPress Slider? Performance + Quality Compared
    • The Truth About Shared WordPress Web Hosting
    • When Do You Really Need Managed WordPress Hosting?
    • 5 Best Drag and Drop WordPress Page Builders Compared
    • How to Switch from Blogger to WordPress without Losing Google Rankings
    • How to Properly Switch From Wix to WordPress (Step by Step)
    • How to Properly Move from Weebly to WordPress (Step by Step)
    • Do You Really Need a VPS? Best WordPress VPS Hosting Compared
    • How to Properly Move from Squarespace to WordPress
    • 5 Best VPN Services for WordPress Users (Compared)
    • HostGator Review - An Honest Look at Speed & Uptime (2018)
    • SiteGround Reviews from 1032 Users & Our Experts (2018)
    • Bluehost Review from Real Users + Performance Stats (2018)
    • How Much Does It Really Cost to Build a WordPress Website?
    • How to Start a Podcast with WordPress (Step by Step)
    • How to Choose the Best Domain Name (8 Tips and Tools)
    • How to Setup a Professional Email Address with Google Apps and Gmail
    • How to Install Google Analytics in WordPress for Beginners
    • How to Move WordPress to a New Host or Server With No Downtime
    • Why is WordPress Free? What are the Costs? What is the Catch?
    • How to Make a Website in 2018 – Step by Step Guide
Deals & Coupons (view all)
MonsterInsights Coupon Code
MonsterInsights Coupon
Get 10% off MonsterInsights, the best Google Analytics plugin for WordPress.
De:comments for WordPress
De:Comments Coupon
Get 30% off on DeComments WordPress plugin. Exclusive offer for WPBeginner users only.
Featured In
About WPBeginner®

WPBeginner is a free WordPress resource site for Beginners. WPBeginner was founded in July 2009 by Syed Balkhi. The main goal of this site is to provide quality tips, tricks, hacks, and other WordPress resources that allows WordPress beginners to improve their site(s).

Site Links
  • About Us
  • Contact Us
  • FTC Disclosure
  • Privacy Policy
  • Terms of Service
  • Free Blog Setup
Our Sites
  • OptinMonster
  • MonsterInsights
  • WPForms
  • List25
  • Awesome Motive
  •  

Copyright © 2009 - 2018 WPBeginner LLC. All Rights Reserved. WPBeginner® is a registered trademark.

WordPress hosting by HostGator | WordPress CDN by MaxCDN | WordPress Security by Sucuri.