WPBeginner

Beginner's Guide for WordPress

  • Blog
    • Beginners Guide
    • News
    • Opinion
    • Showcase
    • Themes
    • Tutorials
    • WordPress Plugins
  • Start Here
    • How to Start a Blog
    • Create a Website
    • Start an Online Store
    • Best Website Builder
    • Email Marketing
    • WordPress Hosting
  • Deals
  • Glossary
  • Videos
  • Products
X
☰
Beginner's Guide for WordPress / Start your WordPress Blog in minutes
Choosing the Best
WordPress Hosting
How to Easily
Install WordPress
Recommended
WordPress Plugins
View all Guides

WPBeginner» Glossary» R» Robots.txt

What is: Robots.txt

33 Shares
Share
Tweet
Share
Pin

Robots.txt is a text file which allows a website to provide instructions to web crawling bots.

Search engines like Google use these web crawlers, sometimes called web robots, to archive and categorize websites. Mosts bots are configured to search for a robots.txt file on the server before it reads any other file from the website. It does this to see if a website’s owner has some special instructions on how to crawl and index their site.

The robots.txt file contains a set of instructions that request the bot to ignore specific files or directories. This may be for the purpose of privacy or because the website owner believes that the contents of those files and directories is irrelevant to the categorization of the website in search engines.

If a website has more than one subdomain, each subdomain must have its own robots.txt file. It is important to note that not all bots will honor a robots.txt file. Some malicious bots will even read the robots.txt file to find which files and directories they should target first. Also, even if a robots.txt file instructs bots to ignore a specific pages on the site, those pages may still appear in search results of they are linked to by other pages that are crawled.

Additional Reading

  • How to Add Your WordPress Site to Google Webmaster Tools
  • SEO
33 Shares
Share
Tweet
Share
Pin
Popular on WPBeginner Right Now!
  • How to Start Your Own Podcast (Step by Step)

    How to Start Your Own Podcast (Step by Step)

  • Revealed: Why Building an Email List is so Important Today (6 Reasons)

    Revealed: Why Building an Email List is so Important Today (6 Reasons)

  • Checklist

    Checklist: 15 Things You MUST DO Before Changing WordPress Themes

  • Google Analytics in WordPress

    How to Install Google Analytics in WordPress for Beginners

About the Editorial Staff

Editorial Staff at WPBeginner is a team of WordPress experts led by Syed Balkhi. Trusted by over 1.3 million readers worldwide.

Over 1,320,000+ Readers

Get fresh content from WPBeginner

Featured WordPress Plugin
MonsterInsights
MonsterInsights
Google Analytics made easy for WordPress. Learn More »
How to Start a Blog How to Start a Blog
I need help with ...
Starting a
Blog
WordPress
Performance
WordPress
Security
WordPress
SEO
WordPress
Errors
Building an
Online Store
Useful WordPress Guides
    • 7 Best WordPress Backup Plugins Compared (Pros and Cons)
    • How to Fix the Error Establishing a Database Connection in WordPress
    • Why You Need a CDN for your WordPress Blog? [Infographic]
    • 25 Legit Ways to Make Money Online Blogging with WordPress
    • Self Hosted WordPress.org vs. Free WordPress.com [Infograph]
    • Free Recording: WordPress Workshop for Beginners
    • 24 Must Have WordPress Plugins for Business Websites
    • How to Properly Move Your Blog from WordPress.com to WordPress.org
    • 5 Best Contact Form Plugins for WordPress Compared
    • Which is the Best WordPress Popup Plugin? (Comparison)
    • Best WooCommerce Hosting in 2019 (Comparison)
    • How to Fix the Internal Server Error in WordPress
    • How to Install WordPress - Complete WordPress Installation Tutorial
    • Why You Should Start Building an Email List Right Away
    • How to Properly Move WordPress to a New Domain Without Losing SEO
    • How to Choose the Best WordPress Hosting for Your Website
    • How to Choose the Best Blogging Platform (Comparison)
    • WordPress Tutorials - 200+ Step by Step WordPress Tutorials
    • 5 Best WordPress Ecommerce Plugins Compared
    • 5 Best WordPress Membership Plugins (Compared)
    • 7 Best Email Marketing Services for Small Business (2019)
    • How to Choose the Best Domain Registrar (Compared)
    • The Truth About Shared WordPress Web Hosting
    • When Do You Really Need Managed WordPress Hosting?
    • 5 Best Drag and Drop WordPress Page Builders Compared
    • How to Switch from Blogger to WordPress without Losing Google Rankings
    • How to Properly Switch From Wix to WordPress (Step by Step)
    • How to Properly Move from Weebly to WordPress (Step by Step)
    • Do You Really Need a VPS? Best WordPress VPS Hosting Compared
    • How to Properly Move from Squarespace to WordPress
    • How to Register a Domain Name (+ tip to get it for FREE)
    • HostGator Review - An Honest Look at Speed & Uptime (2019)
    • SiteGround Reviews from 1032 Users & Our Experts (2019)
    • Bluehost Review from Real Users + Performance Stats (2019)
    • How Much Does It Really Cost to Build a WordPress Website?
    • How to Start a Podcast with WordPress (Step by Step)
    • How to Choose the Best Domain Name (8 Tips and Tools)
    • How to Setup a Professional Email Address with Google Apps and Gmail
    • How to Install Google Analytics in WordPress for Beginners
    • How to Move WordPress to a New Host or Server With No Downtime
    • Why is WordPress Free? What are the Costs? What is the Catch?
    • How to Make a Website in 2019 – Step by Step Guide
Deals & Coupons (view all)
Dreamhost
DreamHost Coupon
Get 40% OFF on DreamHost and get a Free Domain.
Elegant Themes
Elegant Themes Deal
Get all 87 amazingly beautiful WordPress themes by Elegant Themes for only $69. That is like $0.79 per theme!
Featured In
About WPBeginner®

WPBeginner is a free WordPress resource site for Beginners. WPBeginner was founded in July 2009 by Syed Balkhi. The main goal of this site is to provide quality tips, tricks, hacks, and other WordPress resources that allows WordPress beginners to improve their site(s).

Site Links
  • About Us
  • Contact Us
  • FTC Disclosure
  • Privacy Policy
  • Terms of Service
  • Free Blog Setup
Our Sites
  • OptinMonster
  • MonsterInsights
  • WPForms
  • SeedProd
  • Nameboy
  • Awesome Motive

Copyright © 2009 - 2019 WPBeginner LLC. All Rights Reserved. WPBeginner® is a registered trademark.

WordPress hosting by HostGator | WordPress CDN by MaxCDN | WordPress Security by Sucuri.