Site5 - Built For Designers & Developers MENU
Home  ›  Bots  ›  Bots: How to use the robots.txt file

Bots: How to use the robots.txt file

1 Comment

Robot.txt is a way for you to control what search bots (like google, yahoo, and bing) have access to. Please note that not all search bots (email farmers, malware bots, etc) will respect the rules placed in this file.

To use this, please create a file called “robots.txt” into the root web path of your domain (normally public_html).

Here are a few basic examples of what you can do with this file.

1) This will deny all bots from crawling your website.

User-agent: *
Disallow: /

2) This will deny all bots from accessing the web folder called private

User-agent: *
Disallow: /private/

3) This will deny all bots from accessing the web folders private and images

User-agent: *
Disallow: /images/
Disallow: /private/

4) This will deny all bots from accessing the page nobots.html in your web root.

User-agent: *
Disallow: /nobots.html

For more examples of what you can do with the robots.txt file, please see this Wiki Page.

Still have a question? Or need help?
See an error in this article or want to suggest a missing topic? Please leave us a comment below! If you need technical support with your account, please email us, chat live with a representative, or call us!

1 Comment

Money Back Guarantees
Click to verify BBB accreditation and to see a BBB report.