Bots: How to use the robots.txt file
Robot.txt is a way for you to control what search bots (like google, yahoo, and bing) have access to. Please note that not all search bots (email farmers, malware bots, etc) will respect the rules placed in this file. To use this, please create a file called “robots.txt” into the […]
