One way to define a �rogue bot� is �a software robot that does not obey the robots.txt file�.*** Fortunately, there is a fairly simple way to identify these bad bots. The most widely used method is to create a directory on your web server and add an entry like this to your robots.txt [...]