Mark Armendariz wrote: > Basically it would check the user_agent for > >the majority of the robots I could figure out (via the logs - Mr. Shiflett >just gave a good list to start with) > One might also check http://www.robotstxt.org/wc/active.html for a big list of known bots.