[nycphp-talk] Looking for ideas on how to allow spiders to cra wl authenticated pages

DeWitt, Michael mjdewitt at
Mon Feb 24 21:46:50 EST 2003


This was along the lines of what I was thinking and possibly using the
remote_address in conjunction to further limit the access.  I don't know if
this is feasible since from what I have heard of Google, their bots can come
from anywhere.


> Sure, you can check the User-Agent header to see if it matches a known
> spider,
> but your authentication is effectively reduced to someone sending this
> header,
> and if you can find User-Agent strings for known spiders, so can an
> attacker.
> Chris
> --- Unsubscribe at ---

More information about the talk mailing list