[nycphp-talk] Looking for ideas on how to allow spiders to crawl authenticated pages

Chris Shiflett shiflett at
Mon Feb 24 14:01:00 EST 2003

--- "DeWitt, Michael" <mjdewitt at> wrote:
> For some of our sites, many of the pages require registration and login in
> order to view the page.  I would like to open up those pages to spiders in
> an effort to improve our web rankings.

You can either restrict access or not. A spider is basically an anonymous user,
so anything you do to let a spider view restricted resources removes the

Sure, you can check the User-Agent header to see if it matches a known spider,
but your authentication is effectively reduced to someone sending this header,
and if you can find User-Agent strings for known spiders, so can an attacker.


More information about the talk mailing list