[nycphp-talk] Looking for ideas on how to allow spiders to crawl authenticated pages
shiflett at php.net
Mon Feb 24 14:01:00 EST 2003
--- "DeWitt, Michael" <mjdewitt at alexcommgrp.com> wrote:
> For some of our sites, many of the pages require registration and login in
> order to view the page. I would like to open up those pages to spiders in
> an effort to improve our web rankings.
You can either restrict access or not. A spider is basically an anonymous user,
so anything you do to let a spider view restricted resources removes the
Sure, you can check the User-Agent header to see if it matches a known spider,
but your authentication is effectively reduced to someone sending this header,
and if you can find User-Agent strings for known spiders, so can an attacker.
More information about the talk