NYCPHP Meetup

[Mark Armendariz]

> > Is there a (good) way to allow robots to navigate your 
> site, but not 
> > issue them a session?
> 
> Short but not the world's most efficient way to do it:
> 
> if (! preg_match('/googlebot/i', $_SERVER['HTTP_USER_AGENT'])) {
>      session_start();
> }
> 

Using a similar method, I'd once created for a client an include that would
come in above all other code.  Basically it would check the user_agent for
the majority of the robots I could figure out (via the logs - Mr. Shiflett
just gave a good list to start with) and if the current user agent wats one,
it would essentaily spit out a huge list of links to every product
throughout the site with the title of the product as the text of the link.
And then it would exit().  Within 3 weeks the entire site was indexed (some
within a couple days).

You could also put this conditional include at the bottom of your scripts
which would essentially give the search engines an automated and immediate
sitemap, of course then that site map will be cached with the pages.

Not sure if it would still work for all the engines, but it did the job mid
last year.

Mark