NYCPHP Meetup

[Dan Cech]

On 8/13/2010 4:13 PM, Donald J. Organ IV wrote:
> I am using file_get_contents() with a context that defines it should
> be using http 1.1 default is 1.0....
>
> I am using this inside of a loop that is pulling remote product
> images and creating thumbnails....
>
> The reason we had to add a context is because with no context
> specified file_get_contents() used http 1.0 and when looping through
> say 2000 products it creates 2000 connections to a server in under
> say 10 seconds.....Most servers dont like this and most hosting
> companies will see this as a DOS attack.
>
> So Now we've added the context and now each request is taking
> anywhere from 8-15 seconds. From what I can tell it has to do with
> the fact that http 1.1 uses a single connection for all of these
> requests...if I add connection: close to the header then I am right
> back where I started with http 1.0
>
> So I was wondering if there is a way to tel file_get_contents that it
> can use more than one connection but its only allow to use say 10.

I'd be looking into cURL and specifically curl_multi, or forking a bunch
of sub-processes so that each one can maintain its own connection and
process a portion of the requests.

http://us2.php.net/curl
http://us2.php.net/manual/en/book.pcntl.php

Dan