NYCPHP Meetup

NYPHP.org

[nycphp-talk] importing 650,000 records

max goldberg max.goldberg at gmail.com
Sat Dec 31 15:24:40 EST 2005


It depends on where the slowdown is. I'd agree with Chris, it would probably
be best to put an index on the email field (if there isnt one already), and
echo out the INSERTs with a newline. At that point you can just do php
script.php > import.sql. Just keep in mind indexes play a major role when
importing large amounts of data, especially when SELECTs are needed for each
row.

If the table isn't live or in production, it may go a lot faster if you drop
all the indexes on the table other than the primary key once you are ready
to run your INSERTs and re-add them when you are done.


On 12/31/05, Chris Shiflett <shiflett at php.net> wrote:
>
> Joseph Crawford wrote:
> > $link = mysql_connect('localhost', 'root', '********');
>
> You should change this password, even if you only allow local
> connections. You might also want to check to see if you use a similar
> password or naming convention in other places.
>
> As for your question, does fgetcsv() perform better? It would require
> you to process your file a line at a time rather than load it all into
> memory, but the actual parsing should be much faster.
>
> Another idea is to write out a file with your SQL statements. Executing
> each query, one at a time, sounds really slow. Regardless of how
> complicated your queries are, you can still process them in a single
> batch. This would probably be as simple as:
>
> $sql .=
>
> You can write $sql to a file and let MySQL slurp it up in one go.
>
> Hope that helps.
>
> Chris
>
> --
> Chris Shiflett
> Brain Bulb, The PHP Consultancy
> http://brainbulb.com/
> _______________________________________________
> New York PHP Talk Mailing List
> AMP Technology
> Supporting Apache, MySQL and PHP
> http://lists.nyphp.org/mailman/listinfo/talk
> http://www.nyphp.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nyphp.org/pipermail/talk/attachments/20051231/1acec2c1/attachment.html>


More information about the talk mailing list