Jonathan Gardner
2/21/2010 1:45:00 AM
On Sat, Feb 20, 2010 at 5:07 PM, Vincent Davis <vincent@vincentdavis.net> wrote:
>> Code is below, The files are about 5mb and 230,000 rows. When I have 43
>> files of them and when I get to the 35th (reading it in) my system gets so
>> slow that it is nearly functionless. I am on a mac and activity monitor
>> shows that python is using 2.99GB of memory (of 4GB). (python 2.6 64bit).
>> The getsizeof() returns 6424 bytes for the alldata . So I am not sure what
>> is happening.
With this kind of data set, you should start looking at BDBs or
PostgreSQL to hold your data. While processing files this large is
possible, it isn't easy. Your time is better spent letting the DB
figure out how to arrange your data for you.
--
Jonathan Gardner
jgardner@jonathangardner.net