[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

get the size of a dynamically changing file fast ?

Stef Mientki

1/17/2008 9:56:00 PM

hello,

I've a program (not written in Python) that generates a few thousands
bytes per second,
these files are dumped in 2 buffers (files), at in interval time of 50 msec,
the files can be read by another program, to do further processing.

A program written in VB or delphi can handle the data in the 2 buffers
perfectly.
Sometimes Python is also able to process the data correctly,
but often it can't :-(

I keep one of the files open en test the size of the open datafile each
50 msec.
I have tried
os.stat ( ....) [ ST_SIZE]
os.path.getsize ( ... )
but they both have the same behaviour, sometimes it works, and the data
is collected each 50 .. 100 msec,
sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.

I'm using python 2.4 on winXP.

Is there a solution for this problem ?

thanks,
Stef Mientki
8 Answers

Mike Driscoll

1/18/2008 3:55:00 PM

0

On Jan 17, 3:56 pm, Stef Mientki <stef.mien...@gmail.com> wrote:
> hello,
>
> I've a program (not written in Python) that generates a few thousands
> bytes per second,
> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
> the files can be read by another program, to do further processing.
>
> A program written in VB or delphi can handle the data in the 2 buffers
> perfectly.
> Sometimes Python is also able to process the data correctly,
> but often it can't :-(
>
> I keep one of the files open en test the size of the open datafile each
> 50 msec.
> I have tried
> os.stat ( ....) [ ST_SIZE]
> os.path.getsize ( ... )
> but they both have the same behaviour, sometimes it works, and the data
> is collected each 50 .. 100 msec,
> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.
>
> I'm using python 2.4 on winXP.
>
> Is there a solution for this problem ?
>
> thanks,
> Stef Mientki

Tim Golden has a method to watch for changes in a directory on his
website:

http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_directory_for_ch...

This old post also mentions something similar:

http://mail.python.org/pipermail/python-list/2007-October/4...

And here's a cookbook recipe that claims to do it as well using
decorators:

http://aspn.activestate.com/ASPN/Cookbook/Python/Rec...

Hopefully that will get you going.

Mike

Stef Mientki

1/22/2008 9:36:00 PM

0

Mike Driscoll wrote:
> On Jan 17, 3:56 pm, Stef Mientki <stef.mien...@gmail.com> wrote:
>
>> hello,
>>
>> I've a program (not written in Python) that generates a few thousands
>> bytes per second,
>> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
>> the files can be read by another program, to do further processing.
>>
>> A program written in VB or delphi can handle the data in the 2 buffers
>> perfectly.
>> Sometimes Python is also able to process the data correctly,
>> but often it can't :-(
>>
>> I keep one of the files open en test the size of the open datafile each
>> 50 msec.
>> I have tried
>> os.stat ( ....) [ ST_SIZE]
>> os.path.getsize ( ... )
>> but they both have the same behaviour, sometimes it works, and the data
>> is collected each 50 .. 100 msec,
>> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.
>>
>> I'm using python 2.4 on winXP.
>>
>> Is there a solution for this problem ?
>>
>> thanks,
>> Stef Mientki
>>
>
> Tim Golden has a method to watch for changes in a directory on his
> website:
>
> http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_directory_for_ch...
>
> This old post also mentions something similar:
>
> http://mail.python.org/pipermail/python-list/2007-October/4...
>
> And here's a cookbook recipe that claims to do it as well using
> decorators:
>
> http://aspn.activestate.com/ASPN/Cookbook/Python/Rec...
>
> Hopefully that will get you going.
>
> Mike
>
thanks Mike,
sorry for the late reaction.
I've it working perfect now.
After all, os.stat works perfectly well,
the problem was in the program that generated the file with increasing
size,
by truncating it after each block write, it apperently garantees that
the file is flushed to disk and all problems are solved.

cheers,
Stef Mientki

Mike Driscoll

1/22/2008 9:42:00 PM

0

On Jan 22, 3:35 pm, Stef Mientki <stef.mien...@gmail.com> wrote:
> Mike Driscoll wrote:
> > On Jan 17, 3:56 pm, Stef Mientki <stef.mien...@gmail.com> wrote:
>
> >> hello,
>
> >> I've a program (not written in Python) that generates a few thousands
> >> bytes per second,
> >> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
> >> the files can be read by another program, to do further processing.
>
> >> A program written in VB or delphi can handle the data in the 2 buffers
> >> perfectly.
> >> Sometimes Python is also able to process the data correctly,
> >> but often it can't :-(
>
> >> I keep one of the files open en test the size of the open datafile each
> >> 50 msec.
> >> I have tried
> >> os.stat ( ....) [ ST_SIZE]
> >> os.path.getsize ( ... )
> >> but they both have the same behaviour, sometimes it works, and the data
> >> is collected each 50 .. 100 msec,
> >> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.
>
> >> I'm using python 2.4 on winXP.
>
> >> Is there a solution for this problem ?
>
> >> thanks,
> >> Stef Mientki
>
> > Tim Golden has a method to watch for changes in a directory on his
> > website:
>
> >http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_di......
>
> > This old post also mentions something similar:
>
> >http://mail.python.org/pipermail/python-list/2007-October/4...
>
> > And here's a cookbook recipe that claims to do it as well using
> > decorators:
>
> >http://aspn.activestate.com/ASPN/Cookbook/Python/Rec...
>
> > Hopefully that will get you going.
>
> > Mike
>
> thanks Mike,
> sorry for the late reaction.
> I've it working perfect now.
> After all, os.stat works perfectly well,
> the problem was in the program that generated the file with increasing
> size,
> by truncating it after each block write, it apperently garantees that
> the file is flushed to disk and all problems are solved.
>
> cheers,
> Stef Mientki

I almost asked if you were making sure you had flushed the data to the
file...oh well.

Mike

Stef Mientki

1/22/2008 10:23:00 PM

0

Mike Driscoll wrote:
> On Jan 22, 3:35 pm, Stef Mientki <stef.mien...@gmail.com> wrote:
>
>> Mike Driscoll wrote:
>>
>>> On Jan 17, 3:56 pm, Stef Mientki <stef.mien...@gmail.com> wrote:
>>>
>>>> hello,
>>>>
>>>> I've a program (not written in Python) that generates a few thousands
>>>> bytes per second,
>>>> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
>>>> the files can be read by another program, to do further processing.
>>>>
>>>> A program written in VB or delphi can handle the data in the 2 buffers
>>>> perfectly.
>>>> Sometimes Python is also able to process the data correctly,
>>>> but often it can't :-(
>>>>
>>>> I keep one of the files open en test the size of the open datafile each
>>>> 50 msec.
>>>> I have tried
>>>> os.stat ( ....) [ ST_SIZE]
>>>> os.path.getsize ( ... )
>>>> but they both have the same behaviour, sometimes it works, and the data
>>>> is collected each 50 .. 100 msec,
>>>> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.
>>>>
>>>> I'm using python 2.4 on winXP.
>>>>
>>>> Is there a solution for this problem ?
>>>>
>>>> thanks,
>>>> Stef Mientki
>>>>
>>> Tim Golden has a method to watch for changes in a directory on his
>>> website:
>>>
>>> http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_di......
>>>
>>> This old post also mentions something similar:
>>>
>>> http://mail.python.org/pipermail/python-list/2007-October/4...
>>>
>>> And here's a cookbook recipe that claims to do it as well using
>>> decorators:
>>>
>>> http://aspn.activestate.com/ASPN/Cookbook/Python/Rec...
>>>
>>> Hopefully that will get you going.
>>>
>>> Mike
>>>
>> thanks Mike,
>> sorry for the late reaction.
>> I've it working perfect now.
>> After all, os.stat works perfectly well,
>> the problem was in the program that generated the file with increasing
>> size,
>> by truncating it after each block write, it apperently garantees that
>> the file is flushed to disk and all problems are solved.
>>
>> cheers,
>> Stef Mientki
>>
>
> I almost asked if you were making sure you had flushed the data to the
> file...oh well.
>
Yes, that's a small disadavantage of using a "high-level" language,
where there's no flush available, and you assume it'll done
automatically ;-)

cheers,
Stef

Jason

1/23/2008 5:25:00 PM

0

On Jan 22, 3:22 pm, Stef Mientki <stef.mien...@gmail.com> wrote:
> Mike Driscoll wrote:
> > On Jan 22, 3:35 pm, Stef Mientki <stef.mien...@gmail.com> wrote:
>
> >> Mike Driscoll wrote:
>
> >>> On Jan 17, 3:56 pm, Stef Mientki <stef.mien...@gmail.com> wrote:
>
> >>>> hello,
>
> >>>> I've a program (not written in Python) that generates a few thousands
> >>>> bytes per second,
> >>>> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
> >>>> the files can be read by another program, to do further processing.
>
> >>>> A program written in VB or delphi can handle the data in the 2 buffers
> >>>> perfectly.
> >>>> Sometimes Python is also able to process the data correctly,
> >>>> but often it can't :-(
>
> >>>> I keep one of the files open en test the size of the open datafile each
> >>>> 50 msec.
> >>>> I have tried
> >>>> os.stat ( ....) [ ST_SIZE]
> >>>> os.path.getsize ( ... )
> >>>> but they both have the same behaviour, sometimes it works, and the data
> >>>> is collected each 50 .. 100 msec,
> >>>> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.
>
> >>>> I'm using python 2.4 on winXP.
>
> >>>> Is there a solution for this problem ?
>
> >>>> thanks,
> >>>> Stef Mientki
>
> >>> Tim Golden has a method to watch for changes in a directory on his
> >>> website:
>
> >>>http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_di......
>
> >>> This old post also mentions something similar:
>
> >>>http://mail.python.org/pipermail/python-list/2007-October/4...
>
> >>> And here's a cookbook recipe that claims to do it as well using
> >>> decorators:
>
> >>>http://aspn.activestate.com/ASPN/Cookbook/Python/Rec...
>
> >>> Hopefully that will get you going.
>
> >>> Mike
>
> >> thanks Mike,
> >> sorry for the late reaction.
> >> I've it working perfect now.
> >> After all, os.stat works perfectly well,
> >> the problem was in the program that generated the file with increasing
> >> size,
> >> by truncating it after each block write, it apperently garantees that
> >> the file is flushed to disk and all problems are solved.
>
> >> cheers,
> >> Stef Mientki
>
> > I almost asked if you were making sure you had flushed the data to the
> > file...oh well.
>
> Yes, that's a small disadavantage of using a "high-level" language,
> where there's no flush available, and you assume it'll done
> automatically ;-)
>
> cheers,
> Stef

Uhm, there is a flush method for Python's files. From "http://
docs.python.org/lib/bltin-file-objects.html":
flush()
Flush the internal buffer, like stdio's fflush(). This may
be a no-op on some file-like objects.

As for an example:

>>> import os
>>> f = open('vikings.txt', 'wb')
>>> os.stat('vikings.txt').st_size
0L
>>> f.write('Spam, spam, spam, spam! ' * 1000) # Bloody vikings...
>>> os.stat('vikings.txt').st_size
24576L
>>> f.flush()
>>> os.stat('vikings.txt').st_size
25000L
>>>

Is there something that I'm missing here?

--Jason

Stef Mientki

1/23/2008 6:17:00 PM

0


>>>> thanks Mike,
>>>> sorry for the late reaction.
>>>> I've it working perfect now.
>>>> After all, os.stat works perfectly well,
>>>> the problem was in the program that generated the file with increasing
>>>> size,
>>>> by truncating it after each block write, it apperently garantees that
>>>> the file is flushed to disk and all problems are solved.
>>>>
>>>> cheers,
>>>> Stef Mientki
>>>>
>>> I almost asked if you were making sure you had flushed the data to the
>>> file...oh well.
>>>
>> Yes, that's a small disadavantage of using a "high-level" language,
>> where there's no flush available, and you assume it'll done
>> automatically ;-)
>>
>> cheers,
>> Stef
>>
>
> Uhm, there is a flush method for Python's files. From "http://
> docs.python.org/lib/bltin-file-objects.html":
> flush()
> Flush the internal buffer, like stdio's fflush(). This may
> be a no-op on some file-like objects.
>
> As for an example:
>
>
>>>> import os
>>>> f = open('vikings.txt', 'wb')
>>>> os.stat('vikings.txt').st_size
>>>>
> 0L
>
>>>> f.write('Spam, spam, spam, spam! ' * 1000) # Bloody vikings...
>>>> os.stat('vikings.txt').st_size
>>>>
> 24576L
>
>>>> f.flush()
>>>> os.stat('vikings.txt').st_size
>>>>
> 25000L
>
>
> Is there something that I'm missing here?
>
>
hi Jason,
I was talking about a "high-level" language, in which the sending
program was written,
(Delphi, not about Python ;-)
cheers,
Stef

Gabriel Genellina

1/23/2008 10:07:00 PM

0

En Wed, 23 Jan 2008 16:16:58 -0200, Stef Mientki <stef.mientki@gmail.com>
escribió:

>>> Yes, that's a small disadavantage of using a "high-level" language,
>>> where there's no flush available, and you assume it'll done
>>> automatically ;-)
>>
>> Uhm, there is a flush method for Python's files. From "http://
>>
> I was talking about a "high-level" language, in which the sending
> program was written,
> (Delphi, not about Python ;-)

In Delphi, flush(filevar) does work. Or are you using a TFileStream or
similar?

--
Gabriel Genellina

Stef Mientki

1/24/2008 3:30:00 PM

0

Gabriel Genellina wrote:
> En Wed, 23 Jan 2008 16:16:58 -0200, Stef Mientki
> <stef.mientki@gmail.com> escribió:
>
>>>> Yes, that's a small disadavantage of using a "high-level" language,
>>>> where there's no flush available, and you assume it'll done
>>>> automatically ;-)
>>>
>>> Uhm, there is a flush method for Python's files. From "http://
>>>
>> I was talking about a "high-level" language, in which the sending
>> program was written,
>> (Delphi, not about Python ;-)
>
> In Delphi, flush(filevar) does work.
flush is only valid for textfiles, not for binary files (at least in D7)

anyway thanks, cheers,
Stef