Asp Forum - Decorators and buffer flushing

Ethan Metsger

2/28/2008 5:50:00 PM

Hi, all.

I apologize for what is perhaps a newb question. I'm in the process of =
=

transitioning our testing framework from Perl to Python. While that alo=
ne =

probably sets off some red flags, I'm afraid it's what I'm stuck with.

I'm modeling a test with five operations: build, execute, validate, =

publish, and clean. The main loop might look something like this:

with Test(...) as t:
t.build()
t.execute()
t.validate()
t.publish()

At each run, I want to output a '.' to denote completion of that test =

step. I've been able to do this, sort of, using the following decorator=
=

(edited for brevity):

def report(f):

def new(self):
stat =3D f(self);

if stat is True:
sys.stdout.write ('.')

else:
...

sys.stdout.flush()

return new

(Each one of the test functions returns True or False depending on its =

completion status.)

The problem is that rather than outputting a period after each step is =

completed, it outputs all four of them at once. Instead of seeing =

progress as it happens, I get it when it's finished, even though I'm =

flushing the output buffer.

Any thoughts?

Best,

Ethan (emetsger@obj-sys.com)
http://uppertank.n...

5 Answers

Aaron Brady

2/28/2008 7:49:00 PM

On Feb 28, 11:49 am, "Ethan Metsger" <emets...@obj-sys.com> wrote:
> Hi, all.
>
> I apologize for what is perhaps a newb question. I'm in the process of
> transitioning our testing framework from Perl to Python. While that alone
> probably sets off some red flags, I'm afraid it's what I'm stuck with.
>
> I'm modeling a test with five operations: build, execute, validate,
> publish, and clean. The main loop might look something like this:
>
> with Test(...) as t:
> t.build()
> t.execute()
> t.validate()
> t.publish()
>
> At each run, I want to output a '.' to denote completion of that test
> step. I've been able to do this, sort of, using the following decorator
> (edited for brevity):
>
> def report(f):
>
> def new(self):
> stat = f(self);
>
> if stat is True:
> sys.stdout.write ('.')
>
> else:
> ...
>
> sys.stdout.flush()
>
> return new
>
> (Each one of the test functions returns True or False depending on its
> completion status.)
>
> The problem is that rather than outputting a period after each step is
> completed, it outputs all four of them at once. Instead of seeing
> progress as it happens, I get it when it's finished, even though I'm
> flushing the output buffer.
>
> Any thoughts?
>
> Best,
>
> Ethan (emets...@obj-sys.com)http://uppertank.n...

Can you reproduce the bug on the console? Or, try starting with a new
line in the output. Or, try a input().

Ethan Metsger

2/28/2008 8:05:00 PM

On Thu, 28 Feb 2008 14:48:55 -0500, <castironpi@gmail.com> wrote:

> Can you reproduce the bug on the console? Or, try starting with a new=

> line in the output. Or, try a input().

I can reproduce the issue in the console. I'm not convinced it's actual=
ly =

a bug, unless for some reason the interpreter is preventing a buffer flu=
sh.

The new line in the output also doesn't work (and isn't desirable anyway=
), =

and neither does adding input.

Could the problem be that I'm using a with statement? I have output =

flushed in __enter__:

def __enter__(self):
"""Ensures that the test begins appropriately.

In particular, this method will enter the directory specified in =
the
instance."""

self.pardir =3D os.getcwd()
os.chdir(self.path)

sys.stdout.write ("%s" % (self.name.ljust(30),))
sys.stdout.flush()

return self

Is it possible that flushing is prohibited until __exit__ is called?

Best,

Ethan (emetsger@obj-sys.com)
http://uppertank.n...

Gabriel Genellina

2/28/2008 11:53:00 PM

En Thu, 28 Feb 2008 18:04:38 -0200, Ethan Metsger <emetsger@obj-sys.com>
escribió:

> On Thu, 28 Feb 2008 14:48:55 -0500, <castironpi@gmail.com> wrote:
>
>> Can you reproduce the bug on the console? Or, try starting with a new
>> line in the output. Or, try a input().
>
> I can reproduce the issue in the console. I'm not convinced it's
> actually
> a bug, unless for some reason the interpreter is preventing a buffer
> flush.

Try starting the interpreter with python -u xxx.py (-u = unbuffered) but I
don't really think this might be the cause. Are you sure the code actually
says sys.stdout.flush() and not sys.stdout.flush?

> sys.stdout.write ("%s" % (self.name.ljust(30),))

Usually written as:
sys.stdout.write("%-30s" % self.name)

And instead of:

if xxx is True:
...

use:

if xxx:
...

> Is it possible that flushing is prohibited until __exit__ is called?

I don't think so... What's your platform/OS?
Try to post a minimal example showing the problem - copy & paste it, don't
retype. Maybe in the process of reducing your code to find the minimal
example, you find the problem yourself.

Are you aware of the existing framework for unit tests? the unittest
module?

--
Gabriel Genellina

Ethan Metsger

2/29/2008 3:30:00 PM

On Thu, 28 Feb 2008 15:04:38 -0500, Ethan Metsger <emetsger@obj-sys.com>=
=

wrote:

>
> I can reproduce the issue in the console. I'm not convinced it's =

> actually
> a bug, unless for some reason the interpreter is preventing a buffer =

> flush.

Quick question.

Having eliminated some of the other variables in my code, I'm wondering =
if =

the following might cause a problem flushing the stdout buffer:

proc =3D Popen (['ant'] + self.args, stdout=3DPIPE, stderr=3DPIPE)
(out, err) =3D proc.communicate()

Is it possible that the Popen object has somehow mangled things?

For what it's worth, I'm also passing '-u' to the interpreter with no =

effect.

Best,

Ethan (emetsger@obj-sys.com)
http://uppertank.n...

Ethan Metsger

3/3/2008 5:06:00 PM

Hi, Gabriel. I missed this message initially; I apologize for not =

responding sooner.

On Thu, 28 Feb 2008 18:53:28 -0500, Gabriel Genellina =

<gagsl-py2@yahoo.com.ar> wrote:

>> I can reproduce the issue in the console. I'm not convinced it's =

>> actually
>> a bug, unless for some reason the interpreter is preventing a buffer =
=

>> flush.
>
> Try starting the interpreter with python -u xxx.py (-u =3D unbuffered)=
but =

> I don't really think this might be the cause. Are you sure the code =

> actually says sys.stdout.flush() and not sys.stdout.flush?

As I mentioned in another message, I haven't had any luck with adding '-=
u' =

to the invocation of the interpreter. I am sure I'm not looking at the =
=

callable object rather than calling flush().

>> sys.stdout.write ("%s" % (self.name.ljust(30),))
>
> Usually written as:
> sys.stdout.write("%-30s" % self.name)

Thanks for the tips!

[...]

>> Is it possible that flushing is prohibited until __exit__ is called?
>
> I don't think so... What's your platform/OS?

I'm running Ubuntu Feisty with Python 2.5.1:

Python 2.5.1 (r251:54863, May 2 2007, 16:56:35)
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2

> Try to post a minimal example showing the problem - copy & paste it, =

> don't retype. Maybe in the process of reducing your code to find the =

> minimal example, you find the problem yourself.

The minimal example is a little long, but I think in the end I figured o=
ut =

what was happening. The initial build step took the longest amount of =

time to complete, but the other steps were nearly instantaneous.

So the post-build '.' would appear, and the very fast run of the other =

steps made it appear as though everything was being buffered when it =

wasn't. Adding in a short delay (time.sleep(0.05) seemed to work) helpe=
d =

demonstrate that the data actually were unbuffered.

> Are you aware of the existing framework for unit tests? the unittest =

> module?

Yes. I haven't investigated its uses in this context due to the =

constraints of the legacy system and general inertia. I'm trying to =

duplicate functionality while addressing certain annoyances with the =

previous system.

Thanks again for your help!

Best,

Ethan

-- =

Ethan Metsger
http://uppertank.n...

comp.lang.python

Decorators and buffer flushing

Ethan Metsger

Aaron Brady

Ethan Metsger

Gabriel Genellina

Ethan Metsger

Ethan Metsger

x Login to ForumsZone