Asp Forum - Re: Open a List of Files

Fredrik Lundh

1/8/2008 12:17:00 PM

BJ Swope wrote:

> the code looks ok. please define "not working".
>
> Yep, defining "not working" is always helpful! :)
>
> I want to have all 3 files open at the same time. I will write to each
> of the files later in my script but just the last file is open for writing.

to keep more than one file open, you need more than one variable, or a
variable that can hold more than one object.

if the number of files may vary, put the file objects in a list. if the
number of files is constant, you might as well just use three variables
and call open three times; e.g.

def getfilename(name):
return os.path.join(Host_Path, name)

messages_file = open(getfilename("messages"), "w")
recipients_file = open(getfilename("recipients"), "w")
viruses_file = open(getfilename("viruses"), "w")

</F>

11 Answers

Hrvoje Niksic

1/8/2008 12:22:00 PM

Fredrik Lundh <fredrik@pythonware.com> writes:

> BJ Swope wrote:
>
>> the code looks ok. please define "not working".
>>
>> Yep, defining "not working" is always helpful! :)
>>
>> I want to have all 3 files open at the same time. I will write to
>> each of the files later in my script but just the last file is open
>> for writing.
>
> to keep more than one file open, you need more than one variable, or a
> variable that can hold more than one object.
>
> if the number of files may vary, put the file objects in a list.

Or in a dict:

open_files = {}
for fn in ['messages', 'recipients', 'viruses']:
open_files[fn] = open(getfilename(fn), 'w')

# open_files['messages'] is the open file 'messages' etc.

Terry Jones

1/9/2008 2:34:00 AM

Hi BJ

> > Fredrik Lundh <fredrik@pythonware.com> writes:
> > Or in a dict:
> >
> > open_files = {}
> > for fn in ['messages', 'recipients', 'viruses']:
> > open_files[fn] = open(getfilename(fn), 'w')
>
> I decided that I was just trying to be "too smooth by 1/2" so I fell back
> to ...
>
> messages = open(os.path.join(host_path,'messages.txt'), 'wb')
> deliveries = open(os.path.join(host_path,'deliveries.txt'), 'wb')
> actions = open(os.path.join(host_path,'actions.txt'), 'wb')
> parts = open(os.path.join(host_path,'parts.txt'), 'wb')
> recipients = open(os.path.join(host_path,'recipients.txt'), 'wb')
> viruses = open(os.path.join(host_path,'viruses.txt'), 'wb')
> esp_scores = open(os.path.join(host_path,'esp_scores.txt'), 'wb')

I think you should revisit this decision. Something like Fredrik's code is
the way to go. It has multiple advantages:

- It's much shorter.
- It's arguably easier to add/remove to/from.
- It has less risk of error (much less repetition).
- It allows your code to later take a string file tag and
write to that file by looking up its file descriptor in the dict.
- You can close all open files with a trivial loop.

Also, if you start writing code like Fredrik's instead of like what you
fell back on you'll make yourself a better programmer (in general, not just
in Python).

Terry

Tim Chase

1/9/2008 2:41:00 AM

> I decided that I was just trying to be "too smooth by 1/2" so I fell back to
>
> messages = open(os.path.join(host_path,'messages.txt'), 'wb')
> deliveries = open(os.path.join(host_path,'deliveries.txt'), 'wb')
> actions = open(os.path.join(host_path,'actions.txt'), 'wb')
> parts = open(os.path.join(host_path,'parts.txt'), 'wb')
> recipients = open(os.path.join(host_path,'recipients.txt'), 'wb')
> viruses = open(os.path.join(host_path,'viruses.txt'), 'wb')
> esp_scores = open(os.path.join(host_path,'esp_scores.txt'), 'wb')

Another way to write this which reduces some of the code would be

filenames = ['messages', 'deliveries', 'actions', 'parts',
'recipients', 'viruses', 'esp_scores']

(messages, deliveries, actions, parts,
recipients, viruses, esp_scores) = [
open(os.path.join(host_path, '%s.txt' % fn), 'wb')
for fn in filenames
]

It becomes even more clear if you make an intermediate "opener"
function such as

binwriter = lambda fname: open(
os.path.join(host_path, '%s.txt' % fname), 'wb')

(messages, deliveries, actions, parts,
recipients, viruses, esp_scores) = [
binwriter(fn) for fn in filenames]

This makes it easier to change intended functionality in one
place, rather than updating each line. Additionally, it's easy
to add a new entry (just add the filename in the filename list,
and the variable-name in the assignment tuple). It also makes it
clear that each one is getting the same treatment (that there's
not some outlier making you study matters carefully to figure out
that there isn't).

-tkc

Fredrik Lundh

1/9/2008 7:51:00 AM

Terry Jones wrote:

> I think you should revisit this decision. Something like Fredrik's code is
> the way to go.

He used my suggestion, just for a few more files than he had in his
original post.

Seriously, for a limited number of files, the dictionary approach is
mostly pointless; you end up replacing

foo = open("foo")
foo.write(...)

with

somedict["foo"] = open("foo")
somedict["foo"].write(...)

which, frankly, sucks in a whole lot of ways.

> It has multiple advantages:
>
> - It's much shorter.

For a small number of files, it isn't, really.

> - It's arguably easier to add/remove to/from.

Only if you're not using the files you're opening; as soon you try to do
something with them, it's more work.

> - It has less risk of error (much less repetition).

No, it hasn't. There's more to type when using the files, and you have
*two* things you can misspell; that is, you're replacing a NameError
with either a NameError or a KeyError.

> - It allows your code to later take a string file tag and
> write to that file by looking up its file descriptor in the dict.

Instead of allowing your code to take a file object and write to that
file by writing to that file object?

> - You can close all open files with a trivial loop.

Ok, this is actually an advantage. Not that you need to do that very
often, since Python does it for you. And if you find yourself needing
to do this a lot, you can of course stuff all the output files in a list
but *still* use the variables to refer to the file when writing to them.

:::

There is one case where a dictionary of files makes perfect sense, of
course, and that's when you can associate the file with some *other*
value that you're *already* using. (Say, a user name or a machine name
or a severity level or something like that.) But inventing your own
string tags to use instead of variables is just plain bad design.

</F>

Paul Hankin

1/9/2008 9:21:00 AM

On Jan 9, 2:41 am, Tim Chase <python.l...@tim.thechases.com> wrote:
> > I decided that I was just trying to be "too smooth by 1/2" so I fell back to
>
> > messages = open(os.path.join(host_path,'messages.txt'), 'wb')
> > deliveries = open(os.path.join(host_path,'deliveries.txt'), 'wb')
> > actions = open(os.path.join(host_path,'actions.txt'), 'wb')
> > parts = open(os.path.join(host_path,'parts.txt'), 'wb')
> > recipients = open(os.path.join(host_path,'recipients.txt'), 'wb')
> > viruses = open(os.path.join(host_path,'viruses.txt'), 'wb')
> > esp_scores = open(os.path.join(host_path,'esp_scores.txt'), 'wb')
>
> Another way to write this which reduces some of the code would be
>
> filenames = ['messages', 'deliveries', 'actions', 'parts',
> 'recipients', 'viruses', 'esp_scores']
>
> (messages, deliveries, actions, parts,
> recipients, viruses, esp_scores) = [
> open(os.path.join(host_path, '%s.txt' % fn), 'wb')
> for fn in filenames
> ]
>
> It becomes even more clear if you make an intermediate "opener"
> function such as
>
> binwriter = lambda fname: open(
> os.path.join(host_path, '%s.txt' % fname), 'wb')
>
> (messages, deliveries, actions, parts,
> recipients, viruses, esp_scores) = [
> binwriter(fn) for fn in filenames]

This can be more cleanly written using locals()

for fn in filenames:
locals()[fn] = open(os.path.join(host_path, fname + '.txt', 'wb')

--
Paul Hankin

Fredrik Lundh

1/9/2008 10:03:00 AM

Paul Hankin wrote:

> This can be more cleanly written using locals()
>
> for fn in filenames:
> locals()[fn] = open(os.path.join(host_path, fname + '.txt', 'wb')

from the reference manual:

locals()

Update and return a dictionary representing the current
local symbol table.

Warning: The contents of this dictionary should not be
modified; changes may not affect the values of local
variables used by the interpreter.

examples:

>>> def foo():
.... locals()["foo"] = 1
.... print foo
....
>>> foo()
<function foo at 0x00BE1FB0>

>>> def foo():
.... foo = 1
.... locals()["foo"] = 2
.... print foo
....
>>> foo()
1

</F>

Paul Hankin

1/9/2008 10:29:00 AM

On Jan 9, 10:02 am, Fredrik Lundh <fred...@pythonware.com> wrote:
> Paul Hankin wrote:
> > This can be more cleanly written using locals()
>
> > for fn in filenames:
> > locals()[fn] = open(os.path.join(host_path, fname + '.txt', 'wb')
>
> from the reference manual:
>
> locals()
>
> Update and return a dictionary representing the current
> local symbol table.
>
> Warning: The contents of this dictionary should not be
> modified; changes may not affect the values of local
> variables used by the interpreter.

Thanks Fredrik! I learnt something today.

I wonder if there's a reason why it doesn't raise an exception when
you try to write to it? That would seem better to me than having it
sometimes update variables and sometimes not.

--
Paul Hankin

Fredrik Lundh

1/9/2008 10:37:00 AM

Paul Hankin wrote:

> Thanks Fredrik! I learnt something today.
>
> I wonder if there's a reason why it doesn't raise an exception when
> you try to write to it? That would seem better to me than having it
> sometimes update variables and sometimes not.

probably because it returns a standard dictionary object, and Python's
dictionary objects are writable.

(and if someone wants to reopen the old discussion about how Python
absolutely definitely needs frozen dictionaries to be useful at all,
please do so in a new thread ;-)

</F>

Terry Jones

1/9/2008 2:25:00 PM

>>>>> "Fredrik" == Fredrik Lundh <fredrik@pythonware.com> writes:

Fredrik> Seriously, for a limited number of files, the dictionary approach
Fredrik> is mostly pointless; you end up replacing

Fredrik> foo = open("foo")
Fredrik> foo.write(...)

Fredrik> with

Fredrik> somedict["foo"] = open("foo")
Fredrik> somedict["foo"].write(...)

Yes, in some cases. But where you want to do multiple things you can always
pull the file descriptor of of the dict into a local var.

>> - It has less risk of error (much less repetition).

Fredrik> No, it hasn't. There's more to type when using the files, and you
Fredrik> have *two* things you can misspell; that is, you're replacing a
Fredrik> NameError with either a NameError or a Key Error.

Well I meant less error risk in the context of opening the files. And I'm
happy to get a NameError or KeyError if I make the mistake you describe.
But bad code like this:

messages = open(os.path.join(host_path,'messages.txt'), 'wb')
deliveries = open(os.path.join(host_path,'deliveries.txt'), 'wb')
actions = open(os.path.join(host_path,'deliveries.txt'), 'wb')

doesn't give an error at all, and with all that identical and
semi-identical code it's much less obvious where the error is.

Cut & paste errors like this are pretty common, and they can be quite hard
to find (when they don't result in exceptions), in my experience.

>> - It allows your code to later take a string file tag and
>> write to that file by looking up its file descriptor in the dict.

Fredrik> Instead of allowing your code to take a file object and write to that
Fredrik> file by writing to that file object?

No, see below:

>> - You can close all open files with a trivial loop.

Fredrik> Ok, this is actually an advantage. Not that you need to do that very
Fredrik> often, since Python does it for you.

Yes. That's partly why I said it would make him a better programmer in
general, not just in Python.

Fredrik> And if you find yourself needing to do this a lot, you can of
Fredrik> course stuff all the output files in a list but *still* use the
Fredrik> variables to refer to the file when writing to them.

Yes, as above.

Fredrik> There is one case where a dictionary of files makes perfect sense, of
Fredrik> course, and that's when you can associate the file with some *other*
Fredrik> value that you're *already* using. (Say, a user name or a machine name
Fredrik> or a severity level or something like that.)

Yes. That's what I meant by my original - badly worded - remark (see above)
that you could take a "string file tag" and use it to look up its file
descriptor.

Terry

Terry Jones

1/9/2008 2:33:00 PM

>>>>> "BJ" == BJ Swope <bigblueswope@gmail.com> writes:

I (at least) think the code looks much nicer.

BJ> #Referring to files to write in various places...
BJ> open_files['deliveries'].write(flat_line)
BJ> open_files['deliveries'].write('\n')

If you were doing a lot with the deliveries file at some point, you could
put the file descriptor into a local variable

deliveries = open_files['deliveries']
deliveries.write(flat_line)
deliveries.write('\n')

BJ> #And finally to close the opened files
BJ> for fn in open_files.keys():
BJ> open_files[fn].close()

You don't need "for fn in open_files.keys():", you can just use "for fn in
open_files:", but simpler than that is to just use the dictionary values:

for fn in open_files.values():
fn.close()

Regards,
Terry

comp.lang.python

Re: Open a List of Files

Fredrik Lundh

Hrvoje Niksic

Terry Jones

Tim Chase

Fredrik Lundh

Paul Hankin

Fredrik Lundh

Paul Hankin

Fredrik Lundh

Terry Jones

Terry Jones

x Login to ForumsZone