[lnkForumImage]
TotalShareware - Download Free Software

Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

Bizarre behavior with mutable default arguments

bukzor

12/29/2007 5:51:00 PM

I've found some bizzare behavior when using mutable values (lists,
dicts, etc) as the default argument of a function. I want to get the
community's feedback on this. It's easiest to explain with code.

This example is trivial and has design issues, but it demonstrates a
problem I've seen in production systems:

def main(argv = ['TESTING']):
'print out arguments with BEGIN and END'
argv.insert(1, "BEGIN")
argv.append("END")
for arg in argv: print arg

if __name__ == '__main__':
from sys import argv, exit
exit(main(argv))


Now lets try this out at the terminal:

>>> import example
>>> example.main()
BEGIN
TESTING
END
>>> example.main(["TESTING"])
BEGIN
TESTING
END
>>> example.main()
BEGIN
BEGIN
TESTING
END
END
>>> example.main(["TESTING"])
BEGIN
TESTING
END

The function does different things if you call it with ["TESTING"] as
the argument, even though that is identical to the default value!! It
seems that default values are only allocated once. If the default
value is mutable and is changed during the function's execution, this
has the side effect of making the default value change on each
subsequent execution.

Is this functionality intended? It seems very unintuitive. This has
caused a bug in my programs twice so far, and both times I was
completely mystified until I realized that the default value was
changing.

I'd like to draw up a PEP to remove this from py3k, if I can get some
community backing.


--Buck
36 Answers

Martin v. Loewis

12/29/2007 6:12:00 PM

0

> Is this functionality intended?

Google for "Python mutable default arguments" (you can actually
leave out Python).

It's part of the language semantics, yes.

Regards,
Martin

Istvan Albert

12/29/2007 6:17:00 PM

0

On Dec 29, 12:50 pm, bukzor <workithar...@gmail.com> wrote:

> Is this functionality intended? It seems very unintuitive. This has
> caused a bug in my programs twice so far, and both times I was
> completely mystified until I realized that the default value was
> changing.

it is only unintuitive when you do not know about it

once you realize how it works and what it does it can actually be very
useful

i.

Istvan Albert

12/29/2007 6:50:00 PM

0

On Dec 29, 1:11 pm, "Martin v. Löwis" <mar...@v.loewis.de> wrote:

> Google for "Python mutable default arguments"

and a mere 30 minutes later this thread is already one of the results
that come up

bukzor

12/29/2007 7:15:00 PM

0

Here's the answer to the question:
http://www.python.org/doc/faq/general/#why-are-default-values-shared-betwe...

It looks like Guido disagrees with me, so the discussion is closed.




For the record, I still think the following would be an improvement to
py3k:

In python25:
def f(a=None):
if a is None: a = []
...

In py3k becomes:
def f(a=[])
...


In python25 (this function from the FAQ linked above):
def f(a, _cache={}):
# Callers will never provide a third parameter for this function.
(then why is it an argument?)
...

In py3k becomes:
_cache = {}
def f(a):
global _cache
...



This follows the "explicit is better" and "one best way" principles of
Python, and greatly improves the intuitiveness. Also since the first
example is much more common, it reduces the overall verbosity of the
language.

Just my parting two cents,
--Buck

Steven D'Aprano

12/30/2007 12:28:00 AM

0

On Sat, 29 Dec 2007 11:14:30 -0800, bukzor wrote:


> In python25 (this function from the FAQ linked above):
> def f(a, _cache={}):
> # Callers will never provide a third parameter for this function.
> (then why is it an argument?)

The caller might want to provide it's own pre-prepared cache. Say, for
testing.

I think that this behaviour is a little unintuitive, and by a little I
mean a lot. Nevertheless, I am used to it, and I don't see any reason to
change it. There's very little about programming that is intuitive --
there's no intuitive reason to think that dictionary lookups are O(1)
while list lookups are O(n).

In the absence of a better solution, I'm very comfortable with keeping
the behaviour as is. Unfortunately, there's no good solution in Python to
providing functions with local storage that persists across calls to the
function:


(1) Use a global variable.

cache = {}
def foo():
global cache
print cache


(2) Use a function attribute.

def foo():
print foo.cache
foo.cache = {}


def foo():
try:
foo.cache
except AttributeError:
foo.cache = {}
print foo.cache



(3) Use an argument that isn't actually an argument.

def foo(cache={}):
print cache


#1, the global variable, is probably the worst solution of the lot.
Global variables are rightly Considered Harmful.


#2 has the disadvantages that you initialize the value *after* you write
the code that relies on it. Either that, or you waste time on every call
checking to see it if has been initialized. Also, like recursive
functions, it is difficult to rename the function.


#3 is, I think, the least-worse solution, but I would hardly call it
ideal.



> _cache = {}
> def f(a):
> global _cache
> ...
>
> This follows the "explicit is better" and "one best way" principles of
> Python,

Declaring an argument is equally explicit.

And you are confused -- the Zen doesn't say "one best way". People so
often get it wrong.

The Zen says:

"There should be one-- and PREFERABLY only one --OBVIOUS way to do it."
(Emphasis added.)

At least you're not saying "there should be only one way to do it". I
give you credit for that!


> and greatly improves the intuitiveness. Also since the first
> example is much more common, it reduces the overall verbosity of the
> language.

I question that it is "much more common". How do you know? Where's your
data?



--
Steven

Steven D'Aprano

12/30/2007 12:44:00 AM

0

On Sat, 29 Dec 2007 09:50:53 -0800, bukzor wrote:

> I've found some bizzare behavior when using mutable values (lists,
> dicts, etc) as the default argument of a function.

This FAQ is so Frequently Asked that I sometimes wonder if Python should,
by default, print a warning when it compiles a function with a list or
dict as as default value.

There's precedence for such a thing: the sum() built-in (un)helpfully
raises an exception if you try to use it on strings.

I say unhelpfully because the one time I wanted to use sum() on strings
was specifically to demonstrate the difference between O(n**2) behaviour
and O(n). I was quite put out that Python, which normally allows you to
shoot yourself in the foot if you insist, was so unnecessarily protective
in this case. Give me a warning, if you wish, but don't stop me.



--
Steven

bukzor

12/30/2007 3:50:00 AM

0

> I think that this behaviour is a little unintuitive, and by a little I
> mean a lot.

Thanks for acknowledging it.

> I question that it is "much more common". How do you know? Where's your
> data?

I did a dumb grep of my Python25/Lib folder and found 33 occurances of
the first pattern above. (Use None as the default value, then check
for None and assign empty list/dict)

Although I spent at least double the amount of time looking for the
second pattern, I found no occurances. (Use dict/list as default value
and modify it in place.) Every single function that used a list or
dict as a default value treated these variables as read-only. However,
I did find two new ways to accomplish the above (further violating the
Zen).

/c/Python25/Lib/site-packages/wx-2.8-msw-ansi/wx/lib/
customtreectrl.py:
def FillArray(self, item, array=[]):
if not array:
array = []

/c/Python25/Lib/site-packages/wx-2.8-msw-ansi/wx/lib/floatcanvas/
FloatCanvas.py:
def __init__(self, ObjectList=[], InForeground = False, IsVisible =
True):
self.ObjectList = list(ObjectList)

--Buck

bukzor

12/30/2007 4:21:00 AM

0

Just for completeness, the mutable default value problem also affects
classes:

class c:
def __init__(self, list = []):
self.list = list
self.list.append("LIST END")
def __repr__(self):
return "<Class a: %s>" % self.list

>>> import example2
>>> print example2.c()
<Class a: ['LIST END']>
>>> print example2.c([])
<Class a: ['LIST END']>
>>> print example2.c()
<Class a: ['LIST END', 'LIST END']>
>>> print example2.c([])
<Class a: ['LIST END']>

Again, we get different results if we supply an argument that is
identical to the default value. There are many instances in the
standard library where class values are assigned directly from the
initializer, which has list or dict default values, so there is chance
for errors cropping up here.

The error scenario is this:
1. Use a mutable value as default value in a class constructor.
2. Assign class property from constructor arguments.
3. Instantiate class using default value.
4. Modify class property in place.
5. Instantiate (again) class using default value.

The second instance will behave strangely because data from the first
instance has leaked over. The standard library is not affected because
it avoids one of these five steps. Most classes simply don't have
mutable default values (1). Those that do generally treat them as read-
only (4). Some classes are not useful using the default values (3).
Some classes are not useful to be instantiated twice (5). The classes
that don't avoid the problem at one of these four steps have to avoid
it at (2) by using one of the three above patterns.

--Buck

John Machin

12/30/2007 5:17:00 AM

0

On Dec 30, 3:21 pm, bukzor <workithar...@gmail.com> wrote:
> Just for completeness, the mutable default value problem also affects
> classes:

Simply, because methods are functions, and can have default arguments.
You don't need to nail *another* zillion theses to the cathedral
door :-)

thebjorn

12/30/2007 10:24:00 AM

0

On Dec 29, 7:17 pm, Istvan Albert <istvan.alb...@gmail.com> wrote:
> On Dec 29, 12:50 pm, bukzor <workithar...@gmail.com> wrote:
>
> > Is this functionality intended? It seems very unintuitive. This has
> > caused a bug in my programs twice so far, and both times I was
> > completely mystified until I realized that the default value was
> > changing.
>
> it is only unintuitive when you do not know about it
>
> once you realize how it works and what it does it can actually be very
> useful
>
> i.

I agree it is a potentially useful feature, yet it can still bite you
even after a decade of Python... Scenario: long running server
process, Bug report: "people aren't getting older", Code:

def age(dob, today=datetime.date.today()):
...

None of my unit tests caught that one :-)

-- bjorn