[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

app runs fine with interpreter, but not under py2exe

Doug Morse

3/12/2008 3:05:00 AM

Hi,

I have an application that runs just fine using the standard Python distro
interpreter (v2.5.1 on WinXP) but throws the following exception when run as
an executable built with py2exe. My questions are: (a) does anyone have any
thoughts on why this exception is occurring and what to do about it, and (b)
more generally, why would a problem like this occur under py2exe but not with
the standard distro?

Thanks in adavance for any and all help.

Cheers,
Doug


Traceback (most recent call last):
File "VisionTrainer.py", line 49, in <module>
File "SessionController.pyc", line 53, in <module>
File "VisionEgg\__init__.pyc", line 42, in <module>
File "VisionEgg\ParameterTypes.pyc", line 28, in <module>
File "Numeric.pyc", line 93, in <module>
File "Precision.pyc", line 26, in <module>
File "Precision.pyc", line 23, in _fill_table
File "Precision.pyc", line 18, in _get_precisions
TypeError: data type not understood
10 Answers

GHUM

3/12/2008 10:10:00 AM

0

Doug,
> File "VisionTrainer.py", line 49, in <module>
> File "SessionController.pyc", line 53, in <module>
> File "VisionEgg\__init__.pyc", line 42, in <module>
> File "VisionEgg\ParameterTypes.pyc", line 28, in <module>
> File "Numeric.pyc", line 93, in <module>
> File "Precision.pyc", line 26, in <module>
> File "Precision.pyc", line 23, in _fill_table
> File "Precision.pyc", line 18, in _get_precisions
> TypeError: data type not understood

to my knowledge, "data type not understood" is a message not from core
Python, but rather thrown from "your" code. What is happening around
Precision.py, line 18?

What does trigger that exception?

My guess is, you are dealing with some custom imported modules which
define "data type"s.
(PIL does something similiar for supported images)... that is, some
module is trying to import all definitions from a specific directory.

That "dynamic importing" fails within py2exe --- all the importing has
to be definit at py2exing-time. Please ready http://www.py2exe.org/index.cgi/PIL_...
and the other receipe on py2exe.org and try to adapt that knowledge to
your application.

Harald

Harald

Doug Morse

3/13/2008 2:03:00 AM

0

Hi Harald and C.L.P.,

Precision.py is part of the Numeric package. AFAIKT, the problem is during
the module initialization. The first lines of Precision.py are:

from multiarray import zeros
import string

typecodes = {'Character':'c', 'Integer':'1sil', 'UnsignedInteger':'bwu',
'Float':'fd', 'Complex':'FD'}

def _get_precisions(typecodes):
lst = []
for t in typecodes:
lst.append( (zeros( (1,), t ).itemsize()*8, t) ) <-- Line 18
return lst

def _fill_table(typecodes, table={}):
for key, value in typecodes.items():
table[key] = _get_precisions(value)
return table

_code_table = _fill_table(typecodes)


I can't find any reason why line 18 is throwing a "data type not understood"
error. It doesn't seem to have anything to do with dynamic importing, but I
can't be sure. I would note that "zeros" is a built-in function found in the
"python dll" multiarray.pyd (in the Numeric module directory).

Thanks again,
Doug


On Wed, 12 Mar 2008 03:09:59 -0700 (PDT), GHUM <haraldarminmassa@gmail.com>
wrote:
> to my knowledge, "data type not understood" is a message not from core
> Python, but rather thrown from "your" code. What is happening around
> Precision.py, line 18?
>
> What does trigger that exception?
>
> My guess is, you are dealing with some custom imported modules which
> define "data type"s.
> (PIL does something similiar for supported images)... that is, some
> module is trying to import all definitions from a specific directory.
>
> That "dynamic importing" fails within py2exe --- all the importing has
> to be definit at py2exing-time. Please ready http://www.p...
> index.cgi/PIL_and_py2exe
> and the other receipe on py2exe.org and try to adapt that knowledge to
> your application.

GHUM

3/13/2008 6:47:00 PM

0

Doug,

> Precision.py is part of the Numeric package. AFAIKT, the problem is during
> the module initialization. The first lines of Precision.py are:
>
> from multiarray import zeros
> import string
>
[.....]
and your program is crashing on the

lst.append( (zeros( (1,), t ).itemsize()*8, t) ) <-- Line 18

line... Please, try the following:

import multiarray
from multiarray import zeros
import string

(adding the line "import multiarray" before zeros are imported from
it)

I used this workaround with various packages I py2exed - maybe it also
works for you?

please keep us updated,

Harald

Doug Morse

3/14/2008 10:17:00 AM

0

Harald,

Great suggestion, thanks! Unfortunately, "no help there". Adding "import
multiarray" to Precision.py where you suggested (i.e., before the "from
multiarray import zeros" line) did not fix the problem. I'm getting the same
exception and traceback as before except, of course, on line 19 now instead of
line 18.

Doug


On Thu, 13 Mar 2008 11:46:41 -0700 (PDT), GHUM <haraldarminmassa@gmail.com>
wrote:
> Doug,
>
> > Precision.py is part of the Numeric package. AFAIKT, the problem is during
> > the module initialization. The first lines of Precision.py are:
> >
> > from multiarray import zeros
> > import string
> >
> [.....]
> and your program is crashing on the
>
> lst.append( (zeros( (1,), t ).itemsize()*8, t) ) <-- Line 18
>
> line... Please, try the following:
>
> import multiarray
> from multiarray import zeros
> import string
>
> (adding the line "import multiarray" before zeros are imported from
> it)
>
> I used this workaround with various packages I py2exed - maybe it also
> works for you?
>
> please keep us updated,
>
> Harald

Peter Otten

3/14/2008 11:39:00 AM

0

Doug Morse wrote:

> from multiarray import zeros
> import string
>
> typecodes = {'Character':'c', 'Integer':'1sil', 'UnsignedInteger':'bwu',
> 'Float':'fd', 'Complex':'FD'}
>
> def _get_precisions(typecodes):
>     lst = []
>     for t in typecodes:
>         lst.append( (zeros( (1,), t ).itemsize()*8, t) )   <-- Line 18
>     return lst
>
> def _fill_table(typecodes, table={}):
>     for key, value in typecodes.items():
>         table[key] = _get_precisions(value)
>     return table
>
> _code_table = _fill_table(typecodes)
>
> I can't find any reason why line 18 is throwing a "data type not
> understood" error.  It doesn't seem to have anything to do with dynamic
> importing, but I can't be sure.  I would note that "zeros" is a built-in
> function found in the "python dll" multiarray.pyd (in the Numeric module
> directory).

I don't know why, but your module seems to use numpy's multiarray instead of
Numeric's:

>>> from multiarray import zeros
>>> zeros((1,), "1")
array([0], '1')
>>> from numpy.core.multiarray import zeros
>>> zeros((1,), "1")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: data type not understood

Peter

Doug Morse

3/14/2008 2:38:00 PM

0

Peter,

Genius! You nailed it -- thanks!

py2exe is apparently getting confused by the fact that packages "Numeric" and
"numpy" both have files multiarray.pyd and umath.pyd. It copies just one of
each -- from $PYTHONHOME/Lib/site-packages/numpy/core -- and puts both of them
into the top-level of the created "dist" directory. Also, there are no such
files as multiarray.pyc and umath.pyc in my python installation, but py2exe
seems to create these (in the dist and dist/numpy/core directories) -- they
are small (e.g., 526 bytes) and I'm guessing that they are stubs that simply
point python to the matching .pyd that py2exe relocated.

So, by using the --skip-archive option to py2exe (i.e., python setup.py py2exe
--skip-archive) to create the dist and build directories, and then within the
dist directory hierarchy removing all the occurrances of multiarray.pyc and
umath.pyc, and then moving dist/multiarray.pyd and dist/umath.pyd to the
dist/numpy/core directory, and finally by copying multiarray.pyd and umath.pyd
from $PYTHONHOME/Lib/site-packages/Numeric to the dist directory, I am able to
get my application to run correctly and as a standalone executable. HOORAH!!!

Just for completeness and perhaps a bit of additional clarity, here's a
limited directory listing of what the steps in the previous paragraph produce:

morse> find dist | egrep "(multia|umath)" | xargs ls -l
-rwxr-xr-x 1 morse None 26624 Nov 28 2006 dist/multiarray.pyd
-rwxr-xr-x 1 morse None 348160 Nov 8 16:16 dist/numpy/core/multiarray.pyd
-rwxr-xr-x 1 morse None 192512 Nov 8 16:16 dist/numpy/core/umath.pyd
-rwxr-xr-x 1 morse None 54272 Nov 28 2006 dist/umath.pyd
morse>

(This is still WinXP; I'm just using the Cygwin bash and fileutils here.
Also, note that there are NO multiarray.pyc or umath.pyc files.)

So, my next step will be to try to not use the --skip-archive option and then
make these same modifications regarding multiarray.pyd and umath.pyd to the
py2exe-generated library.zip file and see if I can get things running that way
as well (and in so doing reduce my "dist" directory by about 10mg). I may
also try creating a dist/Numeric subdirectory and moving dist/multiarray.pyd
and dist/umath.pyd to this dist/Numeric subdirectory -- for the goal of more
accurately mirroring the actual file/directory structure found in
$PYTHONHOME/Lib/site-packages.

Again, thanks to everyone for their assistance, esp. Harald and Peter.

Is this something I should pursue getting the py2exe folks to fix / work with
them on fixing? In investigating this issue, I noted that a lot of other
py2exe problems seem to revolve around confusions when duplicate files exist
in different modules. I wouldn't thinking that getting py2exe to pay better
attention to the containing modules when building it's dependency tree would
be that difficult (or is it)?

Cheers,
Doug


On Fri, 14 Mar 2008 12:39:23 +0100, Peter Otten <__peter__@web.de> wrote:
> Doug Morse wrote:
>
> > from multiarray import zeros
> > import string
> >
> > typecodes = {'Character':'c', 'Integer':'1sil', 'UnsignedInteger':'bwu',
> > 'Float':'fd', 'Complex':'FD'}
> >
> > def _get_precisions(typecodes):
> >     lst = []
> >     for t in typecodes:
> >         lst.append( (zeros( (1,), t ).itemsize()*8, t) )   <-- Line 18
> >     return lst
> >
> > def _fill_table(typecodes, table={}):
> >     for key, value in typecodes.items():
> >         table[key] = _get_precisions(value)
> >     return table
> >
> > _code_table = _fill_table(typecodes)
> >
> > I can't find any reason why line 18 is throwing a "data type not
> > understood" error.  It doesn't seem to have anything to do with dynamic
> > importing, but I can't be sure.  I would note that "zeros" is a built-in
> > function found in the "python dll" multiarray.pyd (in the Numeric module
> > directory).
>
> I don't know why, but your module seems to use numpy's multiarray instead of
> Numeric's:
>
> >>> from multiarray import zeros
> >>> zeros((1,), "1")
> array([0], '1')
> >>> from numpy.core.multiarray import zeros
> >>> zeros((1,), "1")
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> TypeError: data type not understood
>
> Peter

Doug Morse

3/14/2008 3:04:00 PM

0

Hi,

Well, my attempt to not use the --skip-archive option didn't get very far,
as I quickly noticed that "library.zip" does NOT contain ANY .pyd files.
I'm guessing that they can't be in library.zip for a reason (i.e., they are
DLL files, essentially, and thus must be readily available to be loaded
into memory).

Is there a work-around for this, then? That is, is there a way to either
(a) tell py2exe how to *correctly* handle multiple multiarray.pyd and
umath.pyd files or (b) perhaps rename one set of the .pyd files -- say the
numpy/core versions -- to say multiarray2.pyd and umath2.pyd, and then
manual create the "stub"-like .pyc files that py2exe creates to point to
these alternate .pyd files and then place these stubs in
library.zip/numpy/core? Or am I just hoping for too much here and am going
to be stuck with using the --skip-archive option?

Thanks,
Doug


On Fri, 14 Mar 2008 14:37:32 +0000 (UTC), Doug Morse <morse@edoug.org> wrote:
> Peter,
>
> Genius! You nailed it -- thanks!
>
> py2exe is apparently getting confused by the fact that packages "Numeric"
> ...
> <snip>
>
> So, my next step will be to try to not use the --skip-archive option and
> then make these same modifications regarding multiarray.pyd and umath.pyd
> to the py2exe-generated library.zip file and see if I can get things
> running that way as well (and in so doing reduce my "dist" directory by
> about 10mg). I may also try creating a dist/Numeric subdirectory and
> moving dist/multiarray.pyd and dist/umath.pyd to this dist/Numeric
> subdirectory -- for the goal of more accurately mirroring the actual
> file/directory structure found in $PYTHONHOME/Lib/site-packages.
>
> <snip>

GHUM

3/16/2008 7:34:00 AM

0

Doug,

> as I quickly noticed that "library.zip" does NOT contain ANY .pyd files.
> I'm guessing that they can't be in library.zip for a reason (i.e., they are
> DLL files, essentially, and thus must be readily available to be loaded
> into memory).

.dll and .pyd files CAN be within library.zip and even within the
single-file-exe.

There is some Hellermagic within _memimporter.pyd that loads the .dll
and .pyd from a zipfile. When you scan the build_exe.py, you will
explicitely find:

print "*** finding dlls needed ***"
dlls = self.find_dlls(extensions)
self.plat_finalize(mf.modules, py_files, extensions, dlls)
dlls = [item for item in dlls
if os.path.basename(item).lower() not in
self.dll_excludes]
# should we filter self.other_depends in the same way?

> Is there a work-around for this, then?
-rwxr-xr-x 1 morse None 26624 Nov 28 2006 dist/multiarray.pyd
-rwxr-xr-x 1 morse None 348160 Nov 8 16:16 dist/numpy/core/
multiarray.pyd
-rwxr-xr-x 1 morse None 192512 Nov 8 16:16 dist/numpy/core/
umath.pyd
-rwxr-xr-x 1 morse None 54272 Nov 28 2006 dist/umath.pyd

let's try: your mission is, should you chose to accept it, to make a
py2exed application to load two different multiarray.pyd, who are on
the visible level only different by their place in the file system.

So, please make a minimal script:

import multiarray
import numpy.core.multiarray as ncmultiarray

and, using Peters test
>>> from multiarray import zeros
assert zeros((1,), "1") == array([0], '1')

and check, if those multiarrays are the correct ones. If yes, you can
try to build upon this by making all imports explicit as described
above. (Yes, I know, that's not necessary in plain python ... but it
saved me in some weird "import from .zip" and "import from database"
situations.)

> (a) tell py2exe how to *correctly* handle multiple multiarray.pyd and
> umath.pyd files

somewhere deep inside I have a BAAAD feeling about to files having the
same name but very very different content - but that's not your call
in this situation.
Are you able to build that libraries yourself? Then you could move
down and build the multiarray.pyd as ncmultiarray.pyd and adjust all
imports accordingly. (lots of work)

> manual create the "stub"-like .pyc files that py2exe creates to point to
> these alternate .pyd files and then place these stubs in
> library.zip/numpy/core?  

As much as I know there is only one central "loader stub", which
ensures that .dlls get imported from memory, see above. py2exe does
not create something like p2elma.py containing "import multiarray.pyd"

best luck, and please take your time to scan the py2exe.org wiki,
there are many similiar challanges with various packages, maybe you
get another workaround-idea (and, if we find one, PLEASE put it down
there for others to learn)

best wishes,

Harald

Doug Morse

3/16/2008 9:22:00 AM

0

Hi Harald,

Bond here, James Bond. I accepted your mission. :)

Unfortunately, the mission failed. Creating a "testapp.py" as you described
(i.e., with the suggested import statements) runs just fine with the
interpreter. again, however, py2exe does the same thing as the original
problem -- that is, copies numpy.core's versions of multiarray.pyd and
umath.pyd into dist/, creates what look like stub multiarray.pyc and umath.pyc
files in library.zip/ and library.zip/numpy/core/. when this py2exe version
is run, it throws the same (original) TypeError: data type not understood
error.

Also, with either my original program or testapp.py, I cannot seem to modify
library.zip in such a way as to get the program to run. Basically, I unzipped
library.zip, made the same modifications to it's contents as I did when using
the --skip-archive option (i.e., remove the stub files and put the right .pyd
files in the right locations) and then recreated the library.zip file.
Afterwards, running either origapp.exe or testapp.exe, the program bombs out
with ImportError: no module named multiarray.

In addition, I tried using py2exe to make a simple "Hello, world" program. It
runs just fine. Then, I "zip -m library.zip *.pyd" in the dist/ directory to
move the .pyd files into library.zip. Then, running hello.exe again works
just fine. However, if I add "import bz2; print bz2.__author__" after "print
'Hello, world!'" (the only line in hello.py), hello.exe bombs out with
"ImportError: DLL load failed: The specified module could not be found" when I
run it after having moved *.pyd into library.zip/. (bz2 was one of only two
..pyd files that py2exe put into dist/.)

So, I can seem to figure out how to get .pyd files into library.zip, even when
there are no name conflicts. What am I doing wrong?

Finally, regarding the stub files, if I use vi to edit the py2exe-generated,
536-byte bz2.pyc file (there is no such file in my python install directory
hierarchy), I see the word "import" very close to the word "bz2.pyd". This
suggests to me that py2exe might indeed be creating simple import statements
in a short .py file for each .pyd file and then compiling it into a .pyc stub
file -- or at least doing something similar. Of course I can't be sure of
this, but it seems likely to me.

Thanks again to you and everyone. I'll definitely visit the py2exe wiki and
see what I can come up with (and or course will report back with success /
failures). If you or anyone has any further thoughts based on this post, I'm
all ears and would be most grateful.

Regards,
Doug


On Sun, 16 Mar 2008 00:33:31 -0700 (PDT), GHUM <haraldarminmassa@gmail.com>
wrote:
> Doug,
>
> > as I quickly noticed that "library.zip" does NOT contain ANY .pyd files.
> > I'm guessing that they can't be in library.zip for a reason (i.e., they are
> > DLL files, essentially, and thus must be readily available to be loaded
> > into memory).
>
> .dll and .pyd files CAN be within library.zip and even within the
> single-file-exe.
>
> There is some Hellermagic within _memimporter.pyd that loads the .dll
> and .pyd from a zipfile. When you scan the build_exe.py, you will
> explicitely find:
>
> print "*** finding dlls needed ***"
> dlls = self.find_dlls(extensions)
> self.plat_finalize(mf.modules, py_files, extensions, dlls)
> dlls = [item for item in dlls
> if os.path.basename(item).lower() not in
> self.dll_excludes]
> # should we filter self.other_depends in the same way?
>
> > Is there a work-around for this, then?
> -rwxr-xr-x 1 morse None 26624 Nov 28 2006 dist/multiarray.pyd
> -rwxr-xr-x 1 morse None 348160 Nov 8 16:16 dist/numpy/core/
> multiarray.pyd
> -rwxr-xr-x 1 morse None 192512 Nov 8 16:16 dist/numpy/core/
> umath.pyd
> -rwxr-xr-x 1 morse None 54272 Nov 28 2006 dist/umath.pyd
>
> let's try: your mission is, should you chose to accept it, to make a
> py2exed application to load two different multiarray.pyd, who are on
> the visible level only different by their place in the file system.
>
> So, please make a minimal script:
>
> import multiarray
> import numpy.core.multiarray as ncmultiarray
>
> and, using Peters test
> >>> from multiarray import zeros
> assert zeros((1,), "1") == array([0], '1')
>
> and check, if those multiarrays are the correct ones. If yes, you can
> try to build upon this by making all imports explicit as described
> above. (Yes, I know, that's not necessary in plain python ... but it
> saved me in some weird "import from .zip" and "import from database"
> situations.)
>
> > (a) tell py2exe how to *correctly* handle multiple multiarray.pyd and
> > umath.pyd files
>
> somewhere deep inside I have a BAAAD feeling about to files having the
> same name but very very different content - but that's not your call
> in this situation.
> Are you able to build that libraries yourself? Then you could move
> down and build the multiarray.pyd as ncmultiarray.pyd and adjust all
> imports accordingly. (lots of work)
>
> > manual create the "stub"-like .pyc files that py2exe creates to point to
> > these alternate .pyd files and then place these stubs in
> > library.zip/numpy/core?  
>
> As much as I know there is only one central "loader stub", which
> ensures that .dlls get imported from memory, see above. py2exe does
> not create something like p2elma.py containing "import multiarray.pyd"
>
> best luck, and please take your time to scan the py2exe.org wiki,
> there are many similiar challanges with various packages, maybe you
> get another workaround-idea (and, if we find one, PLEASE put it down
> there for others to learn)
>
> best wishes,
>
> Harald

Thomas Heller

3/16/2008 11:02:00 AM

0

Doug Morse schrieb:
> Peter,
>
> Genius! You nailed it -- thanks!
>
> py2exe is apparently getting confused by the fact that packages "Numeric" and
> "numpy" both have files multiarray.pyd and umath.pyd. It copies just one of
> each -- from $PYTHONHOME/Lib/site-packages/numpy/core -- and puts both of them
> into the top-level of the created "dist" directory. Also, there are no such
> files as multiarray.pyc and umath.pyc in my python installation, but py2exe
> seems to create these (in the dist and dist/numpy/core directories) -- they
> are small (e.g., 526 bytes) and I'm guessing that they are stubs that simply
> point python to the matching .pyd that py2exe relocated.
[...]
>
> Just for completeness and perhaps a bit of additional clarity, here's a
> limited directory listing of what the steps in the previous paragraph produce:
>
> morse> find dist | egrep "(multia|umath)" | xargs ls -l
> -rwxr-xr-x 1 morse None 26624 Nov 28 2006 dist/multiarray.pyd
> -rwxr-xr-x 1 morse None 348160 Nov 8 16:16 dist/numpy/core/multiarray.pyd
> -rwxr-xr-x 1 morse None 192512 Nov 8 16:16 dist/numpy/core/umath.pyd
> -rwxr-xr-x 1 morse None 54272 Nov 28 2006 dist/umath.pyd
>
[...]
> Is this something I should pursue getting the py2exe folks to fix / work with
> them on fixing? In investigating this issue, I noted that a lot of other
> py2exe problems seem to revolve around confusions when duplicate files exist
> in different modules. I wouldn't thinking that getting py2exe to pay better
> attention to the containing modules when building it's dependency tree would
> be that difficult (or is it)?

I have the impression that this issue may already be fixed in py2exe CVS but
noone bothered to do a release. Here is the corresponding changelog entry:

* build_exe.py: Patch from Grant Edwards, slightly adjusted:
py2exe now renames the pyd files that it copies into the dist
directory so that they include the package name. This prevents
name conflicts.

I have created installers and uploaded them (for testing!) to the starship:

http://starship.python.net/crew/theller/py2exe-0.6.7.win32...
http://starship.python.net/crew/theller/py2exe-0.6.7.win32...
http://starship.python.net/crew/theller/py2exe-0.6.7.win32...

Please try them out.

Thanks,
Thomas