Asp Forum - Re: python crash on windows but not on linux

Peter Otten

2/12/2010 2:17:00 PM

hjebbers wrote:

> On Feb 11, 7:01 pm, Peter Otten <__pete...@web.de> wrote:
>> hjebbers wrote:
>> > On Feb 11, 5:45 pm, M3RT <mgul...@gmail.com> wrote:
>> >> The problem may be related to how you treat the EDI file or lets say
>> >> DATA. Also your coding style is important. Can you provide more info?
>>
>> > Yes, a whole lot more; but I do not want to bother you with that now.
>> > I was just wondering if it is possible that the same setup gives a
>> > CRASH! on windows and just works on linux.
>> > (where is the bug?)
>>
>> In the platform-specific code ;)
>>
>> Even on alt.haruspicy they cannot do much without a liver now and then...
>
> if it is in the platform-specific code should I make this into a
> bugreport?

A this stage a bug report would be useless. The details you have provided so
far don't give anyone without physical access to your machine a chance to
reproduce the error.

> (because the crash does not appear at the same place in my code (when
> doing the same test-run) it will be quite hard to point down....)

Indeed. The culprit may be an extension written in C that is overwriting
memory still in use by other parts of your program.

Peter

13 Answers

henk-jan ebbers

2/12/2010 5:21:00 PM

On Feb 12, 3:17 pm, Peter Otten <__pete...@web.de> wrote:
> hjebbers wrote:
> > On Feb 11, 7:01 pm, Peter Otten <__pete...@web.de> wrote:
> >> hjebbers wrote:
> >> > On Feb 11, 5:45 pm, M3RT <mgul...@gmail.com> wrote:
> >> >> The problem may be related to how you treat the EDI file or lets say
> >> >> DATA. Also your coding style is important. Can you provide more info?
>
> >> > Yes, a whole lot more; but I do not want to bother you with that now.
> >> > I was just wondering if it is possible that the same setup gives a
> >> > CRASH! on windows and just works on linux.
> >> > (where is the bug?)
>
> >> In the platform-specific code ;)
>
> >> Even on alt.haruspicy they cannot do much without a liver now and then....
>
> > if it is in the platform-specific code should I make this into a
> > bugreport?
>
> A this stage a bug report would be useless. The details you have provided so
> far don't give anyone without physical access to your machine a chance to
> reproduce the error.
>
> > (because the crash does not appear at the same place in my code (when
> > doing the same test-run) it will be quite hard to point down....)
>
> Indeed. The culprit may be an extension written in C that is overwriting
> memory still in use by other parts of your program.
>
> Peter

No C extensions used. Definitely not.

As these crashes occur during stress tests, I go to the limits of the
memory available.
What strikes me is:
1. the crash on windows, but linux works OK (same test sets)
2. the linux box has 750Mb RAM, the windows box has 1.5Gb (twice as
much).

kind regards, Henk-jan

Dennis Lee Bieber

2/13/2010 9:26:00 AM

On Fri, 12 Feb 2010 09:21:07 -0800 (PST), hjebbers <hjebbers@gmail.com>
declaimed the following in gmane.comp.python.general:

> What strikes me is:
> 1. the crash on windows, but linux works OK (same test sets)
> 2. the linux box has 750Mb RAM, the windows box has 1.5Gb (twice as
> much).
>
Which on its own does not mean much.

Windows in a normal installation only grants 2GB address space to
user code, reserving the other 2GB space for the OS shared libraries. If
your program attempts to allocate over that, it will fail. That the
Windows box has twice the physical memory only means it doesn't resort
to page swapping as soon.

There is a boot parameter switch that toggles Windows into a 3GB
user/1GB OS mode -- it is mainly meant for server machines where there
won't be many disjoint OS libraries loaded, but the server applications
need lots of data space.

What split does the Linux OS use? If it give 3GB to user space,
while you'd start to page swap much soon, you'd also have 50% more
virtual memory that can be allocated than under Windows.
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfraed@ix.netcom.com HTTP://wlfraed.home.netcom.com/

Peter Otten

2/13/2010 10:03:00 AM

hjebbers wrote:

> On Feb 12, 3:17 pm, Peter Otten <__pete...@web.de> wrote:
>> hjebbers wrote:
>> > On Feb 11, 7:01 pm, Peter Otten <__pete...@web.de> wrote:
>> >> hjebbers wrote:
>> >> > On Feb 11, 5:45 pm, M3RT <mgul...@gmail.com> wrote:
>> >> >> The problem may be related to how you treat the EDI file or lets
>> >> >> say DATA. Also your coding style is important. Can you provide more
>> >> >> info?
>>
>> >> > Yes, a whole lot more; but I do not want to bother you with that
>> >> > now. I was just wondering if it is possible that the same setup
>> >> > gives a CRASH! on windows and just works on linux.
>> >> > (where is the bug?)
>>
>> >> In the platform-specific code ;)
>>
>> >> Even on alt.haruspicy they cannot do much without a liver now and
>> >> then...
>>
>> > if it is in the platform-specific code should I make this into a
>> > bugreport?
>>
>> A this stage a bug report would be useless. The details you have provided
>> so far don't give anyone without physical access to your machine a chance
>> to reproduce the error.
>>
>> > (because the crash does not appear at the same place in my code (when
>> > doing the same test-run) it will be quite hard to point down....)
>>
>> Indeed. The culprit may be an extension written in C that is overwriting
>> memory still in use by other parts of your program.
>>
>> Peter
>
> No C extensions used. Definitely not.
>
> As these crashes occur during stress tests, I go to the limits of the
> memory available.
> What strikes me is:
> 1. the crash on windows, but linux works OK (same test sets)
> 2. the linux box has 750Mb RAM, the windows box has 1.5Gb (twice as
> much).

Random thoughts:

If there were not enough memory Python should give a MemoryError rather than
segfault.

Do both systems use the same memory layout (i. e. 32 or 64bit OS, 16 or 32
bit unicode)?

Try an alpha of Python 2.7. If you're lucky "your" bug was found 'n' fixed
independently.

Peter

henk-jan ebbers

2/13/2010 10:49:00 AM

On Feb 13, 11:03 am, Peter Otten <__pete...@web.de> wrote:
> hjebbers wrote:
> > On Feb 12, 3:17 pm, Peter Otten <__pete...@web.de> wrote:
> >> hjebbers wrote:
> >> > On Feb 11, 7:01 pm, Peter Otten <__pete...@web.de> wrote:
> >> >> hjebbers wrote:
> >> >> > On Feb 11, 5:45 pm, M3RT <mgul...@gmail.com> wrote:
> >> >> >> The problem may be related to how you treat the EDI file or lets
> >> >> >> say DATA. Also your coding style is important. Can you provide more
> >> >> >> info?
>
> >> >> > Yes, a whole lot more; but I do not want to bother you with that
> >> >> > now. I was just wondering if it is possible that the same setup
> >> >> > gives a CRASH! on windows and just works on linux.
> >> >> > (where is the bug?)
>
> >> >> In the platform-specific code ;)
>
> >> >> Even on alt.haruspicy they cannot do much without a liver now and
> >> >> then...
>
> >> > if it is in the platform-specific code should I make this into a
> >> > bugreport?
>
> >> A this stage a bug report would be useless. The details you have provided
> >> so far don't give anyone without physical access to your machine a chance
> >> to reproduce the error.
>
> >> > (because the crash does not appear at the same place in my code (when
> >> > doing the same test-run) it will be quite hard to point down....)
>
> >> Indeed. The culprit may be an extension written in C that is overwriting
> >> memory still in use by other parts of your program.
>
> >> Peter
>
> > No C extensions used. Definitely not.
>
> > As these crashes occur during stress tests, I go to the limits of the
> > memory available.
> > What strikes me is:
> > 1. the crash on windows, but linux works OK (same test sets)
> > 2. the linux box has 750Mb RAM, the windows box has 1.5Gb (twice as
> > much).
>
> Random thoughts:
>
> If there were not enough memory Python should give a MemoryError rather than
> segfault.
>
> Do both systems use the same memory layout (i. e. 32 or 64bit OS, 16 or 32
> bit unicode)?
bot use 32bits;
but the windows box uses 16 bits unicode, on linux 32bits unicode is
used.
the memory used by my program is for the larger part unicode strings.
.....well, but that makes the better performance of linux even
stranger...
>
> Try an alpha of Python 2.7. If you're lucky "your" bug was found 'n' fixed
> independently.
well...that's what I thought for version 2.6 ;-)
(I run into this crash a longer time ago)
but seriously, I'll try this.

kind regards,
henk-jan
>
> Peter

henk-jan ebbers

2/13/2010 10:51:00 AM

On Feb 13, 10:25 am, Dennis Lee Bieber <wlfr...@ix.netcom.com> wrote:
> On Fri, 12 Feb 2010 09:21:07 -0800 (PST), hjebbers <hjebb...@gmail.com>
> declaimed the following in gmane.comp.python.general:
>
> > What strikes me is:
> > 1. the crash on windows, but linux works OK (same test sets)
> > 2. the linux box has 750Mb RAM, the windows box has 1.5Gb (twice as
> > much).
>
> Which on its own does not mean much.
>
> Windows in a normal installation only grants 2GB address space to
> user code, reserving the other 2GB space for the OS shared libraries. If
> your program attempts to allocate over that, it will fail. That the
> Windows box has twice the physical memory only means it doesn't resort
> to page swapping as soon.
>
> There is a boot parameter switch that toggles Windows into a 3GB
> user/1GB OS mode --

hey, that would be great!! on my 1,5G mahcine ;-)

> it is mainly meant for server machines where there
> won't be many disjoint OS libraries loaded, but the server applications
> need lots of data space.
>
> What split does the Linux OS use? If it give 3GB to user space,
> while you'd start to page swap much soon, you'd also have 50% more
> virtual memory that can be allocated than under Windows.
> --
> Wulfraed Dennis Lee Bieber KD6MOG
> wlfr...@ix.netcom.com HTTP://wlfraed.home.netcom.com/

I will check this.....any advice on how to check this?

henk-jan

Diez B. Roggisch

2/13/2010 11:24:00 AM

Am 13.02.10 11:50, schrieb hjebbers:
> On Feb 13, 10:25 am, Dennis Lee Bieber<wlfr...@ix.netcom.com> wrote:
>> On Fri, 12 Feb 2010 09:21:07 -0800 (PST), hjebbers<hjebb...@gmail.com>
>> declaimed the following in gmane.comp.python.general:
>>
>>> What strikes me is:
>>> 1. the crash on windows, but linux works OK (same test sets)
>>> 2. the linux box has 750Mb RAM, the windows box has 1.5Gb (twice as
>>> much).
>>
>> Which on its own does not mean much.
>>
>> Windows in a normal installation only grants 2GB address space to
>> user code, reserving the other 2GB space for the OS shared libraries. If
>> your program attempts to allocate over that, it will fail. That the
>> Windows box has twice the physical memory only means it doesn't resort
>> to page swapping as soon.
>>
>> There is a boot parameter switch that toggles Windows into a 3GB
>> user/1GB OS mode --
>
> hey, that would be great!! on my 1,5G mahcine ;-)

You don't get it - it's about the virtual memory address space
available. This has nothing to do with physical memory. Obviously if the
latter runs out, the swapping will make the machine painfully slow. But
that's another issue.

Diez

henk-jan ebbers

2/13/2010 11:36:00 AM

On Feb 13, 12:24 pm, "Diez B. Roggisch" <de...@nospam.web.de> wrote:
> Am 13.02.10 11:50, schrieb hjebbers:
>
>
>
> > On Feb 13, 10:25 am, Dennis Lee Bieber<wlfr...@ix.netcom.com> wrote:
> >> On Fri, 12 Feb 2010 09:21:07 -0800 (PST), hjebbers<hjebb...@gmail.com>
> >> declaimed the following in gmane.comp.python.general:
>
> >>> What strikes me is:
> >>> 1. the crash on windows, but linux works OK (same test sets)
> >>> 2. the linux box has 750Mb RAM, the windows box has 1.5Gb (twice as
> >>> much).
>
> >> Which on its own does not mean much.
>
> >> Windows in a normal installation only grants 2GB address space to
> >> user code, reserving the other 2GB space for the OS shared libraries. If
> >> your program attempts to allocate over that, it will fail. That the
> >> Windows box has twice the physical memory only means it doesn't resort
> >> to page swapping as soon.
>
> >> There is a boot parameter switch that toggles Windows into a 3GB
> >> user/1GB OS mode --
>
> > hey, that would be great!! on my 1,5G mahcine ;-)
>
> You don't get it - it's about the virtual memory address space
> available. This has nothing to do with physical memory. Obviously if the
> latter runs out, the swapping will make the machine painfully slow. But
> that's another issue.
>
> Diez

Diez,
you are right; I misunderstood.
henk-jan

henk-jan ebbers

2/13/2010 11:55:00 AM

I enlarged the windows page file from 750Kb to 1.5Gb .
The crash still happens.
btw, the crash does not happen at a peak memory usage.
According to windows task manager, at the moment of crash mem usage of
my program is 669kb, peak memory usage is 1.136kb

henk-jan

Alf P. Steinbach

2/13/2010 12:15:00 PM

* hjebbers:
> I enlarged the windows page file from 750Kb to 1.5Gb .
> The crash still happens.
> btw, the crash does not happen at a peak memory usage.
> According to windows task manager, at the moment of crash mem usage of
> my program is 669kb, peak memory usage is 1.136kb
>
> henk-jan

Probably you meant to write "M", not "k" or "K"?

I've never managed to make much sense of the displays in Windows' Task Manager,
if that's what you're using, but I think the mem usage it's displaying by
default is the process' working set, or something very similar to that measure.

You can display additional columns in Task Manager, and one useful one is how
much virtual memory is allocated,.

And perhaps then (if that's not what you're looking it already) it'll be closer
to 2 GiB?

Cheers & hth.,

- Alf

henk-jan ebbers

2/13/2010 12:35:00 PM

On Feb 13, 10:25 am, Dennis Lee Bieber <wlfr...@ix.netcom.com> wrote:
> On Fri, 12 Feb 2010 09:21:07 -0800 (PST), hjebbers <hjebb...@gmail.com>
> declaimed the following in gmane.comp.python.general:
>
> > What strikes me is:
> > 1. the crash on windows, but linux works OK (same test sets)
> > 2. the linux box has 750Mb RAM, the windows box has 1.5Gb (twice as
> > much).
>
> Which on its own does not mean much.
>
> Windows in a normal installation only grants 2GB address space to
> user code, reserving the other 2GB space for the OS shared libraries. If
> your program attempts to allocate over that, it will fail. That the
> Windows box has twice the physical memory only means it doesn't resort
> to page swapping as soon.
>
> There is a boot parameter switch that toggles Windows into a 3GB
> user/1GB OS mode -- it is mainly meant for server machines where there
> won't be many disjoint OS libraries loaded, but the server applications
> need lots of data space.
>
> What split does the Linux OS use? If it give 3GB to user space,
> while you'd start to page swap much soon, you'd also have 50% more
> virtual memory that can be allocated than under Windows.
> --
> Wulfraed Dennis Lee Bieber KD6MOG
> wlfr...@ix.netcom.com HTTP://wlfraed.home.netcom.com/

Hi Wulfraed,
I tried this (the 3GB switch).
But python still crashes.

kind regards,
henk-jan

comp.lang.python

Re: python crash on windows but not on linux

Peter Otten

henk-jan ebbers

Dennis Lee Bieber

Peter Otten

henk-jan ebbers

henk-jan ebbers

Diez B. Roggisch

henk-jan ebbers

henk-jan ebbers

Alf P. Steinbach

henk-jan ebbers

x Login to ForumsZone