Asp Forum - The real difference between Mutex and Sync

khaines

9/4/2006 6:12:00 PM

7 Answers

Kent Sibilev

9/4/2006 6:51:00 PM

On 9/4/06, khaines@enigo.com <khaines@enigo.com> wrote:
>
> array.c's behavior is what needs to be examined in greater detail, here.
> Mutex, itself, is not doing anything surprising.
>
>
> Kirk Haines
>

Good findings.

I think the problem lies in the Array#shift (rb_ary_shift)
implementation. Basically, it just increments the internal pointer and
it never modifies the size of an array. This means that if you have an
array with 1000 elements and you 'shift' it 999 times, all these
elements are still visible to the garbage collector, until you modify
this array by triggering rb_ary_store method, for example.

--
Kent
---
http://www.dat...

Kent Sibilev

9/4/2006 6:53:00 PM

On 9/4/06, Kent Sibilev <ksruby@gmail.com> wrote:
> On 9/4/06, khaines@enigo.com <khaines@enigo.com> wrote:
> >
> > array.c's behavior is what needs to be examined in greater detail, here.
> > Mutex, itself, is not doing anything surprising.
> >
> >
> > Kirk Haines
> >
>
> Good findings.
>
> I think the problem lies in the Array#shift (rb_ary_shift)
> implementation. Basically, it just increments the internal pointer and
> it never modifies the size of an array. This means that if you have an
> array with 1000 elements and you 'shift' it 999 times, all these
> elements are still visible to the garbage collector, until you modify
> this array by triggering rb_ary_store method, for example.
>

I meant that rb_ary_shift never modifies the size of allocated memory.

--
Kent
---
http://www.dat...

Kent Sibilev

9/4/2006 7:37:00 PM

This patch will make Mutex a bit slower, but much better in terms of
garbage collection:

Index: lib/thread.rb
===================================================================
RCS file: /src/ruby/lib/thread.rb,v
retrieving revision 1.16.2.2
diff -r1.16.2.2 thread.rb
99c99
< @waiting.push Thread.current
---
> @waiting.unshift Thread.current
115c115
< t = @waiting.shift
---
> t = @waiting.pop

On 9/4/06, Kent Sibilev <ksruby@gmail.com> wrote:
> On 9/4/06, Kent Sibilev <ksruby@gmail.com> wrote:
> > On 9/4/06, khaines@enigo.com <khaines@enigo.com> wrote:
> > >
> > > array.c's behavior is what needs to be examined in greater detail, here.
> > > Mutex, itself, is not doing anything surprising.
> > >
> > >
> > > Kirk Haines
> > >
> >
> > Good findings.
> >
> > I think the problem lies in the Array#shift (rb_ary_shift)
> > implementation. Basically, it just increments the internal pointer and
> > it never modifies the size of an array. This means that if you have an
> > array with 1000 elements and you 'shift' it 999 times, all these
> > elements are still visible to the garbage collector, until you modify
> > this array by triggering rb_ary_store method, for example.
> >
>
> I meant that rb_ary_shift never modifies the size of allocated memory.
>
> --
> Kent
> ---
> http://www.dat...
>
>

--
Kent
---
http://www.dat...

Logan Capaldo

9/4/2006 7:39:00 PM

On Sep 4, 2006, at 2:50 PM, Kent Sibilev wrote:

>
> Good findings.
>
> I think the problem lies in the Array#shift (rb_ary_shift)
> implementation. Basically, it just increments the internal pointer and
> it never modifies the size of an array. This means that if you have an
> array with 1000 elements and you 'shift' it 999 times, all these
> elements are still visible to the garbage collector, until you modify
> this array by triggering rb_ary_store method, for example.
>

Would this help?

class Array
alias rb_ary_shift shift
def shift
@len ||= length
@shift_count ||= 0
res = rb_ary_shift
@shift_count += 1
if @shift_count >= @len / 2
@shift_count = nil
@len = nil
replace(dup)
end
res
end
end

> --
> Kent
> ---
> http://www.dat...
>

khaines

9/5/2006 8:25:00 AM

Bob Hutchison

9/5/2006 12:46:00 PM

On Sep 5, 2006, at 4:25 AM, khaines@enigo.com wrote:

> On Tue, 5 Sep 2006, Kent Sibilev wrote:
>
>> This patch will make Mutex a bit slower, but much better in terms of
>> garbage collection:
>>
>> Index: lib/thread.rb
>> ===================================================================
>> RCS file: /src/ruby/lib/thread.rb,v
>> retrieving revision 1.16.2.2
>> diff -r1.16.2.2 thread.rb
>> 99c99
>> < @waiting.push Thread.current
>> ---
>>> @waiting.unshift Thread.current
>> 115c115
>> < t = @waiting.shift
>> ---
>>> t = @waiting.pop
>
> While I am not benchmarking with performance in mind, that change
> doesn't seem to have any significant effect on overall speed, which
> doesn't surprise me. Speed-wise, it is just flipping the location
> of the expensive array operation from the unlock to the lock.
>
> And yes, that simple change does seem to make a significant
> difference, because pop will realloc the array.

I've attached a trivial script that demonstrates on OS X and likely
linux (unlikely windows because of the ps stuff) the problem. To see
the bug, run it like:

ruby blowup2.rb

A *simple* fix is to just remove the reference to whatever is pointed
to by the first element. To see this work, run the script as:

ruby blowup2.rb fix

This trick will probably lead to a quicker fix (and if you are daring
you might actually patch the C implementation of shift to do it for
you). Might want to hear what Matz has to say about that.

Cheers,
Bob

----
Bob Hutchison -- blogs at <http://www.rec...
hutch/>
Recursive Design Inc. -- <http://www.rec...>
Raconteur -- <http://www.raconteur...
xampl for Ruby -- <http://rubyforge.org/projects/...

$VERBOSE = nil
STDOUT.sync = true
trap('INT'){ exit }

fixit = 0 < ARGV.size

m = 1000000
n = 1000

all_arrays = []
first = true

n.times do |i|
if 0 < m then
a = ["x" * m ]
all_arrays << a
a[0] = nil if fixit and 0 < a.size
a.shift
else
a = []
all_arrays << a
end

if 0 == (all_arrays.size % 10) then
GC.start
stdout = `ps v -p #{ Process.pid }`
stdout = stdout.split(%r/\n/)
if first then
printf("\n %s\n", stdout.first)
first = false
end
printf("%6d:: %s\n", all_arrays.size, stdout.last)
end
end

Jano Svitok

9/5/2006 2:14:00 PM

On 9/5/06, Bob Hutchison <hutch@recursive.ca> wrote:
> I've attached a trivial script that demonstrates on OS X and likely
> linux (unlikely windows because of the ps stuff) the problem. To see

This is 'windows' version, using pslist[1]
And, yes, it does the same under windows as well.

Jano

[1] http://www.sysinternals.com/Utilities/P...

$VERBOSE = nil
STDOUT.sync = true
trap('INT'){ exit }

fixit = 0 < ARGV.size

m = 1000000
n = 1000

all_arrays = []
first = true

n.times do |i|
if 0 < m then
a = ["x" * m ]
all_arrays << a
a[0] = nil if fixit and 0 < a.size
a.shift
else
a = []
all_arrays << a
end

if 0 == (all_arrays.size % 10) then
GC.start
stdout = `pslist -m #{ Process.pid }`
stdout = stdout.split(%r/\n/)
if first then
printf("\n %s\n", stdout[7])
first = false
end
printf("%6d:: %s\n", all_arrays.size, stdout.last)
end
end

comp.lang.ruby

The real difference between Mutex and Sync

khaines

Kent Sibilev

Kent Sibilev

Kent Sibilev

Logan Capaldo

khaines

Bob Hutchison

Jano Svitok

x Login to ForumsZone