Asp Forum - Ruby Threading question Creating 4 threads

Tristin Davis

2/16/2009 9:44:00 PM

Newbie Question:

I have some code where I create 4 threads that pop elements off an
array and process them. The problem is that each thread only pops off
one element, processes it, then returns; they don't continue to
process until the entire array is consumed.

<code>
$jobs = %w[a b c d e f g]
def simulate_cleanup

puts "Checking for #{$jobs.length} jobs in /mnt/gen1/focal/temp"
threads = []

mu = Mutex.new
total = 0

1.upto(4) do |c|
threads << Thread.new do
puts "Thread #{c} created"
job = nil

mu.synchronize do
job = $jobs.pop
puts "Checking for #{job}"
#Exit our thread if we have no more jobs to check
puts "No more jobs" and return if job.nil?
end

t = count_files(job)
mu.synchronize { total += t }
end
end
threads.each { |t| t.join }

puts "Reduced by #{total} files"
end

Example of output:
Thread 1 created
Thread 2 created
Thread 3 created
Thread 4 created
Checking for a
Checking for b
Checking for c
Checking for d

6 Answers

Louis-Philippe

2/16/2009 10:10:00 PM

[Note: parts of this message were removed to make it a legal post.]

what if you put your thread body in an infinite loop, something like:

Thread.new do
loop do
puts "Thread #{c} created"
job = nil

mu.synchronize do
job = $jobs.pop
puts "Checking for #{job}"
#Exit our thread if we have no more jobs to check
puts "No more jobs" and Thread.exit if job.nil?
end

t = count_files(job)
mu.synchronize { total += t }
end
end

and as you may have noticed, I replaced the 'return' statement inside your
thread with a 'Thread.exit' method.
I think it would be the right way to do it...

2009/2/16 <tristin.colby@gmail.com>

> Newbie Question:
>
> I have some code where I create 4 threads that pop elements off an
> array and process them. The problem is that each thread only pops off
> one element, processes it, then returns; they don't continue to
> process until the entire array is consumed.
>
> <code>
> $jobs = %w[a b c d e f g]
> def simulate_cleanup
>
> puts "Checking for #{$jobs.length} jobs in /mnt/gen1/focal/temp"
> threads = []
>
> mu = Mutex.new
> total = 0
>
> 1.upto(4) do |c|
> threads << Thread.new do
> puts "Thread #{c} created"
> job = nil
>
> mu.synchronize do
> job = $jobs.pop
> puts "Checking for #{job}"
> #Exit our thread if we have no more jobs to check
> puts "No more jobs" and return if job.nil?
> end
>
> t = count_files(job)
> mu.synchronize { total += t }
> end
> end
> threads.each { |t| t.join }
>
> puts "Reduced by #{total} files"
> end
>
> Example of output:
> Thread 1 created
> Thread 2 created
> Thread 3 created
> Thread 4 created
> Checking for a
> Checking for b
> Checking for c
> Checking for d
>
>
>

Louis-Philippe

2/16/2009 10:20:00 PM

[Note: parts of this message were removed to make it a legal post.]

and looking at it more, I think it would be better to leave the less stuff
in the mutex block as possible,
making it something something like:

mu.synchronize do
job = $jobs.pop
end

puts "Checking for #{job}"
#Exit our thread if we have no more jobs to check
puts "No more jobs" and Thread.exit if job.nil?

I guess the Thread.exit could have caused a deadlock if inside the mutex,
and since the mutex locks your other threads you better keep it as small as
possible for efficiency sake.

2009/2/16 Louis-Philippe <default@spiralix.org>

> what if you put your thread body in an infinite loop, something like:
>
> Thread.new do
> loop do
> puts "Thread #{c} created"
> job = nil
>
> mu.synchronize do
> job = $jobs.pop
> puts "Checking for #{job}"
> #Exit our thread if we have no more jobs to check
> puts "No more jobs" and Thread.exit if job.nil?
> end
>
> t = count_files(job)
> mu.synchronize { total += t }
> end
> end
>
> and as you may have noticed, I replaced the 'return' statement inside your
> thread with a 'Thread.exit' method.
> I think it would be the right way to do it...
>
> 2009/2/16 <tristin.colby@gmail.com>
>
> Newbie Question:
>>
>> I have some code where I create 4 threads that pop elements off an
>> array and process them. The problem is that each thread only pops off
>> one element, processes it, then returns; they don't continue to
>> process until the entire array is consumed.
>>
>> <code>
>> $jobs = %w[a b c d e f g]
>> def simulate_cleanup
>>
>> puts "Checking for #{$jobs.length} jobs in /mnt/gen1/focal/temp"
>> threads = []
>>
>> mu = Mutex.new
>> total = 0
>>
>> 1.upto(4) do |c|
>> threads << Thread.new do
>> puts "Thread #{c} created"
>> job = nil
>>
>> mu.synchronize do
>> job = $jobs.pop
>> puts "Checking for #{job}"
>> #Exit our thread if we have no more jobs to check
>> puts "No more jobs" and return if job.nil?
>> end
>>
>> t = count_files(job)
>> mu.synchronize { total += t }
>> end
>> end
>> threads.each { |t| t.join }
>>
>> puts "Reduced by #{total} files"
>> end
>>
>> Example of output:
>> Thread 1 created
>> Thread 2 created
>> Thread 3 created
>> Thread 4 created
>> Checking for a
>> Checking for b
>> Checking for c
>> Checking for d
>>
>>
>>
>

Julian Leviston

2/17/2009 3:39:00 AM

Put a while around the pop and block of code to run.

Blog: http://random8.ze...
Learn rails: http://sensei.ze...

On 17/02/2009, at 8:46 AM, tristin.colby@gmail.com wrote:

> Newbie Question:
>
> I have some code where I create 4 threads that pop elements off an
> array and process them. The problem is that each thread only pops off
> one element, processes it, then returns; they don't continue to
> process until the entire array is consumed.
>
> <code>
> $jobs = %w[a b c d e f g]
> def simulate_cleanup
>
> puts "Checking for #{$jobs.length} jobs in /mnt/gen1/focal/temp"
> threads = []
>
> mu = Mutex.new
> total = 0
>
> 1.upto(4) do |c|
> threads << Thread.new do
> puts "Thread #{c} created"
> job = nil
>
> mu.synchronize do
> job = $jobs.pop
> puts "Checking for #{job}"
> #Exit our thread if we have no more jobs to check
> puts "No more jobs" and return if job.nil?
> end
>
> t = count_files(job)
> mu.synchronize { total += t }
> end
> end
> threads.each { |t| t.join }
>
> puts "Reduced by #{total} files"
> end
>
> Example of output:
> Thread 1 created
> Thread 2 created
> Thread 3 created
> Thread 4 created
> Checking for a
> Checking for b
> Checking for c
> Checking for d
>
>

Julian Leviston

2/17/2009 3:41:00 AM

That won't ever exit the thread because puts returns nil.

Blog: http://random8.ze...
Learn rails: http://sensei.ze...

On 17/02/2009, at 9:10 AM, Louis-Philippe <default@spiralix.org> wrote:

> what if you put your thread body in an infinite loop, something like:
>
> Thread.new do
> loop do
> puts "Thread #{c} created"
> job = nil
>
> mu.synchronize do
> job = $jobs.pop
> puts "Checking for #{job}"
> #Exit our thread if we have no more jobs to check
> puts "No more jobs" and Thread.exit if job.nil?
> end
>
> t = count_files(job)
> mu.synchronize { total += t }
> end
> end
>
> and as you may have noticed, I replaced the 'return' statement
> inside your
> thread with a 'Thread.exit' method.
> I think it would be the right way to do it...
>
> 2009/2/16 <tristin.colby@gmail.com>
>
>> Newbie Question:
>>
>> I have some code where I create 4 threads that pop elements off an
>> array and process them. The problem is that each thread only pops
>> off
>> one element, processes it, then returns; they don't continue to
>> process until the entire array is consumed.
>>
>> <code>
>> $jobs = %w[a b c d e f g]
>> def simulate_cleanup
>>
>> puts "Checking for #{$jobs.length} jobs in /mnt/gen1/focal/temp"
>> threads = []
>>
>> mu = Mutex.new
>> total = 0
>>
>> 1.upto(4) do |c|
>> threads << Thread.new do
>> puts "Thread #{c} created"
>> job = nil
>>
>> mu.synchronize do
>> job = $jobs.pop
>> puts "Checking for #{job}"
>> #Exit our thread if we have no more jobs to check
>> puts "No more jobs" and return if job.nil?
>> end
>>
>> t = count_files(job)
>> mu.synchronize { total += t }
>> end
>> end
>> threads.each { |t| t.join }
>>
>> puts "Reduced by #{total} files"
>> end
>>
>> Example of output:
>> Thread 1 created
>> Thread 2 created
>> Thread 3 created
>> Thread 4 created
>> Checking for a
>> Checking for b
>> Checking for c
>> Checking for d
>>
>>
>>

Robert Klemme

2/17/2009 7:22:00 AM

On 17.02.2009 04:38, Julian Leviston wrote:
> Put a while around the pop and block of code to run.

Also class Queue has thread safety built in already.

Kind regards

robert

--
remember.guy do |as, often| as.you_can - without end

7stud --

2/17/2009 2:15:00 PM

Robert Klemme wrote:
> On 17.02.2009 04:38, Julian Leviston wrote:
>> Put a while around the pop and block of code to run.
>
> Also class Queue has thread safety built in already.
>
> Kind regards
>
> robert

...an example of using a Queue:

require 'thread'

arr = %w[a b c d e f g h i j]
num_threads = 4
q = Queue.new

workers = []

num_threads.times do |i|
workers << Thread.new("worker#{i+1}") do |name|
loop do
data = q.pop #pops data off bottom of q, blocks if no data
puts "#{name} retrieved data: #{data}"
break if data == "*END_OF_DATA_FLAG*"
end
end
end

arr.each {|item| q.push(item)}
num_threads.times do
q.push("*END_OF_DATA_FLAG*")
end

workers.each {|thr| thr.join}
puts "Workers are all done working."

--output:--
worker1 retrieved data: a
worker2 retrieved data: b
worker3 retrieved data: c
worker4 retrieved data: d
worker1 retrieved data: e
worker2 retrieved data: f
worker3 retrieved data: g
worker4 retrieved data: h
worker1 retrieved data: i
worker2 retrieved data: j
worker3 retrieved data: *END_OF_DATA_FLAG*
worker4 retrieved data: *END_OF_DATA_FLAG*
worker1 retrieved data: *END_OF_DATA_FLAG*
worker2 retrieved data: *END_OF_DATA_FLAG*
Workers are all done working.

--
Posted via http://www.ruby-....

comp.lang.ruby

Ruby Threading question Creating 4 threads

Tristin Davis

Louis-Philippe

Louis-Philippe

Julian Leviston

Julian Leviston

Robert Klemme

7stud --

x Login to ForumsZone