Asp Forum - options for running code in parallel

Martin DeMello

7/1/2008 5:53:00 PM

Our testing department wants a small tool that can make a number of
http calls simultaneously to the same server, the basic logic being

# -----------------------------------------------------------------
calls = ["call1"..."calln"].map {|x| "http://server/#{x}"}

calls.each do |x|
OnSignal(START) do
http_send call(x)
log_result
end
end

SendSignal(START)
# -----------------------------------------------------------------

What are the options for doing this cleanly and robustly? I haven't
been keeping up with the recent work people have been doing on
distributed/parallel code in ruby, but I get the general impression
there's been a lot of activity in this area.

martin

4 Answers

Igal Koshevoy

7/1/2008 6:01:00 PM

Martin DeMello wrote:
> Our testing department wants a small tool that can make a number of
> http calls simultaneously to the same server
>
Before writing your own, you may want to take a look at existing tools like:
* http://www.hpl.hp.com/research/linu...
* http://httpd.apache.org/docs/2.0/progra...

If you still want to build your own, please review the last couple days
of ruby-talk mailing list archives because there's been some spirited
debate about fork/thread that may be useful.

-igal

ara.t.howard

7/1/2008 6:51:00 PM

On Jul 1, 2008, at 11:52 AM, Martin DeMello wrote:

> Our testing department wants a small tool that can make a number of
> http calls simultaneously to the same server, the basic logic being
>
> # -----------------------------------------------------------------
> calls = ["call1"..."calln"].map {|x| "http://server/#{x}"}
>
> calls.each do |x|
> OnSignal(START) do
> http_send call(x)
> log_result
> end
> end
>
> SendSignal(START)
> # -----------------------------------------------------------------
>
> What are the options for doing this cleanly and robustly? I haven't
> been keeping up with the recent work people have been doing on
> distributed/parallel code in ruby, but I get the general impression
> there's been a lot of activity in this area.
>
> martin
>

give this a whack:

cfp:~> cat a.rb
module Threadify
require 'thread'

@threads = 8
@abort_on_exception = true

class << self
attr_accessor :threads
attr_accessor :abort_on_exception
end
end

module Enumerable
def threadify opts = {}, &block
opts = {:threads => opts} if Numeric === opts
threads = Integer(opts[:threads] || opts['threads'] ||
Threadify.threads)
done = Object.new.freeze
jobs = Queue.new

each_with_index{|elem, i| jobs.push [elem, i]}
threads.times{ jobs.push done} # mark the end

consumers = Array.new threads

threads.times do |i|
consumers[i] = Thread.new do
this = Thread.current
this.abort_on_exception = Threadify.abort_on_exception
loop{
job = jobs.pop
this.exit if job == done
args = job.first
jobs << (job << block.call(*args))
}
end
end

consumers.map{|t| t.join}
jobs.push done

ret = []
while((job = jobs.pop) != done)
elem, i, value = job
ret[i] = value
end
ret
end
end

class Thread
def Thread.ify enumerable, *args, &block
enumerable.send :threadify, *args, &block
end
end

if __FILE__ == $0
require 'open-uri'
require 'yaml'

uris = %w( http://... http:/... http://ruby... http://rub...
)

Thread.ify uris, :threads => 3 do |uri|
body = open(uri){|pipe| pipe.read}
y uri => body.size
end
end

cfp:~> ruby a.rb
---
http:/...: 9562
---
http://...: 6290
---
http://ruby...: 22352
---
http://rub...: 9984

soon to be released as a gem.

a @ http://codeforp...
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama

Martin DeMello

7/1/2008 8:25:00 PM

On Tue, Jul 1, 2008 at 11:50 AM, ara.t.howard <ara.t.howard@gmail.com> wrote:
>
> give this a whack:

Thanks, will take it for a spin!

martin

ara.t.howard

7/1/2008 8:46:00 PM

On Jul 1, 2008, at 2:24 PM, Martin DeMello wrote:

> Thanks, will take it for a spin!

just threw a gem up too - grab it from rubyforge by hand until it
propagates...

cheers.

a @ http://codeforp...
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama

comp.lang.ruby

options for running code in parallel

Martin DeMello

Igal Koshevoy

ara.t.howard

Martin DeMello

ara.t.howard

x Login to ForumsZone