Asp Forum - Newbie on Threads

Nabusman

5/26/2009 8:20:00 AM

I'm creating a screen scraping software and I want to have X (let's say
10 for example) threads running simultaneously doing the scraping. The
program will access a text file containing an unknown number of URLs and
then scrape.

My question is how do I setup the threads so that once a thread finishes
execution it picks up another URL and starts executing again.

Thanks,

Nabs
--
Posted via http://www.ruby-....

12 Answers

Robert Klemme

5/26/2009 8:46:00 AM

2009/5/26 Nabs Kahn <nabusman@gmail.com>:
> I'm creating a screen scraping software and I want to have X (let's say
> 10 for example) threads running simultaneously doing the scraping. The
> program will access a text file containing an unknown number of URLs and
> then scrape.
>
> My question is how do I setup the threads so that once a thread finishes
> execution it picks up another URL and starts executing again.

Create a Queue (require 'thread') and have your worker threads read
URL's from it.

Kind regards

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestprac...

Nabusman

5/26/2009 11:58:00 AM

Thanks for the quick response, this is what I wrote, but it doesn't seem
to work, no errors, but it just finishes the program without doing
anything. Where did I go wrong?

require 'thread'

buffer = SizedQueue.new(10)

producer = Thread.new do
File.open("urls.txt").each do |url|
buffer << url
end
end

consumer = Thread.new do
while buffer.num_waiting != 0
url = buffer.pop
#do screen scraping with url here
end
end

consumer.join

Robert Klemme wrote:
> 2009/5/26 Nabs Kahn <nabusman@gmail.com>:
>> I'm creating a screen scraping software and I want to have X (let's say
>> 10 for example) threads running simultaneously doing the scraping. The
>> program will access a text file containing an unknown number of URLs and
>> then scrape.
>>
>> My question is how do I setup the threads so that once a thread finishes
>> execution it picks up another URL and starts executing again.
>
> Create a Queue (require 'thread') and have your worker threads read
> URL's from it.
>
> Kind regards
>
> robert

--
Posted via http://www.ruby-....

Robert Klemme

5/26/2009 2:55:00 PM

2009/5/26 Nabs Kahn <nabusman@gmail.com>:
> Thanks for the quick response, this is what I wrote, but it doesn't seem
> to work, no errors, but it just finishes the program without doing
> anything. Where did I go wrong?

I am not sure what you expect: your program does not do anything with
the url. Did you try printing it?

> require 'thread'
>
> buffer =3D SizedQueue.new(10)
>
> producer =3D Thread.new do
> =A0File.open("urls.txt").each do |url|
> =A0 =A0buffer << url

For better readability I suggest using buffer.enq.

> =A0end
> end
>
> consumer =3D Thread.new do
> =A0while buffer.num_waiting !=3D 0

Rather use buffer.deq which is a blocking call. Also, in your case,
the consumer will stop working as soon as the queu has run empty
*once*. You rather want a more reliable termination detection.
Usually I put a special value into the queue (more precisely, as many
special values as there are threads). In your case a symbol would
work.

> =A0 =A0url =3D buffer.pop
> =A0 =A0#do screen scraping with url here
> =A0end
> end
>
> consumer.join

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestprac...

Nabusman

5/26/2009 3:06:00 PM

> I am not sure what you expect: your program does not do anything with
> the url. Did you try printing it?

I left that part out, figured it would make the example unnecessarily
complex.

Thanks for the tips.

Nabs
--
Posted via http://www.ruby-....

TPReal

5/26/2009 6:38:00 PM

Nabs Kahn wrote:
> require 'thread'
>
> buffer = SizedQueue.new(10)
>
> producer = Thread.new do
> File.open("urls.txt").each do |url|
> buffer << url
> end
> end
>
> consumer = Thread.new do
> while buffer.num_waiting != 0
> url = buffer.pop
> #do screen scraping with url here
> end
> end
>
> consumer.join
>

Hello. You said that you want X threads, but in your example you have
only one scraping thread. I think this is more what you intended (not
tested):

require 'thread'

buffer = SizedQueue.new(10)

producer = Thread.new do
File.open("urls.txt").each do |url|
buffer << url
end
end

consumers = Array::new(x){
Thread.new do
while url=buffer.deq
#do screen scraping with url here
end
end
}

Now if you want the program to stop after the producer finishes, you
should add

producer.join
consumers.each{|c| c.join}

to make sure all processing is finished.
--
Posted via http://www.ruby-....

Nabusman

5/27/2009 4:04:00 PM

This is what I ended up doing, similar to what was suggested.
(definition of screenScrape method not included)

bufferSize = 10
buffer = SizedQueue.new(bufferSize)
threads = []

producer = Thread.new do
File.open("urls.txt").each do |url|
buffer.enq url
end
bufferSize.times {buffer.enq(:END_OF_WORK)}
end

bufferSize.times do
threads << Thread.new do
url = nil
while(url != :END_OF_WORK)
url = buffer.deq
screenScrape(url)
end
end
end

producer.join
threads.each do |thr|
thr.join
end

Thomas B. wrote:
> Nabs Kahn wrote:
>> require 'thread'
>>
>> buffer = SizedQueue.new(10)
>>
>> producer = Thread.new do
>> File.open("urls.txt").each do |url|
>> buffer << url
>> end
>> end
>>
>> consumer = Thread.new do
>> while buffer.num_waiting != 0
>> url = buffer.pop
>> #do screen scraping with url here
>> end
>> end
>>
>> consumer.join
>>
>
> Hello. You said that you want X threads, but in your example you have
> only one scraping thread. I think this is more what you intended (not
> tested):
>
> require 'thread'
>
> buffer = SizedQueue.new(10)
>
> producer = Thread.new do
> File.open("urls.txt").each do |url|
> buffer << url
> end
> end
>
> consumers = Array::new(x){
> Thread.new do
> while url=buffer.deq
> #do screen scraping with url here
> end
> end
> }
>
> Now if you want the program to stop after the producer finishes, you
> should add
>
> producer.join
> consumers.each{|c| c.join}
>
> to make sure all processing is finished.

--
Posted via http://www.ruby-....

Robert Klemme

5/27/2009 4:12:00 PM

2009/5/27 Nabs Kahn <nabusman@gmail.com>:
> This is what I ended up doing, similar to what was suggested.
> (definition of screenScrape method not included)

Thanks for the update!

> bufferSize =3D 10
> buffer =3D SizedQueue.new(bufferSize)
> threads =3D []
>
> producer =3D Thread.new do
> =A0File.open("urls.txt").each do |url|
> =A0 =A0buffer.enq url
> =A0end
> =A0bufferSize.times {buffer.enq(:END_OF_WORK)}
> end
>
> bufferSize.times do
> =A0threads << Thread.new do
> =A0 =A0url =3D nil
> =A0 =A0while(url !=3D :END_OF_WORK)
> =A0 =A0 =A0url =3D buffer.deq
> =A0 =A0 =A0screenScrape(url)
> =A0 =A0end
> =A0end
> end

The loop above does not work properly because you will hand off
:END_OF_WORK to screenScrape(). Rather do

threads << Thread.new do
while ((url =3D buffer.deq) !=3D :END_OF_WORK)
screenScrape(url)
end
end

or

threads << Thread.new do
until ((url =3D buffer.deq) =3D=3D :END_OF_WORK)
screenScrape(url)
end
end

> producer.join
> threads.each do |thr|
> =A0thr.join
> end

Btw, you do not need a separate producer thread. You can simply do
that in the main thread. But of course you must start worker threads
before you start to fill the queue.

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestprac...

Nabusman

5/27/2009 4:24:00 PM

> Btw, you do not need a separate producer thread. You can simply do
> that in the main thread. But of course you must start worker threads
> before you start to fill the queue.

Oh, I thought it would cause the main thread to pause until there was
room in the queue since there would be more urls in the file, but only
10 in the queue, so I created a new thread for the producer thread.
Thanks for the information.
--
Posted via http://www.ruby-....

Robert Klemme

5/27/2009 5:41:00 PM

On 27.05.2009 18:24, Nabs Kahn wrote:
>> Btw, you do not need a separate producer thread. You can simply do
>> that in the main thread. But of course you must start worker threads
>> before you start to fill the queue.
>
> Oh, I thought it would cause the main thread to pause until there was
> room in the queue since there would be more urls in the file, but only
> 10 in the queue, so I created a new thread for the producer thread.

Well, it will. But so will your producer thread and eventually the main
thread since it joins on the producer. The producer thread brings you
only advantages if you need to do other things in the main thread. But
since you don't in the code you presented you can as well do the queue
filling in the main thread.

Btw, I just notice one thing: you don't chomp the lines read from the
file. So your URL's will still contain the trailing line feed.

> Thanks for the information.

You're welcome!

Kind regards

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestprac...

Nabusman

5/27/2009 7:35:00 PM

> Well, it will. But so will your producer thread and eventually the main
> thread since it joins on the producer. The producer thread brings you
> only advantages if you need to do other things in the main thread. But
> since you don't in the code you presented you can as well do the queue
> filling in the main thread.

I see, that makes sense.

> Btw, I just notice one thing: you don't chomp the lines read from the
> file. So your URL's will still contain the trailing line feed.

I am actually chomping them inside the screenScraper method.

Nabs
--
Posted via http://www.ruby-....

comp.lang.ruby

Newbie on Threads

Nabusman

Robert Klemme

Nabusman

Robert Klemme

Nabusman

TPReal

Nabusman

Robert Klemme

Nabusman

Robert Klemme

Nabusman

x Login to ForumsZone