[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Multiple HTTP Sockets or Threads for file uploads.

gberz3

6/22/2008 2:02:00 AM

Hi All,

I have a client for whom I developed a small application using Ruby
and Curl to upload files to his file server (to and from Windows
boxes). He now wants that application to have multiple sockets in
order to upload more than one file at a time. I have no idea what
course is the safest here. Do I simply create threads that run the
"curl" command and let that be, or do I try a library such as (http://
www.chilkatsoft.com/upload-ruby.asp) and go from there? I'm honestly
not familiar with threading or chilkat.

Any suggestions would be most appreciated.

Regards.
7 Answers

Robert Klemme

6/24/2008 7:06:00 AM

0

2008/6/22 gberz3 <gberz3@gmail.com>:
> I have a client for whom I developed a small application using Ruby
> and Curl to upload files to his file server (to and from Windows
> boxes). He now wants that application to have multiple sockets in
> order to upload more than one file at a time. I have no idea what
> course is the safest here. Do I simply create threads that run the
> "curl" command and let that be,

You just need to fork multiple curl processes.

> or do I try a library such as (http://
> www.chilkatsoft.com/upload-ruby.asp) and go from there? I'm honestly
> not familiar with threading or chilkat.

I do not know that lib. If it does provide multiple concurrent
uploads that's probably a good choice.

If you were doing it in pure Ruby (i.e. using net/http) you could use threads.

Kind regards

robert

--
use.inject do |as, often| as.you_can - without end

Zhukov Pavel

6/24/2008 7:11:00 AM

0

you should use DRb with multi-process architecture since ruby uses
green-threads, that should be much faster.


On Tue, Jun 24, 2008 at 11:06 AM, Robert Klemme
<shortcutter@googlemail.com> wrote:
> 2008/6/22 gberz3 <gberz3@gmail.com>:
>> I have a client for whom I developed a small application using Ruby
>> and Curl to upload files to his file server (to and from Windows
>> boxes). He now wants that application to have multiple sockets in
>> order to upload more than one file at a time. I have no idea what
>> course is the safest here. Do I simply create threads that run the
>> "curl" command and let that be,
>
> You just need to fork multiple curl processes.
>
>> or do I try a library such as (http://
>> www.chilkatsoft.com/upload-ruby.asp) and go from there? I'm honestly
>> not familiar with threading or chilkat.
>
> I do not know that lib. If it does provide multiple concurrent
> uploads that's probably a good choice.
>
> If you were doing it in pure Ruby (i.e. using net/http) you could use threads.
>
> Kind regards
>
> robert
>
> --
> use.inject do |as, often| as.you_can - without end
>
>

Robert Klemme

6/24/2008 11:04:00 AM

0

2008/6/24 Zhukov Pavel <gelios@gmail.com>:
> you should use DRb with multi-process architecture since ruby uses
> green-threads, that should be much faster.

This is not necessarily true for IO bound tasks. Also, if curl is used
you just need multiple child processes but not multiple threads.

Kind regards

robert

--
use.inject do |as, often| as.you_can - without end

gberz3

6/24/2008 8:56:00 PM

0

On Jun 24, 7:03 am, Robert Klemme <shortcut...@googlemail.com> wrote:
>
> This is not necessarily true for IO bound tasks. Also, if curl is used
> you just need multiple child processes but not multiple threads.
>
> Kind regards
>
> robert

Alright, I'm currently using Threads and it has sped things up
tremendously. Thank you both for your suggestions.

One problem I'm having now is that sometimes a file will hang cURL,
and while the other subprocesses finish there might be one or two that
hang up. In this case I'd like to have some kind of granular control
over killing individual items without necessarily killing the whole
thread. I'd be afraid to kill the entire thread as it may not be
finished processing legitimate cURL processes when you send the kill
switch.

Thoughts?

Robert Klemme

6/25/2008 5:29:00 AM

0

On 24.06.2008 22:56, gberz3 wrote:
> On Jun 24, 7:03 am, Robert Klemme <shortcut...@googlemail.com> wrote:
>> This is not necessarily true for IO bound tasks. Also, if curl is used
>> you just need multiple child processes but not multiple threads.
>
> Alright, I'm currently using Threads and it has sped things up
> tremendously. Thank you both for your suggestions.

But note that the speedup likely comes from using multiple curl
processes - not from using multiple Ruby threads.

> One problem I'm having now is that sometimes a file will hang cURL,
> and while the other subprocesses finish there might be one or two that
> hang up. In this case I'd like to have some kind of granular control
> over killing individual items without necessarily killing the whole
> thread. I'd be afraid to kill the entire thread as it may not be
> finished processing legitimate cURL processes when you send the kill
> switch.
>
> Thoughts?

Killing threads does nothing to your external processes. You can either
kill curl and force its termination or you need to investigate whether
curl is built in a way that it responds to certain signals by dropping
the current upload.

Kind regards

robert

Todd Fisher

7/24/2008 2:40:00 AM

0

Michael Williams wrote:
> Hi All,
>
> I have a client for whom I developed a small application using Ruby
> and Curl to upload files to his file server (to and from Windows
> boxes). He now wants that application to have multiple sockets in
> order to upload more than one file at a time. I have no idea what
> course is the safest here. Do I simply create threads that run the
> "curl" command and let that be, or do I try a library such as (http://
> www.chilkatsoft.com/upload-ruby.asp) and go from there? I'm honestly
> not familiar with threading or chilkat.
>
> Any suggestions would be most appreciated.
>
> Regards.

You could try the curb library. This version included support for the
Multi interface, allowing multiple easy handles to perform concurrently.
Check it out: https://github.com/taf2...
--
Posted via http://www.ruby-....

Todd Fisher

7/24/2008 2:43:00 AM

0

Michael Williams wrote:
> On Jun 24, 7:03�am, Robert Klemme <shortcut...@googlemail.com> wrote:
>>
>> This is not necessarily true for IO bound tasks. Also, if curl is used
>> you just need multiple child processes but not multiple threads.
>>
>> Kind regards
>>
>> robert
>
> Alright, I'm currently using Threads and it has sped things up
> tremendously. Thank you both for your suggestions.
>
> One problem I'm having now is that sometimes a file will hang cURL,
> and while the other subprocesses finish there might be one or two that
> hang up. In this case I'd like to have some kind of granular control
> over killing individual items without necessarily killing the whole
> thread. I'd be afraid to kill the entire thread as it may not be
> finished processing legitimate cURL processes when you send the kill
> switch.
>
> Thoughts?

This could be caused by a bad domain name causing a dns to take a long
time... if you install the c-ares library and recompile libcurl you can
at least select and get activity while dns is resolving...
--
Posted via http://www.ruby-....