[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Performance improvement possible?

Phil Rhoades

6/24/2008 4:23:00 PM

People,

I have asked before about Ruby to C conversion programs and other
alternatives with no really satisfactory solution for my particular
situation. I decided to test out a conversion of one of the C programs
and see what sort of results I get. This particular small C program is
called 32,000 times from loops within a shell script. The program then
processes a text file, writes a text file and exits. The same shell
script with a Ruby program replacing the C program does exactly the same
thing but takes 8.5 times as long (27m/227m).

The profile on ONE execution of the Ruby program produced:

% cumulative self self total
time seconds seconds calls ms/call ms/call name
65.18 14.15 14.15 1 14150.00 21700.00 Array#each
12.44 16.85 2.70 93792 0.03 0.03 Array#[]
8.15 18.62 1.77 62108 0.03 0.03 String#split
6.73 20.08 1.46 53215 0.03 0.03 String#==
2.58 20.64 0.56 66 8.48 12.73 Range#each

It would be SO much nicer to rewrite some stuff that needs rewriting and
write ALL new stuff in Ruby but this looks impossible with these times . .

Any suggestions for performance improvements?

I am using F9.

Thanks,

Phil.
--
Philip Rhoades

Pricom Pty Limited (ACN 003 252 275 ABN 91 003 252 275)
GPO Box 3411
Sydney NSW 2001
Australia
E-mail: phil@pricom.com.au
34 Answers

Rob Biedenharn

6/24/2008 4:49:00 PM

0

On Jun 24, 2008, at 12:23 PM, Philip Rhoades wrote:
> People,
>
> I have asked before about Ruby to C conversion programs and other
> alternatives with no really satisfactory solution for my particular
> situation.

Are you looking at all solutions?

> I decided to test out a conversion of one of the C programs and see
> what sort of results I get. This particular small C program is
> called 32,000 times from loops within a shell script. The program
> then processes a text file, writes a text file and exits. The same
> shell script with a Ruby program replacing the C program does
> exactly the same thing but takes 8.5 times as long (27m/227m).

Can you replace the whole shell script with a ruby program? Then
whatever startup cost you have for the Ruby interpreter is paid once
rather than 32_000 times.

> The profile on ONE execution of the Ruby program produced:
>
> % cumulative self self total
> time seconds seconds calls ms/call ms/call name
> 65.18 14.15 14.15 1 14150.00 21700.00 Array#each
> 12.44 16.85 2.70 93792 0.03 0.03 Array#[]
> 8.15 18.62 1.77 62108 0.03 0.03 String#split
> 6.73 20.08 1.46 53215 0.03 0.03 String#==
> 2.58 20.64 0.56 66 8.48 12.73 Range#each
>
> It would be SO much nicer to rewrite some stuff that needs rewriting
> and write ALL new stuff in Ruby but this looks impossible with these
> times . .
>
> Any suggestions for performance improvements?
>
> I am using F9.

F9? Is that some key in an IDE that runs your ruby code?

> Thanks,
>
> Phil.
> --
> Philip Rhoades
>
> Pricom Pty Limited (ACN 003 252 275 ABN 91 003 252 275)
> GPO Box 3411
> Sydney NSW 2001
> Australia
> E-mail: phil@pricom.com.au

Depending on what your C program has to do and why a shell script is
calling it in the first place, you might have other benefits from
using Ruby in place of the shell script. A couple years back, I
worked on a similar kind of project that was replacing shell scripts
with Perl and there were many benefits that Perl could exploit that
just could not be managed by constructs in the shell.

-Rob

Rob Biedenharn http://agileconsult...
Rob@AgileConsultingLLC.com

ara.t.howard

6/24/2008 4:54:00 PM

0


On Jun 24, 2008, at 10:23 AM, Philip Rhoades wrote:

> Any suggestions for performance improvements?

post your code.

a @ http://codeforp...
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama

Tim Pease

6/24/2008 4:57:00 PM

0


On Jun 24, 2008, at 10:23 AM, Philip Rhoades wrote:

> People,
>
> I have asked before about Ruby to C conversion programs and other
> alternatives with no really satisfactory solution for my particular
> situation. I decided to test out a conversion of one of the C
> programs and see what sort of results I get. This particular small
> C program is called 32,000 times from loops within a shell script.
> The program then processes a text file, writes a text file and
> exits. The same shell script with a Ruby program replacing the C
> program does exactly the same thing but takes 8.5 times as long (27m/
> 227m).
>
> The profile on ONE execution of the Ruby program produced:
>
> % cumulative self self total
> time seconds seconds calls ms/call ms/call name
> 65.18 14.15 14.15 1 14150.00 21700.00 Array#each
> 12.44 16.85 2.70 93792 0.03 0.03 Array#[]
> 8.15 18.62 1.77 62108 0.03 0.03 String#split
> 6.73 20.08 1.46 53215 0.03 0.03 String#==
> 2.58 20.64 0.56 66 8.48 12.73 Range#each
>
> It would be SO much nicer to rewrite some stuff that needs rewriting
> and write ALL new stuff in Ruby but this looks impossible with these
> times . .
>
> Any suggestions for performance improvements?
>

The general approach I've followed is to write the code in Ruby, see
where the performance bottlenecks are, and then refactor those parts
of the code into a C-extension for Ruby.

Most recently, refactoring a regexp based string tokenizer into a
Ragel based C-extension gave about 13x performance improvement.

Add salt to taste.

Blessings,
TwP

Robert Klemme

6/24/2008 5:03:00 PM

0

On 24.06.2008 18:23, Philip Rhoades wrote:
> People,
>
> I have asked before about Ruby to C conversion programs and other
> alternatives with no really satisfactory solution for my particular
> situation. I decided to test out a conversion of one of the C programs
> and see what sort of results I get. This particular small C program is
> called 32,000 times from loops within a shell script. The program then
> processes a text file, writes a text file and exits. The same shell
> script with a Ruby program replacing the C program does exactly the same
> thing but takes 8.5 times as long (27m/227m).
>
> The profile on ONE execution of the Ruby program produced:
>
> % cumulative self self total
> time seconds seconds calls ms/call ms/call name
> 65.18 14.15 14.15 1 14150.00 21700.00 Array#each
> 12.44 16.85 2.70 93792 0.03 0.03 Array#[]
> 8.15 18.62 1.77 62108 0.03 0.03 String#split
> 6.73 20.08 1.46 53215 0.03 0.03 String#==
> 2.58 20.64 0.56 66 8.48 12.73 Range#each
>
> It would be SO much nicer to rewrite some stuff that needs rewriting and
> write ALL new stuff in Ruby but this looks impossible with these times . .
>
> Any suggestions for performance improvements?

Difficult without seeing the Ruby program. But I'd also start with
rewriting the script in Ruby so you do not have multiple processes.

Kind regards

robert

Phil Rhoades

6/24/2008 5:34:00 PM

0

Rob,


Rob Biedenharn wrote:
> On Jun 24, 2008, at 12:23 PM, Philip Rhoades wrote:
>> People,
>>
>> I have asked before about Ruby to C conversion programs and other
>> alternatives with no really satisfactory solution for my particular
>> situation.
>
> Are you looking at all solutions?


I guess I was mostly interested in the Ruby to C conversion program but
it is not really ready . .


>> I decided to test out a conversion of one of the C programs and see
>> what sort of results I get. This particular small C program is called
>> 32,000 times from loops within a shell script. The program then
>> processes a text file, writes a text file and exits. The same shell
>> script with a Ruby program replacing the C program does exactly the
>> same thing but takes 8.5 times as long (27m/227m).
>
> Can you replace the whole shell script with a ruby program? Then
> whatever startup cost you have for the Ruby interpreter is paid once
> rather than 32_000 times.


Yes, good point - I should have thought of that - the script is bigger
than the program but that is the next step I guess . .


>> The profile on ONE execution of the Ruby program produced:
>>
>> % cumulative self self total
>> time seconds seconds calls ms/call ms/call name
>> 65.18 14.15 14.15 1 14150.00 21700.00 Array#each
>> 12.44 16.85 2.70 93792 0.03 0.03 Array#[]
>> 8.15 18.62 1.77 62108 0.03 0.03 String#split
>> 6.73 20.08 1.46 53215 0.03 0.03 String#==
>> 2.58 20.64 0.56 66 8.48 12.73 Range#each
>>
>> It would be SO much nicer to rewrite some stuff that needs rewriting
>> and write ALL new stuff in Ruby but this looks impossible with these
>> times . .
>>
>> Any suggestions for performance improvements?
>>
>> I am using F9.
>
> F9? Is that some key in an IDE that runs your ruby code?


Sorry, Fedora 9.


> Depending on what your C program has to do and why a shell script is
> calling it in the first place, you might have other benefits from using
> Ruby in place of the shell script. A couple years back, I worked on a
> similar kind of project that was replacing shell scripts with Perl and
> there were many benefits that Perl could exploit that just could not be
> managed by constructs in the shell.


I would be surprised if there was much improvement in the shell script
part itself but the reduced processes might have a big impact.

Thanks,

Phil.
--
Philip Rhoades

Pricom Pty Limited (ACN 003 252 275 ABN 91 003 252 275)
GPO Box 3411
Sydney NSW 2001
Australia
E-mail: phil@pricom.com.au

Phil Rhoades

6/24/2008 5:35:00 PM

0

ara,


ara.t.howard wrote:
>
> On Jun 24, 2008, at 10:23 AM, Philip Rhoades wrote:
>
>> Any suggestions for performance improvements?
>
> post your code.


I will convert the shell script as well (as others have suggested) and
then post something.

Thanks,

Phil.
--
Philip Rhoades

Pricom Pty Limited (ACN 003 252 275 ABN 91 003 252 275)
GPO Box 3411
Sydney NSW 2001
Australia
E-mail: phil@pricom.com.au

Phil Rhoades

6/24/2008 5:41:00 PM

0

Tim,


Tim Pease wrote:
>
> On Jun 24, 2008, at 10:23 AM, Philip Rhoades wrote:
>
>> People,
>>
>> I have asked before about Ruby to C conversion programs and other
>> alternatives with no really satisfactory solution for my particular
>> situation. I decided to test out a conversion of one of the C
>> programs and see what sort of results I get. This particular small C
>> program is called 32,000 times from loops within a shell script. The
>> program then processes a text file, writes a text file and exits. The
>> same shell script with a Ruby program replacing the C program does
>> exactly the same thing but takes 8.5 times as long (27m/227m).
>>
>> The profile on ONE execution of the Ruby program produced:
>>
>> % cumulative self self total
>> time seconds seconds calls ms/call ms/call name
>> 65.18 14.15 14.15 1 14150.00 21700.00 Array#each
>> 12.44 16.85 2.70 93792 0.03 0.03 Array#[]
>> 8.15 18.62 1.77 62108 0.03 0.03 String#split
>> 6.73 20.08 1.46 53215 0.03 0.03 String#==
>> 2.58 20.64 0.56 66 8.48 12.73 Range#each
>>
>> It would be SO much nicer to rewrite some stuff that needs rewriting
>> and write ALL new stuff in Ruby but this looks impossible with these
>> times . .
>>
>> Any suggestions for performance improvements?
>>
>
> The general approach I've followed is to write the code in Ruby, see
> where the performance bottlenecks are, and then refactor those parts of
> the code into a C-extension for Ruby.


I looked at that but it sort of defeats the purpose of rewriting in Ruby . .


> Most recently, refactoring a regexp based string tokenizer into a Ragel
> based C-extension gave about 13x performance improvement.


Do you mean it was originally a Ruby regexp based string tokenizer?

Thanks,

Phil.
--
Philip Rhoades

Pricom Pty Limited (ACN 003 252 275 ABN 91 003 252 275)
GPO Box 3411
Sydney NSW 2001
Australia
E-mail: phil@pricom.com.au

Phil Rhoades

6/24/2008 5:43:00 PM

0

Robert,


Robert Klemme wrote:
> On 24.06.2008 18:23, Philip Rhoades wrote:
>> People,
>>
>> I have asked before about Ruby to C conversion programs and other
>> alternatives with no really satisfactory solution for my particular
>> situation. I decided to test out a conversion of one of the C
>> programs and see what sort of results I get. This particular small C
>> program is called 32,000 times from loops within a shell script. The
>> program then processes a text file, writes a text file and exits. The
>> same shell script with a Ruby program replacing the C program does
>> exactly the same thing but takes 8.5 times as long (27m/227m).
>>
>> The profile on ONE execution of the Ruby program produced:
>>
>> % cumulative self self total
>> time seconds seconds calls ms/call ms/call name
>> 65.18 14.15 14.15 1 14150.00 21700.00 Array#each
>> 12.44 16.85 2.70 93792 0.03 0.03 Array#[]
>> 8.15 18.62 1.77 62108 0.03 0.03 String#split
>> 6.73 20.08 1.46 53215 0.03 0.03 String#==
>> 2.58 20.64 0.56 66 8.48 12.73 Range#each
>>
>> It would be SO much nicer to rewrite some stuff that needs rewriting
>> and write ALL new stuff in Ruby but this looks impossible with these
>> times . .
>>
>> Any suggestions for performance improvements?
>
> Difficult without seeing the Ruby program. But I'd also start with
> rewriting the script in Ruby so you do not have multiple processes.


Yes, that seems like a good start.

Thanks,

Phil.
--
Philip Rhoades

Pricom Pty Limited (ACN 003 252 275 ABN 91 003 252 275)
GPO Box 3411
Sydney NSW 2001
Australia
E-mail: phil@pricom.com.au

Tim Pease

6/24/2008 6:04:00 PM

0


On Jun 24, 2008, at 11:41 AM, Philip Rhoades wrote:

> Tim,
>
> Do you mean it was originally a Ruby regexp based string tokenizer?
>

Yes

pure ruby regexp => ragel / ruby c-extension


TwP

Phil Rhoades

6/25/2008 10:13:00 PM

0

Rob,


Rob Biedenharn wrote:
> On Jun 24, 2008, at 12:23 PM, Philip Rhoades wrote:
>> People,
>>
>> I have asked before about Ruby to C conversion programs and other
>> alternatives with no really satisfactory solution for my particular
>> situation.
>
> Are you looking at all solutions?
>
>> I decided to test out a conversion of one of the C programs and see
>> what sort of results I get. This particular small C program is called
>> 32,000 times from loops within a shell script. The program then
>> processes a text file, writes a text file and exits. The same shell
>> script with a Ruby program replacing the C program does exactly the
>> same thing but takes 8.5 times as long (27m/227m).
>
> Can you replace the whole shell script with a ruby program? Then
> whatever startup cost you have for the Ruby interpreter is paid once
> rather than 32_000 times.
>
>> The profile on ONE execution of the Ruby program produced:
>>
>> % cumulative self self total
>> time seconds seconds calls ms/call ms/call name
>> 65.18 14.15 14.15 1 14150.00 21700.00 Array#each
>> 12.44 16.85 2.70 93792 0.03 0.03 Array#[]
>> 8.15 18.62 1.77 62108 0.03 0.03 String#split
>> 6.73 20.08 1.46 53215 0.03 0.03 String#==
>> 2.58 20.64 0.56 66 8.48 12.73 Range#each
>>
>> It would be SO much nicer to rewrite some stuff that needs rewriting
>> and write ALL new stuff in Ruby but this looks impossible with these
>> times . .
>>
>> Any suggestions for performance improvements?
>>
>> I am using F9.
>
> F9? Is that some key in an IDE that runs your ruby code?
>
>> Thanks,
>>
>> Phil.
>> --
>> Philip Rhoades
>>
>> Pricom Pty Limited (ACN 003 252 275 ABN 91 003 252 275)
>> GPO Box 3411
>> Sydney NSW 2001
>> Australia
>> E-mail: phil@pricom.com.au
>
> Depending on what your C program has to do and why a shell script is
> calling it in the first place, you might have other benefits from using
> Ruby in place of the shell script. A couple years back, I worked on a
> similar kind of project that was replacing shell scripts with Perl and
> there were many benefits that Perl could exploit that just could not be
> managed by constructs in the shell.


I tried:

- Using v1.9: This was actually a little bit worse than the already 8.4
times slower v1.8

- Replacing the shell scripts as well and having one consolidated Ruby
script: This improved the slowness from a factor of 8.4 to 7.4

- Replacing the array in the Ruby script (I was reading a large file
into an array for processing): This improved the slowness from a factor
of 7.4 to 4.0

It would be nice if I could improve the Ruby script still further but
the profile now looks like:

% cumulative self self total
time seconds seconds calls ms/call ms/call name
59.94 8.44 8.44 1 8440 14080 Object#statz
10.44 9.91 1.47 58318 0.03 0.03 Array#[]
8.1 11.05 1.14 26630 0.04 0.04 String#split
4.05 11.62 0.57 70 8.14 616.14 Range#each
4.05 12.19 0.57 17737 0.03 0.03 String#==
3.05 12.62 0.43 8872 0.05 0.05 IO#gets
2.98 13.04 0.42 8866 0.05 0.05 Fixnum#-
2.34 13.37 0.33 9737 0.03 0.03 Fixnum#+
2.06 13.66 0.29 8916 0.03 0.03 String#to_i
1.92 13.93 0.27 9955 0.03 0.03 Array#[]=

The change to a single Ruby script simplified the previous setup quite
markedly but I'm not sure that much more can be done with the relatively
small script now. I logged into "ruby Talk" on
http://codeforp... but I couldn't see anywhere to post the code.
I can mail the files to anyone who is interested in having a look.

Thanks,

Phil.
--
Philip Rhoades

Pricom Pty Limited (ACN 003 252 275 ABN 91 003 252 275)
GPO Box 3411
Sydney NSW 2001
Australia
E-mail: phil@pricom.com.au