[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Re: Benchmarking whole programs

Berger, Daniel

4/5/2005 4:43:00 PM

> -----Original Message-----
> From: gabriele renzi [mailto:surrender_it@remove-yahoo.it]
> Sent: Tuesday, April 05, 2005 10:35 AM
> To: ruby-talk ML
> Subject: Benchmarking whole programs
>
>
> Hi gurus and nubys,
>
> I was playing with the ruby programs in the alioth language
> shootout and I realized I have no idea on how to benchmark
> them all. What I mean is: If I want to benchmark a single
> function I can use the
> usual benchmark code also shown in ri, but what if I want to
> check two
> slightly different implementations of a a single program?
>
> I can't just stick them all in a #meth1 and #meth2 thing, since there
> may be constants defined (actually they almost surely will, at least
> classes and modules), I suppose I could do something like:
> Benchmark.bm do |x|
> x.report {require 'first_impl'; main()}
> x.report {require 'second_impl'; main()}
> end
>
> but this seems kludgy and loading the thing at each run could mess up
> the results, I think.

I don't know. Does the benchmark package take startup time into
account? You can always reverse the tests to ensure that they aren't
being tainted by startup time couldn't you?

Another thing you can do is to run your implementation through the
profiler. I also see ruby-prof on the RAA which you could try.

I also see YABM, but there are no links so I can't comment on that
package.

Dan
3 Answers

Dick Davies

4/5/2005 4:47:00 PM

0

* Berger, Daniel <Daniel.Berger@qwest.com> [0442 17:42]:
> > -----Original Message-----
> > From: gabriele renzi [mailto:surrender_it@remove-yahoo.it]
> > Sent: Tuesday, April 05, 2005 10:35 AM
> > To: ruby-talk ML
> > Subject: Benchmarking whole programs
> >
> >
> > Hi gurus and nubys,
> >
> > I was playing with the ruby programs in the alioth language
> > shootout and I realized I have no idea on how to benchmark
> > them all. What I mean is: If I want to benchmark a single
> > function I can use the
> > usual benchmark code also shown in ri, but what if I want to
> > check two
> > slightly different implementations of a a single program?
> >
> > I can't just stick them all in a #meth1 and #meth2 thing, since there
> > may be constants defined (actually they almost surely will, at least
> > classes and modules), I suppose I could do something like:
> > Benchmark.bm do |x|
> > x.report {require 'first_impl'; main()}
> > x.report {require 'second_impl'; main()}
> > end
> >
> > but this seems kludgy and loading the thing at each run could mess up
> > the results, I think.
>
> I don't know. Does the benchmark package take startup time into
> account? You can always reverse the tests to ensure that they aren't
> being tainted by startup time couldn't you?
>
> Another thing you can do is to run your implementation through the
> profiler. I also see ruby-prof on the RAA which you could try.

For those who don't know -
Don't use that for a walltime, it slows things right down.

For a full app, what's wrong with

time first_impl.rb
time second_impl.rb

--
'Tempers are wearing thin. Let's hope some robot doesn't kill everybody.'
-- Bender
Rasputin :: Jack of All Trades - Master of Nuns

Jeremy Kemper

4/5/2005 5:05:00 PM

0

Dick Davies wrote:
> * Berger, Daniel <Daniel.Berger@qwest.com> [0442 17:42]:
>>Another thing you can do is to run your implementation through the
>>profiler. I also see ruby-prof on the RAA which you could try.
>
> For those who don't know -
> Don't use that for a walltime, it slows things right down.

The recently-posted ruby-prof extension by Shugo Maeda is absolutely
wonderful. Such a simple idea; why did it take so long? It times and
counts method calls with very low overhead using clock, gettimeofday, or
the CPU timer (on Pentium or PowerPC only). Unfortunely, it requires
Ruby 1.8 newer than 2005-03-22, which means CVS, or Ruby 1.9 newer than
2005-03-17.

I've seen a couple of other fast Ruby profilers, but they're either
closed-source or Windows-only. This contribution deserves some hearty
pimping action: go forth and profile with impunity, for it is no longer
intolerably slow.

Thanks Shugo!
jeremy

gabriele renzi

4/5/2005 5:16:00 PM

0

Dick Davies ha scritto:

>>I don't know. Does the benchmark package take startup time into
>>account? You can always reverse the tests to ensure that they aren't
>>being tainted by startup time couldn't you?

yup, I was thinking of doing the same, but see later

>>Another thing you can do is to run your implementation through the
>>profiler. I also see ruby-prof on the RAA which you could try.

I recall rbprof as part of the AspectR distribution, not sure if it
still works. Anyhow, again, see later..


> For those who don't know -
> Don't use that for a walltime, it slows things right down.
>
> For a full app, what's wrong with
>
> time first_impl.rb
> time second_impl.rb

nothing, actually, even if I think that should be something like

time first_impl.rb
time first_impl.rb
time first_impl.rb
time second_impl.rb
time second_impl.rb
time second_impl.rb

to warm up the cache :)

And nothing is wrong with the ideas that Daniel Berger pointed out, I
was just wondering if there was something builtin in ruby's benchmark.rb
or if people had wrote something like this, since it seem something
quite common to think about.
Some changes like "use an array instead of a whole class" or "use two
structs and a mixin instead of two classes", I mean, stuff that changes
larger things that simple methods, are imho quite common, and even if
probably the timing differences could be unvaluable it would be
interesting to look at them :)

Thanks both.