Asp Forum - Benchmark Mono - Ruby

Michael Gebhart

2/6/2005 2:59:00 AM

Hi,

because of my interest in mono and ruby I have done a small benchmark.
These are the results:

Mono Code:

using System;

class Bench {
public static void Main() {
double d = 0;

for (int i = 0; i < 1000000000; i++) {
d = d + i;
}

Console.WriteLine(d);
}

Needs 10.8 Seconds.

In Ruby:

d=0
1000000000.times {
d = d + 1
}

puts d

Needs: 8 minutes and 20 seconds.

Does not look very good :( Is there a possibility to tune my ruby-program,
to be as fast as mono?

Greetings

Mike

19 Answers

Jamis Buck

2/6/2005 3:32:00 AM

On 12:05 Sun 06 Feb , Michael Gebhart wrote:
> Hi,
>
> because of my interest in mono and ruby I have done a small benchmark.
> These are the results:
>
> Mono Code:
>
> using System;
>
> class Bench {
> public static void Main() {
> double d = 0;
>
> for (int i = 0; i < 1000000000; i++) {
> d = d + i;
> }
>
> Console.WriteLine(d);
> }
>
> Needs 10.8 Seconds.
>
>
> In Ruby:
>
> d=0
> 1000000000.times {
> d = d + 1
> }
>
> puts d
>
>
> Needs: 8 minutes and 20 seconds.
>
>
> Does not look very good :( Is there a possibility to tune my ruby-program,
> to be as fast as mono?

At the risk of sounding defensive...

My first question when seeing benchmarks like this: how many
real-world programs have, as their performance bottleneck, a massive
loop that just increments numbers? I'm sure there are some--I don't
deny it--but is your application one of them?

If you want to compare performance between two programming languages
and execution environments, the first rule is to find a benchmark that
properly measures the performance of those languages in the correct
domain. Figure out what you want those languages to do, for real, and
write a simple test that does that thing. Then run your test.

Second rule: figure out what to measure. Is execution time the only
important factor? Or is memory consumption an issue as well? Network
traffic? Development time? Maintenence effort? Portability? Etc.

If execution time and memory consumption are both the most important,
you'd do well to abandon both Ruby and Mono and code in C. Or assembly
code.

You can *always* find a situation in which language X will
"outperform" language Y (for some definition of "outperform"). There
are things that Ruby does better than Mono, I'm sure (but not being
familiar with Mono, I'm not qualified to say what they might be).
Although, at a glance, it looks like the Mono sample took twice as
many lines of code as the Ruby sample...

Just some things to think about. Certainly continue your
investigations--please don't think I'm discouraging that--but make
sure you are testing the right things, and comparing the right
metrics.

- Jamis

--
Jamis Buck
jamis_buck@byu.edu
http://jamis.jam...
------------------------------
"I am Victor of Borge. You will be assimil-nine-ed."

Navindra Umanee

2/6/2005 4:49:00 AM

Michael Gebhart <mail@miketech.net> wrote:
> d=0
> 1000000000.times {
> d = d + 1
> }
>
> puts d

1000000000.times --> 1000000000 method calls to block
d = d + 1 --> 1000000000 method calls to d
puts d --> 2 method calls
-----------------------------------------------------
2000000002 method calls
=====================================================

Well, it sucks, but I guess you might want to realise that you are
performing *at least* 2000000001 method calls in Ruby, probably *a
factor* more at the C-level implementation, using the above code.

I don't think you can really get around that, since even using a for
loop probably requires that you use a range which might involve more
method calls.

I guess that's the price of pervasive and flexible OO throughout the
language. I wonder if ruby2c might be able to optimise this sort of
thing.

How does Mono fare when performing 2000000002 method calls?

Cheers,
Navin.

Kent Sibilev

2/6/2005 5:00:00 AM

Consider this 'tuning':

$ cat t.rb
require 'ruby_to_c'

module Inline
class Ruby < Inline::C
def initialize(mod)
super
end

def optimize(meth)
src = RubyToC.translate(@mod, meth)
@mod.class_eval "alias :#{meth}_slow :#{meth}"
@mod.class_eval "remove_method :#{meth}"
c src
end
end
end

class Test
def run
d=0
1000000000.downto(1) {
d = d + 1
}
return d
end

inline(:Ruby) do |builder|
builder.optimize :run
end
end

puts Test.new.run

$ time ruby t.rb
1000000000

real 0m3.118s
user 0m2.670s
sys 0m0.050s

$ cat t.cs
using System;

class Bench {
public static void Main() {
double d = 0;

for (int i = 0; i < 1000000000; i++) {
d = d + i;
}

Console.WriteLine(d);
}
}

$ mcs t.cs
Compilation succeeded

$ time mono t.exe
4.99999999067109E+17

real 0m37.692s
user 0m34.540s
sys 0m0.040s

Cheers,
Kent.

On Feb 5, 2005, at 10:05 PM, Michael Gebhart wrote:

> Hi,
>
> because of my interest in mono and ruby I have done a small benchmark.
> These are the results:
>
> Mono Code:
>
> using System;
>
> class Bench {
> public static void Main() {
> double d = 0;
>
> for (int i = 0; i < 1000000000; i++) {
> d = d + i;
> }
>
> Console.WriteLine(d);
> }
>
> Needs 10.8 Seconds.
>
>
> In Ruby:
>
> d=0
> 1000000000.times {
> d = d + 1
> }
>
> puts d
>
>
> Needs: 8 minutes and 20 seconds.
>
>
> Does not look very good :( Is there a possibility to tune my
> ruby-program,
> to be as fast as mono?
>
> Greetings
>
> Mike
>

Alexander Staubo

2/6/2005 6:14:00 AM

Kent Sibilev wrote:
> Consider this 'tuning':
[snip]
>
> class Test
> def run
> d=0
> 1000000000.downto(1) {
> d = d + 1
> }
[snip]
> double d = 0;
>
> for (int i = 0; i < 1000000000; i++) {
> d = d + i;
> }

To be fair, the type of the variable in the Mono example is "double",
but the Ruby example uses an int. A more appropriate Ruby version:

def run
d = 0.0
1000000000.downto(1) {
d = d + 1.0
}
return d
end

Alexander.

Kent Sibilev

2/6/2005 6:40:00 AM

That's true, but this example is meaningless anyway. The point is that
if you find more than a couple of places in your application where Ruby
does not provide an acceptable performance, you probably should use
another language for the task. I just wanted to demonstrate that with
the help of RubyInline or even Ruby2C you can translate these
bottleneck spots without much of the hassle.

Cheers,
Kent.

On Feb 6, 2005, at 1:14 AM, Alexander Staubo wrote:

> Kent Sibilev wrote:
>> Consider this 'tuning':
> [snip]
>> class Test
>> def run
>> d=0
>> 1000000000.downto(1) {
>> d = d + 1
>> }
> [snip]
>> double d = 0;
>> for (int i = 0; i < 1000000000; i++) {
>> d = d + i;
>> }
>
> To be fair, the type of the variable in the Mono example is "double",
> but the Ruby example uses an int. A more appropriate Ruby version:
>
> def run
> d = 0.0
> 1000000000.downto(1) {
> d = d + 1.0
> }
> return d
> end
>
> Alexander.
>

Zach Dennis

2/6/2005 7:12:00 AM

Kent,

this was pretty hella cool. I'm going to check out Inline::C now that I
see an easy example and your benchmark. =) Gratzi,

Zach

George Moschovitis

2/6/2005 9:34:00 AM

WOW,

I am impressed!

George

Sam Roberts

2/6/2005 3:44:00 PM

Quoteing mail@miketech.net, on Sun, Feb 06, 2005 at 12:05:12PM +0900:
> Hi,
>
> because of my interest in mono and ruby I have done a small benchmark.

Ruby is an interpreted language. Its strengths are speed of development
and flexibility of implementation. C# is a compiled language.

Java and C# exist in a kind of half-world - more flexible than C++ in
some ways (GC, at least), but not as fast as the lower-level languages.

If you want to do fast math, you should be working in C or C++. If you
want both speed and a beautiful OO language, you write the math in C,
and bind it into ruby.

People doing cryptography in .Net do the same -> all the crypto code is
C/assembler, and it is bound in.

Cheers,
Sam

Alexander Kellett

2/6/2005 9:49:00 PM

On Feb 6, 2005, at 10:07 PM, Caio Tiago Oliveira wrote:
> I've made this benchmarking in some more languages (using d=d+1):
>
> In my Pentium 4 2Ghz 256Mb SDRAM 7200rpm:
>
> Java: 00m14,45s (d as long)
> Java: 00m12,70s (d as double)
> C: 00m03,27s (d as long int)
> C: 00m15,90s (d as double)
> Squeak Smalltalk: 01m48,60s
> Ruby: 18m12,21s
> Python: 14m04,83s

with c+llvm it is instantaneous.
Alex

Kaspar Schiess

2/6/2005 10:39:00 PM

(In response to news:pan.2005.02.06.02.58.29.950821@miketech.net by
Michael Gebhart)

> In Ruby:
>
> d=0
> 1000000000.times {
> d = d + 1
> }

As far as I can see, no one tried YARV. I didn't run these tests myself,
they are the current benchmarks from the YARV-devel list. And yes, the
numbers different, but you get the picture:

whileloop:
i = 0
while i<10000000
i+=1
end
--
user system total real
ruby 18.780000 0.050000 18.830000 ( 18.944069)
yarv 1.210000 0.000000 1.210000 ( 1.215278)

simpleiter:
1000000.times{|simpleiter|
simpleiter
}
--
user system total real
ruby 1.130000 0.000000 1.130000 ( 1.122475)
yarv 0.470000 0.000000 0.470000 ( 0.473542)

--
kaspar

code manufacture - ruby lab
www.tua.ch/ruby

comp.lang.ruby

Benchmark Mono - Ruby

Michael Gebhart

Jamis Buck

Navindra Umanee

Kent Sibilev

Alexander Staubo

Kent Sibilev

Zach Dennis

George Moschovitis

Sam Roberts

Alexander Kellett

Kaspar Schiess

x Login to ForumsZone