[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Newb math question

brian

6/9/2005 12:41:00 AM

Can someone tell me why this code:
puts (9.0-8.9).to_s
results in:
0.0999999999999996

I'm expecting 0.01 as the result. Do I have to specify the precision?
Even with the full precision available on the platform (Windows XP), the
correct answer is still 0.01, not what's printed. What's the deal? I
like this language so far, but this is confusing...

Brian
4 Answers

ES

6/9/2005 12:57:00 AM

0


Le 9/6/2005, "brian" <brian@le-roy.org> a écrit:
>Can someone tell me why this code:
> puts (9.0-8.9).to_s
>results in:
> 0.0999999999999996
>
>I'm expecting 0.01 as the result. Do I have to specify the precision?
>Even with the full precision available on the platform (Windows XP), the
>correct answer is still 0.01, not what's printed. What's the deal? I
>like this language so far, but this is confusing...

Floating point math :/ You can use BigDecimal to avoid most problems.

>Brian

E

--
template<typename duck>
void quack(duck& d) { d.quack(); }

Bill Kelly

6/9/2005 1:58:00 AM

0

Hi Brian,

From: "ES" <ruby-ml@magical-cat.org>
>
> Le 9/6/2005, "brian" <brian@le-roy.org> a écrit:
> >Can someone tell me why this code:
> > puts (9.0-8.9).to_s
> >results in:
> > 0.0999999999999996
> >
> >I'm expecting 0.01 as the result. Do I have to specify the precision?
> >Even with the full precision available on the platform (Windows XP), the
> >correct answer is still 0.01, not what's printed. What's the deal? I
> >like this language so far, but this is confusing...
>
> Floating point math :/ You can use BigDecimal to avoid most problems.

If you'd like more info about the problem, the first
paragraphs of the section entitled Rounding Error in
this document give the gist of it:
http://docs.sun.com/source/806-3568/ncg_gol...



Regards,

Bill

brian

6/9/2005 2:21:00 AM

0

Thanks, I'll give it a read. Thanks to Bill too. I have to say this is
one of the friendlier communities I've come across. The BigDecimal is
working out. Newb's not getting burned with RTFM's, tends to keep them
coming back for more. Thanks again!


Gary Wright wrote:

>
> On Jun 8, 2005, at 8:40 PM, brian wrote:
>
>> Can someone tell me why this code:
>> puts (9.0-8.9).to_s
>> results in:
>> 0.0999999999999996
>>
>> I'm expecting 0.01 as the result. Do I have to specify the
>> precision? Even with the full precision available on the platform
>> (Windows XP), the correct answer is still 0.01, not what's printed.
>> What's the deal? I like this language so far, but this is confusing...
>
>
> This is a floating point math issue not a ruby language issue.
> Some base 10 floating point numbers can not be represented exactly
> when converted to binary (and vice versa). This often leads to small
> differences between the arithmetic results expected by you (in base 10
> format) and calculated by the computer (in binary).
>
> The solution is to look for "close answers" or switch to a different
> encoding (i.e. not binary floating point) for your math. In many
> cases just using integers with an appropriate scaling factor solves
> the problem.
>
> If anyone is interested in all the gory details of this topic here is
> a good read: "What Every Computer Scientist Should Know About
> Floating-Point
> Arithmetic".
>
> http://docs.sun.com/source/806-3568/ncg_gol...
>

Jacob Fugal

6/9/2005 4:28:00 PM

0

On 6/8/05, brian <brian@le-roy.org> wrote:
> Can someone tell me why this code:
> puts (9.0-8.9).to_s
> results in:
> 0.0999999999999996
>
> I'm expecting 0.01 as the result. Do I have to specify the precision?
> Even with the full precision available on the platform (Windows XP), the
> correct answer is still 0.01, not what's printed. What's the deal? I
> like this language so far, but this is confusing...
>
> Brian

Along with all the other good responses, I thought I'd just take the
moment to (once again) state that this behaviour isn't unique to ruby:

ruby: "%.16f" % (9.0-8.9) ==> 0.0999999999999996
perl: sprintf("%.16f", 9.0-8.9) ==> 0.0999999999999996
python: "%.16f" % (9.0-8.9) ==> 0.0999999999999996
php: sprintf("%.16f", 9.0-8.9) ==> 0.0999999999999996
C: sprintf(dest, "%.16f", 9.0-8.9) ==> 0.0999999999999996

In all of the cases, specifying a lower precision allows the
float->string conversion to round to the "expected" result:

ruby: "%.10f" % (9.0-8.9) ==> 0.1000000000
perl: sprintf("%.10f", 9.0-8.9) ==> 0.1000000000
python: "%.10f" % (9.0-8.9) ==> 0.1000000000
php: sprintf("%.10f", 9.0-8.9) ==> 0.1000000000
C: sprintf(dest, "%.10f", 9.0-8.9) ==> 0.1000000000

(The source for round_error_c_2 is identical to round_error_c but with
only 10 instead of 16 digits of precision in the printf.)

The major difference is the default precision used by languages in the
float->string conversion. From experimentation (not actually looking
at sources), I came to the following conclusions on the default
precision:

* ruby and perl both use up to 15 significant digits
* python uses up to 12 significant digits
* php uses up to 14 significant digits
* C uses 6 places after the decimal, always
(with %f instead of %.10f in the printf)

For all of ruby, perl, python and php I say "up to X significant
digits" because significant 0's at the end are truncated (e.g. (1.0 -
0.5).to_s returns '0.5', not '0.500000000000000') and significant
digit rules are obeyed (significant digits before the decimal point
reduce the number of digits after the decimal point, leading zeros
after the decimal point are not counted).

Jacob Fugal