Chris Pearl
12/1/2006 1:17:00 AM
Someone else might wish to give a more concrete explanation, but as a
rule, floating-point calculations - using the usual Float datatype -
are not accurate, and prone to rounding errors such as the above.
If you're dealing with limited precision (i.e. X digits after the
decimal point) you should consider using the Decimal datatype instead.
On 12/1/06, teeshift <teeshift@gmail.com> wrote:
> I recently wrote a code and haven't been found the bug and start
> suspecting that it was
> the floating computation that causing the problem. I did a little
> experiement and found the
> following phenomena:
>
> I am using
>
> Linux source 2.6.15-27-386 #1 PREEMPT Sat Sep 16 01:51:59 UTC 2006 i686
> GNU/Linux
> Ubuntu is the OS.
>
> #####################
> #!/usr/bin/ruby1.9
>
> a = 1.0
>
> a.to_f
>
> while(a < 15.0)
> a = a + 0.1
> print a,"\t",a * a,"\n"
> end
> #####################
>
> and the output is...(as following) quite odd. Anyone can explain it?
> or a way to avoid it (while
> keeping the precision)
>
>
> 1.1 1.21
> 1.2 1.44
> 1.3 1.69
> 1.4 1.96
> 1.5 2.25
> 1.6 2.56
> 1.7 2.89
> 1.8 3.24
> 1.9 3.61
> 2.0 4.0
> 2.1 4.41
> 2.2 4.84
> 2.3 5.29000000000001
> 2.4 5.76000000000001
> 2.5 6.25000000000001
> 2.6 6.76000000000001
> 2.7 7.29000000000001
> 2.8 7.84000000000001
> 2.9 8.41000000000001
> 3.0 9.00000000000001
> 3.1 9.61000000000001
> 3.2 10.24
> 3.3 10.89
> 3.4 11.56
> 3.5 12.25
> 3.6 12.96
> 3.7 13.69
> 3.8 14.44
> 3.9 15.21
> 4.0 16.0
> 4.1 16.81
> 4.2 17.64
> 4.3 18.49
> 4.4 19.36
> 4.5 20.25
> 4.6 21.16
> 4.7 22.09
> 4.8 23.04
> 4.9 24.01
> 5.0 25.0
> 5.1 26.01
> 5.2 27.04
> 5.3 28.09
> 5.4 29.16
> 5.5 30.25
> 5.6 31.36
> 5.7 32.49
> 5.8 33.64
> 5.9 34.81
> 6.0 35.9999999999999
> 6.1 37.2099999999999
> 6.19999999999999 38.4399999999999
> 6.29999999999999 39.6899999999999
> 6.39999999999999 40.9599999999999
> 6.49999999999999 42.2499999999999
> 6.59999999999999 43.5599999999999
>
>
>