[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Negative Float conversion to Fixnum

Drew Raines

1/19/2007 8:05:00 PM

I'm sure this has been hashed out somewhere on the list before, but
my searches have turned up fruitless. I had a strange issue come up
while diagnosing a unit test failure.

Notice the first dialog compared with the rest of this IRB session.
What causes -3930.0 to be converted to a Fixnum as -3929 only after
it's been multiplied by 100 or 100.0? Even weirder is that it seems
to only be *this* number. Can I workaround this?

>> (-39.30 * 100).to_i
=> -3929
>> (-39.30 * 100.0).to_i
=> -3929
>> -3930.0.to_i
=> -3930
>> (-39.31 * 100).to_i
=> -3931
>> (-9999.99 * 100).to_i
=> -999999
$ ruby -v
ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-linux]

Thanks.

-Drew
8 Answers

WoNáDo

1/19/2007 8:23:00 PM

0

Drew Raines schrieb:
> I'm sure this has been hashed out somewhere on the list before, but
> my searches have turned up fruitless. I had a strange issue come up
> while diagnosing a unit test failure.
>
> Notice the first dialog compared with the rest of this IRB session.
> What causes -3930.0 to be converted to a Fixnum as -3929 only after
> it's been multiplied by 100 or 100.0? Even weirder is that it seems
> to only be *this* number. Can I workaround this?
>
> >> (-39.30 * 100).to_i
> => -3929
> >> (-39.30 * 100.0).to_i
> => -3929
> >> -3930.0.to_i
> => -3930
> >> (-39.31 * 100).to_i
> => -3931
> >> (-9999.99 * 100).to_i
> => -999999
> $ ruby -v
> ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-linux]
>
> Thanks.
>
> -Drew

irb(main):001:0> 3930-39.30*100
=> 4.54747350886464e-013

Rounding errors are typical for floating point numbers. In this case the result
of -39.30*100 ist a little bit smaller than 3930, so to_i works correct.

WoNáDo

1/19/2007 9:29:00 PM

0

Drew Raines schrieb:
> Can I workaround this?

If you always expect a result for which "x * 100" is a number without a
fractional part, you can use "(-39.30 * 100).round" instead.

Wolfgang Nádasi-Donner

Drew Raines

1/19/2007 9:46:00 PM

0

Wolfgang Nádasi-Donner wrote:

> irb(main):001:0> 3930-39.30*100
> => 4.54747350886464e-013
>
> Rounding errors are typical for floating point numbers. In this case
> the result of -39.30*100 ist a little bit smaller than 3930, so to_i
> works correct.

When I think of rounding error, I think of lost precision due to
rounding too early in a series of floating point calculations. When
multiplying a non-repeating Float point by 100, shouldn't the decimal
move over two places without introducing extra precision?

And how can 39.30*100 really be 3930.000000000000454747350886464...,
but 39.31*100 be 3931.0?

BigDecimal seems to be unaffected, so I guess I'll use that as a
workaround:

>> (39.30*100).to_i
=> 3929
>> (39.30*100).to_d.to_i
=> 3930

Thanks for your response.

-Drew

Ara.T.Howard

1/19/2007 9:54:00 PM

0

Gary Wright

1/20/2007 12:33:00 AM

0


On Jan 19, 2007, at 4:50 PM, Drew Raines wrote:
> And how can 39.30*100 really be 3930.000000000000454747350886464...,
> but 39.31*100 be 3931.0?

Because 39.30 is a base 10 representation of a number and there
is no exact representation of that number in base 2. The Ruby
parser has to do a conversion to base 2 for 39.30 long before
the multiplication occurs so you aren't really multiplying
39.30*100 but instead you are multiplying

39.2999999999999971578290569595992565155029296875 * 100

to get

3929.99999999999971578290569595992565155029296875

which when truncated to an integer is

3929


Gary Wright

William James

1/20/2007 7:02:00 AM

0

Drew Raines wrote:
> Wolfgang Nádasi-Donner wrote:
>
> > irb(main):001:0> 3930-39.30*100
> > => 4.54747350886464e-013
> >
> > Rounding errors are typical for floating point numbers. In this case
> > the result of -39.30*100 ist a little bit smaller than 3930, so to_i
> > works correct.
>
> When I think of rounding error, I think of lost precision due to
> rounding too early in a series of floating point calculations. When
> multiplying a non-repeating Float point by 100, shouldn't the decimal
> move over two places without introducing extra precision?
>
> And how can 39.30*100 really be 3930.000000000000454747350886464...,

Almost all programming languages use binary floating point, so 0.30
can't be
represented exactly. A very few languages use binary-coded decimal.
Examples: Decimal BASIC and Business BASIC (I think). The BASIC
that came with the old 8-bit Atari used binary-coded decimal, so it
would
have had no problem with 3930-39.30*100.

Since computers of today are so much faster than 8-bit computers,
one would think that they could afford to incur the speed penalty
associated with binary-coded decimal.

Drew Raines

2/8/2007 6:48:00 PM

0

Wolfgang Nádasi-Donner wrote:

> Drew Raines schrieb:
>> Can I workaround this?
>
> If you always expect a result for which "x * 100" is a number without
> a fractional part, you can use "(-39.30 * 100).round" instead.

Hal Fulton[1] gave me a good idea:

class Float
FUDGE = 1e-3
def ==(x)
(self-x).abs < FUDGE
end
end

This is better for my tests anyway. assert_equal now works on its
own.

-Drew

Footnotes:
[1] http://lnk.nu/amaz...

James Gray

2/8/2007 6:55:00 PM

0

On Feb 8, 2007, at 12:50 PM, Drew Raines wrote:

> Wolfgang Nádasi-Donner wrote:
>
>> Drew Raines schrieb:
>>> Can I workaround this?
>>
>> If you always expect a result for which "x * 100" is a number without
>> a fractional part, you can use "(-39.30 * 100).round" instead.
>
> Hal Fulton[1] gave me a good idea:
>
> class Float
> FUDGE = 1e-3
> def ==(x)
> (self-x).abs < FUDGE
> end
> end
>
> This is better for my tests anyway. assert_equal now works on its
> own.

Test::Unit already includes an assertion just for this. It's called
assert_in_delta().

James Edward Gray II