Asp Forum - Integer/Float oddity

horati0

3/31/2005 2:05:00 PM

not really an oddity, more a lack of understanding on my part regarding
ruby's internal type conversion...

(100*(1.15-1))
=> 15.0

(100*(1.15-1)).to_int
=> 14

(100*(1.15-1)).to_int.round
=> 15

15.0.to_int
=> 15

what am i missing here? i never would have thought to use .round to
get the 'correct' result unless someone told me. by correct i mean if
i was doing this with a paper and pencil, i would expect to get 15, not
14.

any insight would be appreciated, especially a way (if possible) to
override this behaviour so i dont run into this in the future.

thanks,
horati0

11 Answers

Patrick Hurley

3/31/2005 2:27:00 PM

On Thu, 31 Mar 2005 23:09:46 +0900, horati0@linuxmail.org
<horati0@linuxmail.org> wrote:
> not really an oddity, more a lack of understanding on my part regarding
> ruby's internal type conversion...
>
> (100*(1.15-1))
> => 15.0
>
> (100*(1.15-1)).to_int
> => 14
>
> (100*(1.15-1)).to_int.round
> => 15
>
> 15.0.to_int
> => 15
>
> what am i missing here? i never would have thought to use .round to
> get the 'correct' result unless someone told me. by correct i mean if
> i was doing this with a paper and pencil, i would expect to get 15, not
> 14.
>
> any insight would be appreciated, especially a way (if possible) to
> override this behaviour so i dont run into this in the future.
>
> thanks,
> horati0
>
>

Two quick notes - I think you meant
> (100*(1.15-1)).round.to_int
> => 15

See if this clears anything up:
printf("%.15f", (100*(1.15-1)))
14.999999999999991=> nil

The result is not really 15, but just really really close, due to the
imprecision of floats - always round moving to ints - to_i is a
"floor" function. Similarly be very careful when comparing floats.

Patrick

Robert Klemme

3/31/2005 2:30:00 PM

<horati0@linuxmail.org> schrieb im Newsbeitrag
news:1112277873.331258.267880@f14g2000cwb.googlegroups.com...
> not really an oddity, more a lack of understanding on my part regarding
> ruby's internal type conversion...
>
> (100*(1.15-1))
> => 15.0
>
> (100*(1.15-1)).to_int
> => 14
>
> (100*(1.15-1)).to_int.round
> => 15
>
> 15.0.to_int
> => 15
>
> what am i missing here? i never would have thought to use .round to
> get the 'correct' result unless someone told me. by correct i mean if
> i was doing this with a paper and pencil, i would expect to get 15, not
> 14.
>
> any insight would be appreciated, especially a way (if possible) to
> override this behaviour so i dont run into this in the future.
>
> thanks,
> horati0

This is a typical limitation of numeric calculations. The reason for what
you see is that 1.15-1 is smaller than 0.15:

>> (1.15-1) - 0.15
=> -8.32667268468867e-17

consequently

>> (100 * (1.15-1))-15
=> -8.88178419700125e-15

this is negative => (100 * (1.15-1)) is smaller than 15 and thus it's
correctly converted to 14 by to_int.

Kind regards

robert

Glenn Parker

3/31/2005 2:53:00 PM

horati0@linuxmail.org wrote:
> not really an oddity, more a lack of understanding on my part regarding
> ruby's internal type conversion...
>
> (100*(1.15-1))
> => 15.0
>
> (100*(1.15-1)).to_int
> => 14
>
> (100*(1.15-1)).to_int.round
> => 15

FWIW, I get 14. Perhaps you meant (100*(1.15-1)).round.to_int ?

Float#to_int simply truncates, while Float#round goes to the nearest
integer.

Try this: "%.20f" % (100*(1.15-1))

--
Glenn Parker | glenn.parker-AT-comcast.net | <http://www.tetrafoi...

michael.b.masi

3/31/2005 3:57:00 PM

Glenn Parker wrote:
> horati0@linuxmail.org wrote:
> > not really an oddity, more a lack of understanding on my part
regarding
> > ruby's internal type conversion...
> >
> > (100*(1.15-1))
> > => 15.0
> >
> > (100*(1.15-1)).to_int
> > => 14
> >
> > (100*(1.15-1)).to_int.round
> > => 15
>
> FWIW, I get 14. Perhaps you meant (100*(1.15-1)).round.to_int ?
>

oops, yep thats what i meant. thanks for all the replies to my query;
i figured it had something to do with the internal representation of fp
numbers.

this whole thing started as a conversation i was having with my dad
about the relative merits of various programming languages. he is a
die-hard xbasic user, and pointed me to the following post on an xb
list:

---------------------------------------------------------------------

[snip]

Message: 7
Date: Wed, 30 Mar 2005 12:59:42 -0000
From: "Bruno Schaefer" <bup.schaefer@...>
Subject: XBasic is one of the best!

I'm back now to work with XBasic, because there are strange things
with other languages.

If you try to calculate the following simple expression

INT(100*(1.15-1))

XBasic produces the correct result: 15.

This is a calculation, which is solved by children easy.

But try this with other languages
(e.g. newLISP, Visual Basic, Liberty Basic, IBasic, Yabasic, Python)
you will get the result

14 !!!!?

What a nonsense ! Does somebody understand this ?

This is one the reason for me to use XBasic!

---------------------------------------------------------------------
i'm assuming that since this behaviour is exhibited in ruby and the
rest of the above "other languages" and NOT xbasic that it is really
due to an idiosyncracy of xbasic that the "right" answer comes out.

so the question becomes, what is different about xbasic's internal
numeric representation that allows it to give a correct answer to this
seemingly simple number manipulation?

horati0

Glenn Parker

3/31/2005 4:32:00 PM

michael.b.masi wrote:
>
> so the question becomes, what is different about xbasic's internal
> numeric representation that allows it to give a correct answer to this
> seemingly simple number manipulation?

Probably nothing about the internal representation is different. The
difference would have to be in xbasic's "INT()" operator, which
apparently performs rounding.

Dealing with floating point numbers can often produce counter-intuitive
results, mostly because simple analogies based on fixed-point math fail.
For example, the concept of "equality" in floating point math often
clashes with common sense. You might assume that the following
expression is true:

(100*(1.15-1)) - 15 == 0

The computer would disagree. What you really want to test is:

abs( (100*(1.15-1)) - 15 ) < TOLERANCE

Where the choice of TOLERANCE depends very much on the application.

--
Glenn Parker | glenn.parker-AT-comcast.net | <http://www.tetrafoi...

Steven Jenkins

3/31/2005 7:23:00 PM

michael.b.masi@gmail.com wrote:
> [snip]
> i'm assuming that since this behaviour is exhibited in ruby and the
> rest of the above "other languages" and NOT xbasic that it is really
> due to an idiosyncracy of xbasic that the "right" answer comes out.
>
> so the question becomes, what is different about xbasic's internal
> numeric representation that allows it to give a correct answer to this
> seemingly simple number manipulation?

Nothing. It's just that Xbasic (apparently) defines INT() to mean
"closest integer" and some other languages define it to mean "largest
integer not exceeding". Both give results consistent with their
definitions. Either one gives you what you need to round *or* truncate.

round(x) is equivalent to truncate(x + .5)

Steve

Mark Hubbart

3/31/2005 8:45:00 PM

On Fri, 1 Apr 2005 00:59:45 +0900, michael.b.masi@gmail.com
<michael.b.masi@gmail.com> wrote:
> i'm assuming that since this behaviour is exhibited in ruby and the
> rest of the above "other languages" and NOT xbasic that it is really
> due to an idiosyncracy of xbasic that the "right" answer comes out.
>
> so the question becomes, what is different about xbasic's internal
> numeric representation that allows it to give a correct answer to this
> seemingly simple number manipulation?
>
> horati0

While the behavior of xbasic in this circumstance would seem to be
"correct", and is probably useful in simple calculations, it would
make it unsuitable for serious calculations. The "correct" answer is
actually not. This is due to the way floating point operations are
calculated by computers.

Without going into too much depth, lets just say that floats are
stored as fractions in the computer's memory. Specifically, as
fractions with denominators that are powers of 2; (x/2^n). Just as
some fractions don't translate precisely to decimal numbers (1/2
becomes 0.5, but 1/7 becomes 0.142857142857142...), many decimal
numbers don't translate precisely to binary numbers (0.5 becomes
0.1b2, but 0.3 becomes 0.00100110011b2...). This causes rounding
errors.

To avoid this, use rational numbers, which are precise fractions:
require 'mathn'
==> true
num = 115/100
==>23/20
num -= 1
==>3/20
num * 100
==>15

cheers,
Mark

Mark Hubbart

3/31/2005 8:52:00 PM

On Fri, 1 Apr 2005 04:23:00 +0900, Steven Jenkins
<steven.jenkins@ieee.org> wrote:
> michael.b.masi@gmail.com wrote:
> > [snip]
> > i'm assuming that since this behaviour is exhibited in ruby and the
> > rest of the above "other languages" and NOT xbasic that it is really
> > due to an idiosyncracy of xbasic that the "right" answer comes out.
> >
> > so the question becomes, what is different about xbasic's internal
> > numeric representation that allows it to give a correct answer to this
> > seemingly simple number manipulation?
>
> Nothing. It's just that Xbasic (apparently) defines INT() to mean
> "closest integer" and some other languages define it to mean "largest
> integer not exceeding". Both give results consistent with their
> definitions. Either one gives you what you need to round *or* truncate.
>
> round(x) is equivalent to truncate(x + .5)

Still, one is considered the standard (which can be translated as
"correct"), and the other isn't. The standardized behavior is to
truncate, if you count on that and it rounds, your computations are
not going to come out the same. IIRC, the IEEE standard for floating
point numbers specifies how they should be converted to integers.

cheers,
Mark

Steven Jenkins

3/31/2005 9:18:00 PM

Mark Hubbart wrote:
>>Nothing. It's just that Xbasic (apparently) defines INT() to mean
>>"closest integer" and some other languages define it to mean "largest
>>integer not exceeding". Both give results consistent with their
>>definitions. Either one gives you what you need to round *or* truncate.
>>
>>round(x) is equivalent to truncate(x + .5)
>
>
> Still, one is considered the standard (which can be translated as
> "correct"), and the other isn't. The standardized behavior is to
> truncate, if you count on that and it rounds, your computations are
> not going to come out the same. IIRC, the IEEE standard for floating
> point numbers specifies how they should be converted to integers.

I don't think it does. IEEE 754 specifies how rounding is applied in
floating-point calculations, but does not define an API.

No standard requires *every* language to have an operator called INT()
that truncates. (C doesn't, for example.) Absent that, Xbasic can define
INT() however it wants. You might wish they'd chosen differently, but
that doesn't make it wrong.

Also note that ISO C defines various math library functions (e.g.,
floor()), but does not mandate IEEE 754 compliance.

Steve

Lutz

4/24/2005 12:46:00 AM

michael.b.masi@gmail.com wrote:
> Glenn Parker wrote:
> > horati0@linuxmail.org wrote:
> > > not really an oddity, more a lack of understanding on my part
> regarding
> > > ruby's internal type conversion...
> > >
> > > (100*(1.15-1))
> > > => 15.0
> > >
> > > (100*(1.15-1)).to_int
> > > => 14
> > >
> > > (100*(1.15-1)).to_int.round
> > > => 15
> >
> > FWIW, I get 14. Perhaps you meant (100*(1.15-1)).round.to_int ?
> >
>
> oops, yep thats what i meant. thanks for all the replies to my
query;
> i figured it had something to do with the internal representation of
fp
> numbers.
>
> this whole thing started as a conversation i was having with my dad
> about the relative merits of various programming languages. he is a
> die-hard xbasic user, and pointed me to the following post on an xb
> list:
>
> ---------------------------------------------------------------------
>
> [snip]
>
> Message: 7
> Date: Wed, 30 Mar 2005 12:59:42 -0000
> From: "Bruno Schaefer" <bup.schaefer@...>
> Subject: XBasic is one of the best!
>
>
> I'm back now to work with XBasic, because there are strange things
> with other languages.
>
> If you try to calculate the following simple expression
>
> INT(100*(1.15-1))
>
> XBasic produces the correct result: 15.
>
> This is a calculation, which is solved by children easy.
>
> But try this with other languages
> (e.g. newLISP, Visual Basic, Liberty Basic, IBasic, Yabasic, Python)
> you will get the result
>
> 14 !!!!?
>
> What a nonsense ! Does somebody understand this ?
>
> This is one the reason for me to use XBasic!
>
> ---------------------------------------------------------------------
> i'm assuming that since this behaviour is exhibited in ruby and the
> rest of the above "other languages" and NOT xbasic that it is really
> due to an idiosyncracy of xbasic that the "right" answer comes out.
>
> so the question becomes, what is different about xbasic's internal
> numeric representation that allows it to give a correct answer to
this
> seemingly simple number manipulation?
>
> horati0

comp.lang.ruby

Integer/Float oddity

horati0

Patrick Hurley

Robert Klemme

Glenn Parker

michael.b.masi

Glenn Parker

Steven Jenkins

Mark Hubbart

Mark Hubbart

Steven Jenkins

Lutz

x Login to ForumsZone