Asp Forum - what's wrong with these calculations?

Ivo Dancet

12/5/2007 11:31:00 AM

Hi

I thought, why search for two hours being totally frustrated if I can
just ask...

why does this happen:

>> 73.07-63.00
=> 10.07 # ok, normal!
>> 73.07-64.00
=> 9.06999999999999 # why oh why?

What should I do about it?

Thanks
-c

8 Answers

Ivo Dancet

12/5/2007 11:48:00 AM

Ok

I found this thread which explains this problem: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby...

So, now I understand why this happens, but I still would like to get
the proper result. How can I achieve that?

Thanks
-c

Op 5-dec-07, om 12:30 heeft Ivo Dancet het volgende geschreven:

> Hi
>
> I thought, why search for two hours being totally frustrated if I
> can just ask...
>
> why does this happen:
>
> >> 73.07-63.00
> => 10.07 # ok, normal!
> >> 73.07-64.00
> => 9.06999999999999 # why oh why?
>
> What should I do about it?
>
> Thanks
> -c
>
>

hemant

12/5/2007 11:52:00 AM

On Dec 5, 2007 5:00 PM, Ivo Dancet <caifara.subscribe@gmail.com> wrote:
> Hi
>
> I thought, why search for two hours being totally frustrated if I can
> just ask...
>
> why does this happen:
>
> >> 73.07-63.00
> => 10.07 # ok, normal!
> >> 73.07-64.00
> => 9.06999999999999 # why oh why?

Thats folly of floats, you can't do much apart from :

sprintf("%.2f",(73.07-64.00)).to_f

--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://g...

hemant

12/5/2007 11:54:00 AM

On Dec 5, 2007 5:22 PM, hemant <gethemant@gmail.com> wrote:
> On Dec 5, 2007 5:00 PM, Ivo Dancet <caifara.subscribe@gmail.com> wrote:
> > Hi
> >
> > I thought, why search for two hours being totally frustrated if I can
> > just ask...
> >
> > why does this happen:
> >
> > >> 73.07-63.00
> > => 10.07 # ok, normal!
> > >> 73.07-64.00
> > => 9.06999999999999 # why oh why?
>
> Thats folly of floats, you can't do much apart from :
>
> sprintf("%.2f",(73.07-64.00)).to_f
>

generally, I add a method to Float class like this:

class Float
def r2p places
sprintf("%.#{places}f",self).to_f
end
end

Jason Roelofs

12/5/2007 12:34:00 PM

Note: parts of this message were removed by the gateway to make it a legal Usenet post.

On Dec 5, 2007 6:54 AM, hemant <gethemant@gmail.com> wrote:

> On Dec 5, 2007 5:22 PM, hemant <gethemant@gmail.com> wrote:
> > On Dec 5, 2007 5:00 PM, Ivo Dancet <caifara.subscribe@gmail.com> wrote:
> > > Hi
> > >
> > > I thought, why search for two hours being totally frustrated if I can
> > > just ask...
> > >
> > > why does this happen:
> > >
> > > >> 73.07-63.00
> > > => 10.07 # ok, normal!
> > > >> 73.07-64.00
> > > => 9.06999999999999 # why oh why?
> >
> > Thats folly of floats, you can't do much apart from :
> >
> > sprintf("%.2f",(73.07-64.00)).to_f
> >
>
> generally, I add a method to Float class like this:
>
> class Float
> def r2p places
> sprintf("%.#{places}f",self).to_f
> end
> end
>
>
Also look into using the BigDecimal class. It does special processing to
keep floating point precision.

Jason

Phrogz

12/5/2007 2:10:00 PM

On Dec 5, 4:54 am, hemant <gethem...@gmail.com> wrote:
> On Dec 5, 2007 5:22 PM, hemant <gethem...@gmail.com> wrote:
>
>
>
> > On Dec 5, 2007 5:00 PM, Ivo Dancet <caifara.subscr...@gmail.com> wrote:
> > > Hi
>
> > > I thought, why search for two hours being totally frustrated if I can
> > > just ask...
>
> > > why does this happen:
>
> > > >> 73.07-63.00
> > > => 10.07 # ok, normal!
> > > >> 73.07-64.00
> > > => 9.06999999999999 # why oh why?
>
> > Thats folly of floats, you can't do much apart from :
>
> > sprintf("%.2f",(73.07-64.00)).to_f
>
> generally, I add a method to Float class like this:
>
> class Float
> def r2p places
> sprintf("%.#{places}f",self).to_f
> end
> end

Note that, due to the nature of floats, calling to_f on that string
brings you back into the problem again:

x = ( 73.07 - 64.00 )
y = x.r2p( 2 )
p x, y, "%.20f" % y

#=> 9.06999999999999
#=> 9.07
#=> "9.07000000000000028422"

Jason Roelofs

12/5/2007 2:31:00 PM

Note: parts of this message were removed by the gateway to make it a legal Usenet post.

On Dec 5, 2007 9:15 AM, Phrogz <phrogz@mac.com> wrote:

> On Dec 5, 4:54 am, hemant <gethem...@gmail.com> wrote:
> > On Dec 5, 2007 5:22 PM, hemant <gethem...@gmail.com> wrote:
> >
> >
> >
> > > On Dec 5, 2007 5:00 PM, Ivo Dancet <caifara.subscr...@gmail.com>
> wrote:
> > > > Hi
> >
> > > > I thought, why search for two hours being totally frustrated if I
> can
> > > > just ask...
> >
> > > > why does this happen:
> >
> > > > >> 73.07-63.00
> > > > => 10.07 # ok, normal!
> > > > >> 73.07-64.00
> > > > => 9.06999999999999 # why oh why?
> >
> > > Thats folly of floats, you can't do much apart from :
> >
> > > sprintf("%.2f",(73.07-64.00)).to_f
> >
> > generally, I add a method to Float class like this:
> >
> > class Float
> > def r2p places
> > sprintf("%.#{places}f",self).to_f
> > end
> > end
>
> Note that, due to the nature of floats, calling to_f on that string
> brings you back into the problem again:
>
> x = ( 73.07 - 64.00 )
> y = x.r2p( 2 )
> p x, y, "%.20f" % y
>
> #=> 9.06999999999999
> #=> 9.07
> #=> "9.07000000000000028422"
>
>
require 'bigdecimal'
require 'bigdecimal/math'
include BigMath

(BigDecimal("73.07") - BigDecimal("64.00")).to_f #=> 9.07

http://www.ruby-doc.org/stdlib/libdoc/bigdecimal/rdoc/...

Jason

Phrogz

12/5/2007 2:55:00 PM

On Dec 5, 7:31 am, Jason Roelofs <jameskil...@gmail.com> wrote:
> Note: parts of this message were removed by the gateway to make it a legal Usenet post.
>
> On Dec 5, 2007 9:15 AM, Phrogz <phr...@mac.com> wrote:
>
>
>
> > On Dec 5, 4:54 am, hemant <gethem...@gmail.com> wrote:
> > > On Dec 5, 2007 5:22 PM, hemant <gethem...@gmail.com> wrote:
>
> > > > On Dec 5, 2007 5:00 PM, Ivo Dancet <caifara.subscr...@gmail.com>
> > wrote:
> > > > > Hi
>
> > > > > I thought, why search for two hours being totally frustrated if I
> > can
> > > > > just ask...
>
> > > > > why does this happen:
>
> > > > > >> 73.07-63.00
> > > > > => 10.07 # ok, normal!
> > > > > >> 73.07-64.00
> > > > > => 9.06999999999999 # why oh why?
>
> > > > Thats folly of floats, you can't do much apart from :
>
> > > > sprintf("%.2f",(73.07-64.00)).to_f
>
> > > generally, I add a method to Float class like this:
>
> > > class Float
> > > def r2p places
> > > sprintf("%.#{places}f",self).to_f
> > > end
> > > end
>
> > Note that, due to the nature of floats, calling to_f on that string
> > brings you back into the problem again:
>
> > x = ( 73.07 - 64.00 )
> > y = x.r2p( 2 )
> > p x, y, "%.20f" % y
>
> > #=> 9.06999999999999
> > #=> 9.07
> > #=> "9.07000000000000028422"
>
> require 'bigdecimal'
> require 'bigdecimal/math'
> include BigMath
>
> (BigDecimal("73.07") - BigDecimal("64.00")).to_f #=> 9.07
>
> http://www.ruby-doc.org/stdlib/libdoc/bigdecimal/rdoc/...

irb(main):002:0> require 'bigdecimal'
irb(main):003:0> require 'bigdecimal/math'
irb(main):004:0> include BigMath
irb(main):005:0> y = (BigDecimal("73.07") - BigDecimal("64.00")).to_f
=> 9.07
irb(main):006:0> "%.20f" % y
=> "9.07000000000000028422"

You might not see the problem on that particular output, but the
potential inaccuracy still exists as long as you're in the Float
domain.

Ivo Dancet

12/5/2007 3:28:00 PM

Thanks to everyone helping to find me/us a nice solution.

I'm using a method to 'round' the float now.

btw, I noticed the same 'problem' in mysql, so just because someone
may be searching here, the page handling that info is here: http://dev.mysql.com/doc/refman/5.0/en/problems-with-...

-c

comp.lang.ruby

what's wrong with these calculations?

Ivo Dancet

Ivo Dancet

hemant

hemant

Jason Roelofs

Phrogz

Jason Roelofs

Phrogz

Ivo Dancet

x Login to ForumsZone