[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Floating point calculation problem

Hyun chul Park

7/8/2008 8:40:00 AM

hi.
i'm confused by float point calculation.
like this!

------------------------------------------
irb(main):001:0> 123.6 - 123
=> 0.599999999999994

------------------------------------------

how to i get result 0.6 ?
--
Posted via http://www.ruby-....
3 Answers

Zoran Regvart

7/8/2008 8:51:00 AM

0

Hi,

On Tue, Jul 8, 2008 at 10:39 AM, Hyun chul Park <paxcholy@gmail.com> wrote:
> hi.
> i'm confused by float point calculation.
> like this!
>
> ------------------------------------------
> irb(main):001:0> 123.6 - 123
> => 0.599999999999994
>
> ------------------------------------------
>
> how to i get result 0.6 ?

try using BigDecimal, like:

irb(main):001:0> require 'bigdecimal'
=> true
irb(main):002:0> x = BigDecimal("123.6") - BigDecimal("123")
=> #<BigDecimal:8b13c,'0.6E0',4(16)>
irb(main):003:0> x.to_f
=> 0.6

zoran
--
Human by day user by night

Hyun chul Park

7/8/2008 9:09:00 AM

0

Zoran Regvart wrote:
> Hi,
>
> On Tue, Jul 8, 2008 at 10:39 AM, Hyun chul Park <paxcholy@gmail.com>
> wrote:
>> how to i get result 0.6 ?
> try using BigDecimal, like:
>
> irb(main):001:0> require 'bigdecimal'
> => true
> irb(main):002:0> x = BigDecimal("123.6") - BigDecimal("123")
> => #<BigDecimal:8b13c,'0.6E0',4(16)>
> irb(main):003:0> x.to_f
> => 0.6
>
> zoran

thank you, you're reply.
--
Posted via http://www.ruby-....

Axel Etzold

7/8/2008 9:16:00 AM

0


-------- Original-Nachricht --------
> Datum: Tue, 8 Jul 2008 17:39:50 +0900
> Von: Hyun chul Park <paxcholy@gmail.com>
> An: ruby-talk@ruby-lang.org
> Betreff: Floating point calculation problem

Hi --

> hi.
> i'm confused by float point calculation.
> like this!
>
> ------------------------------------------
> irb(main):001:0> 123.6 - 123
> => 0.599999999999994
>
> ------------------------------------------
>
> how to i get result 0.6 ?

you may want to take a look at this:

http://docs.sun.com/source/806-3568/ncg_gol...

If you want to do exact calculations, you can use fractions

http://www.ruby-doc.org/stdlib/libdoc/rational/rdoc/classes/Rat...

To convert between the two, for each real number (Float), you can use its continued fraction
approximation.

http://www.math.mtu.edu/mathlab/COURSES/holt/dnt/...

Here's a link to a Ruby program that calculates digits of Pi from its continued fraction representation
(and also a Java one, which is much longer):

http://www.cs.utsa.edu/~wagner/pi/pi...

Best regards,

Axel

--
Ist Ihr Browser Vista-kompatibel? Jetzt die neuesten
Browser-Versionen downloaden: http://www.gmx.net/de/...