Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

Decimal accuracy

 [1/4] from: chalz:earthlink at: 10-Jan-2003 22:53


Okay, I don't think I like this.
>> 261424513284453.055884726735293525 - 261424513284453.0
== 0.0625 Huh? How about this one:
>> 61424513284453.055884726735293525 - 61424513284453.0
== 0.0546875 Closer, but... Something's obviously not right here. In fact, it makes me curious as to whether or not
>> pi ** 29
== 261424513284461 .. is true. Tried it out in Windows /Core 2.5.3 and RedHat /Core 2.5.0. Am I missing something, or does REBOL just decide that, once you reach a certain value, accuracy is a moot point? (In my first example, the first value is pi ** 29 in Windows calculator, while the second is pi ** 29 in REBOL..)

 [2/4] from: chalz:earthlink at: 10-Jan-2003 22:55


Whoops. No, it's not pi ** 29 in REBOL; it's pi ** 29 in a different interpreter. Sorry. --Charles

 [3/4] from: sunandadh:aol at: 11-Jan-2003 5:35


Chatz:
> I missing something, or does REBOL just decide that, once you reach a
certain
> value, accuracy is a moot point?
I think you are missing the spec for decimal! From the Core manual: <<The decimal! data type includes 64-bit standard IEEE floating point numbers. They are distinguished from integer numbers by a decimal point. [snip] Decimal numbers span from 2.2250738585072e-308 up to 1.7976931348623e+308 and can contain up to 15 digits of precision.>> Your example is up beyond the 15 digit accuracy so rounding will occur. As far as I know, there is no precise spec for how Rebol handles that rounding. As far as I know, there is no native Rebol support for a number type that carries more digits of accuracy than decimal! Sunanda.

 [4/4] from: joel::neely::fedex::com at: 11-Jan-2003 10:12


Hi, Sunanda, [SunandaDH--aol--com] wrote:
> Chatz: > > I missing something, or does REBOL just decide that, once you
<<quoted lines omitted: 8>>
> ... As far as I know, there is no precise spec for how Rebol > handles that rounding.
IIRC the IEEE Standard (754) itself addresses rounding. A large collection of related reference links can be found at http://cch.loria.fr/documentation/IEEE754/ -jn-

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted