Decimal accuracy
[1/4] from: chalz:earthlink at: 10-Jan-2003 22:53
Okay, I don't think I like this.
>> 261424513284453.055884726735293525 - 261424513284453.0
== 0.0625
Huh? How about this one:
>> 61424513284453.055884726735293525 - 61424513284453.0
== 0.0546875
Closer, but... Something's obviously not right here. In fact, it makes me
curious as to whether or not
>> pi ** 29
== 261424513284461
.. is true. Tried it out in Windows /Core 2.5.3 and RedHat /Core 2.5.0. Am
I missing something, or does REBOL just decide that, once you reach a certain
value, accuracy is a moot point?
(In my first example, the first value is pi ** 29 in Windows calculator,
while the second is pi ** 29 in REBOL..)
[2/4] from: chalz:earthlink at: 10-Jan-2003 22:55
Whoops. No, it's not pi ** 29 in REBOL; it's pi ** 29 in a different
interpreter. Sorry.
--Charles
[3/4] from: sunandadh:aol at: 11-Jan-2003 5:35
Chatz:
> I missing something, or does REBOL just decide that, once you reach a
certain
> value, accuracy is a moot point?
I think you are missing the spec for decimal! From the Core manual:
<<The decimal! data type includes 64-bit standard IEEE floating point
numbers. They are distinguished from integer numbers by a decimal point.
[snip] Decimal numbers span from 2.2250738585072e-308 up to
1.7976931348623e+308 and can contain up to 15 digits of precision.>>
Your example is up beyond the 15 digit accuracy so rounding will occur. As
far as I know, there is no precise spec for how Rebol handles that rounding.
As far as I know, there is no native Rebol support for a number type that
carries more digits of accuracy than decimal!
Sunanda.
[4/4] from: joel::neely::fedex::com at: 11-Jan-2003 10:12
Hi, Sunanda,
[SunandaDH--aol--com] wrote:
> Chatz:
> > I missing something, or does REBOL just decide that, once you
<<quoted lines omitted: 8>>
> ... As far as I know, there is no precise spec for how Rebol
> handles that rounding.
IIRC the IEEE Standard (754) itself addresses rounding. A large
collection of related reference links can be found at
http://cch.loria.fr/documentation/IEEE754/
-jn-
Notes
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted