[REBOL] Re: strange results with decimal!
From: joel:neely:fedex at: 9-Jan-2002 23:59
Hi, Thorsten,
Thorsten Moeller wrote:
> Hi rebolers,
>
> i have a script that gives to produce an output when there is a
> difference between two values. Therefore i read a textfile,
> convert some fields to decimal values and add them together to
> one or another sum. if a specific condition comes true they are
> are compared should be 0 (checked manually with a small amount).
> For some reason i get results like "-1.3476653466456E-12".
>
> Does anybody understand this?
>
> Any suggestions will be appreciated!!
>
I strongly suspect you're falling victim to binary round-off.
In general it is A Bad Idea to perform floating-point (decimal!)
calculations and then expect to get an exact equality; rather
you should check for the difference between two values being
very small. For example, instead of
a: ...lots of decimal! calculation...
b: ...more decimal! calculation or literal...
if a = b ...
you should try
a: ...as above...
b: ...as above...
if 1.0e-10 > a - b ...
This is a well-known phenomenon in numerical analysis; to see
why it's the case, imagine performing the following decimal
calculation on a calculator with a fixed number of digits (or
with pencil and paper using only e.g., 5 digits):
a: 1.0 / 3.0
b: 1.0 / 3.0
c: 1.0 / 3.0
print a + b + c
If you use exactly 5 digits of precision, you'll get
a = 0.3333
b = 0.3333
c = 0.3333
a + b + c = 0.9999
since 1/3 does not have a terminating fractional expansion.
The only divisors which produce terminating fractions are
those containing only powers of 2 and/or 5, since those are
the only divisors of 10.
Since nothing but 2 divides 2, the only terminating binary
fractions are those with divisors of powers of two. Since
we use decimal externally, the conversion to binary and
subsequent calculation are likely to produce the phenomenon.
The reason why most desk calculators will perform the above
computation and return 1.0 is that they keep additional
hidden
digits beyond what is displayed, and then round to
the display size. If we had kept 6 digits and rounded to 5
digits for display, we'd have gotten a result of
0.99999
and rounded that to the (displayable)
1.0000
HTH!
-jn-
--
Shouting at the ground doesn't enable it to hear any better.
-- Henry Warwick
joel^dot^FIX^PUNCTUATION^neely^at^fedex^dot^com
