Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search

[REBOL] Re: strange results with decimal!

From: philb:upnaway at: 10-Jan-2002 22:43

Hi Thorsten, I would think that the problem is the way computers hold decimals. As a decimal is held as a binary representaion of the decimal in a finite number of digits (usually 16,32 or 64) in most cases it cannot hold a decimal exactly and some rounding occurs. We dont know how internally Rebol hold decimal numbers but as its C based it may well hold them in base 2 decimal form. For example 9.3 = 9 * 10 + 3 * 0.1 has the binary form 1001.01001 (recurring) meaning 1*2^3 + 0*2^2 + 0*2^1 + 1*2^0 + 0*2^-1 + 1*2^-2 + 0*2^-3 + 0*2^-4 + 1*2^-5 As this is a recurring binary decimal some rounding will occur so the decimal number is not held exactly. Consider the following (typed at the console)
>> 0.33333333333333 + 0.333333333333333 + 0.333333333333333
== 0.999999999999996 This indicates that the number 0.33333333333333 was truncated. I would suggest that one or both of numbers in your example was truncated and gave you the unexpected result. Cheers Phil === Original Message === Hi Joel, thanks for your reply. If devision or multiplication is involvolved i could follow your samples. Perhaps i should be more detailed, as i am only using addition and substraction. File sheme: 15888;some text;1.000,50;1.000,50;some text 15889;some text;50,49;50,48;some text As the first line contains german values, i have to parse out the "." and the convert the field to a decimal like 1.000,50 to 1000,50. After that i add the values to a sum with a foreach loop. the result is: sum-a: 1050,99 sum-b: 1050,98 Now i want the difference to be displayed. print sum-a - sum-b The result is, even with this example, "9.99999999999091E-3" instead of 0,01. If you just type in the lines with the sums and the print statement you can follow the example. Is that the same problem as you described? Thorsten