Strange multiply phenomenon
[1/4] from: swhite::ci::bloomington::mn::us at: 4-Aug-2009 16:20
I have been trying to convert a decimal number, representing dollars and cents, into
a fixed-length string for use in building a fixed-format data file, for sending to a
bank. I have been off by a penny on my current attempt and I have narrowed it down to
the following REBOL statement that I can run at the interpreter prompt, giving the following
result:
>> print to-integer multiply 76.71 100
7670
>>
Can anyone explain this?
Thank you.
Steven White
City of Bloomington
1800 W Old Shakopee Rd
Bloomington MN 55431-3096
USA
952-563-4882 (voice)
952-563-4672 (fax)
steven.white-ci.bloomington.mn.us
[2/4] from: compkarori::gmail::com at: 5-Aug-2009 9:58
Steven
Looks like floating point math
try print round 76.71 * 100
On Wed, Aug 5, 2009 at 9:20 AM, Steven White<swhite-ci.bloomington.mn.us> wrote:
> I have been trying to convert a decimal number, representing dollars and cents, into
a fixed-length string for use in building a fixed-format data file, for sending to a
bank. =A0I have been off by a penny on my current attempt and I have narrowed it down
to the following REBOL statement that I can run at the interpreter prompt, giving the
following result:
>>> print to-integer multiply 76.71 100
<<quoted lines omitted: 13>>
> To unsubscribe from the list, just send an email to
> lists at rebol.com with unsubscribe as the subject.
--
Graham Chiu
http://www.synapsedirect.com
Synapse - the use from anywhere EMR.
[3/4] from: dhsunanda:gm:ail at: 4-Aug-2009 23:01
> print to-integer multiply 76.71 100
You are a victim of imprecise conversions btewen floating poiut numbers
and integers:
>> 7671 - ( 76.71 * 100)
== 9.09494701772928E-13
One work around: use ROUND
>> round 76.71 * 100
== 7671
Sunanda.
[4/4] from: swhite:ci:bloomington:mn:us at: 5-Aug-2009 10:00
This is a pretty amazing mailing list. I don't use it a lot, but I have a few times,
and every time I have gotten a correct answer faster than some support sites we use here
that we pay for.
And, except for the incident of a few years ago that I suspect everyone remembers, it
is an amazingly polite one.
Thank you all.
>>> "Steven White" <swhite-ci.bloomington.mn.us> 8/4/2009 4:20 PM >>>
I have been trying to convert a decimal number, representing dollars and cents, into
a fixed-length string for use in building a fixed-format data file, for sending to a
bank. I have been off by a penny on my current attempt and I have narrowed it down to
the following REBOL statement that I can run at the interpreter prompt, giving the following
result:
>> print to-integer multiply 76.71 100
7670
>>
Can anyone explain this?
Thank you.
Steven White
City of Bloomington
1800 W Old Shakopee Rd
Bloomington MN 55431-3096
USA
952-563-4882 (voice)
952-563-4672 (fax)
steven.white-ci.bloomington.mn.us
Notes
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted