LB Booster
Programming >> Language extensions >> Floating point precision
http://lbb.conforums.com/index.cgi?board=extensions&action=display&num=1420582924
Floating point precision
Post by Richard Russell on Jan 6th, 2015, 9:22pm

There's currently a thread at the LB Community Forum discussing the precision of floating-point numbers. Somebody has discovered a strange feature of LB:

Code:
print using( "#.######################", (1/7)*7 )
print using( "#.#######################", (1/7)*7 )

LB 4.04 gives:

1.0000000000000000000000
0.99999999999999991611392

In other words attempting to print 23 decimal places rather than 22 produces a less accurate result!

LBB behaves sensibly, the output being:

1.0000000000000000000000
1.00000000000000000000000

I was aware of an anomaly when USING is used to format integers in LB:

Code:
n = 12345678901234567890
print using( "#######################", n )

LB 4.04 outputs:

12345678901234567168

whilst LBB outputs:

12345678901234567890

but the misbehaviour with floating-point numbers is a new one on me.

Richard.