LB Booster
Programming >> Language extensions >> Maximum size of Random Access File
http://lbb.conforums.com/index.cgi?board=extensions&action=display&num=1460651047
Maximum size of Random Access File
Post by Richard Russell on Apr 14th, 2016, 4:24pm

There's a comment at the LB Community Forum which refers to a 4 Gigabytes "Windows imposed limit" for a Random Access File. I don't know whether LB 4 imposes such a limit, but Windows certainly doesn't (at least, so long as the storage device is formatted NTFS rather than FAT).

In LBB not only can the file be of any length, up to the capacity of the storage device, but even the record number isn't restricted to a 4G limit!

Try this, if you have enough free space on your drive. It first writes and then reads the 5-billionth record in a file defined as having records which are 1 byte long (it will take a few seconds to run):

Code:
    open "hugeraf.dat" for random as #f len = 1
    field #f, 1 as v$
    v$ = "x"
    put #f, 5000000000
    close #f
    
    open "hugeraf.dat" for random as #g len = 1
    field #g, 1 as w$
    print "File size is "; lof(#g)
    get #g, 5000000000
    close #g
    
    print "Data read = "; w$
    kill "hugeraf.dat"

Richard.
Re: Maximum size of Random Access File
Post by RobM on Apr 14th, 2016, 5:16pm

As I was typing that I wasn't sure if it was correct for the modern file standard embarassed

I would doubt anyone working with LB or LBB would want to work with a database that large. As your code proves, it is possible but exceedingly slow.
Re: Maximum size of Random Access File
Post by Richard Russell on Apr 14th, 2016, 5:36pm

on Apr 14th, 2016, 5:16pm, RobM wrote:
As your code proves, it is possible but exceedingly slow.

Creating a 5 Gbyte file from scratch is slow, but that's not what typically happens with a database. Once the file is created you should find that seeking and reading/writing is in fact exceedingly fast, and having a large, sparse, file may be easier than deriving the record number with a hash and having to deal with collisions.

With modern storage devices having typically hundreds of Gigabytes, or even Terabytes, of capacity I certainly wouldn't rule out using a RAF considerably bigger than 4 Gbytes. LBB is entirely suitable for that.

The real point I was trying to make is that LB 4 is very limiting. Having a 4 Gbyte file-size limit or a 32-bit record-number limit is not really acceptable in 2016. LBB has neither of those limits.

Richard.
Re: Maximum size of Random Access File
Post by Richard Russell on Apr 14th, 2016, 9:42pm

on Apr 14th, 2016, 5:36pm, Richard Russell wrote:
Having a 4 Gbyte file-size limit or a 32-bit record-number limit is not really acceptable in 2016.

Chris Iverson writes at the Community Forum that "The file size parameter in many of the Win32 file access APIs is still 32-bit for compatibility reasons" but what APIs are those? GetFileSize, SetFilePointer and the WIN32_FIND_DATA structure have all been 64-bit-size compatible since at least Windows 95 - that's over twenty years!

Whatever the reason for LB 4 having a 4 Gbyte file-size limit might be, it's not that the Win32 API ever enforced it.

Richard.
Re: Maximum size of Random Access File
Post by RobM on Apr 14th, 2016, 10:29pm

Chris recently mentioned that much of what he writes in regards to how LB works in the background is just speculation wink
Re: Maximum size of Random Access File
Post by Richard Russell on Apr 15th, 2016, 10:03am

on Apr 14th, 2016, 10:29pm, RobM wrote:
Chris recently mentioned that much of what he writes in regards to how LB works in the background is just speculation wink

Well indeed it must be, since only Carl knows what actually happens (and even he may not always know if it's buried within SmallTalk). Similarly any comments I make on the internal workings of LB are just intelligent guesses based on the symptoms and experience gained from writing LBB.

But in this case Chris was referring not to the internals of LB but to the Windows API. Claiming that many Win32 APIs support only a 32-bit file-size is, I'm afraid, plain wrong. Yes it can be easier to use a 32-bit size value, and I've done that many times myself (indeed it was a limitation of LBB pre v3.00), but it's not actually enforced by the API.

Richard.