Author |
Topic: LB 4.5.0 memory allocation failure (Read 1064 times) |
|
Richard Russell
Administrator
member is offline
 
Posts: 1348
|
 |
LB 4.5.0 memory allocation failure
« Thread started on: Aug 17th, 2015, 8:28pm » |
|
As I have said before, LBB v3.00 allocates 512 Mbytes of memory rather than 1 Gbyte (which LB 4.5.0 tries to allocate) because of the likelihood of not being able to get that much contiguous virtual memory. Indeed LB 4.5.0 won't run on my Windows XP machine for that reason, and other people reported the same thing during the beta test period.
Judging by some comments I have seen, Carl doesn't have a terribly good grasp of the mechanics of virtual memory in Windows, because he seems to think the problem is related to the amount of physical memory available, when it has nothing to do with that. It's perfectly possible that you can allocate 1 Gbyte of contiguous virtual memory even if only (say) 512 Mbytes of RAM is fitted, and it's equally possible that you cannot even if 3 Gbytes of physical memory is available.
Fortunately Microsoft changed the behaviour in Windows Vista and later, so you are less likely to encounter the problem now (although it's not impossible).
Richard.
|
|
 Logged
|
|
|
|
tsh73
Full Member
member is offline
Gender: 
Posts: 210
|
|
Re: LB 4.5.0 memory allocation failure
« Reply #1 on: Aug 17th, 2015, 9:10pm » |
|
I dunno about what LB4.5.0 allocates/tries to, but:
I have XP with 4GB RAM and it used to be more then enough.
I used to JB taking very little memory - so for example then measuring program performance, I run it several times leaving completed run windows (mainwin) on-screen.
10 JB programs have no impact on Windows memory.
like this simple program
Code:print time$(), time$("ms")
for i = 1 to 10
print rnd(1)
next
Now, LB 4.5.0 behaves for me the same - I can run 10 instances no problem.
More. I have virtual machine (VMWare Player) to fiddle with - it is set up with 512 MB RAM and Windows 2000. And I still could run 10 instances of JB/ LB4.04/LB 4.5.0
Now that might be crazy thing to do but it used to work pretty good.
Now, with LBB 3, each program run allocates 512 MB - I see it in Process Explorer in "private bytes".
And actual 4GB RAM Windows starts to complain about lack of virtual memory after 4th program instance. Happen to be my fault, see EDIT below
It looks like Windows doesn't care "private bytes" is "virtual" or "never actually used" - it reserves it and refuses to give more.
So I keep copy of LBB 2.51 just in case - it's last version which allocates only 100 mb per instance (and I don't think I've bumped that limit, ever).
It runs 4 instances under 512 MB virtual machine, getting "no room" on 5th
(LB 3 could run only 1 instance on same virtual machine)
EDIT
about crazy things to do.
It looks like my 4GB XP has virtual memory disabled altogether.
I have no idea if it was me and if it was, why did I do that.
|
| « Last Edit: Aug 17th, 2015, 9:19pm by tsh73 » |
Logged
|
|
|
|
Richard Russell
Administrator
member is offline
Posts: 1348
|
|
Re: LB 4.5.0 memory allocation failure
« Reply #2 on: Aug 17th, 2015, 10:01pm » |
|
on Aug 17th, 2015, 9:10pm, tsh73 wrote:| Now, LB 4.5.0 behaves for me the same - I can run 10 instances no problem. |
|
The problem with LB 4.5.0 is not the availability of memory, but that it attempts to reserve 1 Gbyte of contiguous virtual address space. If the memory map is fragmented by DLLs (in the case of my XP PC it's iertutil.dll that is the major source of the problem) it makes no difference how much physical memory is fitted.
Quote:| So I keep copy of LBB 2.51 just in case - it's last version which allocates only 100 mb per instance |
|
Except of course that LBB v2.51 lacks many features of v3.00, and is unsupported. Any anyway 100 Mbytes is still a lot of memory for a small program to use up (see below).
Quote:| It looks like my 4GB XP has virtual memory disabled altogether. |
|
LBB - and many other Windows applications - rely on virtual memory being enabled. It is much easier to allocate a contiguous chunk of virtual memory, and let Windows work out for itself what is actually 'used' (so must be mapped to physical memory) and what is 'allocated but unused' (which need not use up any RAM).
This technique also ensures that any memory allocation error will occur at startup, when the LBB program is initially run, and not at some unpredictable future time after the program has been running for hours or days or weeks.
But, yes, it does mean that if you have disabled virtual memory then you will be able to run only a small number of instances of LBB before it runs out of memory.
With virtual memory enabled, which is usual, a small LBB program should only show up in Task Manager as using a few megabytes of actual memory, and you should be able to run hundreds of instances.
Richard.
|
|
|
|
Richard Russell
Administrator
member is offline
Posts: 1348
|
|
Re: LB 4.5.0 memory allocation failure
« Reply #3 on: Aug 19th, 2015, 04:59am » |
|
Incidentally Linux makes no distinction between reserving memory (which is what LB 4.5.0 tries to do on startup) and allocating memory (which is what LBB does). So if you run them both under Wine the only difference you will see is that LB 4.5.0 tries to allocate twice as much address space (1 Gbyte) as LBB (500 Mbytes).
Richard.
|
|
Logged
|
|
|
|
tsh73
Full Member
member is offline
Gender:
Posts: 210
|
|
Re: LB 4.5.0 memory allocation failure
« Reply #4 on: Aug 19th, 2015, 5:57pm » |
|
Richard,
I enabled virtual memory in XP
Now I have 4 gb real and Windows took some 3gb virtual (Windows-selected default).
And I can run 10 instances of LBB program
- but really, it looks like each instance still really tooks 500mb (seeing "Commit charge" in ProcessExplorer).
Ok may be it is XP that is old and all that.
I fired up Win 7 in VMWare, 1GB ram (VMWare supposed default)
so it shows 2GB memory - 1 read and 1 virtual.
I run a program from LBB - see bbcwrun6.exe in a task manager - and it shows "private working set" (sorry backwards translation from Russian) about 3Mb only - but Process Explorer shows Private Bytes 514 844 bytes (just like in XP)
And I can run only one instance of LBB, no matter that.
It just don't start.
So may be it really takes 500 MB - and you can somewhat decrease that amount in next version?
|
|
Logged
|
|
|
|
Richard Russell
Administrator
member is offline
Posts: 1348
|
|
Re: LB 4.5.0 memory allocation failure
« Reply #5 on: Aug 19th, 2015, 7:12pm » |
|
on Aug 19th, 2015, 5:57pm, tsh73 wrote:| So may be it really takes 500 MB |
|
No, it doesn't. That would be totally unacceptable.
Did you see my earlier post in which I pointed out that Linux (and thus Wine) makes no distinction between 'reserving' and 'committing' virtual memory? If you think LBB is "really" taking 500 Mbytes try running LB 4.5.0 in Wine - it will "really" use 1 Gbyte of RAM!!
But of course it won't. You are reading too much into the 'commit charge' figure reported by Process Explorer. What that figure means is not that LBB is using 500 Mbytes of 'real' memory but that it has 'requested' that 500 Mbytes be made available by Windows if necessary.
As I said before, this has real practical advantages. Specifically it means that if the user's program DIMs or REDIMs a very large array it is guaranteed that an 'out of memory' error will not occur (assuming the 500 Mbytes is sufficient, of course) - although swapping to disk might result if there's not enough physical RAM.
The best test of how much "real" memory LBB uses is to see how many instances of a program can be run. I tried it with this program:
Code: print "Hello world!"
wait
I got bored after running 20 copies, with still only 33% of the memory used:
Richard.
|
|
Logged
|
|
|
|
tsh73
Full Member
member is offline
Gender:
Posts: 210
|
|
Re: LB 4.5.0 memory allocation failure
« Reply #6 on: Aug 19th, 2015, 8:10pm » |
|
Quote:| with still only 33% of the memory used |
|
Just for the record, how much memory your computer has?
I just checked - minimal for Win 7 8 8.1 10 still 1Gb for 32 bit Windows.
|
| « Last Edit: Aug 19th, 2015, 8:16pm by tsh73 » |
Logged
|
|
|
|
Richard Russell
Administrator
member is offline
Posts: 1348
|
|
Re: LB 4.5.0 memory allocation failure
« Reply #7 on: Aug 19th, 2015, 9:31pm » |
|
on Aug 19th, 2015, 8:10pm, tsh73 wrote:| Just for the record, how much memory your computer has? |
|
4 Gbytes I think.
To put the 33% memory usage into context, when I am running no foreground applications at all, apart from Task Manager itself, it reports 28% usage. When I am also running Internet Explorer Edge (which I was at the time, as you can see) it reports 30% usage. So the 20 instances of bbcwrun6.exe were using maybe 3% of the available memory. In fact you can see how much memory each instance was using from the screenshot - only about 2 Megabytes each!
All modern operating systems work the same way. When a program writes to (allocated, but as yet non-existent) memory it generates a 'page fault'. This causes the OS kernel to 'page in' some real memory at that address - a page usually being 4 Kbytes. After 4K of memory has been written another page fault will be generated, and another chunk of real memory paged in, and so on.
Using that strategy it really doesn't matter how much virtual memory your program initially allocates, because it doesn't use up any 'real' memory until it is actually written to. What does matter however is the amount of contiguous address space you try to reserve, because if the memory map has been fragmented (typically by DLLs) it will be impossible to allocate more than the largest fragment. This is completely independent of the amount of physical RAM fitted.
So I have set the amount of memory allocated by LBB to 500 Mbytes because that is around the largest amount of contiguous address space that you can be fairly confident will be available (under XP, anyway). But (initially) none of that 500 Mbytes is mapped by the operating system to physical memory. As I explain above, only when memory is actually written to does it get mapped to real RAM.
Windows, unlike Linux, makes a distinction between 'reserving' and 'committing' virtual memory. It so happens that LB 4.5.0 'reserves' 1 Gbyte of memory whereas LBB 'commits' 500 Mbytes. This will cause Process Explorer to report the memory usage differently, but in all normal circumstances you don't need to worry about the difference. And in Linux/Wine there is no difference anyway.
Richard.
|
|
|
|
tsh73
Full Member
member is offline
Gender:
Posts: 210
|
|
Re: LB 4.5.0 memory allocation failure
« Reply #8 on: Aug 20th, 2015, 01:19am » |
|
Thanks for clarifying that.
RE:
Quote:I fired up Win 7 in VMWare, 1GB ram (VMWare supposed default)
so it shows 2GB memory - 1 read and 1 virtual.
I run a program from LBB (...)
And I can run only one instance of LBB, no matter that.
It just don't start. |
|
After Win7 ends its update cycle, I was able to run 3 instances of LBB program.
Then 4th said "no room".
On next attempt, 4th instance run.
(probably Windows just made more virtual memory - it now shows "peak" commit more then 2 GB - at 2.7)
(and Task Manager shows "memory" - it says "physical memory" at 36%, Process Explorer Commit charge at 80%
Empty values are 32% Task Manager and 28% Process Explorer )
|
| « Last Edit: Aug 20th, 2015, 01:23am by tsh73 » |
Logged
|
|
|
|
Richard Russell
Administrator
member is offline
Posts: 1348
|
|
Re: LB 4.5.0 memory allocation failure
« Reply #9 on: Aug 20th, 2015, 09:30am » |
|
on Aug 20th, 2015, 01:19am, tsh73 wrote:After Win7 ends its update cycle, I was able to run 3 instances of LBB program.
Then 4th said "no room". |
|
That may be because you are running in a VM. This is what it says at MSDN:
"As an alternative to dynamic allocation, the process can simply commit the entire region instead of only reserving it. Both methods result in the same physical memory usage because committed pages do not consume any physical storage until they are first accessed. The advantage of dynamic allocation is that it minimizes the total number of committed pages on the system. For very large allocations, pre-committing an entire allocation can cause the system to run out of committable pages, resulting in virtual memory allocation failures."
It is this "running out of committable pages" that you are probably experiencing. In practice this is very unlikely to be a problem, but if you have specific issues (for example because you are running in a VM) what I can do is to make the amount of committed memory configurable by a setting in the INI file. Would that be helpful?
Richard.
|
|
Logged
|
|
|
|
tsh73
Full Member
member is offline
Gender:
Posts: 210
|
|
Re: LB 4.5.0 memory allocation failure
« Reply #10 on: Aug 20th, 2015, 10:35am » |
|
Quote:what I can do is to make the amount of committed memory configurable by a setting in the INI file. Would that be helpful?
|
|
Yes it definitely will be.
I would be very grateful if it wil became configurable somehow.
|
|
Logged
|
|
|
|
|
