Re: Help with Multi-user application
Post by Richard Russell on Sep 5th, 2016, 3:34pm
There's a thread at the Liberty BASIC Yahoo! Group about sharing files between multiple users. The underlying problem is that in LB 4.04 and LB 4.5.0 a random access file is always opened for both reading and writing - even if you only need to do one or the other!
This is a serious design flaw, because it makes it impossible for one LB program to be writing a random file whilst another LB program concurrently reads that same file, even though there's no fundamental reason why that should not work.
Fortunately LBB provides a solution. An undocumented feature (I really should add it to the help file) is that you can open a random file in a read-only mode, and in so doing allow concurrent access from multiple programs (a maximum of one of which is writing to the file while the others read from it).
Here's a simplistic example; in this primitive case you must start the 'writing' program before you start the 'reading' program, but it demonstrates the principle:
Writing program Code: open "SharedFile.dat" for output as #f len = 30
field #f, 20 as string$, 10 as number
do
i += 1
string$ = "My string ";i
number = i
put #f,i
call delay 100
loop until 0
end
sub delay d
timer d, [pause]
wait
[pause]
timer 0
end sub
Reading program Code: open "SharedFile.dat" for input as #f len = 30
field #f, 20 as string$, 10 as number
do
i += 1
gettrim #f,i
print string$; " "; number
call delay 110
loop until 0
end
sub delay d
timer d, [pause]
wait
[pause]
timer 0
end sub
Perhaps somebody would be kind enough to post something to the Yahoo! group about this capability of LBB.
Richard.
Re: Help with Multi-user application
Post by Alincon on Sep 5th, 2016, 8:09pm
I think I understand the point of your 'primitive' example is to demonstrate the 'for output' and 'for input' phrases for random files.
But, please explain some things that I don't recall seeing before.
I assume that 'i += 1' is equivalent to 'i=i+1'
What is it that becomes 0 and ends the do loop? A binary condition?
How many records are written to the file?
What does the delay sub do that a timer statement does not do?
r.m.
Re: Help with Multi-user application
Post by Richard Russell on Sep 5th, 2016, 8:51pm
on Sep 5th, 2016, 8:09pm, Alincon wrote:| I assume that 'i += 1' is equivalent to 'i=i+1' |
|
Correct.
Quote:| What is it that becomes 0 and ends the do loop? |
|
Nothing - that's the point: it's an infinite loop. Some people like to emphasise this by setting a variable to zero, such as:
Code: the.cows.come.home = 0
do
' an infinite loop
loop until the.cows.come.home
Actually in LBB there's a shorthand way of creating an infinite loop but it's not standard Liberty BASIC code:
Code: do
' an infinite loop
loop
Quote:| How many records are written to the file? |
|
It writes ten per second, so it depends on how long the program is left running. As I said, it was a primitive example to emphasise the principle.
Quote:| What does the delay sub do that a timer statement does not do? |
|
It's just an example of making the program more modular. If you want different delays in different places, calling a common subroutine is an elegant way to achieve it. However it's a technique that cannot safely be used in a program that uses branch labels as event handlers (because whilst the delay subroutine is executing the label will be out-of-scope). So it's almost always better to use SUB event handlers to avoid this problem.
Richard.
Re: Help with Multi-user application
Post by Rod on Sep 6th, 2016, 08:03am
Richard, is it not the writing conflict that needs a solution. In a multiuser environment there will be multiple write attempts that need managed. A solution would be a welcome addition.
Perhaps I am misunderstanding the example.
Re: Help with Multi-user application
Post by Richard Russell on Sep 6th, 2016, 11:41am
on Sep 6th, 2016, 08:03am, Rod wrote:| Richard, is it not the writing conflict that needs a solution. In a multiuser environment there will be multiple write attempts that need managed. |
|
A common usage case is that multiple users won't be simultaneously attempting to write; rather only one will be responsible for writing and the rest will only need to read. The point is that this case should be simple to implement, because it doesn't violate the usual file sharing provisions (a file may be concurrently open for writing once and for reading multiple times) but it isn't because LB 4 doesn't allow you to open a random file for input only.
That's the scenario that LBB has a simple solution for. If multiple users need to write to the file then that will require either a file locking protocol (such that only one user at a time has acquired the right to write) or opening the file in a shared-write mode. Contrary to popular perception, Windows does allow a file to be opened for shared writes (pass the FILE_SHARE_WRITE flag to the CreateFile API) but it can be tricky to guarantee data integrity.
Obviously both those scenarios are more complex to implement than the simple 'single writer' case. File locking is likely to require error trapping and retries (LBB's Structured Exception Handling will make this nicer to code) and the shared-write case will require opening the file using the Windows API. Unlike LB 4, LBB will allow you to use the native file-handling statements (like FIELD and PUT) with a file opened using the API, although I'd need to explain how it's done.
But there comes a point when it would be better to use a proper database manager designed to provide shared access (ODBC, MySQL, SQLite etc.) rather than to reinvent the wheel.
Richard.
Re: Help with Multi-user application
Post by Richard Russell on Sep 6th, 2016, 2:51pm
Here's one way to acquire write access using SEH:
Code: do
try
open "SharedFile.dat" for output as #f len = 30
success = 1
catch
call delay rnd(1) * 100
success = 0
end try
loop until success
Much nicer than ON ERROR GOTO!
To ensure that the file is locked for as short a time as possible it should be closed again as soon as the record(s) have been written. The file can be opened for input on a different handle, and kept open continuously, for reads to take place.
Something to bear in mind when implementing any kind of file sharing is that there may be various levels of data-buffering, with the result that a PUT statement will not necessarily cause the data to reach the file immediately! If this is an issue, following it with a second, dummy, PUT (perhaps you can reserve record 1 for this purpose) will usually be sufficient to flush the previously-written data to the file.
Richard.
Re: Help with Multi-user application
Post by Mystic on Sep 6th, 2016, 3:23pm
Love the SEH concept. I will now attempt to implement this in a few of my programs. Elegant.
Unfortunately, where I work I cannot spin up any form of database server (long story), so I am stuck with writing my own databases, and do have a few I'm currently working on that may encounter the multiple user write issue.
I was going to try to do a crude form of checking if the file is being written to, such as create a lock-check file, if it's "1", try again in a few seconds sort of thing.
Or, in the case of the SEH, this might work better because if a user attempt to write to the file it SHOULD error, and the program can just retry until successful.
Because my programs are multiple user across a network, I always open and close my files only when actually required to be reading or writing to them. I never leave one hanging open like in a single user usage.
Re: Help with Multi-user application
Post by Richard Russell on Sep 6th, 2016, 4:22pm
on Sep 6th, 2016, 3:23pm, Mystic wrote:| I was going to try to do a crude form of checking if the file is being written to, such as create a lock-check file, if it's "1", try again in a few seconds sort of thing. |
|
To avoid race hazards, you really need to test for write access being available and acquire that write access atomically (otherwise somebody else might have acquired the lock in the short period between the two). So a separate lock file or record won't really do the job, rather the only safe way is to try to open the file for output and retry if you can't.
It's the same issue as trying to protect against an error occurring when you attempt to delete or rename a file, by testing for the existence of the relevant file beforehand. That will succeed most of the time, but one day you will be unlucky and another process will jump in between the test and the delete/rename and it will fail.
The only safe way is to attempt the operation and accept that an error might occur. I've seen some people argue that trapping errors is a sign of bad coding, and that there is always a better way. But in fact in this case the converse is true: not doing it by error trapping is a sign of sloppy coding!
That's one reason why having a structured means of trapping and recovering from errors (such as SEH) is desirable, because it may be unavoidable.
Richard.
Re: Help with Multi-user application
Post by michael on Sep 6th, 2016, 5:36pm
Interesting concept.
Re: Help with Multi-user application
Post by roxyryan on Sep 7th, 2016, 4:57pm
Richard good news it seems as though I finally have a multi-user program without having to use any third party database software, and without any API calls!
The final clincher was the try/catch clause. I could not it working with 'on error goto' which does not seem to like jumping back into subroutines. But as soon as I used the try/catch clause it started working! It is an amazing enhancement!
I have been running my Pub software from 2 programs, one with my left hand and one with my right, deliberately trying to break it, but so far it has has got through without crashing.
I want to do some more testing and then will send my code, which is really quite simple at the end of the day.
Dermot
Re: Help with Multi-user application
Post by Richard Russell on Sep 7th, 2016, 9:02pm
on Sep 7th, 2016, 4:57pm, roxyryan wrote:| 'on error goto' which does not seem to like jumping back into subroutines. |
|
Once you've exited the scope of a subroutine you can't jump back in. That is, the GOTO may seem to work but something bad is sure to happen before too long! So if, when an error occurs, you need to jump to a label within a subroutine, put the ON ERROR statement itself within the scope of that subroutine.
Of course I'd rather that you use SEH (try...catch...end try) instead. There's no GOTO involved so no danger that you might, deliberately or by accident, try to jump somewhere you shouldn't.
Richard.
Re: Help with Multi-user application
Post by roxyryan on Sep 8th, 2016, 09:01am
Richard
I cannot break the program with the code below, it really works. I am sure there is prettier code to achieve the same, but this is simple and it works.
While testing I had displays to show when it was "caught" in a conflict, and I only had that happen in the second "catch" shown in the routine. Maybe I dont need the first catch?
' the example below is the code that has to be done for each file, in this case the orders file, #10
Code: ' -------------------------------------------------------
' If a file is to be accessed for read-only then:
gosub [openorders]
' do processing
close #10
' -------------------------------------------------------
' If a file is to be accessed for writing then:
gosub [set.orders.lock]
' do processing
gosub [rel.orders.lock]
' -------------------------------------------------------
[openorders] ' opens the file in the appropriate mode
if file.mode$ = "OUT" then
OPEN ordersname$ FOR random AS #10 len=215
else
OPEN ordersname$ FOR input AS #10 len=215
end if
file.mode$ = "" ' set flag back to default, ie input
FIELD #10,_
1 AS ORlock$,_ ' only used on record 1 (no data held on record 1)
1 AS ORuser$,_ ' only used on record 1
etc other fields
return
' -------------------------------------------------------
[set.orders.lock] ' This subroutine called to open the file and set the lock indicator
waited = 0
gosub [openorders] ' file.mode$ not set, so it defaults to open input mode
[set.orders.lock.repeat]
get #10, 1
if ORlock$ = "Y" then ' Indicator on record 1 of random file
close #10
waited += 1
if waited < 11 then
call delaysub, 500 ' keep trying 10 times
gosub [openorders] ' open in input mode
goto [set.orders.lock.repeat]
end if
for x = 1 to max.users
if ORuser$ = userscodes$(x) then
other.user$ = usersnames$(x)
end if
next x
noticetext1$ = "Orders File is locked by " + other.user$ ' This message should never normally appear
notice.x = 20: notice.y = 60: gosub [notice.box] ' (my own notice routine)
waited = 0
gosub [openorders] ' open in input mode
goto [set.orders.lock.repeat]
else
close #10
try
file.mode$ = "OUT": gosub [openorders] ' open in read/write mode
catch
call delaysub, 100
gosub [openorders]
goto [set.orders.lock.repeat]
end try
try
ORlock$ = "Y" ' set the lock now to prohibit other users
ORuser$ = user.code$ ' the user code for this user
put #10, 1
catch
close #10
call delaysub, 100
gosub [openorders] ' open in input mode
goto [set.orders.lock.repeat]
end try
end if
return
' -------------------------------------------------------
[rel.orders.lock] ' Release the lock and close the file
get #10, 1
ORlock$ = "N" ' Indicates no longer locked
ORuser$ = ""
put #10, 1
close #10
return
' -------------------------------------------------------
sub delaysub delaytime
timer delaytime, [pause]
wait
[pause]
timer 0
end sub
' -------------------------------------------------------
Re: Help with Multi-user application
Post by Richard Russell on Sep 8th, 2016, 1:29pm
on Sep 8th, 2016, 09:01am, roxyryan wrote:| I only had that happen in the second "catch" shown in the routine. Maybe I dont need the first catch? |
|
So, that means the OPEN never failed but the PUT did? I expect the explanation is that you opened the file for RANDOM (in order to preserve the existing contents) which probably succeeds even if the file is currently being written by another user. Then when you try to write to the file the sharing violation is triggered.
Personally I'd be inclined to leave the first TRY clause in place. It doesn't do any harm and you might find that the behaviour is different if (for example) the file is stored on a shared network server running a different OS.
I notice that you haven't randomized the delays. You'll probably get away with it, but it does increase the probability of a deadlock if two users keep retrying simultaneously.
Incidentally I don't know if this limitation of LBB is going to hit you, but there is a maximum of eight simultaneously-open random files.
Richard.
Re: Help with Multi-user application
Post by roxyryan on Sep 8th, 2016, 2:36pm
Richard
Thanks for the reminder, yes I will randomise the delays.
Also now that I am only opening when needed, I wont be hitting that 8 file limit, I have a maximum of 3 or 4 open at a time
Dermot
Re: Help with Multi-user application
Post by Mystic on Sep 8th, 2016, 2:39pm
on Sep 8th, 2016, 1:29pm, Richard Russell wrote:I notice that you haven't randomized the delays. You'll probably get away with it, but it does increase the probability of a deadlock if two users keep retrying simultaneously.
|
|
Ah ha! Was wondering about the randomization of the delay. Now it makes sense! I didn't even think of the potential user-standoff occurring.
I love learning new things.
Thanks Richard, and everyone for a great discussion.
Re: Help with Multi-user application
Post by Richard Russell on Sep 11th, 2016, 01:08am
on Sep 8th, 2016, 09:01am, roxyryan wrote:| I cannot break the program with the code below, it really works. |
|
I've only just noticed this:
Code: try
file.mode$ = "OUT"
gosub [openorders] ' open in read/write mode
catch
call delaysub, 100
gosub [openorders]
goto [set.orders.lock.repeat]
end try
I need hardly say that if the GOTO here is ever taken the program will almost certainly suffer a slow and painful death. It takes some bravado to jump out of a Structured Exception Handling clause with a GOTO - the antithesis of structured programming!!
Richard.
Re: Help with Multi-user application
Post by roxyryan on Sep 11th, 2016, 12:05pm
Richard
Thank you for the kind words, but I think I told you previously that I had displays in my testing to prove which path the program followed. The program most certainly goes through this "goto" statement, and it works, every time. As I said previously, there are no doubt prettier statements to achieve the same, but this works.
Dermot
Re: Help with Multi-user application
Post by Richard Russell on Sep 11th, 2016, 1:06pm
on Sep 11th, 2016, 12:05pm, roxyryan wrote:| The program most certainly goes through this "goto" statement, and it works, every time. |
|
Well, it rather depends on how you define "works". It is certainly the case that executing a TRY statement, but not executing the matching END TRY statement, will leave LBB in an unstable state. For example, if a subsequent error were to occur (for any reason) it would not be reported, and LBB would most probably crash. So at the very least debugging the program would be made much more difficult because even a trivial typo could result in an unexplained non-obvious failure.
It's a bit like jumping out of a loop (like a FOR loop or a WHILE loop) with a GOTO. That may appear to 'work', and indeed the program may keep running apparently correctly for hours or more, but eventually it will crash out for no obvious reason. In those cases there is an 'official' way to exit the loop prematurely, but there's no legitimate way of terminating a TRY clause other than executing the END TRY.
It's the very fact that the incautious use of GOTO is so prone to creating a 'fragile' program that many people (including me) will say it should ideally never be used at all. It's a subject that creates a lot of heated argument, and some people vociferously defend the use of GOTO, but even its most ardent supporters have to admit that it must be used with extreme care.
It's a basic tenet of Software Engineering that you cannot ascertain whether a program works or not by testing alone. Testing is valuable, but in most cases it cannot hope to exercise every possible path through a program, nor confirm that a program will remain stable even if left running for a very long time. 'Following the rules' is just as important as testing for ensuring that a program is reliable and resilient.
Richard.