Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

Concurrent access to data file

 [1/35] from: coussement::c::itc::mil::be at: 19-Feb-2001 16:30


> Hi, > > the problem was discussed here some time before.
[sorry, I didn't got that]
> You need some atomic > operation, which cannot be successfully done by two processes at the
<<quoted lines omitted: 3>>
> file locking suggested by Brett Handley is reliable only with a local > file, not a file on a network disk.
[Now, I've learn something ! Thanks !]
> I think that you should use TCP port on the server as suggested by > Brett Handley. There are also several distributed mutual exclusion > algorithms that don't require a central server, look at this paper for > more information: > http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps
[I'm not familiar with post-script files. Which soft do you use to read it ?]
> HTH, > > -- > Michal Kracik
[ Thanks for your answer, chr== ]

 [2/35] from: kracik:mbox:dkm:cz at: 19-Feb-2001 17:40


Hi Ladislav, sorry, I looked at it and it is atomic in both NFS and SMB. Next time I'll check before confusing anyone. Still there's a possibility that a client does not know that it succeeded or failed if the reply is lost and the connection time-outs. After the connection is re-estabilished, recovery should consider this situation, but that can happen even with a local file when power fails at the "right" time. Regards, -- Michal Kracik Ladislav Mecir wrote:

 [3/35] from: kracik:mbox:dkm:cz at: 19-Feb-2001 17:43


Hi, CRS - Psy Sel/SPO, COUSSEMENT Christophe, CPN wrote:
> > Hi, > >
<<quoted lines omitted: 8>>
> > file, not a file on a network disk. > [Now, I've learn something ! Thanks !]
Sorry for confusion, rename operation is atomic even on network drives. Now I've learnt something from Ladislav ;-) I'm not sure about file locking though.
> > I think that you should use TCP port on the server as suggested by > > Brett Handley. There are also several distributed mutual exclusion
<<quoted lines omitted: 14>>
> [rebol-request--rebol--com] with "unsubscribe" in the > subject, without the quotes.
-- Michal Kracik

 [4/35] from: coussement:c:itc:mil:be at: 19-Feb-2001 10:39


Hi REBOLians: I'm confronted with a little but important problem... I'm using a block structure for storing data on a server. 40 PC's -soon 80- could potentialy dump their data on the same time into one single data file on the server. Because we using Win NT -which I do not trust for reability-, I use a system of token to insure concurrent access to the file:
<snip>
forever [ either all [(count<= 20) token?] [ count: count + 1 ;--- attempt to reserve token if error? err: try [write token-file now (true)][ disarm err return false ] wait 5 ;--- take delay to be sure token is written ;--- try write operations answers: load local-file either error? err: try [save remote-file append load remote-file answers (true)][ ;--- release token delete token-file disarm err return false ][ ;--- release token delete token-file delete local-file return true ] ][ return false ] ] token?: func [][ return not exists? rejoin [global-answer-path global-token] ] </snip> The token is just a text file in this case, and gives the PC the right to talk to the server. The proc checks for the existence of a token on the server, for a maximum of 20 times. If there's no token on the server, it writes its own one, dump the data and release the token. Empirically, I had to set a delay of 5 sec to ensure the token is written, because of the slow network connection. Well, my question is not about straight REBOL, but more a matter of methodology : The procedure works, but, as you can imagine, it is not very efficient for so many PC working and dumping together. How can I achieve a better performence regarding concurrent access in REBOL on WinNT, which can garanty me no data will be lost ? Best regard, chr==

 [5/35] from: petr:krenzelok:trz:cz at: 19-Feb-2001 11:29


Hi, I don't have currently enough free time to read whole your email, but I would like to suggest you to use semaphore like solution: sem: open/new %semaphore.sem ->> error? try [delete %semaphore.sem] == true You can't delete your semaphore file while OS holds lock on it .... but be carefull and don't open several instances of 'sem ... The advantage here is - even if your app crashes, the solution will work, because OS will free lock it holds upon the file and rebol will succesfully delete it ... or so I think :-) Hope this helps, Cheers, -pekr- CRS - Psy Sel/SPO, COUSSEMENT Christophe, CPN wrote:

 [6/35] from: coussement:c:itc:mil:be at: 19-Feb-2001 11:51


Thanks Petr, I will test it at once ! chr==

 [7/35] from: brett:codeconscious at: 19-Feb-2001 22:58


Hi Christophe, Firstly I assume this bit of code is going to be running on each of the 80 Pcs, not as some sort of server process. Is this the case? (1) If so, I think your proposed method is not going give you any of the protection you expect. There could be a "window of opportunity" for another process to get in even though it should not. The weakness is that two processes could both succeed in "reserving" the token. (2) The other problem is the possibility that a program will fail due to an error yet leave the token in place so that other processes believe it is locked. I suspect that you really need to trust file locking, but the problem is I haven't been able to work out if there is a way to get Rebol to lock files. I attempted this: x: open/write/direct/binary %lock-file.dat and again in another rebol session which did not return an error so it seems that Rebol appears to opening the file without exclusive write locking. With them still open I attempted to open the file with Wordpad.exe. Wordpad did not open the file - complaining that another application had the file open - correct. To verify I close the opened ports and tried wordpad.exe again - this time it opened. So I believe that it would be good if rebol had file locking abilities. However I found that if you open a file like I did above, and try to rename it you will cause an error - indicating the file is in use. This was confirmed when I belatedly read my site :-\ below http://www.codeconscious.com/rebol/rebol-net.html#Filelocking This indicates that a good idea might be for the client to open a tcp/ip connection with a rebol process on the server. This way if the client process dies, the connection should (?) timeout - thus dealing with problem (2). I haven't tried it so ... Hope it helps. Brett. ----- Original Message ----- From: "CRS - Psy Sel/SPO, COUSSEMENT Christophe, CPN" <[COUSSEMENT--C--ITC--mil--be]>

 [8/35] from: rgombert:essentiel at: 19-Feb-2001 13:38


If the system does not have to be full realtime, each PC can write data in his own file and an administartor (or a cron) can check for all of these files and update the main one... then delete each individual files. This way, each PC could know if his file have allready be processed, and concurent acces is no more a problem. Renaud

 [9/35] from: coussement:c:itc:mil:be at: 19-Feb-2001 13:56


> This indicates that a good idea might be for the client to open a tcp/ip > connection with a rebol process on the server. This way if the client > process dies, the connection should (?) timeout - thus dealing with > problem > (2). I haven't tried it so ...
[good suggestion: it's IMHO the most safe way to do it. I will test it ASAP. Thanks a lot ! chr== ]

 [10/35] from: coussement:c:itc:mil:be at: 19-Feb-2001 13:52


Thanks for the suggestion, but the system HAS to work in real time: the results file have to be processed immediatly. Thanks anyway ! Regards, chr==

 [11/35] from: kracik:mbox:dkm:cz at: 19-Feb-2001 15:36


Hi, the problem was discussed here some time before. You need some atomic operation, which cannot be successfully done by two processes at the same time. One such operation is in most OSes (including Windows NT) local file rename. Any operation on network drives (NetBios or NFS) is not atomic, so you cannot rely on it when synchronizing multiple PCs - file locking suggested by Brett Handley is reliable only with a local file, not a file on a network disk. I think that you should use TCP port on the server as suggested by Brett Handley. There are also several distributed mutual exclusion algorithms that don't require a central server, look at this paper for more information: http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps HTH, -- Michal Kracik CRS - Psy Sel/SPO, COUSSEMENT Christophe, CPN wrote:

 [12/35] from: lmecir:mbox:vol:cz at: 19-Feb-2001 16:12


Hi Michal, Did I understand correctly, that RENAME is not an atomic operation for network drives? Regards Ladislav

 [13/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 9:01


> Hi, > >
<<quoted lines omitted: 14>>
> drives. Now I've learnt something from Ladislav ;-) I'm not sure about > file locking though.
[Thanks for the tip. I will try to test such a solution.]
> > > I think that you should use TCP port on the server as suggested by > > > Brett Handley. There are also several distributed mutual exclusion > > > algorithms that don't require a central server, look at this paper for > > > more information: > > > http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps
[I'm still interrested in your paper, but still don't know how to read it. I apologize for the simplicity of my request, but how can I access a readable content ?]
> > > > > HTH, > > > > > > -- > > > Michal Kracik
[Best Regards, chr==]

 [14/35] from: brett:codeconscious at: 20-Feb-2001 19:33


> > > http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps > [I'm still interrested in your paper, but still don't know how to > read it. > I apologize for the simplicity of my request, but how can I access a > readable content ?]
Try Ghostscript and GSView http://www.cs.wisc.edu/~ghost/ They worked for me. Brett.

 [15/35] from: andrew:wxc at: 20-Feb-2001 21:31


> > > > more information: > > > > http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps > [I'm still interrested in your paper, but still don't know how to read it.
I apologize for the simplicity of my request, but how can I access a readable content ?] Try the GhostScript viewer available from here: http://www.cs.wisc.edu/~ghost/gsview/ for various computer systems. It should be able to read the file. Andrew Martin ICQ: 26227169 http://members.nbci.com/AndrewMartin/

 [16/35] from: sharriff:aina:med-iq at: 20-Feb-2001 8:39


Hi Michal! I think youŽll have to download a Postscipt viewer for it, for windows use Ghostscript , its freeware... Sharriff Aina CRS - Psy Sel/SPO, An: [rebol-list--rebol--com] COUSSEMENT Kopie: Christophe, CPN Thema: [REBOL] Re: Concurrent access to data file <[COUSSEMENT--C--I] TC.mil.be> Gesendet von: [rebol-bounce--re] bol.com 20.02.01 08:01 Bitte antworten an rebol-list
> Hi, > >
<<quoted lines omitted: 6>>
> > > same time. One such operation is in most OSes (including Windows NT) > > > local file rename. Any operation on network drives (NetBios or NFS)
is
> > > not atomic, so you cannot rely on it when synchronizing multiple PCs
-
> > > file locking suggested by Brett Handley is reliable only with a local > > > file, not a file on a network disk.
<<quoted lines omitted: 3>>
> drives. Now I've learnt something from Ladislav ;-) I'm not sure about > file locking though.
[Thanks for the tip. I will try to test such a solution.]
> > > I think that you should use TCP port on the server as suggested by > > > Brett Handley. There are also several distributed mutual exclusion > > > algorithms that don't require a central server, look at this paper
for
> > > more information: > > > http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps
[I'm still interrested in your paper, but still don't know how to read it. I apologize for the simplicity of my request, but how can I access a readable content ?]
> > > > > HTH, > > > > > > -- > > > Michal Kracik
[Best Regards, chr==]

 [17/35] from: andrew:wxc at: 20-Feb-2001 21:58


> I'm confronted with a little but important problem... > > I'm using a block structure for storing data on a server. 40 PC's -soon
80- could potentialy dump their data on the same time into one single data file on the server. Perhaps better would be for the 40 (or 80) PCs to each write their own data into a file on a shared folder on their Hard Disk Drive. That way the PCs don't have to _compete_ for network access at the same time. Then a program on the server can periodically read each PC (say, every ten seconds or when convenient) and accumulate the data files, one at a time. Andrew Martin Asbestos-suited Rebol... ICQ: 26227169 http://members.nbci.com/AndrewMartin/

 [18/35] from: timewarp:sirius at: 20-Feb-2001 1:16


i solved this problem with poem access on my site by creating new read and write functions that "lock" the file while it's being read from or written to. the lock is just a temporary file that is removed as soon as the read or write is finished. the lock exists only as long as it takes to grab the data out of the file or write to the file ... so the lock does not exists the whole time the script is running, just for about a fraction of a second, if that. read locks are time stamped so there may be multiples of them, this is because a read lock only prevents writing, not reading. a write lock is just a single file that prevents other processes from writing to or reading from the file. the read and write functions recheck for the lock file's existence about every 100th of a second. the read function only checks for a write lock while the write function checks for a read or write lock. i'd share the functions with you, but i wrote them in perl. should be very easy to do in rebol. cheerfulness, -----EAT Andrew Martin wrote:

 [19/35] from: timewarp:sirius at: 20-Feb-2001 1:18


another solution, one that bo told me about, is to write a serialization server. then send all the data to the server script and let it handle writing/reading to/from the one data file on it's own. Andrew Martin wrote:

 [20/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 10:27


> > I'm confronted with a little but important problem... > >
<<quoted lines omitted: 9>>
> when > convenient) and accumulate the data files, one at a time.
[Thanks for answering. I did consider this solution. It's indeed IMHO a safe one, and easy-to-implement, but I would like to go for a more "elegant" solution. If I do not find anything else I will use that. Best Regards, chr== ]

 [21/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 10:39


> another solution, one that bo told me about, is to write > a serialization server. then send all the data to the server > script and let it handle writing/reading to/from the one > data file on it's own.
[I didn't already consider this one. It seem to be an usefull one in my case. I will launch some tests ASAP. Thanks ! Best Regards, chr== ]

 [22/35] from: andrew:wxc at: 20-Feb-2001 22:56


Probably best to find out if the network between the PCs and the server is a limiting factor. I was thinking it was an internet or slow LAN network and big messages. If it's a fast network with small messages, then Erin's solution is better. Andrew Martin Taking a cold shower... ICQ: 26227169 http://members.nbci.com/AndrewMartin/

 [23/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 10:53


> > > > > more information: > > > > > http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps
<<quoted lines omitted: 5>>
> http://www.cs.wisc.edu/~ghost/gsview/ > for various computer systems. It should be able to read the file.
[Thanks to all for answering ! There seem to be some problem with their ftp server now, but I'll download and test it as soon as possible. Best Regards, chr== ]

 [24/35] from: andrew:wxc at: 20-Feb-2001 22:59


> I would like to go for a more "elegant" solution.
Rebol/Express is an elegant solution. Andrew Martin ICQ: 26227169 http://members.nbci.com/AndrewMartin/

 [25/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 10:37


> i solved this problem with poem access on my site by > creating new read and write functions that "lock" the
<<quoted lines omitted: 16>>
> i'd share the functions with you, but i wrote them > in perl. should be very easy to do in rebol.
[Thanks Erin for answering. I think the solution you propose here is the one I presented when initiating this thread. It's a -relative- safe one (see thread for discussion about it) but will give poor performences when 40 to 80 PC's you want to dump their data together ... Thanks anyway ;-) Best Regards, chr== ]

 [26/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 11:06


> Probably best to find out if the network between the PCs and the server is > a > limiting factor. I was thinking it was an internet or slow LAN network and > big messages. If it's a fast network with small messages, then Erin's > solution is better.
[It's a slow LAN and about 5kb messages. Although I will test them all I think your solution is the most robust one. Thanks, CU chr== ]

 [27/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 11:08


> > I would like to go for a more "elegant" solution. > > Rebol/Express is an elegant solution.
[ Right ! I don't know how much RT will ask for this, but refering to a thread from some time ago, I think it will be far above the budgets I could get for the project ! Bad luck for me ;-\ Cheers, chr=== ]

 [28/35] from: g:santilli:tiscalinet:it at: 20-Feb-2001 12:06


CRS - Psy Sel/SPO, COUSSEMENT Christophe, CPN wrote:
> Thanks for the suggestion, but the system HAS to work in real time: the > results file have to be processed immediatly.
The simplest way might be to serialize access using a server. Your server process takes care of updating your file; it stays there listening to a TCP port and accepts one connection at a time only. Your clients just connect to the server and send it the data. HTH, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [29/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 12:47


> > Thanks for the suggestion, but the system HAS to work in real time: the > > results file have to be processed immediatly. > > The simplest way might be to serialize access using a server. Your > server process takes care of updating your file; it stays there > listening to a TCP port and accepts one connection at a time only. > Your clients just connect to the server and send it the data.
[ Thanks for sharing ;-) I will try a the serialization server solution ASAP. Cheers, chr== ]

 [30/35] from: sharriff:aina:med-iq at: 20-Feb-2001 12:58


Excuse my question, but could this slow down the workflow if only one connection is allowed to the server in the case of 20 or more PCs? Regards Sharriff Aina Gabriele Santilli <[g--santilli--tisca] An: [rebol-list--rebol--com] linet.it> Kopie: Gesendet von: Thema: [REBOL] Re: Concurrent access to data file [giesse--dsiaq1--ing] .univaq.it 20.02.01 11:06 Bitte antworten an rebol-list CRS - Psy Sel/SPO, COUSSEMENT Christophe, CPN wrote:
> Thanks for the suggestion, but the system HAS to work in real time: the > results file have to be processed immediatly.
The simplest way might be to serialize access using a server. Your server process takes care of updating your file; it stays there listening to a TCP port and accepts one connection at a time only. Your clients just connect to the server and send it the data. HTH, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [31/35] from: timewarp:sirius at: 20-Feb-2001 7:04


it seems to me the server can accept more than one connection at a time, but it would "serialize" the data being output to the data file. i don't see why it would accept only one connection at a time. [Sharriff--Aina--med-iq--de] wrote:

 [32/35] from: jeff:rebol at: 20-Feb-2001 8:44


Depends on the platform, but here under linux:
>> x: open tcp://:9922 >> y: open tcp://localhost:9922
<<quoted lines omitted: 3>>
>> c: open tcp://localhost:9922 >> d: open tcp://localhost:9922
-\|/-\|/-|/-\|/-\|/\|/-\|/-\|-\|/-\|/-\/-\|/-\|/-|/-\|/-\|/\|/-\|/-\ . . . (will eventually time out) The listen port has a certain fixed sized queue, but if it can dispatch connections faster than the queue can fill up it can handle an unlimited amount of connections.

 [33/35] from: g:santilli:tiscalinet:it at: 21-Feb-2001 12:41


[Sharriff--Aina--med-iq--de] wrote:
> Excuse my question, but could this slow down the workflow if only one > connection is allowed to the server in the case of 20 or more PCs?
It depends on the kind of access you are doing. File locking would cause waiting too. The efficient way to handle concurrency on a set of data is to use transactions and a good scheduler. This is not very simple, and it does not seem to be applicable to the case discussed here, at least as I understood it. A simple serialization server should be fine in this case IMHO. Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [34/35] from: sharriff:aina:med-iq at: 21-Feb-2001 12:43


Anybody with a URL where I can RTFM on "Serialisation on Servers" for dummies? further is there some way to implement "Object serialisation" in REBOL without doing handstands? Sharriff Aina Gabriele Santilli <[g--santilli--tisca] An: [rebol-list--rebol--com] linet.it> Kopie: Gesendet von: Thema: [REBOL] Re: Antwort: Re: Concurrent access to data file [giesse--dsiaq1--ing] .univaq.it 21.02.01 11:41 Bitte antworten an rebol-list [Sharriff--Aina--med-iq--de] wrote:
> Excuse my question, but could this slow down the workflow if only one > connection is allowed to the server in the case of 20 or more PCs?
It depends on the kind of access you are doing. File locking would cause waiting too. The efficient way to handle concurrency on a set of data is to use transactions and a good scheduler. This is not very simple, and it does not seem to be applicable to the case discussed here, at least as I understood it. A simple serialization server should be fine in this case IMHO. Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [35/35] from: g:santilli:tiscalinet:it at: 22-Feb-2001 20:18


Hello [Sharriff--Aina--med-iq--de]! On 21-Feb-01, you wrote: S> Anybody with a URL where I can RTFM on "Serialisation on S> Servers" for dummies? further is there some way to implement S> "Object serialisation" S> in REBOL without doing handstands? With "serialization" I intended "turning concurrent accesses into serial accesses"; that has nothing to do with object serialization. Sorry for the confusion. Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted