Hi,
I am creating a progress bar for file upload for which I have CGI script which copies the data and depending on certain bytes it increments the progress bar. Here, I am writing the incremented value to a file which is read by Ajax at server end. However, here I want to ask that, is it possible for two different processes to write and read data simultaneously from the same file?
If not is there any other way by which I can write and read simultaneously?
Thanks
Short answer, yes, it is possible.
In order to keep the reader consistent you have to employ some kind of IPC - sockets, shared memory, mutex, file locking. Example: the reader tries to read whenever it feels like it, it will probably hit an EOF, which is not the real EOF, because reader did not wait for writer to finish.
Check out the flock() (advisory lock) call.
edit -
My bad - you are writing perl (probably). A better perler than me will have to advise about flock.
Well I do not wish to use IPC at this time.
So without that it is not possible, right?
while (cgiFormFileRead(file, b, sizeof(b), &got_count) == cgiFormSuccess)
{
fptr = fopen("progress_bar.txt", "w");
fwrite(b,sizeof(char),got_count,output);
i++;
if(i == inc && j<=100)
{
fprintf(fptr, "%d" ,j);
fseek(fptr, 0, SEEK_SET);
i = 0;
//fflush(fptr);
fclose(fptr);
j++;
}
}
So, my ajax reads the text file after every 1 sec or so, however the data is not available until whole file is written by the CGI. Any suggestion?
---------- Post updated at 01:24 PM ---------- Previous update was at 01:11 PM ----------
No I am using C.
Without some form of communication or synchronization between the reader and the writer the reader will not be able to tell the difference between the real end of the file, a temporary underrun, a broken socket, etc, etc.
Well the ajax reads the data synchronously ie
function send_request()
{
request.onreadystatechange = request_handler;
request.open("GET","progress_bar.txt",true); //true means that Ajax will get data asynchronously
request.send(null);
}
function request_handler() //function to check if data is available for read
{
if (request.readyState == 4 && request.status == 200)
{
document.getElementById("progress").innerHTML= request.responseText + '%';
document.getElementById("progress").style.width = request.responseText + '%';
document.getElementById("progress").style.backgroundColor = "green";
}
}
So, here the send_request() method is called after every 500ms(changeable). It can read the data only when request.readyState = 4 and this gets true only when whole file is written NOT at each data write.
I hope you understood what I mean to say.
Is the object request.readyState in shared memory - can both reader and writer process see exactly the same object? If not then there needs to be a way using IPC to get the other process to see request.readyState.
If this is so and you don't like IPC:
You have to use advisory file locking - period. Since you're in C it ain't hard.
I think I get what you're saying, it doesn't change the problem. The AJAX can't tell it the difference between an EOF before the file is finished and the real EOF.
Thanks for your reply. So, let me see what I can do, will get back again.
---------- Post updated at 02:12 PM ---------- Previous update was at 02:09 PM ----------
So, do you have any idea about Ajax? Any suggestion? I need to send a variable basically from server to client. Ajax can read .xml and .txt files. But I think it is not working the way I want. As, I want to read the data by Ajax each time the server writes to the file, instead when it has completed the process of writing. To add here, I REPLACE my text not add the data, so at a time my text file just have one value out of 1 to 100.
Moreover, although I am replacing my text and data could be read only once whole values are written, but the strange part here is that my Ajax could still read all the values one by one.
For this I tried putting the sleep(1); in between data write to file, and here my progress bar incremented after each 1 second. However this is not desirable as this process initiated after the whole file write not at the moment it was written
Even if you do manage to transmit this variable to the AJAX script somehow it's going to be a very awkward solution involving lots of retries, handfuls to dozens to hundreds more HTML retrievals than really necessary, etc. This should be solved on the server end, not the client. Instead of a rube goldberg system to check if an EOF was the real EOF, just don't EOF unless you mean it when sending the file to the client!
Advisory locking, as already said, is the way to do this. The server-side script that writes the file should always keep the very last section of the file locked. When it finishes writing another block, lock it, unlock the one before, write more, lock, unlock, etc. Once it's finally done,unlock the file completely.
The server-side script that reads the file should do much the same: always lock the section before it reads it. If the writer program doesn't have data ready for it, it'll be made to wait until the writer unlocks it. So lock a section, read it, unlock it, lock the next section, read it, unlock, repeat until EOF.
The relevant function call is flock.
I do not want to lock anything. That is not the issue. Issue is server doesn't release the file to be read.
Clearly. You should, though. Otherwise you're going to hit all the pitfalls we've been telling you about repeatedly, and all you'll be doing by doing it in AJAX instead is building your own slower, buggier, flock-over-http protocol from the ground up.
Doesn't release the file? What error do you get when you try to retrieve it -- does it think the file's not present at all, or give 'permission denied', etc. Either is quite odd, most UNIX systems don't do mandatory locking of open files, nor play tricks to hide files until they're complete; what system are you using?
My issue is not that my data is getting corrupted, my issue is that my data is not written and read the way I want.
My server writes to the file and client reads the file. As ajax at the client could ONLY read the file, there is no point of locking.
Well regarding "file release" thing: I mean that the data is not available while it is getting updates, so it seems that the file is being held until whole process is completed.
ie:
if i write 1,2....100 in text file , intead of the changing values 1,2,3 etc; i get
1,2,..100 at ONCE not one by one ie not at the time when it is UPDATED.
I am using Linux, you are write that Linux doesn't hold/hide the file. So, may be this is not a Linux issue. In case you have knowledge of Ajax, let me know if can discuss this further.
Thanks
hope this makes sense to you.
I have written AJAX-driven systems callign linux scripts, yes.
We've told you over and over what the locking's for and why you need it for reading a file that something else is appending to.
Again, linux doesn't hold the file. Locking of that sort is opt-in and you're not opting. The behavior you describe seems more and more strange...
By file do you mean a real, genuine, unparsed, honest-to-god file? Or do you actually mean your AJAX is reading output fetched from a server-side script? They act very differently, and the latter would act like what you're saying, and wouldn't need locking like you're saying. The web browser would just wait for its output. If you're reading script output, and it's arriving all at once, it's probably line-buffering. Flush the output on the server side whenever you really want the script to send. Or, just write a newline, which should flush it too.
Oh, you are writing a real file. And making new AJAX requests every time to download the complete file; you might have said so. And replacing the entire file each time, not writing "1,2,3,..."
That fseek does nothing, delete it; you start over at the beginning of the file every time you re-open it anyway.
Make sure your logic's working. I think there might be precedence problems in
(i == inc && j<=100)
, make it
((i == inc) && (j<=100))
...and make sure that j ever gets incremented at all. You could test your server-side code by opening the file for append instead of overwrite, and doing fprintf(file, "%d\n", j); so that it adds numbers line by line. If it works right the .txt should contain the numbers in sequence by the end.
yes I tried that, it do print 1,2...100.....No problem in logic.....rather I also put sleep(1); and the progress bar increments after 1 sec till 100, but this happens after the file has actually being copied. So, seems there is some buffering issue.
The OS doesn't buffer unless you ask it to, and you're not, so it may be your web server. What is it?
Ya, web server or CGI, web server is lighttpd.
---------- Post updated at 01:29 PM ---------- Previous update was at 01:25 PM ----------
Now I was following a different approach. I am using a CGI which outputs
system ("ls -la /tmp/lighttpd-upload* | awk {'print $5'} | wc -l");
and my ajax reads something like this 0,1,2,3.....22,23(=MAX),22,....2,1,0 (depending on the file size). However, this is not an efficient way as MAX couldn't be known beforehand so progress bar couldn't be incremented appropriately. Therefore, I like the previous approach, but that is not working at all.Do you have any other idea??
I really appreciate your help.
I am not familiar with lighttpd, but it does a lot of I/O tricks to keep performance low, that'd be the place to start looking. Try and disable caching/optimization for certain files or folders.