Hi all, I was just wondering if there are any consequences, or if its a problem to have a multiple scripts parsing (reading) the same file simultaneously.
For example, I have file.txt with lots of information.
cat script1.sh
grep "awesome" file1.txt > awesome.txt
cat script2.sh
grep -v "awesome" file1.txt > notawesome.txt
Can both scripts run at the same time and read from file1.txt simultaneously?
Many thanks in advance,
Torch
Unsynchronized simultaneous reads are not a problem.
Regarding the example you gave, though, it could be accomplished in a single pass by a single process (sed or awk, to name a couple tools).
Regards,
Alister
Thanks for your response, Alister.
What do you mean by unsynchronized? The example I gave was just for simplicity, but typically the scripts are perl or python programs that read the file (gigabites in size). But don't write to it or append anything.
Just wondering if I am screwing things up by accessing the file with multiple programs.
Thanks again,
Torch
Unsynchronized means the readers of the file read whatever they want to read whenever they need to read it.
You can have large number of processes reading a file at exactly the same time. If you have too many you can reach I/O saturation on huge files. On medium sized files the I/O systems kernel memory cache and disk controller cache can easily hold several GB's of data for a group of files on a system that is not starved for memory. On small desktops with 2GB of RAM this will not fly.
That said: two scripts scripta scriptb running in parallel launched from another bigscript
bigscript:
scripta giantfile >output1 &
scriptb giantfile >output2 &
wait
put bigscript in background so you can keep working:
bigscript 1>bigscript.log 2>&1 &
Or. run it with at:
at -k now <<!
cd /home/me
/home/me/bigscript
!
Any output (like errors) will be emailed to your UNIX account.
Thank you so much for your very clear reply! That was very helpful.