Hi,
The below command takes more than 2hrs.
result=$(find . -name "star_st*" -exec head -1 {} \; | grep "1175 876330")
Is there a way I can tweak the command to get better performance / quicker results.
Suggestions would be appreciated.
run it against a smaller file system?
It's actually quite an efficient way of capturing any opening lines with that string.
Try changing your script to:
result=$(find . -name "star_st*" -exec head -1 {} + | grep "1175 876330")
It should run a lot faster because it runs head for groups of files instead of having to run head for each file to be processed. The fork() and exec*() system calls needed to invoke head are expensive, relatively slow operations.
But, this assumes that a lot of files have names starting with star_st .
If . is on a remote filesystem, network issues could also have a significant impact.
Is there other heavy load on the system?
How big is the file hierarchy rooted in . ?
How many files have names starting with star_st ?
Hi, I have 99583 files in the directory with start_st
ls -l start_st* | wc -l
99583
I m on Linux
uname -a
Linux my_machine1 2.6.32-431.5.1.el6.x86_64 #1 SMP Fri Jan 10 14:46:43 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
Is there other heavy load on the system? Answer: No
How big is the file hierarchy rooted in . ? All the files are in the same directory.Answer: There are NO subdirectories.
If . is on a remote filesystem, network issues could also have a significant impact.: Answer: It is on the same Local File System.
It is depends on from which directory you are executing this find command and the number of files exists on your machine with the name of "star_st*".
if that . is / (root directory) and if your machine is configured with automount then it will try to mount all the auto mount filesystems and will perform find operation on them.
Hello mohtashims,
Could you please try following command and let me know if this helps.
According to your statement alll files are in same directory without any sub directories so following may help you then.
awk '(NR==1 && $0 ~ /1175 876330/){print FILENAME}' /tmp/test13/star_st*
Above is just an example, so in spite of using /tmp/test13/star_st* use Actual_path/star_st* and let us know if this helps.
Thanks,
R. Singh
here was a wrong suggestion 
@ agent.kgb - Which one was the wrong suggestion ?
Could you please be specific.
While find . -name "star_st*" -exec head -1 {} + | grep "1175 876330" helped reduce the time take to half it is still considered very slow.
I am answering the requested question so it help find a clue n solution.
If . is on a remote filesystem, network issues could also have a significant impact?
Its on the same file system not Remote.
Is there other heavy load on the system? Yes Here is the output of TOP showing high CPU.
top - 04:40:06 up 35 days, 13:15, 7 users, load average: 8.86, 8.03, 7.22
Tasks: 1844 total, 1 running, 1843 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.9%us, 2.9%sy, 0.0%ni, 93.7%id, 1.5%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 132141488k total, 88842372k used, 43299116k free, 887360k buffers
Swap: 16777208k total, 0k used, 16777208k free, 43330260k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30350 myadmin 20 0 2892m 2.6g 13m S 111.0 2.1 338:07.13 mohton
121341 myadmin 20 0 1126m 670m 99m S 99.9 0.5 794:02.69 lobster
29107 myadmin 20 0 167m 58m 900 S 31.7 0.0 0:22.00 find
53199 myadmin 20 0 982m 737m 93m S 4.9 0.6 20:32.78 lobster
However, one of the colleges says that CPU is common in this case and shouldn't affect the find command.
How big is the file hierarchy rooted in . ? There is no hierarchy. I m in the same directory in which the find command runs
How many files have names starting with star_st ? 180954
There is a HUGE difference between your command above:
find . -name start_st* -exec head -1 {} + | grep "1175 876330"
and the command I suggested:
find . -name "start_st*" -exec head -1 {} + | grep "1175 876330"
If you put in the double-quotes I suggested (or the single-quotes agent.kgb suggested), it should work.
But if the ls you showed us above worked and all of the files are in a single directory, try just using:
head -1 start_st* | grep "1175 876330"
And, if I'm reading your code correctly, RavinderSingh13's awk script can be modified to be more efficient than the above suggestion:
cd /directory/containing/your/files
awk '$0 ~ /1175 876330/
{nextfile}' star_st*
The nextfile command in awk is an extension to the standards, but I believe it is present in awk on Linux systems. If your awk doesn't include nextfile and your star_st* files are small, you could try:
awk 'FNR == 1 && $0 ~ /1175 876330/' star_st*
The head and grep pipeline above may be faster if your files are larger than one block, depending on your average file size and the block size used on the filesystem containing your files. (Note that the above awk uses FNR == 1 , not the NR == 1 in RavinderSingh13's script (which would only look at the 1st line in the 1st file instead of looking at the 1st line in each file).
I get an error while ruuning your suggestion bash: /bin/awk: Argument list too long
Hello Don,
Thank you for correcting me, I think your code should have included !~ instead of ~ as follows.
cd /directory/containing/your/files
awk '$0 !~ /1175 876330/
{nextfile}' star_st*
Please do correct me if I am worng here.
Thanks,
R. Singh
Sorry, but I think you're wrong. The:
$0 ~ /1175 876330/
in awk (since there is no action part, uses the default print the current line when the line contains the string 1175 876330 ). This simulates the action of the grep . With !~ instead of ~ , it would simulate grep -v ... .
The second line of the script:
{nextfile}
(with no condition, so it applies to every input line) skips to the next input file after processing the 1st line in a file (which duplicates the action of:
head -1
on each file processed.
And, when we're processing almost 100,000 files, we need to run this command in the directory where the files are located and just pass filenames as operands. Passing the absolute pathnames of 100,000 files runs a MUCH higher chance of exceeding ARG_MAX limits when execing awk . (Which mohtashims reported as a problem in post #11 in this thread.)
We are still waiting for mohtashims to tell us if:
cd /directory/containing/your/files
awk '$0 ~ /1175 876330/
{nextfile}' star_st*
works.
If this has still "too many arguments" then combine with find
find . -name "star_st*" -exec awk '/1175 876330/;{nextfile}' {} +
This find . -name star_st* -exec head -1 {} + | grep "1175 876330" gives output however, this awk '$0 ~ /1175 876330/ {nextfile}' star_st* does not show any Output
Put a ; or a <newline> after the /.../ . This is about 50% faster on my machine.
That is strange, before you said that:
find . -name star_st* -exec head -1 {} + | grep "1175 876330"
gave you a syntax error (which it would unless you ran it in a directory where there is no more than one file with a name starting with star_st ). We told you before that the -name primary's argument has to be quoted to work properly.
I see that RudiC has already explained that I didn't put that newline in the middle of my awk script just of the fun of it.
@MadeInGermany: I was testing your suggestion verses that of Don Cragun's find . -name "star_st*" -exec head -1 {} + | grep "1175 876330"
I check if the result is found or not using if [ $? -eq 0 ]; then
I m not able to test n compare the performace of both as MadeInGermany's command always passes the condition of $? -eq 0 even if there are no results found. May be you [MadeInGermany] can provide a fix there ?
---------- Post updated at 09:18 AM ---------- Previous update was at 09:11 AM ----------
ok, I put a ; but it gives me error
awk '$0 ~ /1175 876330/; {nextfile}' star_st*
bash: /bin/awk: Argument list too long
---------- Post updated at 09:23 AM ---------- Previous update was at 09:18 AM ----------
I am sorry Don i meant "star_st*" and not star_st*
Pipe to grep, that gives an exit status
find . -name "star_st*" -exec awk '/1175 876330/;{nextfile}' {} + | grep ^