Is there a way to have awk output its regular output to a file and some other stuff to a log file to monitor progress?
ie: print first field of a file and for every 100000 lines dealt with, print time in file.log
Is there a way to have awk output its regular output to a file and some other stuff to a log file to monitor progress?
ie: print first field of a file and for every 100000 lines dealt with, print time in file.log
awk ' { print $0
if(FNR % 1000000 ==0 ) {printf ("processed %d lines", FNR) > "./file.log" } } ' inputfilename
here's something I've been using for quite some time for 'large' files to entertain a user:
nawk -f myAWK.awk myFile
OR
nawk -v twirle="${USER}" -f myAWK.awk myFile
OR
nawk -v twirle='str' -f myAWK.awk myFile
myAWK.awk:
#!/usr/bin/nawk -f
#
#---------------------------------------------------------------------------
function makeTwirleS(twirle, sep, s,twLength,twSlength,i,left,l,r,right,arrN ,arr)
{
s=twirle
twLength=length(twirle)
twSlength=length(twirle) * ((length(twirle) <= 3) ? 5 : 2)
for(right=twLength+1; right<=twSlength; right++)
s=s "."
# left -> right
for(i=1; i+twLength<=twSlength; i++) {
l=r=""
for(left=1; left<=i; left++)
l=(l=="")? "." : l "."
for(right=i+twLength+1; right<=twSlength; right++)
r=(r=="")? "." : r "."
s=s sep l twirle r
}
# reverse the direction - right -> left
arrN=split(s, arr, sep)
for(i=arrN-1; i>1; i--)
s=s sep arr
return s
}
#---------------------------------------------------------------------------
BEGIN {
# Progress bar
PROGRESSdiv=1000
if (!twirle) twirle="*"
# deal with the TWIRLE
twirleS=makeTwirleS(twirle, SUBSEP)
ntwirleA=split(twirleS, twirleA, SUBSEP)
stderr="cat 1>&2"
}
#---------------------------------------------------------------------------
!( FNR % PROGRESSdiv) {
printf("\rProgress [%s]-> [%d] lines [%s]", FILENAME, FNR, twirleA[(++_twirle % ntwirleA+1)]) | stderr;
}
#
# here goes the rest of the awk script......
#---------------------------------------------------------------------------