Extract count of string in all files and display on date wise

VasuKukkapalli · November 7, 2015, 7:04pm

Hi All,
hope you all are doing well!
I kindly ask you for shell scripting help, here is the description:

I have huge number of files shown below on date wise, which contains different strings(numbers you can say) including 505001 and 602001.

exp.interface.1303242.transactions.cor.2015-05-01.002403.tar.gz
transactions.cor.2015-05-01.002403.207051e2a0acff0661020f0506a3463c.tgz
transactions.cor.2015-05-01.002403.chk.812e2b458a6cba3fbbeb0f97731ad986.tgz
exp.interface.1303950.transactions.cor.2015-05-01.002405.tar.gz
transactions.cor.2015-05-01.002405.40e7d04b837c53571944b7c1bc82c4ae.tgz
transactions.cor.2015-05-01.002405.chk.56f70fd05a63775c27345d308b441735.tgz
exp.interface.1304599.transactions.cor.2015-05-01.002407.tar.gz
transactions.cor.2015-05-01.002407.a2948151d638a5eeca4aaed134bd41b9.tgz
transactions.cor.2015-05-01.002407.chk.8ac3628a11d38228402ef314ef49beae.tgz
exp.interface.1305211.transactions.cor.2015-05-01.002409.tar.gz
transactions.cor.2015-05-01.002409.b880f665515e6d18c056ce6483d5ff12.tgz
transactions.cor.2015-05-01.002409.chk.f66026a5bc2810f7a7147e12bf8e1f4e.tgz
exp.interface.1305794.transactions.cor.2015-05-01.002411.tar.gz
transactions.cor.2015-05-01.002411.97595eaf67a0333c40af3f589029f0d7.tgz
transactions.cor.2015-05-01.002411.chk.6fe34a59fff56fa3a3aa03a6fbf1bdea.tgz
exp.interface.1306377.transactions.cor.2015-05-01.002413.tar.gz
transactions.cor.2015-05-01.002413.042a33aa055bdf572529cf23f6fac000.tgz
transactions.cor.2015-05-01.002413.chk.99c1fa81839ab23357d41c09a88d4367.tgz
exp.interface.1306936.transactions.cor.2015-05-01.002415.tar.gz
transactions.cor.2015-05-01.002415.3934512e8bc98df67a34a3f90dc96340.tgz
transactions.cor.2015-05-01.002415.chk.966a39a2493c1b5503547909ee9609fc.tgz
exp.interface.1307493.transactions.cor.2015-05-01.002417.tar.gz
transactions.cor.2015-05-01.002417.4f07864d29eb53dc8eb43f4de4f1e711.tgz
transactions.cor.2015-05-01.002417.chk.6f9bf302c7cf298565f1b8b131afa7f4.tgz
exp.interface.1308085.transactions.cor.2015-05-02.002419.tar.gz
transactions.cor.2015-05-02.002419.eeb2e0d6616c3cf443f77c34c2a27bd5.tgz
transactions.cor.2015-05-02.002419.chk.e3b894ff0eb31786d78a2774d4e7316c.tgz
exp.interface.1308789.transactions.cor.2015-05-02.002421.tar.gz
transactions.cor.2015-05-02.002421.7969fdfb132dabe6c200e2287f685cfe.tgz
transactions.cor.2015-05-02.002421.chk.2a8554aae7caed7c3887f90ca4fd5b8f.tgz
exp.interface.1309434.transactions.cor.2015-05-02.002423.tar.gz
transactions.cor.2015-05-02.002423.678ee6d0f05ce0f727108b0229ef14c2.tgz
transactions.cor.2015-05-02.002423.chk.d8bcb91a5eb5691face8d1f59a26ace5.tgz
exp.interface.1310048.transactions.cor.2015-05-02.002425.tar.gz
transactions.cor.2015-05-02.002425.f49937f7a93af6300ad93ed7bf57423a.tgz
transactions.cor.2015-05-02.002425.chk.d46740d8acc82b221dfa7b27389391c1.tgz
exp.interface.1310626.transactions.cor.2015-05-02.002427.tar.gz
transactions.cor.2015-05-02.002427.6527b1cab4530f8b91d6f6aa9cc37952.tgz
transactions.cor.2015-05-02.002427.chk.304d0babcccb8f293618d59fd8983884.tgz
exp.interface.1311208.transactions.cor.2015-05-02.002429.tar.gz
transactions.cor.2015-05-02.002429.b0ccb8dd11bb106b335915b8fb4985ce.tgz
transactions.cor.2015-05-02.002429.chk.2612b2f02abe419ea774efe75b1495b1.tgz
exp.interface.1311770.transactions.cor.2015-05-02.002431.tar.gz
transactions.cor.2015-05-02.002431.ca996b5cc8f29828ad62b40c390da738.tgz
transactions.cor.2015-05-02.002431.chk.a151e1837ed7ec4d530399bee61be07e.tgz
exp.interface.1312325.transactions.cor.2015-05-02.002433.tar.gz
transactions.cor.2015-05-02.002433.4ba12a47b9f44fe76a760a89d3e6f31e.tgz
transactions.cor.2015-05-02.002433.chk.398df00f78dd0049de322e01fe6746ca.tgz
exp.interface.1312919.transactions.cor.2015-05-03.002435.tar.gz
transactions.cor.2015-05-03.002435.09eeef9eb51974b1527078189c13ef8e.tgz
transactions.cor.2015-05-03.002435.chk.b506f65bf3385486d2170796f762a13e.tgz
exp.interface.1313624.transactions.cor.2015-05-03.002437.tar.gz
exp.interface.1314270.transactions.cor.2015-05-03.002439.tar.gz
transactions.cor.2015-05-03.002437.99f16075cba422ed1d94c74f8d07c2f4.tgz
transactions.cor.2015-05-03.002437.chk.ec7d341056ee26e7415eefa51f4e3288.tgz
transactions.cor.2015-05-03.002439.028a0e7f0721d31caa5441c42aa525de.tgz
transactions.cor.2015-05-03.002439.chk.a15749dec5b30b251e12623799f4dcac.tgz
exp.interface.1314880.transactions.cor.2015-05-03.002441.tar.gz
transactions.cor.2015-05-03.002441.3f550a3c8375bcf30e98cdc61569f608.tgz
transactions.cor.2015-05-03.002441.chk.839c3e5a4412a9dfd08c0066aae8689b.tgz
exp.interface.1315468.transactions.cor.2015-05-03.002443.tar.gz
transactions.cor.2015-05-03.002443.853d1ebc1aca469219eeac4703bbf79b.tgz
transactions.cor.2015-05-03.002443.chk.edad6e9122b540c65df8e8c750825456.tgz
exp.interface.1316054.transactions.cor.2015-05-03.002445.tar.gz
transactions.cor.2015-05-03.002445.5670aa3c41371b4b0fe92c0ab44327c1.tgz
transactions.cor.2015-05-03.002445.chk.bcce7f279b3ae9b4eba37876207ae182.tgz
exp.interface.1316608.transactions.cor.2015-05-03.002447.tar.gz
transactions.cor.2015-05-03.002447.5f9ef688d904286bb9b06af7ede4c0be.tgz
transactions.cor.2015-05-03.002447.chk.0295b8d5a628e23e4f3465a3bfcaa8d1.tgz
exp.interface.1317160.transactions.cor.2015-05-03.002449.tar.gz
transactions.cor.2015-05-03.002449.bcb75d6bd1b2215707f888f624805f55.tgz
transactions.cor.2015-05-03.002449.chk.56f92b5bd0d6380ab191c7e55a3db3e5.tgz
exp.interface.1317760.transactions.cor.2015-05-04.002451.tar.gz
transactions.cor.2015-05-04.002451.6edcbbb58ff42f989c56749ea8f86b80.tgz
transactions.cor.2015-05-04.002451.chk.af02de8f4f0f897ba8b736d766306eeb.tgz
exp.interface.1318464.transactions.cor.2015-05-04.002453.tar.gz
transactions.cor.2015-05-04.002453.a9a94fbf15fadd6a78744501c4a6ba49.tgz
transactions.cor.2015-05-04.002453.chk.f2b0cead4e5456bd86d962674268dfa8.tgz
exp.interface.1319107.transactions.cor.2015-05-04.002455.tar.gz
transactions.cor.2015-05-04.002455.4f43e8a048050da400731d329847f5d7.tgz
transactions.cor.2015-05-04.002455.chk.b9ccf5ad05f922e9d7636cfa7318106e.tgz
exp.interface.1319722.transactions.cor.2015-05-04.002457.tar.gz
transactions.cor.2015-05-04.002457.02aef18177c7ee7019ead0287b958461.tgz
transactions.cor.2015-05-04.002457.chk.16dcdd6e11f7dd0ebe90c08d660e0995.tgz
exp.interface.1320309.transactions.cor.2015-05-04.002459.tar.gz
transactions.cor.2015-05-04.002459.5f51a68060af20116c2930a3e174bf4f.tgz
transactions.cor.2015-05-04.002459.chk.53be659573fb8adbdb08ed2bf360c23b.tgz
exp.interface.1320888.transactions.cor.2015-05-04.002461.tar.gz
transactions.cor.2015-05-04.002461.7963565bd984bccc9cd6b44f95c4fd58.tgz
transactions.cor.2015-05-04.002461.chk.00b7543c2fdf50f8853abb8cd4d616c9.tgz
exp.interface.1321445.transactions.cor.2015-05-04.002463.tar.gz
transactions.cor.2015-05-04.002463.fb298cc433d4b6946d6bb1179c3352ff.tgz
transactions.cor.2015-05-04.002463.chk.3ac335da4046211e1afe88565e5290a1.tgz
exp.interface.1321993.transactions.cor.2015-05-04.002465.tar.gz
transactions.cor.2015-05-04.002465.127e5c7af469c167114b62ef562ce883.tgz
transactions.cor.2015-05-04.002465.chk.1eeb9366c85b2ee6efcc51c3da6c1fa2.tgz
exp.interface.1322596.transactions.cor.2015-05-05.002467.tar.gz
transactions.cor.2015-05-05.002467.41ae0e29651fba731c2501301d655efa.tgz
transactions.cor.2015-05-05.002467.chk.c6321b180a78a94f58d071fbc01e650c.tgz
exp.interface.1323303.transactions.cor.2015-05-05.002469.tar.gz
transactions.cor.2015-05-05.002469.ab4439f269c30c2764d50b74271d45f3.tgz
transactions.cor.2015-05-05.002469.chk.1e8a386198470edcd9cf43ed70a4168e.tgz
exp.interface.1323944.transactions.cor.2015-05-05.002471.tar.gz
transactions.cor.2015-05-05.002471.dd73123909967f907e5692347b01e11b.tgz
transactions.cor.2015-05-05.002471.chk.b3ed1a47580b50e61dfa08f11792f6b6.tgz
exp.interface.1324570.transactions.cor.2015-05-05.002473.tar.gz
transactions.cor.2015-05-05.002473.b81d210295ca5cd25289a68aa112b575.tgz
transactions.cor.2015-05-05.002473.chk.44369c78affd438bc3bfcc3e8a3fd80b.tgz
exp.interface.1325157.transactions.cor.2015-05-05.002475.tar.gz
transactions.cor.2015-05-05.002475.843a216425681d6778894b7812b684f2.tgz
transactions.cor.2015-05-05.002475.chk.153f1c4957fa8f093e59830a66e63473.tgz
exp.interface.1325738.transactions.cor.2015-05-05.002477.tar.gz
transactions.cor.2015-05-05.002477.1137cf1010af69a8b41f6a4ff6189f9a.tgz
transactions.cor.2015-05-05.002477.chk.5bb9e1dfaaf1ff24b73daf76339cd62c.tgz
exp.interface.1326288.transactions.cor.2015-05-05.002479.tar.gz
transactions.cor.2015-05-05.002479.8b088f749460b93a183799d417b0092c.tgz
transactions.cor.2015-05-05.002479.chk.a88dda5fcaa4fa33a59cfae969dc5e1c.tgz
exp.interface.1326834.transactions.cor.2015-05-05.002481.tar.gz
transactions.cor.2015-05-05.002481.0ee0bfaea345c3167ed7b30ff34a98d9.tgz
transactions.cor.2015-05-05.002481.chk.b9ae8b670e091cd75b3a6ce9c4b3c036.tgz
exp.interface.1327436.transactions.cor.2015-05-06.002483.tar.gz
transactions.cor.2015-05-06.002483.af6549f393d27dc78a7544f42d6dd570.tgz
transactions.cor.2015-05-06.002483.chk.26b3627a58b70d82bf666d9cbc4e7d98.tgz
exp.interface.1328144.transactions.cor.2015-05-06.002485.tar.gz
exp.interface.1328781.transactions.cor.2015-05-06.002487.tar.gz
transactions.cor.2015-05-06.002485.2bd23fb03fe6b48b4110a35b40da7dd0.tgz
transactions.cor.2015-05-06.002485.chk.222dcabc5ccfa6c0fe61b13be1ac3b06.tgz
transactions.cor.2015-05-06.002487.cec42a84aaf31048713bc83ffa78a2c3.tgz
transactions.cor.2015-05-06.002487.chk.417aea0ad4f733f005462785cf8b8aa3.tgz
exp.interface.1329384.transactions.cor.2015-05-06.002489.tar.gz
transactions.cor.2015-05-06.002489.9c4b19e84be4d1d1482cfbac473d7764.tgz
transactions.cor.2015-05-06.002489.chk.0f914c2e77efee534746acb03e576618.tgz
exp.interface.1329972.transactions.cor.2015-05-06.002491.tar.gz
transactions.cor.2015-05-06.002491.8b9db656a66629b47219f979418c6541.tgz
transactions.cor.2015-05-06.002491.chk.d9f756faef0c67bac6099d82f1a7b98c.tgz
exp.interface.1330551.transactions.cor.2015-05-06.002493.tar.gz
transactions.cor.2015-05-06.002493.13ef91ca2fada5ecc972011bfff6fbed.tgz
transactions.cor.2015-05-06.002493.chk.f8a0f747daafda74f5559fed9455d787.tgz
exp.interface.1331106.transactions.cor.2015-05-06.002495.tar.gz
transactions.cor.2015-05-06.002495.665139f339f29ff7fdcaaf2a5c56bcc9.tgz
transactions.cor.2015-05-06.002495.chk.033689678f3bffdd1c78be2064cb3681.tgz
exp.interface.1331666.transactions.cor.2015-05-06.002497.tar.gz
transactions.cor.2015-05-06.002497.a925363d76a3d97ac364544ae4118679.tgz
transactions.cor.2015-05-06.002497.chk.a2f855be7d6ad0cbde459154c7ad6f84.tgz
exp.interface.1332271.transactions.cor.2015-05-07.002499.tar.gz
transactions.cor.2015-05-07.002499.adafe2b73a0c5c653046672131dc021f.tgz
transactions.cor.2015-05-07.002499.chk.1109302d88f1da586925698df2d10a7d.tgz
exp.interface.1332978.transactions.cor.2015-05-07.002501.tar.gz
transactions.cor.2015-05-07.002501.7b557e32b07ae9a5105c365c8517f147.tgz
transactions.cor.2015-05-07.002501.chk.b62afb04835adac1e247007508a7ad37.tgz
exp.interface.1333623.transactions.cor.2015-05-07.002503.tar.gz
transactions.cor.2015-05-07.002503.7b1a24ce9847ede90c412d1c121b4144.tgz
transactions.cor.2015-05-07.002503.chk.854720b16cea034d14d0d52295631191.tgz
exp.interface.1334242.transactions.cor.2015-05-07.002505.tar.gz
transactions.cor.2015-05-07.002505.254ded0792208442326837f4219a3c51.tgz
transactions.cor.2015-05-07.002505.chk.d4c91b98bf4c7bce0a13cb5ca3637cd5.tgz
exp.interface.1334837.transactions.cor.2015-05-07.002507.tar.gz
transactions.cor.2015-05-07.002507.fc0a2fd466d34298334b9208453d371e.tgz
transactions.cor.2015-05-07.002507.chk.7e7df789b4fb9640df6cf29e55335059.tgz
exp.interface.1335410.transactions.cor.2015-05-07.002509.tar.gz
transactions.cor.2015-05-07.002509.23137badbc153e06945f0b1733c4ea20.tgz
transactions.cor.2015-05-07.002509.chk.18607fc2043a8ee2d86e9888157df0b5.tgz
exp.interface.1335960.transactions.cor.2015-05-07.002511.tar.gz
transactions.cor.2015-05-07.002511.945eb9dffe79e6be1852e3ded2e87a9a.tgz
.
.
.
...2015-10-31...
...2015-10-31...
...2015-10-31...
...2015-10-31...
...2015-10-31...

I can straight a way ignore the files starting with exp and transactions files which contains .chk.
In rest of all other files starting from ...2015-05-01... to ...2015-10-31... , i need the count of string (or number) 505001 and 602001 and display on date wise. here i am using the shell script, but some where i am missing the logic.

#!/bin/bash
count_505001=0
count_602001=0
tempdate=${file:-"2015-05-01"}
for file in transactions.cor.2015-0[56789]; do 
	if [[ $tempdate = ${file:18:10} ]]; then
		tempdate=${file:18:10}
		count_505001=count+`zless $file | egrep "505001" | wc -l`;
		count_602001=count+`zless $file | egrep "602001" | wc -l`;
	else
		echo "Count for 505001 :"$count_505001" on date: "$tempdate
		echo "Count for 602001 :"$count_602001" on date: "$tempdate
		tempdate=${file:18:10}
		count_505001=`zless $file | egrep "505001" | wc -l`;
		count_602001=`zless $file | egrep "602001" | wc -l`;
	fi
done;

here is the example output:

Count for 505001 :236 on date: 2015-05-01
Count for 602001 :371 on date: 2015-05-01

Count for 505001 :9 on date: 2015-05-02
Count for 602001 :42 on date: 2015-05-02

Count for 505001 :53 on date: 2015-05-03
Count for 602001 :63 on date: 2015-05-03
.
.
.
Count for 505001 :423 on date: 2015-10-31
Count for 602001 :745 on date: 2015-10-31

RudiC · November 7, 2015, 7:43pm

Not sure I understand ... above is a list of file names, of which roughly two thirds are to be ignored? In the remaining files (which I infer are tar red and gzip ped?) , you want to count the occurrences of two strings?

Don_Cragun · November 8, 2015, 1:10am

It would seem that the following script should do what you were trying to do:

#!/bin/bash
shopt -s nullglob
for file in transactions.cor.2015-0[56789]*
do	[[ "$file" != ${file##*.chk.} ]] && continue
	if [[ $tempdate = ${file:17:10} ]]
	then	count_505001=$((count_505001 + $(zless "$file" | fgrep -c 505001)))
		count_602001=$((count_602001 + $(zless "$file" | fgrep -c 602001)))
		continue
	fi
	if [[ $tempdate != "" ]]
	then	echo "Count for 505001: $count_505001 on date: $tempdate"
		echo "Count for 602001: $count_602001 on date: $tempdate"
	fi
	tempdate=${file:17:10}
	count_505001=$(zless "$file" | fgrep -c 505001)
	count_602001=$(zless "$file" | fgrep -c 602001)
done
if [[ $tempdate != "" ]]
then	echo "Count for 505001: $count_505001 on date: $tempdate"
	echo "Count for 602001: $count_602001 on date: $tempdate"
fi

Note that several syntax errors have been corrected, it now skips *.chk.* files, it prints results from the sum of the counts for each date (instead of the count from the last file on each date), and it prints results from the last date processed.

Note also that fgrep is usually faster than egrep and fgrep -c is definitely faster than egrep | wc -l .

This could all be built into an awk script that would only need to zless each file once (instead of twice) and could eliminate the grep (and wc ) invocations completely.

Late fixes added:

Changed [[ template != "" ]] to [[ $template != "" ]] in two places.
Added shopt -s nullglob .

VasuKukkapalli · November 8, 2015, 3:06am

Hello Rudi,

Thanks for checking for me.

Yes, all files (file name start with exp and file name contains .chk. ) should be ignored.
The .tgz file = gzip (tar (text file) )
i would like to count of the 1st string in all rest of the files as date wise and count of the 2nd string in all rest of files as date wise

Please find my sample output in above post

Don_Cragun · November 8, 2015, 4:56am

Hello VasuKukkapalli,
Since you didn't give us your compressed files, it is difficult to reproduce your results, but doesn't the script I provided in post #3 in this thread do what you requested?

RudiC · November 8, 2015, 5:06am

As the files are tar red, I don't think zless will yield correct results dependably. As you seem to be using bash , the extglob option might be available. Try

shopt -s extglob
for FN in tra!(*chk*)z
  do    tar -Oxzf  $FN | 
        awk -vdate=${FN:17:10} '/505001/ {CNT1++} /602001/ {CNT2++} END {print "Count 505:", CNT1, ", date:", date; print "Count 602:", CNT2, ", date:", date}'
  done

Were the files available clear coded, awk could handle all files in one go, the shell for loop were unnecessay.

VasuKukkapalli · November 8, 2015, 12:05pm

Thank you So much Don Cragun Sir, it works for me and giving the correct results.

Hello Rudi,

Thank you for spending your time!
i am getting the count from individual files, not on date wise, please find the output below:

Count 505:  , date: 2015-06-21
Count 602: 351 , date: 2015-06-21
Count 505:  , date: 2015-06-21
Count 602: 89 , date: 2015-06-21
Count 505:  , date: 2015-06-21
Count 602: 167 , date: 2015-06-21
Count 505: 10 , date: 2015-06-21
Count 602: 460 , date: 2015-06-21
Count 505: 14 , date: 2015-06-21
Count 602: 612 , date: 2015-06-21
Count 505: 16 , date: 2015-06-21
Count 602: 547 , date: 2015-06-21
Count 505: 7 , date: 2015-06-21
Count 602: 469 , date: 2015-06-21
Count 505: 2 , date: 2015-06-21
Count 602: 464 , date: 2015-06-21
Count 505: 1 , date: 2015-06-22
Count 602: 201 , date: 2015-06-22
Count 505:  , date: 2015-06-22
Count 602: 47 , date: 2015-06-22
Count 505: 1 , date: 2015-06-22
Count 602: 127 , date: 2015-06-22
Count 505: 15 , date: 2015-06-22
Count 602: 492 , date: 2015-06-22
Count 505: 41 , date: 2015-06-22
Count 602: 557 , date: 2015-06-22
Count 505: 27 , date: 2015-06-22
Count 602: 475 , date: 2015-06-22
Count 505: 23 , date: 2015-06-22
Count 602: 411 , date: 2015-06-22
Count 505: 4 , date: 2015-06-22
Count 602: 382 , date: 2015-06-22
Count 505: 1 , date: 2015-06-23
Count 602: 217 , date: 2015-06-23
Count 505:  , date: 2015-06-23
Count 602: 139 , date: 2015-06-23
Count 505: 17 , date: 2015-06-23
Count 602: 450 , date: 2015-06-23
Count 505: 27 , date: 2015-06-23
Count 602: 526 , date: 2015-06-23
Count 505: 32 , date: 2015-06-23
Count 602: 461 , date: 2015-06-23
Count 505: 22 , date: 2015-06-23
Count 602: 457 , date: 2015-06-23
Count 505: 8 , date: 2015-06-23
Count 602: 446 , date: 2015-06-23
Count 505:  , date: 2015-06-24
Count 602: 165 , date: 2015-06-24
Count 505:  , date: 2015-06-24
Count 602: 60 , date: 2015-06-24
Count 505:  , date: 2015-06-24
Count 602: 147 , date: 2015-06-24
Count 505: 26 , date: 2015-06-24
Count 602: 485 , date: 2015-06-24
Count 505: 31 , date: 2015-06-24
Count 602: 522 , date: 2015-06-24
Count 505: 33 , date: 2015-06-24
Count 602: 486 , date: 2015-06-24
Count 505: 13 , date: 2015-06-24
Count 602: 353 , date: 2015-06-24
Count 505: 3 , date: 2015-06-24
Count 602: 352 , date: 2015-06-24

But i need output like below:

Count for 505001: 49 on date: 2015-06-21
Count for 602001: 3159 on date: 2015-06-21
Count for 505001: 112 on date: 2015-06-22
Count for 602001: 2692 on date: 2015-06-22
Count for 505001: 107 on date: 2015-06-23
Count for 602001: 2696 on date: 2015-06-23
Count for 505001: 106 on date: 2015-06-24
Count for 602001: 2570 on date: 2015-06-24

Basically the consolidate count on date wise...

Aia · November 8, 2015, 12:15pm

don cragun:

[...]

#!/bin/bash
for file in transactions.cor.2015-0[56789]*
do	[[ "$file" != ${file##*.chk.} ]] && continue
	if [[ $tempdate = ${file:17:10} ]]
	then	count_505001=$((count_505001 + $(zless "$file" | fgrep -c 505001)))
		count_602001=$((count_602001 + $(zless "$file" | fgrep -c 602001)))
		continue
	fi
	if [[ tempdate != "" ]]
	then	echo "Count for 505001: $count_505001 on date: $tempdate"
		echo "Count for 602001: $count_602001 on date: $tempdate"
	fi
	tempdate=${file:17:10}
	count_505001=$(zless "$file" | fgrep -c 505001)
	count_602001=$(zless "$file" | fgrep -c 602001)
done
if [[ tempdate != "" ]]
then	echo "Count for 505001: $count_505001 on date: $tempdate"
	echo "Count for 602001: $count_602001 on date: $tempdate"
fi

Hi Don,

[[ tempdate != "" ]] will always be true since the string tempdate is never "" .

Don_Cragun · November 8, 2015, 2:07pm

aia:

#!/bin/bash
for file in transactions.cor.2015-0[56789]*
do	[[ "$file" != ${file##*.chk.} ]] && continue
	if [[ $tempdate = ${file:17:10} ]]
	then	count_505001=$((count_505001 + $(zless "$file" | fgrep -c 505001)))
		count_602001=$((count_602001 + $(zless "$file" | fgrep -c 602001)))
		continue
	fi
	if [[ tempdate != "" ]]
	then	echo "Count for 505001: $count_505001 on date: $tempdate"
		echo "Count for 602001: $count_602001 on date: $tempdate"
	fi
	tempdate=${file:17:10}
	count_505001=$(zless "$file" | fgrep -c 505001)
	count_602001=$(zless "$file" | fgrep -c 602001)
done
if [[ tempdate != "" ]]
then	echo "Count for 505001: $count_505001 on date: $tempdate"
	echo "Count for 602001: $count_602001 on date: $tempdate"
fi

Hi Don,
[[ tempdate != "" ]] will always be true since the string tempdate is never "" .

Hi Aia,
$tempdate will expand to an empty string inside the for loop the first time a filename matching transactions.cor.2015-0[56789]* is found that does not contain the string .chk. .

$tempdate will expand to an empty string after the for loop completes if, and only if, no filenames matching transactions.cor.2015-0[56789]* were found that did not contain the string .chk. , or no filenames matching transactions.cor.2015-0[56789]* were found.

There should also be a check to be sure that at least one file matched the pattern transactions.cor.2015-0[56789]* (or the bash option that causes a non-matching pattern to expand to an empty list should be specified). I normally use ksh instead of bash . With ksh , I would avoid this problem by changing:

for file in transactions.cor.2015-0[56789]*

to:

for file in ~(N:transactions.cor.2015-0[56789]*)

which will not execute the loop if there are no matches. I always forget the option that enables this feature in bash and have to look it up, but for recent versions of bash , it is:

shopt -s nullglob
for file in transactions.cor.2015-0[56789]*

Aia · November 8, 2015, 2:17pm

Hi Don,

Then, shouldn't be: if [[ $tempdate != "" ]] ?

Don_Cragun · November 8, 2015, 2:58pm

With single square brackets, you need the <dollar-sign> and quotes:

if [ "$template" != "" ]

because the shell is parsing operands to a built-in. But, double square brackets are part of the shell command language, not a built-in, so the <dollar-sign> and quotes are optional. All of the following produce the same results:

if [[ template != "" ]]
if [[ $template != "" ]]
if [[ "$template" != "" ]]

taking the then branch if $template expands to a non-empty string. But the following are equivalent to each other, but, unlike the above three commands, always take the then branch:

if [[ "template" != "" ]]
if [[ 'template' != "" ]]

Aia · November 8, 2015, 3:44pm

don cragun:

[...] But, double square brackets are part of the shell command language, not a built-in, so the <dollar-sign> and quotes are optional. All of the following produce the same results:
if [[ template != "" ]]
if [[ $template != "" ]]
if [[ "$template" != "" ]]

Hi Don,
First, thank you for taking the time.

I understand that the properties of the double [[ does not suffer of the same issues that the single [ or test . It does not do word splitting nor globing, therefore you do not need to double quote the left-hand side of it and you only quote the right side if you do not want a pattern match, instead of a string comparison.

However, if [[ template != "" ]] does not expand since it is a literal string. How can it be the same that $template with or without quotes?

Perhaps, I can demonstrate my question better with a test.

$ cat test_test
#!/bin/bash
set -x
template=""
[[ template != "" ]] && echo "success!" || echo "failure!"
[[ $template != "" ]] && echo "success!" || echo "failure!"

$ ./test_test
+ template=
+ [[ template != '' ]] <-- it is the literal string template
+ echo 'success!'
success! <-- this should be failure! if it were dereferencing 
+ [[ '' != '' ]] <-- dereferenced to the value in template
+ echo 'failure!'
failure!

drl · November 8, 2015, 4:52pm

Hi.

Arithmetic expressions work that way:

Within an expression,
       shell variables may also be referenced by name without using the
       parameter expansion syntax.

-- man bash , ARITHMETIC EVALUATION

Best wishes ... cheers, drl

Aia · November 8, 2015, 5:23pm

Hi drl,

Could you tell me when this if [[ template != "" ]] became an arithmetic expression?
As far as I understand it it is a string conditional expression. Maybe that's what I am missing. But if this were the case, then I would have to still believe that:

it is not so?

Don_Cragun · November 8, 2015, 5:44pm

Hi Aia & drl,
Sorry. Yes, you are correct. I have updated the script in post #3 of this thread to:

Change [[ template != "" ]] to [[ $template != "" ]] .
Add shopt -s nullglob before the for loop.

I'm most familiar with the features in the standards. And, although arithmetic expansion (i.e., $((expression)) ) is in the standards; arithmetic commands (i.e., ((expression)) ) and compound conditions (i.e., [[ expression ]] ) are not in the standards (at least not yet, and not currently planned even for the next revision). I believe the treatment of variables in arithmetic expansions and arithmetic commands are the same; I incorrectly assumed that that behavior carried over into compound conditions as well.

Thanks you both for clarifying the distinction for me.

Don