# Extract count of string in all files and display on date wise

**URL:** <https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631>\
**Category:** Shell Programming and Scripting\
**Created:** [November 7, 2015, 7:04pm UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631 "2015-11-07T19:04:58Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![VasuKukkapalli](https://community.unix.com/letter_avatar/vasukukkapalli/32/5_5575768a8748004e209b776fc1b2916d.png) [@VasuKukkapalli](https://community.unix.com/u/VasuKukkapalli)\
**Post date:** [November 7, 2015, 7:04pm UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/1 "2015-11-07T19:04:58Z")

</div>

Hi All,  
hope you all are doing well!  
I kindly ask you for shell scripting help, here is the description:

I have huge number of files shown below on date wise, which contains different strings(numbers you can say) including 505001 and 602001.

```nohighlight
exp.interface.1303242.transactions.cor.2015-05-01.002403.tar.gz
transactions.cor.2015-05-01.002403.207051e2a0acff0661020f0506a3463c.tgz
transactions.cor.2015-05-01.002403.chk.812e2b458a6cba3fbbeb0f97731ad986.tgz
exp.interface.1303950.transactions.cor.2015-05-01.002405.tar.gz
transactions.cor.2015-05-01.002405.40e7d04b837c53571944b7c1bc82c4ae.tgz
transactions.cor.2015-05-01.002405.chk.56f70fd05a63775c27345d308b441735.tgz
exp.interface.1304599.transactions.cor.2015-05-01.002407.tar.gz
transactions.cor.2015-05-01.002407.a2948151d638a5eeca4aaed134bd41b9.tgz
transactions.cor.2015-05-01.002407.chk.8ac3628a11d38228402ef314ef49beae.tgz
exp.interface.1305211.transactions.cor.2015-05-01.002409.tar.gz
transactions.cor.2015-05-01.002409.b880f665515e6d18c056ce6483d5ff12.tgz
transactions.cor.2015-05-01.002409.chk.f66026a5bc2810f7a7147e12bf8e1f4e.tgz
exp.interface.1305794.transactions.cor.2015-05-01.002411.tar.gz
transactions.cor.2015-05-01.002411.97595eaf67a0333c40af3f589029f0d7.tgz
transactions.cor.2015-05-01.002411.chk.6fe34a59fff56fa3a3aa03a6fbf1bdea.tgz
exp.interface.1306377.transactions.cor.2015-05-01.002413.tar.gz
transactions.cor.2015-05-01.002413.042a33aa055bdf572529cf23f6fac000.tgz
transactions.cor.2015-05-01.002413.chk.99c1fa81839ab23357d41c09a88d4367.tgz
exp.interface.1306936.transactions.cor.2015-05-01.002415.tar.gz
transactions.cor.2015-05-01.002415.3934512e8bc98df67a34a3f90dc96340.tgz
transactions.cor.2015-05-01.002415.chk.966a39a2493c1b5503547909ee9609fc.tgz
exp.interface.1307493.transactions.cor.2015-05-01.002417.tar.gz
transactions.cor.2015-05-01.002417.4f07864d29eb53dc8eb43f4de4f1e711.tgz
transactions.cor.2015-05-01.002417.chk.6f9bf302c7cf298565f1b8b131afa7f4.tgz
exp.interface.1308085.transactions.cor.2015-05-02.002419.tar.gz
transactions.cor.2015-05-02.002419.eeb2e0d6616c3cf443f77c34c2a27bd5.tgz
transactions.cor.2015-05-02.002419.chk.e3b894ff0eb31786d78a2774d4e7316c.tgz
exp.interface.1308789.transactions.cor.2015-05-02.002421.tar.gz
transactions.cor.2015-05-02.002421.7969fdfb132dabe6c200e2287f685cfe.tgz
transactions.cor.2015-05-02.002421.chk.2a8554aae7caed7c3887f90ca4fd5b8f.tgz
exp.interface.1309434.transactions.cor.2015-05-02.002423.tar.gz
transactions.cor.2015-05-02.002423.678ee6d0f05ce0f727108b0229ef14c2.tgz
transactions.cor.2015-05-02.002423.chk.d8bcb91a5eb5691face8d1f59a26ace5.tgz
exp.interface.1310048.transactions.cor.2015-05-02.002425.tar.gz
transactions.cor.2015-05-02.002425.f49937f7a93af6300ad93ed7bf57423a.tgz
transactions.cor.2015-05-02.002425.chk.d46740d8acc82b221dfa7b27389391c1.tgz
exp.interface.1310626.transactions.cor.2015-05-02.002427.tar.gz
transactions.cor.2015-05-02.002427.6527b1cab4530f8b91d6f6aa9cc37952.tgz
transactions.cor.2015-05-02.002427.chk.304d0babcccb8f293618d59fd8983884.tgz
exp.interface.1311208.transactions.cor.2015-05-02.002429.tar.gz
transactions.cor.2015-05-02.002429.b0ccb8dd11bb106b335915b8fb4985ce.tgz
transactions.cor.2015-05-02.002429.chk.2612b2f02abe419ea774efe75b1495b1.tgz
exp.interface.1311770.transactions.cor.2015-05-02.002431.tar.gz
transactions.cor.2015-05-02.002431.ca996b5cc8f29828ad62b40c390da738.tgz
transactions.cor.2015-05-02.002431.chk.a151e1837ed7ec4d530399bee61be07e.tgz
exp.interface.1312325.transactions.cor.2015-05-02.002433.tar.gz
transactions.cor.2015-05-02.002433.4ba12a47b9f44fe76a760a89d3e6f31e.tgz
transactions.cor.2015-05-02.002433.chk.398df00f78dd0049de322e01fe6746ca.tgz
exp.interface.1312919.transactions.cor.2015-05-03.002435.tar.gz
transactions.cor.2015-05-03.002435.09eeef9eb51974b1527078189c13ef8e.tgz
transactions.cor.2015-05-03.002435.chk.b506f65bf3385486d2170796f762a13e.tgz
exp.interface.1313624.transactions.cor.2015-05-03.002437.tar.gz
exp.interface.1314270.transactions.cor.2015-05-03.002439.tar.gz
transactions.cor.2015-05-03.002437.99f16075cba422ed1d94c74f8d07c2f4.tgz
transactions.cor.2015-05-03.002437.chk.ec7d341056ee26e7415eefa51f4e3288.tgz
transactions.cor.2015-05-03.002439.028a0e7f0721d31caa5441c42aa525de.tgz
transactions.cor.2015-05-03.002439.chk.a15749dec5b30b251e12623799f4dcac.tgz
exp.interface.1314880.transactions.cor.2015-05-03.002441.tar.gz
transactions.cor.2015-05-03.002441.3f550a3c8375bcf30e98cdc61569f608.tgz
transactions.cor.2015-05-03.002441.chk.839c3e5a4412a9dfd08c0066aae8689b.tgz
exp.interface.1315468.transactions.cor.2015-05-03.002443.tar.gz
transactions.cor.2015-05-03.002443.853d1ebc1aca469219eeac4703bbf79b.tgz
transactions.cor.2015-05-03.002443.chk.edad6e9122b540c65df8e8c750825456.tgz
exp.interface.1316054.transactions.cor.2015-05-03.002445.tar.gz
transactions.cor.2015-05-03.002445.5670aa3c41371b4b0fe92c0ab44327c1.tgz
transactions.cor.2015-05-03.002445.chk.bcce7f279b3ae9b4eba37876207ae182.tgz
exp.interface.1316608.transactions.cor.2015-05-03.002447.tar.gz
transactions.cor.2015-05-03.002447.5f9ef688d904286bb9b06af7ede4c0be.tgz
transactions.cor.2015-05-03.002447.chk.0295b8d5a628e23e4f3465a3bfcaa8d1.tgz
exp.interface.1317160.transactions.cor.2015-05-03.002449.tar.gz
transactions.cor.2015-05-03.002449.bcb75d6bd1b2215707f888f624805f55.tgz
transactions.cor.2015-05-03.002449.chk.56f92b5bd0d6380ab191c7e55a3db3e5.tgz
exp.interface.1317760.transactions.cor.2015-05-04.002451.tar.gz
transactions.cor.2015-05-04.002451.6edcbbb58ff42f989c56749ea8f86b80.tgz
transactions.cor.2015-05-04.002451.chk.af02de8f4f0f897ba8b736d766306eeb.tgz
exp.interface.1318464.transactions.cor.2015-05-04.002453.tar.gz
transactions.cor.2015-05-04.002453.a9a94fbf15fadd6a78744501c4a6ba49.tgz
transactions.cor.2015-05-04.002453.chk.f2b0cead4e5456bd86d962674268dfa8.tgz
exp.interface.1319107.transactions.cor.2015-05-04.002455.tar.gz
transactions.cor.2015-05-04.002455.4f43e8a048050da400731d329847f5d7.tgz
transactions.cor.2015-05-04.002455.chk.b9ccf5ad05f922e9d7636cfa7318106e.tgz
exp.interface.1319722.transactions.cor.2015-05-04.002457.tar.gz
transactions.cor.2015-05-04.002457.02aef18177c7ee7019ead0287b958461.tgz
transactions.cor.2015-05-04.002457.chk.16dcdd6e11f7dd0ebe90c08d660e0995.tgz
exp.interface.1320309.transactions.cor.2015-05-04.002459.tar.gz
transactions.cor.2015-05-04.002459.5f51a68060af20116c2930a3e174bf4f.tgz
transactions.cor.2015-05-04.002459.chk.53be659573fb8adbdb08ed2bf360c23b.tgz
exp.interface.1320888.transactions.cor.2015-05-04.002461.tar.gz
transactions.cor.2015-05-04.002461.7963565bd984bccc9cd6b44f95c4fd58.tgz
transactions.cor.2015-05-04.002461.chk.00b7543c2fdf50f8853abb8cd4d616c9.tgz
exp.interface.1321445.transactions.cor.2015-05-04.002463.tar.gz
transactions.cor.2015-05-04.002463.fb298cc433d4b6946d6bb1179c3352ff.tgz
transactions.cor.2015-05-04.002463.chk.3ac335da4046211e1afe88565e5290a1.tgz
exp.interface.1321993.transactions.cor.2015-05-04.002465.tar.gz
transactions.cor.2015-05-04.002465.127e5c7af469c167114b62ef562ce883.tgz
transactions.cor.2015-05-04.002465.chk.1eeb9366c85b2ee6efcc51c3da6c1fa2.tgz
exp.interface.1322596.transactions.cor.2015-05-05.002467.tar.gz
transactions.cor.2015-05-05.002467.41ae0e29651fba731c2501301d655efa.tgz
transactions.cor.2015-05-05.002467.chk.c6321b180a78a94f58d071fbc01e650c.tgz
exp.interface.1323303.transactions.cor.2015-05-05.002469.tar.gz
transactions.cor.2015-05-05.002469.ab4439f269c30c2764d50b74271d45f3.tgz
transactions.cor.2015-05-05.002469.chk.1e8a386198470edcd9cf43ed70a4168e.tgz
exp.interface.1323944.transactions.cor.2015-05-05.002471.tar.gz
transactions.cor.2015-05-05.002471.dd73123909967f907e5692347b01e11b.tgz
transactions.cor.2015-05-05.002471.chk.b3ed1a47580b50e61dfa08f11792f6b6.tgz
exp.interface.1324570.transactions.cor.2015-05-05.002473.tar.gz
transactions.cor.2015-05-05.002473.b81d210295ca5cd25289a68aa112b575.tgz
transactions.cor.2015-05-05.002473.chk.44369c78affd438bc3bfcc3e8a3fd80b.tgz
exp.interface.1325157.transactions.cor.2015-05-05.002475.tar.gz
transactions.cor.2015-05-05.002475.843a216425681d6778894b7812b684f2.tgz
transactions.cor.2015-05-05.002475.chk.153f1c4957fa8f093e59830a66e63473.tgz
exp.interface.1325738.transactions.cor.2015-05-05.002477.tar.gz
transactions.cor.2015-05-05.002477.1137cf1010af69a8b41f6a4ff6189f9a.tgz
transactions.cor.2015-05-05.002477.chk.5bb9e1dfaaf1ff24b73daf76339cd62c.tgz
exp.interface.1326288.transactions.cor.2015-05-05.002479.tar.gz
transactions.cor.2015-05-05.002479.8b088f749460b93a183799d417b0092c.tgz
transactions.cor.2015-05-05.002479.chk.a88dda5fcaa4fa33a59cfae969dc5e1c.tgz
exp.interface.1326834.transactions.cor.2015-05-05.002481.tar.gz
transactions.cor.2015-05-05.002481.0ee0bfaea345c3167ed7b30ff34a98d9.tgz
transactions.cor.2015-05-05.002481.chk.b9ae8b670e091cd75b3a6ce9c4b3c036.tgz
exp.interface.1327436.transactions.cor.2015-05-06.002483.tar.gz
transactions.cor.2015-05-06.002483.af6549f393d27dc78a7544f42d6dd570.tgz
transactions.cor.2015-05-06.002483.chk.26b3627a58b70d82bf666d9cbc4e7d98.tgz
exp.interface.1328144.transactions.cor.2015-05-06.002485.tar.gz
exp.interface.1328781.transactions.cor.2015-05-06.002487.tar.gz
transactions.cor.2015-05-06.002485.2bd23fb03fe6b48b4110a35b40da7dd0.tgz
transactions.cor.2015-05-06.002485.chk.222dcabc5ccfa6c0fe61b13be1ac3b06.tgz
transactions.cor.2015-05-06.002487.cec42a84aaf31048713bc83ffa78a2c3.tgz
transactions.cor.2015-05-06.002487.chk.417aea0ad4f733f005462785cf8b8aa3.tgz
exp.interface.1329384.transactions.cor.2015-05-06.002489.tar.gz
transactions.cor.2015-05-06.002489.9c4b19e84be4d1d1482cfbac473d7764.tgz
transactions.cor.2015-05-06.002489.chk.0f914c2e77efee534746acb03e576618.tgz
exp.interface.1329972.transactions.cor.2015-05-06.002491.tar.gz
transactions.cor.2015-05-06.002491.8b9db656a66629b47219f979418c6541.tgz
transactions.cor.2015-05-06.002491.chk.d9f756faef0c67bac6099d82f1a7b98c.tgz
exp.interface.1330551.transactions.cor.2015-05-06.002493.tar.gz
transactions.cor.2015-05-06.002493.13ef91ca2fada5ecc972011bfff6fbed.tgz
transactions.cor.2015-05-06.002493.chk.f8a0f747daafda74f5559fed9455d787.tgz
exp.interface.1331106.transactions.cor.2015-05-06.002495.tar.gz
transactions.cor.2015-05-06.002495.665139f339f29ff7fdcaaf2a5c56bcc9.tgz
transactions.cor.2015-05-06.002495.chk.033689678f3bffdd1c78be2064cb3681.tgz
exp.interface.1331666.transactions.cor.2015-05-06.002497.tar.gz
transactions.cor.2015-05-06.002497.a925363d76a3d97ac364544ae4118679.tgz
transactions.cor.2015-05-06.002497.chk.a2f855be7d6ad0cbde459154c7ad6f84.tgz
exp.interface.1332271.transactions.cor.2015-05-07.002499.tar.gz
transactions.cor.2015-05-07.002499.adafe2b73a0c5c653046672131dc021f.tgz
transactions.cor.2015-05-07.002499.chk.1109302d88f1da586925698df2d10a7d.tgz
exp.interface.1332978.transactions.cor.2015-05-07.002501.tar.gz
transactions.cor.2015-05-07.002501.7b557e32b07ae9a5105c365c8517f147.tgz
transactions.cor.2015-05-07.002501.chk.b62afb04835adac1e247007508a7ad37.tgz
exp.interface.1333623.transactions.cor.2015-05-07.002503.tar.gz
transactions.cor.2015-05-07.002503.7b1a24ce9847ede90c412d1c121b4144.tgz
transactions.cor.2015-05-07.002503.chk.854720b16cea034d14d0d52295631191.tgz
exp.interface.1334242.transactions.cor.2015-05-07.002505.tar.gz
transactions.cor.2015-05-07.002505.254ded0792208442326837f4219a3c51.tgz
transactions.cor.2015-05-07.002505.chk.d4c91b98bf4c7bce0a13cb5ca3637cd5.tgz
exp.interface.1334837.transactions.cor.2015-05-07.002507.tar.gz
transactions.cor.2015-05-07.002507.fc0a2fd466d34298334b9208453d371e.tgz
transactions.cor.2015-05-07.002507.chk.7e7df789b4fb9640df6cf29e55335059.tgz
exp.interface.1335410.transactions.cor.2015-05-07.002509.tar.gz
transactions.cor.2015-05-07.002509.23137badbc153e06945f0b1733c4ea20.tgz
transactions.cor.2015-05-07.002509.chk.18607fc2043a8ee2d86e9888157df0b5.tgz
exp.interface.1335960.transactions.cor.2015-05-07.002511.tar.gz
transactions.cor.2015-05-07.002511.945eb9dffe79e6be1852e3ded2e87a9a.tgz
.
.
.
...2015-10-31...
...2015-10-31...
...2015-10-31...
...2015-10-31...
...2015-10-31...

```

I can straight a way ignore the files starting with `exp` and `transactions` files which contains `.chk.`  
In rest of all other files starting from `...2015-05-01...` to `...2015-10-31...` , i need the count of string (or number) 505001 and 602001 and display on date wise. here i am using the shell script, but some where i am missing the logic.

```nohighlight
#!/bin/bash
count_505001=0
count_602001=0
tempdate=${file:-"2015-05-01"}
for file in transactions.cor.2015-0[56789]; do 
	if [[$tempdate = ${file:18:10}]]; then
		tempdate=${file:18:10}
		count_505001=count+`zless $file | egrep "505001" | wc -l`;
		count_602001=count+`zless $file | egrep "602001" | wc -l`;
	else
		echo "Count for 505001 :"$count_505001" on date: "$tempdate
		echo "Count for 602001 :"$count_602001" on date: "$tempdate
		tempdate=${file:18:10}
		count_505001=`zless $file | egrep "505001" | wc -l`;
		count_602001=`zless $file | egrep "602001" | wc -l`;
	fi
done;

```

here is the example output:

```nohighlight
Count for 505001 :236 on date: 2015-05-01
Count for 602001 :371 on date: 2015-05-01

Count for 505001 :9 on date: 2015-05-02
Count for 602001 :42 on date: 2015-05-02

Count for 505001 :53 on date: 2015-05-03
Count for 602001 :63 on date: 2015-05-03
.
.
.
Count for 505001 :423 on date: 2015-10-31
Count for 602001 :745 on date: 2015-10-31

```

---

<div class="post-metadata">

**Author:** ![RudiC](https://community.unix.com/letter_avatar/rudic/32/5_5575768a8748004e209b776fc1b2916d.png) [@RudiC](https://community.unix.com/u/RudiC)\
**Post date:** [November 7, 2015, 7:43pm UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/2 "2015-11-07T19:43:13Z")

</div>

Not sure I understand ... above is a list of file names, of which roughly two thirds are to be ignored? In the remaining files (which I infer are `tar` red and `gzip` ped?) , you want to count the occurrences of two strings?

---

<div class="post-metadata">

**Author:** ![Don\_Cragun](https://community.unix.com/user_avatar/community.unix.com/don_cragun/32/2682_2.png) [@Don\_Cragun](https://community.unix.com/u/Don_Cragun)\
**Post date:** [November 8, 2015, 1:10am UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/3 "2015-11-08T01:10:31Z")

</div>

It would seem that the following script should do what you were trying to do:

```nohighlight
#!/bin/bash
shopt -s nullglob
for file in transactions.cor.2015-0[56789]*
do	[["$file" != ${file##*.chk.}]] && continue
	if [[$tempdate = ${file:17:10}]]
	then	count_505001=$((count_505001 + $(zless "$file" | fgrep -c 505001)))
		count_602001=$((count_602001 + $(zless "$file" | fgrep -c 602001)))
		continue
	fi
	if [[$tempdate != ""]]
	then	echo "Count for 505001: $count_505001 on date: $tempdate"
		echo "Count for 602001: $count_602001 on date: $tempdate"
	fi
	tempdate=${file:17:10}
	count_505001=$(zless "$file" | fgrep -c 505001)
	count_602001=$(zless "$file" | fgrep -c 602001)
done
if [[$tempdate != ""]]
then	echo "Count for 505001: $count_505001 on date: $tempdate"
	echo "Count for 602001: $count_602001 on date: $tempdate"
fi

```

Note that several syntax errors have been corrected, it now skips `*.chk.*` files, it prints results from the sum of the counts for each date (instead of the count from the last file on each date), and it prints results from the last date processed.

Note also that `fgrep` is usually faster than `egrep` and `fgrep -c` is definitely faster than `egrep | wc -l` .

This could all be built into an `awk` script that would only need to `zless` each file once (instead of twice) and could eliminate the `grep` (and `wc` ) invocations completely.

Late fixes added:

1. Changed `[[template != ""]]` to `[[$template != ""]]` in two places.
2. Added `shopt -s nullglob` .

---

<div class="post-metadata">

**Author:** ![VasuKukkapalli](https://community.unix.com/letter_avatar/vasukukkapalli/32/5_5575768a8748004e209b776fc1b2916d.png) [@VasuKukkapalli](https://community.unix.com/u/VasuKukkapalli)\
**Post date:** [November 8, 2015, 3:06am UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/4 "2015-11-08T03:06:25Z")

</div>

Hello Rudi,

Thanks for checking for me.

1. Yes, all files (file name start with exp and file name contains `.chk.` ) should be ignored.
2. The .tgz file = gzip (tar (text file) )
3. i would like to count of the 1st string in all rest of the files as date wise and count of the 2nd string in all rest of files as date wise

Please find my sample output in above post

---

<div class="post-metadata">

**Author:** ![Don\_Cragun](https://community.unix.com/user_avatar/community.unix.com/don_cragun/32/2682_2.png) [@Don\_Cragun](https://community.unix.com/u/Don_Cragun)\
**Post date:** [November 8, 2015, 4:56am UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/5 "2015-11-08T04:56:56Z")

</div>

Hello VasuKukkapalli,  
Since you didn't give us your compressed files, it is difficult to reproduce your results, but doesn't the script I provided in post #3 in this thread do what you requested?

---

<div class="post-metadata">

**Author:** ![RudiC](https://community.unix.com/letter_avatar/rudic/32/5_5575768a8748004e209b776fc1b2916d.png) [@RudiC](https://community.unix.com/u/RudiC)\
**Post date:** [November 8, 2015, 5:06am UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/6 "2015-11-08T05:06:30Z")

</div>

As the files are `tar` red, I don't think `zless` will yield correct results dependably. As you seem to be using `bash` , the `extglob` option might be available. Try

```nohighlight
shopt -s extglob
for FN in tra!(*chk*)z
  do tar -Oxzf $FN | 
        awk -vdate=${FN:17:10} '/505001/ {CNT1++} /602001/ {CNT2++} END {print "Count 505:", CNT1, ", date:", date; print "Count 602:", CNT2, ", date:", date}'
  done

```

Were the files available clear coded, `awk` could handle all files in one go, the shell for loop were unnecessay.

---

<div class="post-metadata">

**Author:** ![VasuKukkapalli](https://community.unix.com/letter_avatar/vasukukkapalli/32/5_5575768a8748004e209b776fc1b2916d.png) [@VasuKukkapalli](https://community.unix.com/u/VasuKukkapalli)\
**Post date:** [November 8, 2015, 12:05pm UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/7 "2015-11-08T12:05:56Z")

</div>

Thank you So much Don Cragun Sir, it works for me and giving the correct results.

Hello Rudi,

Thank you for spending your time!  
i am getting the count from individual files, not on date wise, please find the output below:

```nohighlight
Count 505: , date: 2015-06-21
Count 602: 351 , date: 2015-06-21
Count 505: , date: 2015-06-21
Count 602: 89 , date: 2015-06-21
Count 505: , date: 2015-06-21
Count 602: 167 , date: 2015-06-21
Count 505: 10 , date: 2015-06-21
Count 602: 460 , date: 2015-06-21
Count 505: 14 , date: 2015-06-21
Count 602: 612 , date: 2015-06-21
Count 505: 16 , date: 2015-06-21
Count 602: 547 , date: 2015-06-21
Count 505: 7 , date: 2015-06-21
Count 602: 469 , date: 2015-06-21
Count 505: 2 , date: 2015-06-21
Count 602: 464 , date: 2015-06-21
Count 505: 1 , date: 2015-06-22
Count 602: 201 , date: 2015-06-22
Count 505: , date: 2015-06-22
Count 602: 47 , date: 2015-06-22
Count 505: 1 , date: 2015-06-22
Count 602: 127 , date: 2015-06-22
Count 505: 15 , date: 2015-06-22
Count 602: 492 , date: 2015-06-22
Count 505: 41 , date: 2015-06-22
Count 602: 557 , date: 2015-06-22
Count 505: 27 , date: 2015-06-22
Count 602: 475 , date: 2015-06-22
Count 505: 23 , date: 2015-06-22
Count 602: 411 , date: 2015-06-22
Count 505: 4 , date: 2015-06-22
Count 602: 382 , date: 2015-06-22
Count 505: 1 , date: 2015-06-23
Count 602: 217 , date: 2015-06-23
Count 505: , date: 2015-06-23
Count 602: 139 , date: 2015-06-23
Count 505: 17 , date: 2015-06-23
Count 602: 450 , date: 2015-06-23
Count 505: 27 , date: 2015-06-23
Count 602: 526 , date: 2015-06-23
Count 505: 32 , date: 2015-06-23
Count 602: 461 , date: 2015-06-23
Count 505: 22 , date: 2015-06-23
Count 602: 457 , date: 2015-06-23
Count 505: 8 , date: 2015-06-23
Count 602: 446 , date: 2015-06-23
Count 505: , date: 2015-06-24
Count 602: 165 , date: 2015-06-24
Count 505: , date: 2015-06-24
Count 602: 60 , date: 2015-06-24
Count 505: , date: 2015-06-24
Count 602: 147 , date: 2015-06-24
Count 505: 26 , date: 2015-06-24
Count 602: 485 , date: 2015-06-24
Count 505: 31 , date: 2015-06-24
Count 602: 522 , date: 2015-06-24
Count 505: 33 , date: 2015-06-24
Count 602: 486 , date: 2015-06-24
Count 505: 13 , date: 2015-06-24
Count 602: 353 , date: 2015-06-24
Count 505: 3 , date: 2015-06-24
Count 602: 352 , date: 2015-06-24

```

But i need output like below:

```nohighlight
Count for 505001: 49 on date: 2015-06-21
Count for 602001: 3159 on date: 2015-06-21
Count for 505001: 112 on date: 2015-06-22
Count for 602001: 2692 on date: 2015-06-22
Count for 505001: 107 on date: 2015-06-23
Count for 602001: 2696 on date: 2015-06-23
Count for 505001: 106 on date: 2015-06-24
Count for 602001: 2570 on date: 2015-06-24

```

Basically the consolidate count on date wise...

---

<div class="post-metadata">

**Author:** ![Aia](https://community.unix.com/user_avatar/community.unix.com/aia/32/917_2.png) [@Aia](https://community.unix.com/u/Aia)\
**Post date:** [November 8, 2015, 12:15pm UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/8 "2015-11-08T12:15:03Z")

</div>

> [@don cragun](#):
>
> [...]
> 
> ```plaintext
> #!/bin/bash
> for file in transactions.cor.2015-0[56789]*
> do	[["$file" != ${file##*.chk.}]] && continue
> if [[$tempdate = ${file:17:10}]]
> then	count_505001=$((count_505001 + $(zless "$file" | fgrep -c 505001)))
> count_602001=$((count_602001 + $(zless "$file" | fgrep -c 602001)))
> continue
> fi
> if [[tempdate != ""]]
> then	echo "Count for 505001: $count_505001 on date: $tempdate"
> echo "Count for 602001: $count_602001 on date: $tempdate"
> fi
> tempdate=${file:17:10}
> count_505001=$(zless "$file" | fgrep -c 505001)
> count_602001=$(zless "$file" | fgrep -c 602001)
> done
> if [[tempdate != ""]]
> then	echo "Count for 505001: $count_505001 on date: $tempdate"
> echo "Count for 602001: $count_602001 on date: $tempdate"
> fi
> 
> ```

Hi Don,

`[[tempdate != ""]] ` will always be true since the string `tempdate` is never `""` .

---

<div class="post-metadata">

**Author:** ![Don\_Cragun](https://community.unix.com/user_avatar/community.unix.com/don_cragun/32/2682_2.png) [@Don\_Cragun](https://community.unix.com/u/Don_Cragun)\
**Post date:** [November 8, 2015, 2:07pm UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/9 "2015-11-08T14:07:17Z")

</div>

> [@aia](#):
>
> ```plaintext
> #!/bin/bash
> for file in transactions.cor.2015-0[56789]*
> do	[["$file" != ${file##*.chk.}]] && continue
> if [[$tempdate = ${file:17:10}]]
> then	count_505001=$((count_505001 + $(zless "$file" | fgrep -c 505001)))
> count_602001=$((count_602001 + $(zless "$file" | fgrep -c 602001)))
> continue
> fi
> if [[tempdate != ""]]
> then	echo "Count for 505001: $count_505001 on date: $tempdate"
> echo "Count for 602001: $count_602001 on date: $tempdate"
> fi
> tempdate=${file:17:10}
> count_505001=$(zless "$file" | fgrep -c 505001)
> count_602001=$(zless "$file" | fgrep -c 602001)
> done
> if [[tempdate != ""]]
> then	echo "Count for 505001: $count_505001 on date: $tempdate"
> echo "Count for 602001: $count_602001 on date: $tempdate"
> fi
> 
> ```
> 
> Hi Don,  
> `[[tempdate != ""]] ` will always be true since the string `tempdate` is never `""` .

Hi Aia,  
`$tempdate` will expand to an empty string inside the `for` loop the first time a filename matching `transactions.cor.2015-0[56789]*` is found that does not contain the string `.chk.` .

`$tempdate` will expand to an empty string after the `for` loop completes if, and only if, no filenames matching `transactions.cor.2015-0[56789]*` were found that did not contain the string `.chk.` , or no filenames matching `transactions.cor.2015-0[56789]*` were found.

There should also be a check to be sure that at least one file matched the pattern `transactions.cor.2015-0[56789]*` (or the `bash` option that causes a non-matching pattern to expand to an empty list should be specified). I normally use `ksh` instead of `bash` . With `ksh` , I would avoid this problem by changing:

```plaintext
for file in transactions.cor.2015-0[56789]*

```

to:

```plaintext
for file in ~(N:transactions.cor.2015-0[56789]*)

```

which will not execute the loop if there are no matches. I always forget the option that enables this feature in `bash` and have to look it up, but for recent versions of `bash` , it is:

```plaintext
shopt -s nullglob
for file in transactions.cor.2015-0[56789]*

```

---

<div class="post-metadata">

**Author:** ![Aia](https://community.unix.com/user_avatar/community.unix.com/aia/32/917_2.png) [@Aia](https://community.unix.com/u/Aia)\
**Post date:** [November 8, 2015, 2:17pm UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/10 "2015-11-08T14:17:58Z")

</div>

Hi Don,

Then, shouldn't be: ` if [[$tempdate != ""]]` ?

---

<div class="post-metadata">

**Author:** ![Don\_Cragun](https://community.unix.com/user_avatar/community.unix.com/don_cragun/32/2682_2.png) [@Don\_Cragun](https://community.unix.com/u/Don_Cragun)\
**Post date:** [November 8, 2015, 2:58pm UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/11 "2015-11-08T14:58:01Z")

</div>

> [@aia](#):
>
> Hi Don,
> 
> Then, shouldn't be: ` if [[$tempdate != ""]]` ?

With single square brackets, you need the \<dollar-sign\> and quotes:

```plaintext
if ["$template" != ""]

```

because the shell is parsing operands to a built-in. But, double square brackets are part of the shell command language, not a built-in, so the \<dollar-sign\> and quotes are optional. All of the following produce the same results:

```plaintext
if [[template != ""]]
if [[$template != ""]]
if [["$template" != ""]]

```

taking the `then` branch if `$template` expands to a non-empty string. But the following are equivalent to each other, but, unlike the above three commands, always take the `then` branch:

```plaintext
if [["template" != ""]]
if [['template' != ""]]

```

---

<div class="post-metadata">

**Author:** ![Aia](https://community.unix.com/user_avatar/community.unix.com/aia/32/917_2.png) [@Aia](https://community.unix.com/u/Aia)\
**Post date:** [November 8, 2015, 3:44pm UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/12 "2015-11-08T15:44:58Z")

</div>

> [@don cragun](#):
>
> [...] But, double square brackets are part of the shell command language, not a built-in, so the \<dollar-sign\> and quotes are optional. All of the following produce the same results:
> 
> ```plaintext
> if [[template != ""]]
> if [[$template != ""]]
> if [["$template" != ""]]
> 
> ```

Hi Don,  
First, thank you for taking the time.

I understand that the properties of the double `[[` does not suffer of the same issues that the single `[` or `test` . It does not do word splitting nor globing, therefore you do not need to double quote the left-hand side of it and you only quote the right side if you do not want a pattern match, instead of a string comparison.

However, `if [[template != ""]]` does not expand since it is a literal string. How can it be the same that `$template` with or without quotes?

Perhaps, I can demonstrate my question better with a test.

```plaintext
$ cat test_test
#!/bin/bash
set -x
template=""
[[template != ""]] && echo "success!" || echo "failure!"
[[$template != ""]] && echo "success!" || echo "failure!"

```

```plaintext
$ ./test_test
+ template=
+ [[template != '']] <-- it is the literal string template
+ echo 'success!'
success! <-- this should be failure! if it were dereferencing 
+ [['' != '']] <-- dereferenced to the value in template
+ echo 'failure!'
failure!

```

---

<div class="post-metadata">

**Author:** ![drl](https://community.unix.com/user_avatar/community.unix.com/drl/32/542_2.png) [@drl](https://community.unix.com/u/drl)\
**Post date:** [November 8, 2015, 4:52pm UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/13 "2015-11-08T16:52:24Z")

</div>

Hi.

Arithmetic expressions work that way:

```nohighlight
Within an expression,
       shell variables may also be referenced by name without using the
       parameter expansion syntax. 

```

-- `man bash` , ARITHMETIC EVALUATION

Best wishes ... cheers, drl

---

<div class="post-metadata">

**Author:** ![Aia](https://community.unix.com/user_avatar/community.unix.com/aia/32/917_2.png) [@Aia](https://community.unix.com/u/Aia)\
**Post date:** [November 8, 2015, 5:23pm UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/14 "2015-11-08T17:23:10Z")

</div>

> [@drl](#):
>
> [...]
> 
> Arithmetic expressions work that way:  
> [...]

Hi drl,

Could you tell me when this `if [[template != ""]] ` became an arithmetic expression?  
As far as I understand it it is a string conditional expression. Maybe that's what I am missing. But if this were the case, then I would have to still believe that:

> [@](#):
>
> ```plaintext
> if [[template != ""]]
> if [[$template != ""]]
> if [["$template" != ""]]
> 
> ```
> 
> All of the following produce the same results

it is not so?

---

<div class="post-metadata">

**Author:** ![Don\_Cragun](https://community.unix.com/user_avatar/community.unix.com/don_cragun/32/2682_2.png) [@Don\_Cragun](https://community.unix.com/u/Don_Cragun)\
**Post date:** [November 8, 2015, 5:44pm UTC](https://community.unix.com/t/extract-count-of-string-in-all-files-and-display-on-date-wise/356631/15 "2015-11-08T17:44:28Z")

</div>

Hi Aia & drl,  
Sorry. Yes, you are correct. I have updated the script in post #3 of this thread to:

1. Change `[[template != ""]]` to `[[$template != ""]]` .
2. Add `shopt -s nullglob` before the `for` loop.

I'm most familiar with the features in the standards. And, although arithmetic expansion (i.e., `$((expression))` ) is in the standards; arithmetic commands (i.e., `((expression))` ) and compound conditions (i.e., `[[expression]]` ) are not in the standards (at least not yet, and not currently planned even for the next revision). I believe the treatment of variables in arithmetic expansions and arithmetic commands are the same; I incorrectly assumed that that behavior carried over into compound conditions as well.

Thanks you both for clarifying the distinction for me.

- Don