Count Unique values from multiple lists of files

Shell Programming and Scripting
1

Looking for a little help here.

I have 1000's of text files within a multiple folders.

YYYY/
       /MM
             /1000's Files

Eg.

2014/01/1000 files
2014/02/1237 files
2014/03/1400 files

There are folders for each year and each month, and within each monthly folder there are 1000's of txt files.

Within each txt file there are 1000's of values.

eg values

546546541121321321142839741ACME183012839081
56412314235412563416243banana31231928301928
239187239719231231231273192Orange1283018231

What I want to do is for each year/month is to find and count all the Unique Alpha upper of lowercase values and get a count for that value.

EG.

Year: 2008
Month: 01 

ACME: 40057
banana: 20253
Orange:14321
etc..

Then move on to the second monthly folder in 2008 and so on until all years and months parsed.

It would fine all the find unique values and count but do this for all the year and monthly folders.

Any help would be greatly appreciated, as I'd rather not do this manually.

2

Please show the input that would cause that output.

What have you tried?

3

My input looks like this

Files

20140902114511523AAA112.txt
20140902114511523BBB113.txt
20140902114511523XXX114.txt

Within each file there values look like this.

01650126861409662618TUCAAAA_____#20140902075659
01650018671409678351TUCABBB_____#20140902121932
01650070281409691297TUCARSA_____#20140902155457
01650133911409663109TUCAPDS_____#20140902155457

Initially was i just going to cat the files and grep the out put and count

eg. cat *.txt|grep TUCAAAA|wc -l for each folder and year.

I know there must be a much simpler way then pasting in hundreds of count lines into a .sh script and running that.

Another thought i had was to get a unique values, then do something like this.

while read line ; do
        a=`cat /home/test/2014/01/*.txt|grep $line|wc -l`
        echo $line $a >> /home/test/counts.txt
done < /home/test/values.txt

It's getting the unique values in the first place that is holding me up.

4

How about putting the to-be-found values into a file and then run

grep -of file /home/test/2014/01/*.txt | sort | uniq -c
      3 TUCAAAA
      3 TUCARSA

on every directory? Would you accept a different output layout?

5

My issue right now is gathering the unique values, if all values were the same fixed length I'd just use cut and sort to get all the unique values, but not all the values are.