Help finding non duplicates

Shell Programming and Scripting
1

I am currently creating a script to find filenames that are listed once in an input file (find non duplicates). I then want to report those single files in another file. Here is the function that I have so far:

function dups_filenames
{
file2=""
file1=""
file=""
dn=""
ch=""
pn=""
 
while read file dn ch pn
do
if [[ $file != $file1 && $file1 != $file2 ]]; then
echo "FILE \t\t\t\t\t CHECKSUM" >> "$dirs"_"$host"_singlefilelog
echo "---- \t\t\t\t\t --------" >> "$dirs"_"$host"_singlefilelog
printf "%-40s%-50s\n" $file1  $ch1 >> "$dirs"_"$host"_singlefilelog
printf "%-20s%-20s\n" "PATH-> "$dn1 >> "$dirs"_"$host"_singlefilelog
echo  >> "$dirs"_"$host"_singlefilelog
fi
 
file2=$file1
file1=$file
dn1=$dn
ch1=$ch
pn1=$pn
done < "$dirs"_"$host"_filelists
}

"$dirs"_"$host"_singlefilelog = the output file

The above code does find single occurrances, but it does not report single files at the bottom of the text file ("$dirs"_"$host"_filelists). If I printf "$file" instead of "$file1," the output does not report the top file in the text file.

Any suggestions?
Thank you

2

Not sure what your conveying here. What does but it does not report single files... mean..?

Yes. Its because you have assigned null value to file1 as file1="" hence it prints nothing to the output file when the first line of "$dirs"_"$host"_filelists is read. But when 2nd and rest of the lines are read you have assigned the values as below in script so you get the subsequent filenames reported/printed.

...
file2=$file1 
file1=$file 
dn1=$dn 
ch1=$ch 
pn1=$pn 
done < "$dirs"_"$host"_filelists
3

Please provide and example of your initial input file.

... depending on how it looks like, maybe you can just use the command :

uniq -u
4

"What does but it does not report single files... mean..?" If the last file in the input file is a single file, the script will not report it.

Here is an example of the input file:

filename size checksum path

file1 23 72625276372 /dir/dir1/dir2
file2 38 93939209302 /dir/dir1/dir2/dir3
file2 38 93939209302 /dir/dir1
file3 10 82828282282 /dir/dir1/dir5

The code I pasted earlier will report file1 but not file3 (since it is the last entry in the input file).

5 
awk '{print$1}' infile | uniq -u | sed 's/^/^/' | egrep -f - infile

awk '{a[$1]++;b[$1]=$0}END{for(i in a) if(a<2) print b}' infile

---------- Post updated at 12:34 PM ---------- Previous update was at 12:32 PM ----------

For a better pattern matching :

awk '{print$1}' infile | uniq -u | sed 's/.*/^& /' | egrep -f - infile
6

Thank you ctsgnb.

The second code

awk '{a[$1]++;b[$1]=$0}END{for(i in a) if(a<2) print b}' infile

works. When I run the first and third codes I receive "egrep: can't open - " errors.

I have another input file that is sorted by checksums instead of filenames but the above awk code still finds the non duplicates according to filename, not checksum. I am just starting to dive into awk so I have no clue what to change in the code to make it look at the checksums instead of filenames. Any suggestions?

Thank you

---------- Post updated at 10:23 PM ---------- Previous update was at 09:51 PM ----------

Nevermind...my input file was not properly named. Just had to change $2 to $3 in my below awk print.

awk '{print $1,"\n""PATH-> "$3}'

Thanks again!

7

I tested it on a FreeBSD machine, maybe you egrep implementation has a different behaviour and cannot deal with the hyphen