I am new with awk scripting. I am trying to do the following:
1.) Parse various files with multiple extensions (.csv, .txt, .nbe)
2.) Print a field=>Sort via Uniq
3.) I am trying print a line that shows the total
4.) Output to a console and text file
- Will I be able to read via 'cat' a group of files in a directory.
awk
BEGIN {
if ( $0 ~ /|/) {
FS=":"; }{
print "Tool #1 Scan Results"
print --------------------
print $2 | Sort | uniq
print "Total IP Count: $count" # Not sure how to come up with the $count of IPs 
}
else ($0 ~ /Interesting/) {
print "Tool #2 Scan Results"
print ------------------
print $4 | Sort | uniq
print " Total IP Count: $count" # Not sure how to come up with the $count of IPs
}
}
Could you post sample input and an example of the desired output?
Sure Thanks.
.NBE File Format
Tool #1 Raw Results
results|xxx.xxx.3|xxx.xxx.3.85|microsoft-ds (445/tcp)|11119|
.txt File Format
Tool #2 Raw Results
Interesting xxx.xxx.3 microsoft-ds (445/tcp) xxx.xxx.3.85 11119 x
Desire Output:
Tool #1 Scan Results
-----------------------
xx.xx.xx.23
xx.xx.xx.24
xx.xx.xx.24
Total IP Count: 3
Same for Tool #2
Thanks!
... Also, is it possible to create this shell script (bash) to parse all files in the directory? Thanks in advance for your help.
Do you want to count the unique IPs accross different files or you want the count and unique per file? Could you please post a bigger part of at least three files (change all sensible data IPs etc).
Could you also provide the platform (uname -a) and the awk version (awk --version) ?
I would like the IP count for each file not combine.
Total #1 Raw Results (.NBE) (I just need the IP address)
results|xx.xx.3|xx.xx.3.85|microsoft-ds (445/tcp)|11119|Security Note
results|xx.xx.3|xx.xx.3.85|ms-wbt-server (3389/tcp)|22964|Security Note
timestamps||xx.xx.3.5|host_end|Fri Sep 26 14:20:20 2008
results|xx.xx.3|xx.xx.3.245|ntp (123/udp)|10884|Security Note
results|xx.xx.3|xx.xx.3.245|general/tcp|19506|Security Note
Tool #2 Raw Results (I just need the IP addresses)
Interesting ports on xx.xx.1.99:
Not shown: 1710 closed ports
PORT STATE SERVICE VERSION
All 1715 scanned ports on xx.xx.1.100 are filtered
Too many fingerprints match this host to give specific OS details
All 1715 scanned ports on xx.xx.1.100 are filtered
Too many fingerprints match this host to give specific OS details
Interesting ports on xx.xx.1.99:
Not shown: 1710 closed ports
PORT STATE SERVICE VERSION
Interesting ports on xx.xx.1.99:
Not shown: 1710 closed ports
PORT STATE SERVICE VERSION
******
I was using the following commands individually:
cat *.nbe | awk -F"|" '{print$3}' | sort | uniq
cat *.txt | grep Interesting | awk '{ print $4}'|sort|uniq
I would like a script for this and additional information. I perfer a bash shell for this and a summary as well. Later, I would like to add one additional file format.
Thanks!!!!!!!!!
You can try something like this (the code may need some corrections).
Just copy/paste the code in your terminal.
(use nawk or /usr/xpg4/bin/awk on Solaris)
awk 'BEGIN { cmd = "sort -t. -k1n -k2n -k3n -k4n " }
FNR == 1 {
FS = FILENAME ~ /\.nbe$/ ? "|" : OFS
if (f) {
for (k in ip) {
print k | cmd
c++
}
close(cmd)
printf "\nTotal IP Count: %d\n\n", c
c = f = 0
split("", ip)
}
print FNR == 1 ? $0 RS : RS $0 RS
next
}
(FILENAME ~ /\.nbe$/ && $2)|| /Interesting/ {
/Interesting/ && sub(/:$/,"")
ip[FILENAME ~ /\.nbe$/ ? $2 : $NF]
f = 1
}
END {
if (f) {
for (k in ip) {
print k | cmd
c++
}
close(cmd)
printf "\n\nTotal IP Count: %d\n\n", c
}
}' *.nbe *.txt
I got multiple lines or error.
Show the error please (I assume you've used nawk or XPG4 awk on Solaris, as I suggested).
Consider also that the above code is very expensive because of the continuous forks for the sorting routine. If you don't really need the sort (for example if unique is sufficient) or the IPs first 3 bits match and you have GNU awk, the code could be modified and become much faster.
Edit: Ah, and, of course, the code will be much more efficient if re-written in Perl (no need to go out for the sort routine).