Each row represents a packet with a time stamp (epoch time) and the source IP address. As the granularity level is in "seconds", hence there are multiple entries for the same time stamp. So in 1st second there are two packets (from 1 IP), 2nd second three (from 3 IPs), 3rd second five (from 3 IPs) and so on.
I want to have a script using sed/awk (as the log files are quite big) which takes "time (seconds)" as the user input and gives the number of packets (instances of epoch time) and number of unique IP address within that specified time as the output.
So for e.g., if user gives 1 second as the input, the output file (3 columns) should be like:
#Time No of Packets No of Unique IPs
1 2 1
2 3 3
3 5 3
4 2 2
Similarly for user input as 2 second the output file should be like:
#Time No of Packets No of Unique IPs
1 5 4
2 7 4
PS: here 1 and 2 in the first column means first two seconds and next two seconds respectively.
Sorry, I forgot to point out that the script assumes input from stdin -- I'm an old school programmer and generally write code to take input from stdin as this allows preprocessing of the input (through sed or grep) if needed without having to make a change to the script, or code the script to do extra work.
The script could also be modified slightly (the very last line) to allow the name of the input file to be supplied on the command line to read from stdin:
awk -v bin_size=${1:-5} '
# body of awk from previous example
' $2
Assuming the script is saved in foo.ksh, this allows it to be invoked both ways:
In place of "number of unique IPs" for that timeinterval, if I would like to calculate "number of NEW IPs (comparing the IPs in the current time interval to the IPs in the previous time interval), could someone suggest the modifications to the script?
Small tweeks to the original awk to show number of new IPs in the current bin compared to the previous.
#!/usr/bin/env ksh
awk -v bin_size=${1:-5} '
function dump( )
{
if( NR == 1 )
return;
new_count = 0;
for( u in unique ) # compute total in this bin that were not in last bin
if( last_bin == 0 )
new_count++;
printf( "%3d %3d %3d\n", bin+1, total, new_count );
bin++;
}
{
if( $1+0 >= next_bin )
{
dump( );
next_bin = $1 + bin_size;
delete last_bin;
for( u in unique ) # copy hits from this bin
last_bin = 1;
delete unique;
total = 0;
}
unique[$2]++
total++;
}
END {
if( total )
dump( );
}
'
exit
Glad it worked for you. I've added some comments. I'll watch the thread if you have specific questions.
awk -v bin_size=${1:-5} '
# use a function so we can call as we process input and at the end without duplicating the code
function dump( ) # dump out the information that we collected about the last bin
{
if( NR == 1 ) # we will call this for the first record;
return; # if this is the first record (NR equals 1) then we skip the print
new_count = 0;
for( u in unique ) # look at each unique IP we saved
if( last_bin == 0 ) # if it wasnt noticed last time, count it
new_count++;
bin++;
printf( "%3d %3d %3d\n", bin, total, new_count ); # print all of the counts
}
{ # process for each record in the file (impled true condition)
if( $1+0 >= next_bin ) # if timestamp (col 1) is in the next bin
{
dump( ); # print data from the previous bin
next_bin = $1 + bin_size; # mark the start of the next bin
delete last_bin; # must delete contents of last bin
for( u in unique ) # copy hits from this bin
last_bin = 1; # for comparison when we see the start of next bin
delete unique; # must delete the list of unique IPs from the current bin before we start
total = 0; # zero number of hits in the bin
}
unique[$2]++ # count the number of times this IP address was seen in the bin
total++; # total number of entries in the bin
}
END { # at the end of the file, one last print if we saw something in the previous bin
if( total )
dump( );
}
'
Can you modify it to compute the 'new IPs in the current bin compared to the ENTIRE HISTORY upto that interval instead of just the previous interval' ?
I have another column to the input file and would like to sum-up and print the entries of that column for the user specified time interval. For e.g. if the user specifies 5 second as the input, the script should add all the entries of the third column for this 5 sec interval and print it alongside the other information being currently printed by the above script i.e. Time-stamp, number of packets, number of uniq IPs in that interval and number of new IP in that interval as compared to the previous interval.
Okay. I have a file with three columns. Time-stamp, IP Address and Bytes. Time-stamp is in epoch and there are multiple entries for same time-stamp, each representing a packet.
I need a awk/sed script which takes time (in secs) as the user input and give an output file with five columns (tab separated). Time-stamp, Number of packets in that interval (essentially a count of number of entries within that time interval), number of unique IPs in that interval, number of NEW IPs in that interval (as compared to the previous interval) and sum of bytes in that interval.
I go through the whole post and don't see any source files which have three columns, and if you need the result as 5 columns, the best way is to show the sample source file, and your expect output directly.
When you reply that others' codes don't resolve your problem with your non-write-down purpose, how can we think about you?
