Is there a BASH script allowing me to grep specifics from /var/log/messages?

vgplayer54 · March 24, 2017, 12:50pm

I am wondering if there is a script (if one exists, not confident in my own scripting ability) that is able to bring up specified information from the /var/log/messages. I need to show logged traffic on specific dates and times and protocols (ie. Show all insecure FTP traffic (most likely via port number) on October 23 between 12:30pm and 12:35pm). Is there a script that can do this? No matter what I try, mine aren't working too well

RudiC · March 24, 2017, 1:59pm

Almost sure: yes.

Had we a specification worth its name, we could help further.
And, post your attempts so they can be analysed, discussed, or even improved.

Corona688 · March 24, 2017, 2:31pm

Show some lines from your log file. Hopefully the dates look like YYYY MM DD HH MM SS, that kind of timestamp is really easy to compare, everything else will mean parsing every single line to compute its date.

vgplayer54 · March 27, 2017, 10:25am

Sure, here is the code I have so far that is not working:

#!/bin/bash

read -p "Enter month (first 3 letters): " month

read -p "Enter day of month: " day

read -p "Enter starting time (HH:MM:SS): " stime

read -p "Enter ending time (HH:MM:SS): " ftime

read -p "Enter chain: " chain

read -p "Enter any other modifiers (TTL, SRC, DSP, SPT, DPT, IN, OUT, etc): " modifier

if [ -z "$month" ]; then
    month='IN='
fi

if [ -z "$chain" ]; then
    chain='IN='
fi

if [ -z "$modifier" ]; then
    modifier='IN='
fi

if [ -z "$stime" ] && [ -z "$ftime" ]; then

cat /var/log/messages | grep -i "$month $day" | grep -i $chain | grep -i $modifier

else

cat /var/log/messages | grep -i "$month $day" | grep -i $chain | grep -i $modifier | sed -n "/$stime/,/$ftime/p"

fi

And here is an example of the /var/log/messages traffic:

Mar 27 10:24:29 router kernel: [ 2298.775662] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1791 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:31 router kernel: [ 2301.516556] FORWARD-ACCEPTEDIN=eth1 OUT=eth3 MAC=00:0c:29:91:18:81:00:0c:29:02:2c:a0:08:00 SRC=195.165.11.5 DST=195.165.11.70 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=5181 PROTO=UDP SPT=59696 DPT=53 LEN=49 
Mar 27 10:24:32 router kernel: [ 2301.619601] FORWARD-DROPPEDIN=eth3 OUT=eth0 MAC=00:0c:29:91:18:95:00:0c:29:46:29:d3:08:00 SRC=195.165.11.70 DST=199.7.83.42 LEN=80 TOS=0x00 PREC=0x00 TTL=127 ID=32098 PROTO=UDP SPT=57894 DPT=53 LEN=60 
Mar 27 10:24:32 router kernel: [ 2301.619641] FORWARD-DROPPEDIN=eth3 OUT=eth0 MAC=00:0c:29:91:18:95:00:0c:29:46:29:d3:08:00 SRC=195.165.11.70 DST=202.12.27.33 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=26672 PROTO=UDP SPT=57894 DPT=53 LEN=49 
Mar 27 10:24:34 router kernel: [ 2304.527738] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1792 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:35 router kernel: [ 2305.243755] FORWARD-ACCEPTEDIN=eth3 OUT=eth1 MAC=00:0c:29:91:18:95:00:0c:29:46:29:d3:08:00 SRC=195.165.11.70 DST=195.165.11.5 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=8386 DF PROTO=UDP SPT=53 DPT=59696 LEN=49 
Mar 27 10:24:35 router kernel: [ 2305.244995] FORWARD-ACCEPTEDIN=eth1 OUT=eth3 MAC=00:0c:29:91:18:81:00:0c:29:02:2c:a0:08:00 SRC=195.165.11.5 DST=195.165.11.70 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=5182 PROTO=UDP SPT=57560 DPT=53 LEN=49 
Mar 27 10:24:35 router kernel: [ 2305.245255] FORWARD-DROPPEDIN=eth3 OUT=eth0 MAC=00:0c:29:91:18:95:00:0c:29:46:29:d3:08:00 SRC=195.165.11.70 DST=192.58.128.30 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=7802 PROTO=UDP SPT=58259 DPT=53 LEN=49 
Mar 27 10:24:35 router kernel: [ 2305.277867] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1793 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:36 router kernel: [ 2306.030264] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1794 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:36 router kernel: [ 2306.249173] FORWARD-ACCEPTEDIN=eth1 OUT=eth3 MAC=00:0c:29:91:18:81:00:0c:29:02:2c:a0:08:00 SRC=195.165.11.5 DST=195.165.11.70 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=5183 PROTO=UDP SPT=57560 DPT=53 LEN=49 
Mar 27 10:24:37 router kernel: [ 2306.885204] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1795 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:37 router kernel: [ 2307.264584] FORWARD-ACCEPTEDIN=eth1 OUT=eth3 MAC=00:0c:29:91:18:81:00:0c:29:02:2c:a0:08:00 SRC=195.165.11.5 DST=195.165.11.70 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=5184 PROTO=UDP SPT=57560 DPT=53 LEN=49 
Mar 27 10:24:38 router kernel: [ 2307.634718] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1796 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:38 router kernel: [ 2308.384824] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1797 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:39 router kernel: [ 2309.279134] FORWARD-ACCEPTEDIN=eth1 OUT=eth3 MAC=00:0c:29:91:18:81:00:0c:29:02:2c:a0:08:00 SRC=195.165.11.5 DST=195.165.11.70 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=5185 PROTO=UDP SPT=57560 DPT=53 LEN=49 
Mar 27 10:24:39 router kernel: [ 2309.339155] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1798 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:40 router kernel: [ 2309.773505] FORWARD-DROPPEDIN=eth3 OUT=eth0 MAC=00:0c:29:91:18:95:00:0c:29:46:29:d3:08:00 SRC=195.165.11.70 DST=198.41.0.4 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=10024 PROTO=UDP SPT=58259 DPT=53 LEN=49 
Mar 27 10:24:40 router kernel: [ 2310.089359] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1799 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:41 router kernel: [ 2310.839629] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1800 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:42 router kernel: [ 2311.991505] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1801 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:43 router kernel: [ 2312.741286] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1802 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:43 router kernel: [ 2313.292604] FORWARD-ACCEPTEDIN=eth1 OUT=eth3 MAC=00:0c:29:91:18:81:00:0c:29:02:2c:a0:08:00 SRC=195.165.11.5 DST=195.165.11.70 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=5186 PROTO=UDP SPT=57560 DPT=53 LEN=49 
Mar 27 10:24:43 router kernel: [ 2313.396563] FORWARD-DROPPEDIN=eth3 OUT=eth0 MAC=00:0c:29:91:18:95:00:0c:29:46:29:d3:08:00 SRC=195.165.11.70 DST=199.7.83.42 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=32099 PROTO=UDP SPT=58259 DPT=53 LEN=49 
Mar 27 10:24:43 router kernel: [ 2313.396740] FORWARD-DROPPEDIN=eth3 OUT=eth0 MAC=00:0c:29:91:18:95:00:0c:29:46:29:d3:08:00 SRC=195.165.11.70 DST=193.0.14.129 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=26639 PROTO=UDP SPT=58259 DPT=53 LEN=49 
Mar 27 10:24:43 router kernel: [ 2313.493634] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1803 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:45 router kernel: [ 2315.046166] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1804 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:46 router kernel: [ 2315.796555] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1805 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:46 router kernel: [ 2316.546721] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1806 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 27 10:24:47 router kernel: [ 2317.292821] FORWARD-ACCEPTEDIN=eth1 OUT=eth3 MAC=00:0c:29:91:18:81:00:0c:29:02:2c:a0:08:00 SRC=195.165.11.5 DST=195.165.11.70 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=5187 PROTO=UDP SPT=64648 DPT=53 LEN=49 
root@router:/home/ajfoncec/Checkpoint_4_Scripts#

Corona688 · March 27, 2017, 12:06pm

Code tags for code please.

OK, some sadist printed the dates in MON D HH:MM:SS order and we'll need to convert them before comparing.

$ awk -v FIRST='03 27 10:24:30' -v LAST='03 27 10:24:35'  'BEGIN {
        split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", A); # A[1]=Jan, etc
        for(X in A) A[A[X]]=sprintf("%02d",X) # Convert A[1]=Jan to A[Jan]=01
}
{ $1=A[$1] ; $2=sprintf("%02d", $2); } # Substitute two digit months and days into first two fields
($0 >= FIRST) && ($0 <= LAST)' logentryfile

03 27 10:24:31 router kernel: [ 2301.516556] FORWARD-ACCEPTEDIN=eth1 OUT=eth3 MAC=00:0c:29:91:18:81:00:0c:29:02:2c:a0:08:00 SRC=195.165.11.5 DST=195.165.11.70 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=5181 PROTO=UDP SPT=59696 DPT=53 LEN=49
03 27 10:24:32 router kernel: [ 2301.619601] FORWARD-DROPPEDIN=eth3 OUT=eth0 MAC=00:0c:29:91:18:95:00:0c:29:46:29:d3:08:00 SRC=195.165.11.70 DST=199.7.83.42 LEN=80 TOS=0x00 PREC=0x00 TTL=127 ID=32098 PROTO=UDP SPT=57894 DPT=53 LEN=60
03 27 10:24:32 router kernel: [ 2301.619641] FORWARD-DROPPEDIN=eth3 OUT=eth0 MAC=00:0c:29:91:18:95:00:0c:29:46:29:d3:08:00 SRC=195.165.11.70 DST=202.12.27.33 LEN=69 TOS=0x00 PREC=0x00 TTL=127 ID=26672 PROTO=UDP SPT=57894 DPT=53 LEN=49
03 27 10:24:34 router kernel: [ 2304.527738] INPUT-DROPPEDIN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:c0:00:08:08:00 SRC=192.168.48.1 DST=192.168.48.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=1792 PROTO=UDP SPT=137 DPT=137 LEN=58

$

Code in red is where you change the date range to what you want. Dates are in MM DD HH:MM:SS order.

Code in blue is the code which does the actual work, everything else is putting the date in proper MM DD HH:MM:SS order.

vgplayer54 · March 29, 2017, 7:40am

Okay so...do I add the red part to the top of the code I posted earlier? or is this a stand-alone script? Also, is it possible to allow me to input the ports/times/dates outside the script rather than editing the script each time I want to use it?

Also, the bottom lines of your code appear to be copied from my /var/log/messages example. I'm not sure if that would work as I would be monitoring traffic from a variety of machines, varying MAC addresses, etc. So I don't think that would work within the code if it's looking for the same machine each time it's run (unless that wasn't meant to be within the code?)

Don_Cragun · March 29, 2017, 8:23am

No. The bottom lines in the Corona688's post were the output produced by running the script he provided with an input file named logentryfile containing the sample data you showed us in post #4. If you want to parameterize his suggestion and read data from standard input (instead of from a file named logentryfile ), change your script to something like:

#!/bin/bash
IAm=${0##*/}
if [ $# -ne 2 ]
then	printf 'Usage: %s start end
	where start and end are starting and ending dates and times in the format
		"MM DD hh:mm:ss"
	representing the start and end times to be selected from the logfile found
	on standard input.
'	"$IAm" >&2
	exit 1
fi
awk -v FIRST="$1" -v LAST="$2"  '
BEGIN {
	split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", A); # A[1]=Jan, etc
	for(X in A) A[A[X]]=sprintf("%02d",X) # Convert A[1]=Jan to A[Jan]=01
}
{	$1=A[$1]
	$2=sprintf("%02d", $2)
} # Substitute two digit months and days into first two fields
($0 >= FIRST) && ($0 <= LAST) # Select and print entries in range.'

When you invoke this script, give it two quoted operands containing your desired start and end dates and times and pipe the logfile you want it to process into it or, if the data is in a file, redirect the input to the script from that file.

vgplayer54 · March 29, 2017, 8:45am

don cragun:

No. The bottom lines in the Corona688's post were the output produced by running the script he provided with an input file named logentryfile containing the sample data you showed us in post #4. If you want to parameterize his suggestion and read data from standard input (instead of from a file named logentryfile ), change your script to something like:
#!/bin/bash
IAm=${0##*/}
if [ $# -ne 2 ]
then    printf 'Usage: %s start end
   where start and end are starting and ending dates and times in the format
   "MM DD hh:mm:ss"
   representing the start and end times to be selected from the logfile found
   on standard input.
'    "$IAm" >&2
   exit 1
fi
awk -v FIRST="$1" -v LAST="$2"  '
BEGIN {
   split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", A); # A[1]=Jan, etc
   for(X in A) A[A[X]]=sprintf("%02d",X) # Convert A[1]=Jan to A[Jan]=01
}
{    $1=A[$1]
   $2=sprintf("%02d", $2)
} # Substitute two digit months and days into first two fields
($0 >= FIRST) && ($0 <= LAST) # Select and print entries in range.'
When you invoke this script, give it two quoted operands containing your desired start and end dates and times and pipe the logfile you want it to process into it or, if the data is in a file, redirect the input to the script from that file.

Edit: Sorry I didn't see the text at the bottom of your code tags. Thank you so much for your help. So, as an example, assuming I wanted to find all IIS network traffic (port 80) I would type: scriptname "Mar27 10:00:00 10:10:00 DPT=80" <--- is this the correct format?

Don_Cragun · March 29, 2017, 9:12am

If your script is named myscript , is located in some directory in your utility search path (as indicated by the expansion of $PATH ), and you want it to select records from a file named /var/log/messages and you want it to print out records with timestamps between 7:15 and 8:30 this morning, invoke it with:

myscript '03 29 07:15:00' '03 29 08:30:59' < /var/log/messages

As long as myscript is on your utility search path, it doesn't make any difference whether or not the script and the files it will be processing are in the same directory.

vgplayer54 · March 29, 2017, 9:16am

don cragun:

If your script is named myscript , is located in some directory in your utility search path (as indicated by the expansion of $PATH ), and you want it to select records from a file named /var/log/messages and you want it to print out records with timestamps between 7:15 and 8:30 this morning, invoke it with:
myscript '03 29 07:15:00' '03 29 08:30:59' < /var/log/messages
As long as myscript is on your utility search path, it doesn't make any difference whether or not the script and the files it will be processing are in the same directory.

Ah perfect! But what of the port number variable? Would it work if I used the command you provided and piped it with a grep statement stating the port number? ie.

myscript '03 29 07:15:00' '03 29 08:30:59' < /var/log/messages | grep 80

Don_Cragun · March 29, 2017, 2:24pm

vgplayer54:

Ah perfect! But what of the port number variable? Would it work if I used the command you provided and piped it with a grep statement stating the port number? ie.
myscript '03 29 07:15:00' '03 29 08:30:59' < /var/log/messages | grep 80

Obviously, you can search for the string 80 in awk (if you'd like to pass your script some options or some more operands) or you can grep the output produced by the code I suggested, but I don' understand how searching for 80 in your sample input has anything to do with a port number. The string 80 in the sample data you provided will match on LEN=80 and on several different ID values such as ID=7802 and ID=1800 through ID=1806 . Although it doesn't happen to match any parts of your sample data like the numbers between square brackets; parts of the SRC and DEST IP addresses; parts of the MAC addresses; and parts of the TOS , PREC , SPT , and DPT fields it certainly looks like any of those fields could provide a match and none of them seem to have anything to do with a port number.

Your original code also had searches for chain s (which do not have any obvious entries in your sample data) and it prompts for an IN modifier, but has INPUT-DROPPEDIN=value and FORWARD-ACCEPTEDIN=value (but not just IN=value ).

Note also that most lines in your sample data contain two LEN=value fields. If you wanted to match a LEN field, would it matter which one matched?

Before we talk about adding a chain of grep commands to a pipeline or adding code to the awk script to search for more patterns, please clearly describe what each of your patterns is supposed to match. (Are you hoping to match a basic regular expression, an extended regular expression, or a fixed string? Are you hoping to match a pattern if it matches any substring in a line, or to only match an entire "word" in a line?)

vgplayer54 · March 30, 2017, 10:10am

don cragun:

Obviously, you can search for the string 80 in awk (if you'd like to pass your script some options or some more operands) or you can grep the output produced by the code I suggested, but I don' understand how searching for 80 in your sample input has anything to do with a port number. The string 80 in the sample data you provided will match on LEN=80 and on several different ID values such as ID=7802 and ID=1800 through ID=1806 . Although it doesn't happen to match any parts of your sample data like the numbers between square brackets; parts of the SRC and DEST IP addresses; parts of the MAC addresses; and parts of the TOS , PREC , SPT , and DPT fields it certainly looks like any of those fields could provide a match and none of them seem to have anything to do with a port number.

Your original code also had searches for chain s (which do not have any obvious entries in your sample data) and it prompts for an IN modifier, but has INPUT-DROPPEDIN=value and FORWARD-ACCEPTEDIN=value (but not just IN=value ).

Note also that most lines in your sample data contain two LEN=value fields. If you wanted to match a LEN field, would it matter which one matched?

Before we talk about adding a chain of grep commands to a pipeline or adding code to the awk script to search for more patterns, please clearly describe what each of your patterns is supposed to match. (Are you hoping to match a basic regular expression, an extended regular expression, or a fixed string? Are you hoping to match a pattern if it matches any substring in a line, or to only match an entire "word" in a line?)

Sorry for not being overly clear - what I am attempting to do is run traffic through my network on a variety of different ports (ie. HTTP traffic on port 80, SMTP on port 143, MySQL on port 1306, etc). Then, I will be asked a variety of questions including as an example: "Show me all MySQL traffic that went through your network on March 27 between 12:30pm and 12:35pm". This is what I need the script to do: I enter the name of the script on the command line followed by variables that specify the requested information. (ie. myscript Mar 27 12:30:00 12:35:00 port=1306) and it will display all MySQL traffic on that date between those times from the /var/log/messages file. Will the above posted script be able to do this? I am..not very good at scripting awk, not sure where to use that or how to make it work for port numbers.

Also you mentioned the INPUT-DROPPED and FORWARD-ACCEPTED before from my first script, those are just chains within my iptables. I do not think they should be related to the script as it would be locating all network traffic, both dropped and accepted.

Don_Cragun · March 30, 2017, 1:47pm

vgplayer54:

Sorry for not being overly clear - what I am attempting to do is run traffic through my network on a variety of different ports (ie. HTTP traffic on port 80, SMTP on port 143, MySQL on port 1306, etc). Then, I will be asked a variety of questions including as an example: "Show me all MySQL traffic that went through your network on March 27 between 12:30pm and 12:35pm". This is what I need the script to do: I enter the name of the script on the command line followed by variables that specify the requested information. (ie. myscript Mar 27 12:30:00 12:35:00 port=1306) and it will display all MySQL traffic on that date between those times from the /var/log/messages file. Will the above posted script be able to do this? I am..not very good at scripting awk, not sure where to use that or how to make it work for port numbers.

Also you mentioned the INPUT-DROPPED and FORWARD-ACCEPTED before from my first script, those are just chains within my iptables. I do not think they should be related to the script as it would be locating all network traffic, both dropped and accepted.

I don't care whether or not you know awk (although I would hope that you are making an attempt to learn how to use it if you want to process the types of data you're asking us to help you learn how to handle). But, I do expect you to know be able to answer questions about your data and I expect you to be able to describe how you want to process your data.

Given your data (which you provided in post #4 in this thread), please show us where the port number you want to use to select records is located within those records.
If you want to select records based on chains, please explain how a chain is identified in your data.
The code you showed us used case-insensitive searches, but there doesn't seem to be any inconsistency in case in your data. Do you need case-insensitive search capabiliities?
When you pass search criteria to your script, do you expect the script to treat those criteria as regular expressions or as fixed strings?

I can almost write awk scripts in my sleep, but I can't write a script to process data in any language if I don't understand what it is that I'm trying to do.

vgplayer54 · March 30, 2017, 3:21pm

don cragun:

I don't care whether or not you know awk (although I would hope that you are making an attempt to learn how to use it if you want to process the types of data you're asking us to help you learn how to handle). But, I do expect you to know be able to answer questions about your data and I expect you to be able to describe how you want to process your data.

Given your data (which you provided in post #4 in this thread), please show us where the port number you want to use to select records is located within those records.

If you want to select records based on chains, please explain how a chain is identified in your data.

The code you showed us used case-insensitive searches, but there doesn't seem to be any inconsistency in case in your data. Do you need case-insensitive search capabiliities?

When you pass search criteria to your script, do you expect the script to treat those criteria as regular expressions or as fixed strings?

I can almost write awk scripts in my sleep, but I can't write a script to process data in any language if I don't understand what it is that I'm trying to do.

Sure, thank you for your help.

1.The port numbers are located within the /var/log/messages divided by how the port was used (ie. DPT=80 for incoming HTTP traffic or SPT=143 for outgoing mail traffic) Using only those two prefixes should be fine...such as

*PT=$portnumber

and have the user enter the variable for the port number.
2.No need for this, /var/log/messages includes all chain traffic which is what I want. No need to divide them by chain.
3.Case insensitive capabilities would be a bonus if you can make it possible! But I can definitely live without it.
4. I'm going to say fixed strings here, I don't believe there's a need for the script to search using regular expressions.