How to find ip addresses in logfiles?

Shell Programming and Scripting
1

Hi guys,

I need to check a few log files as below to find out whether certain ip addresses is present on these log files.

type8code0: ls -alt
-rw-r--r-- 1 root other 796219588 Mar 20 02:25 logfile
drwxr-xr-x 2 root root 1536 Mar 20 02:00 .
-rw-r--r-- 1 root other 1854093343 Mar 20 02:00 logfile.hour02
-rw-r--r-- 1 root other 366729263 Mar 20 01:00 logfile.hour01
-rw-r--r-- 1 root other 9001399293 Mar 20 00:47 logfile.20.Z
-rw-r--r-- 1 root other 8267721901 Mar 19 00:45 logfile.19.Z
-rw-r--r-- 1 root other 7498682761 Mar 18 00:39 logfile.18.Z
-rw-r--r-- 1 root other 6196926607 Mar 17 00:31 logfile.17.Z
-rw-r--r-- 1 root other 4794493570 Mar 16 00:23 logfile.16.Z

I've saved the list of ip addresses in �iplist� file.
cat iplist
10.10.10.10
10.10.10.11
10.10.10.12
10.10.10.13
10.10.10.14

What is the best command to do this?
This is what I do now, but it takes sometime. Hopefully there is easy way to do this.

grep 10.10.10.10 logfile > output_logfile_10.10.10.10
grep 10.10.10.11 logfile > output_ logfile_10.10.10.11

and so on

zcat logfile.16.Z | grep 10.10.10.10 > output_logfile.16.Z_10.10.10.10
zcat logfile.16.Z | grep 10.10.10.11 > output_logfile.16.Z_10.10.10.11

and so on

Thanks

2

My first guess would be:

#! /bin/bash

logfile=/path/to/logfile
while read ip; do
	zgrep "$ip" $logfile > output_logfile_${ip}
done <iplist

It could be done more elegantly, and maybe more efficiently, if you post more details/requirements. This should work for a quick'n'dirty script, though.

1 Like
3

Thanks LivinFree for the bash script. I�m new in a bash script, but I�ll try to learn this.
There are a few log files I need to check. All of them located in /logs folder and there are (from March 16 till today):

# pwd
  /logs

-rw-r--r-- 1 root other 1854093343 Mar 20 02:00 logfile.hour02
-rw-r--r-- 1 root other 366729263 Mar 20 01:00 logfile.hour01
-rw-r--r-- 1 root other 9001399293 Mar 20  00:47 logfile.20.Z
-rw-r--r-- 1 root other 8267721901 Mar 19  00:45 logfile.19.Z
-rw-r--r-- 1 root other 7498682761 Mar 18  00:39 logfile.18.Z
-rw-r--r-- 1 root other 6196926607 Mar 17  00:31 logfile.17.Z
-rw-r--r-- 1 root other 4794493570 Mar 16  00:23 logfile.16.Z

# cat iplist
10.10.10.10
10.10.10.11
10.10.10.12
10.10.10.13
10.10.10.14

I would like to send output of this file into my home directory which is in ~/result

Thanks again for your help

---------- Post updated at 01:07 PM ---------- Previous update was at 12:57 PM ----------

I've modified the bash script based on log files location. Do you think this will work? Thanks

#! /bin/bash

logfile=/logs/logfile.20.Z
logfile=/logs/logfile.19.Z
logfile=/logs/logfile.18.Z
logfile=/logs/logfile.17.Z
logfile=/logs/logfile.16.Z

while read ip; do
    zgrep "$ip" $logfile > ~/acs/output_logfile_${ip}
done <iplist
4

OK, here's try #2:

#! /bin/bash

for file in /logs/logfile.*; do
	while read ip; do
		if [[ "${file#${file%??}}" == ".Z" ]]; then
			# This is a compressed file - it ends with .Z - use zgrep
			zgrep ${ip} ${file} >> ~/results/output_log_${ip}
		else
			# Not a .Z file - regular ol' grep
			grep ${ip} ${file} >> ~/results/output_log_${ip}
		fi
	done <iplist
done

In your example, you're redefining the "logfile" variable - only the last one will count. You could set an array or a simple list of files to loop through, though.

See mine above - it gathers the list of logfiles at run time and loops over each one, checks to see if it has a .Z (I assume you use that to mean compressed - it's typical but not necessarily true) to determine if it should run grep or zgrep, appends the output to the output_log_$ip file (append will create if necessary).

I haven't really tested it - does it work on your system with your data?