Search text pattern

Shell Programming and Scripting
1

Hello folks,

I have a text file of apache logs, that have robots.txt text with logs, i want to search all all words that are with "anyword.txt" but i dont want to search robots.txt in file.

Like i have abcd.txt file

X.X.X.X - - [25/May/2010:11:12:10+0100] "GET /myimage/index.php?page=http://abcd.com/language/id.txt??? HTTP/1.1" 200 17264 "-" "libwww-perl/5.813" "-"

So i want two things from abcd.txt

  1. -> IP-Address [X.X.X.X]
  2. -> textfile=id.txt [dont want to search robots.txt]
2 
sed -n "/robots.txt/!{s/.*\([0-9][0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\).*\/\(.*\.txt\).*/\1 \2/p;}" file
3 
grep -v "robots\.txt" |
        egrep -o "(^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\s)|([^ /]+\.txt)"
4

Not Working below error

-bash: !{s/.*\: event not found

---------- Post updated at 02:02 PM ---------- Previous update was at 02:00 PM ----------

[/COLOR]

It is also searching robots.txt as well. I have total 4 lines of log file, it will search all ips and all txt file including robots.txt and after that it hangup, it will not exit automatically.

5

Try single quotes

sed -n '/robots.txt/!{s/.*\([0-9][0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\).*\/\(.*\.txt\).*/\1 \2/p;}' file
6

the ip address is this

010.020.232.231 but it is showing -> 0.020.232.231

should i use below one?


sed -n '/robots.txt/!{s/.*\([0-9][0-9][0-9][0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\).*\/\(.*\.txt\).*/\1 \2/p;}'

if i want to search *.txt and *.php what change i should do?

7 
sed -n '/robots.txt/!{s/\([0-9]\{1,\}\.[0-9]*\.[0-9]*\.[0-9]*\).*\/\(.*\.[tp][hx][tp]\).*/\1 \2/p;}' file
8

sorry for bothering again, if i want to add robots.txt with kf4sd.php what changes are require, so it not search robots.txt and kf4sd.php and search other txt and php files.

9 
sed -e '/robots.txt/d' -e '/kf4sd.php/d' -e 's/\([0-9]\{1,\}\.[0-9]*\.[0-9]*\.[0-9]*\).*\/\(.*\.[tp][hx][tp]\).*/\1 \2/' file

Add this -e '/robots.txt/d' if you want to ignore anything else

10

Now it is showing complete line of apache log,

202.1.1.1 - - [25/May/2010:22:13:02 +0100] "GET /images/pic01..jpg HTTP/1.0" 304 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" "1.1.6.5"

second if i want to search all *.pl *.php *.txt what should change in second part.

Thanks