Extract IP from logs and

Shell Programming and Scripting
1

story
------------------------
The file is a.out, it consist a set of logs from user access to my system ( email system)

question
--------------------------
using shell script, how can we extract 2 sets of IP output from the a.out log by separate the IP, determine human and non-human access base on rules of multiple same IP access within maximum 4 sec time.

the rules,
1) multiple same IP access that maximum time more than 4 second time = human

2) multiple same IP access within maximum 4 second time = non-human

output
-------------------
output should be in b.out and c.out

meaning
1) b.out = IP list with rule 1
2) c.out = IP list with rule 2

more a.out log details,

Jan 9, 2013 2:32:29 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.108:null:login successful
Jan 9, 2013 2:34:31 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:116.197.5.39:null:login successful
Jan 9, 2013 2:36:51 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.186.188.172:null:login successful
Jan 9, 2013 2:37:00 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.162.27.126:null:login successful
Jan 9, 2013 2:37:12 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.128:null:login successful
Jan 9, 2013 2:38:11 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.186.241.248:null:login successful
Jan 9, 2013 2:41:06 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.159:null:login successful
Jan 9, 2013 2:42:34 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:124.13.171.171:null:login successful
Jan 9, 2013 2:45:28 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.188.41.226:null:login successful
Jan 9, 2013 2:48:08 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.195.222.207:null:login successful
Jan 9, 2013 2:49:22 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:60.52.28.59:null:login successful
Jan 9, 2013 2:52:39 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.175.227:null:login successful
Jan 9, 2013 2:55:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:116.197.5.39:null:login successful
Jan 9, 2013 2:56:04 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.144.156.85:null:login successful
Jan 9, 2013 2:58:17 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.71.157.170:null:login successful
Jan 9, 2013 3:01:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:08:25 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:17:03 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:19:10 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:26:57 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:28:53 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:49.249.114.228:null:login successful
Jan 9, 2013 3:30:30 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:43:08 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:188.52.46.85:null:login successful
Jan 9, 2013 3:47:46 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:47:47 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.162.27.126:null:login successful
Jan 9, 2013 3:49:22 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.140.126.74:null:login successful
Jan 9, 2013 3:49:35 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:49:54 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:51:45 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:60.50.3.143:null:login successful
Jan 9, 2013 3:52:32 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.141.95.105:null:login successful
Jan 9, 2013 3:54:25 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.143.242.152:null:login successful
Jan 9, 2013 3:56:09 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.143.242.152:null:login successful
Jan 9, 2013 3:59:38 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:124.13.188.59:null:login successful
2

So, just to be clear, what you want given the a.out in your example is for b.out to be an exact copy of a.out, and either there will be no c.out or c.out will be an empty file. Is that what you want? If that isn't what you want please explain in more detail what the output is supposed to be.

3

Although not entirely clear about what you want, this

awk     '{gsub (/:*login:*|:*null:*/,"")
          if (!max[$9] || $4>max[$9]) max[$9]=$4
          if (!min[$9] || $4<min[$9]) min[$9]=$4
         }
         END    {for (i in max) {split (max, MX, ":")
                                 split (min, MN, ":")
                                 Delta = (MX[1]-MN[1]) * 3600 + (MX[2]-MN[2]) * 60 + MX[3]-MN[3]
                                 print i, min, max > ((Delta > 4)?"b":"c")".out"
                                }
                }
        ' OFS="\t" a.out

will list logins with a span larger than 4 sec between earliest and latest access to b.out , and the rest to c.out . It doesn't care if there's four accesses within four sec, indicating a non-human attempt. And, it does NOT take care of periods spanning across midnight - this would open a wide field, then.

4

nope, a.out is the existing file. after we ran the shell script the command will generate 2 set of files, one is b.out which contain only IP from the a.out file and the other one is c.out which also contain only IP list from a.out file.

5

And, (since there are only two records in your sample a.out with timestamps within 4 seconds of each other and those two records have different IPs), nothing will be written to c.out.

6

I modified your file to make the test posible:

# cat a.txt
Jan 9, 2013 23:59:59 PM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.108:null:login successful
Jan 9, 2013 0:00:01 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.108:null:login successful
Jan 9, 2013 2:32:29 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.108:null:login successful
Jan 9, 2013 2:34:31 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:116.197.5.39:null:login successful
Jan 9, 2013 2:36:51 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.186.188.172:null:login successful
Jan 9, 2013 2:37:00 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.162.27.126:null:login successful
Jan 9, 2013 2:37:12 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.128:null:login successful
Jan 9, 2013 2:38:11 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.186.241.248:null:login successful
Jan 9, 2013 2:41:06 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.159:null:login successful
Jan 9, 2013 2:42:34 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:124.13.171.171:null:login successful
Jan 9, 2013 2:45:28 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.188.41.226:null:login successful
Jan 9, 2013 2:48:08 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.195.222.207:null:login successful
Jan 9, 2013 2:49:22 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:60.52.28.59:null:login successful
Jan 9, 2013 2:52:39 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.175.227:null:login successful
Jan 9, 2013 2:55:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:116.197.5.39:null:login successful
Jan 9, 2013 2:56:04 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.144.156.85:null:login successful
Jan 9, 2013 2:58:17 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.71.157.170:null:login successful
Jan 9, 2013 2:58:18 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.71.157.170:null:login successful
Jan 9, 2013 2:58:19 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.71.157.170:null:login successful
Jan 9, 2013 3:01:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:08:25 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:17:03 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:19:10 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:26:57 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:28:53 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:49.249.114.228:null:login successful
Jan 9, 2013 3:30:30 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:43:08 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:188.52.46.85:null:login successful
Jan 9, 2013 3:47:46 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:47:47 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.162.27.126:null:login successful
Jan 9, 2013 3:49:22 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.140.126.74:null:login successful
Jan 9, 2013 3:49:35 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:49:54 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:51:45 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:60.50.3.143:null:login successful
Jan 9, 2013 3:52:32 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.141.95.105:null:login successful
Jan 9, 2013 3:52:34 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.141.95.105:null:login successful
Jan 9, 2013 3:52:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.141.95.105:null:login successful
Jan 9, 2013 3:54:25 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.143.242.152:null:login successful
Jan 9, 2013 3:56:09 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.143.242.152:null:login successful
Jan 9, 2013 3:59:38 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:124.13.188.59:null:login successful

Then (if python is ok):

# python -V
Python 2.4.2

cat ips.py 
#!/usr/bin/python

import re
import sys
from datetime import datetime 
import time
strptime = lambda date_string, format: datetime(*(time.strptime(date_string, format)[0:6]))

try:
   f = open('./a.txt', 'r')
   out_fic=f.read()
   f.close()
except IOError:
   print "File error"
   sys.exit(5)

FMT = '%b %d, %Y %H:%M:%S %p'
pat='^(.+\s+(AM|PM)).+login:(\d+\.\d+\.\d+\.\d+):'
seek_p=re.compile(pat,re.MULTILINE)

res_h={}
res_f={}
for match_p in seek_p.findall(out_fic):
   time_p=match_p[0]
   ip_p=match_p[2]
   try:
      last=res_h[ip_p]
      res_h[ip_p]=time_p
      delta=strptime(time_p,FMT) - strptime(last,FMT)
      delta_segs=delta.seconds 
      if delta_segs <= 4 : 
          res_f[ip_p]=True
   except KeyError:
      res_h[ip_p]=time_p
      res_f[ip_p]=False

for ip in res_f: 
    if res_f[ip]:print 'Not human ip connect : %s' % ip

./ips.py
Not human ip connect : 58.71.157.170
Not human ip connect : 220.255.2.108
Not human ip connect : 175.141.95.105
1 Like
7

thanks, need to test it out first. will let you know the result soon.

---------- Post updated at 03:40 PM ---------- Previous update was at 03:01 PM ----------

seems there is no python in my unix server

[root|webmail5.tm.net.my:/tmp] python -V
ksh: python:  not found

---------- Post updated at 03:46 PM ---------- Previous update was at 03:40 PM ----------

how to test this,

already put the code in code1.sh with the same directory to a.out file,

then enable all access to the code file

chmod 777 code1.sh

finally execute this with

./code1.sh

result,

[root|webmail5.tm.net.my:/tmp] ./code1.sh
awk: syntax error near line 1
awk: illegal statement near line 1
awk: syntax error near line 8
awk: illegal statement near line 8
8

I've tested this on my linux platform; it works fine. What system /awk version do you have? You may need to explicitly invoke e.g. nawk...

9

Hi RudiC,
While trying to run your code on OS/X, I got the error:

awk: syntax error at source line 8
 context is
	                                 print i, min, max > ((Delta > >>>  4)?"b":"c")".out" <<< 
awk: illegal statement at source line 9

I reformatted your script and changed line 8 to the following:

awk '
{       gsub (/:*login:*|:*null:*/,"")
        if (!max[$9] || $4>max[$9]) max[$9]=$4
        if (!min[$9] || $4<min[$9]) min[$9]=$4
}
END {   for (i in max) {
                split (max, MX, ":")
                split (min, MN, ":")
                Delta = (MX[1]-MN[1]) * 3600 + (MX[2]-MN[2]) * 60 + MX[3]-MN[3]
                if(Delta > 4)   print i, min, max > "b.out"
                else            print i, min, max > "c.out"
        }
}' OFS="\t" a.out

That produces the output files:
b.out:

116.197.5.39    2:34:31 2:55:36
175.143.242.152 3:54:25 3:56:09
41.138.180.156  3:17:03 3:30:30
202.162.27.126  2:37:00 3:47:47
58.27.85.20     3:47:46 3:49:54

and c.out:

202.188.41.226  2:45:28 2:45:28
175.141.95.105  3:52:32 3:52:32
188.52.46.85    3:43:08 3:43:08
210.186.188.172 2:36:51 2:36:51
220.255.2.159   2:41:06 2:41:06
175.140.126.74  3:49:22 3:49:22
210.186.241.248 2:38:11 2:38:11
210.195.222.207 2:48:08 2:48:08
220.255.2.128   2:37:12 2:37:12
41.203.67.54    3:01:36 3:01:36
58.71.157.170   2:58:17 2:58:17
60.50.3.143     3:51:45 3:51:45
49.249.114.228  3:28:53 3:28:53
60.52.28.59     2:49:22 2:49:22
175.136.58.228  3:08:25 3:08:25
41.138.175.227  2:52:39 2:52:39
175.144.156.85  2:56:04 2:56:04
124.13.171.171  2:42:34 2:42:34
124.13.188.59   3:59:38 3:59:38
220.255.2.108   2:32:26 2:32:29

I would have thought that the IP addresses that appeared in the input file only once and the IP addresses where all of the timestamps of consecutive entries for that IP address were each more than 4 seconds apart should appear in b.out rather than c.out. But, I agree that the requirements are not clear.

Note also that your calculation of Delta assumes 24 hour time notation in field 4. But with field 5 being "AM" in all of the sample input, I believe the timestamps are 12 hour time notation using fields 4 and 5.

Hi Mr_47,
I am still not clear as to what output you want. Exactly what output do you want in b.out and c.out if a.out contains the following:

Jan 9, 2013 3:01:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:42 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:01:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:02:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:03:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
1 Like
10

@Don Cragun: LOL - that was my first attempt

and I thought: "I can do this smarter!"

And, you're right - that "AM" time postfix slipped through. Sorry and thanks for that, too. But there's even more to correct date/time arithmetics, as I stated in my post.

After all I share the point of view that the specification should be way more precise...

11

if a.out contains,

Jan 9, 2013 3:01:36 AM   com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:42 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:01:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:02:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:03:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful

then the output b.out should be

b.out

175.136.58.228

c.out

41.203.67.54
12

Give this a shot:

awk     '       {gsub (/:*login:*|:*null:*/,"")
                 split ($4, TMP, ":")
                 TIME = TMP[1] * 3600 + TMP[2] * 60 + TMP[3]
                 if ($5 = "PM") TIME = TIME + 43200
                 if (LAST[$9])  {DELTA = TIME - LAST[$9]
                                 if (!DT[$9] || (DELTA < DT[$9])) DT[$9] = DELTA
                                }
                 LAST[$9] = TIME
                }
         END    {for (i in DT)  if (DT > 4)  print i > "b.out"
                                  else          print i > "c.out"
                }
        ' OFS="\t" file
13

Given that there is more than 4 seconds between:

Jan 9, 2013 3:01:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful

and less than 4 seconds between:

Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful

why isn't IP address 41.203.67.54 in both b.out and c.out?

14

Hi,

from

Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful

the IP was confirmed a non-human access because it authenticate more than 1 time within less than 4 second. so it will straight go to non human list in c.out.

---------- Post updated at 02:16 PM ---------- Previous update was at 02:14 PM ----------

this wont work as using,

more code1.sh

awk     '       {gsub (/:*login:*|:*null:*/,"")
                 split ($4, TMP, ":")
                 TIME = TMP[1] * 3600 + TMP[2] * 60 + TMP[3]
                 if ($5 = "PM") TIME = TIME + 43200
                 if (LAST[$9])  {DELTA = TIME - LAST[$9]
                                 if (!DT[$9] || (DELTA < DT[$9])) DT[$9] = DELTA
                                }
                 LAST[$9] = TIME
                }
         END    {for (i in DT)  if (DT > 4)  print i > "b.out"
                                  else          print i > "c.out"
                }
        ' OFS="\t" file

then

./code1.sh
 ./code1.sh
awk: syntax error near line 1
awk: illegal statement near line 1

---------- Post updated at 05:08 PM ---------- Previous update was at 02:16 PM ----------

well this simply works, but without a list in b.out ( human access) and c.out ( non human access)

15

It's pretty straightforward to add the file writting:

#!/usr/bin/python

import re
import sys
from datetime import datetime 
import time
strptime = lambda date_string, format: datetime(*(time.strptime(date_string, format)[0:6]))

try:
   f = open('./a.txt', 'r')
   out_fic=f.read()
   f.close()
except IOError:
   print "File error"
   sys.exit(5)

FMT = '%b %d, %Y %H:%M:%S %p'
pat='^(.+\s+(AM|PM)).+login:(\d+\.\d+\.\d+\.\d+):'
seek_p=re.compile(pat,re.MULTILINE)

res_h={}
res_f={}
for match_p in seek_p.findall(out_fic):
   time_p=match_p[0]
   ip_p=match_p[2]
   try:
      last=res_h[ip_p]
      res_h[ip_p]=time_p
      delta=strptime(time_p,FMT) - strptime(last,FMT)
      delta_segs=delta.seconds 
      if delta_segs <= 4 : 
          res_f[ip_p]=True
   except KeyError:
      res_h[ip_p]=time_p
      res_f[ip_p]=False

human = open('./b.out','w')
not_human = open('./c.out','w')

for ip in res_f: 
    if res_f[ip]:
        not_human.write('%s\n' % ip)
    else:
        human.write('%s\n' % ip)

human.close()
not_human.close()
sys.exit(0)
1 Like
16

yup thats works great, anyway anyone can do it in shell script as my unix server did not have any python inside.

17

Here is a shell script (bash/ksh) version

#!/bin/bash

rm -f b.out c.out
human=0
machine=0

while read line
do
   TM=${line%%AM*}AM
   [ ${#TM} -ge ${#line} ] && TM=${line%%PM*}PM
   IP=${line##*login:}
   IP=${IP%%:*}
   printf "%s %d\n" "$IP" $(date -d"$TM" +%s)
done < a.out | sort -t' ' -k1,1 -k2,2n | while read IP time
do
   if [ "$IP" = "$prev_ip" ]
   then
      if [ $human -eq 1 -o $(( time - prev_time )) -gt 4 ]
      then
         human=1
      else
         machine=1
      fi
   else
      [ $human -eq 1 ] && echo $prev_ip >> b.out
      [ $machine -eq 1 ] && echo $prev_ip >> c.out
      prev_ip=$IP
      prev_time=$time
      human=0
      machine=0
   fi
done
[ $human -eq 1 ] && echo $prev_ip >> b.out
[ $machine -eq 1 ] && echo $prev_ip >> c.out
18

got error when run this,

bash-3.00# more code1.out

rm -f b.out c.out
human=0
machine=0

while read line
do
   TM=${line%%AM*}AM
   [ ${#TM} -ge ${#line} ] && TM=${line%%PM*}PM
   IP=${line##*login:}
   IP=${IP%%:*}
   printf "%s %d\n" "$IP" $(date -d"$TM" +%s)
done < a.out | sort -t' ' -k1,1 -k2,2n | while read IP time
do
   if [ "$IP" = "$prev_ip" ]
   then
      if [ $human -eq 1 -o $(( time - prev_time )) -gt 4 ]
      then
         human=1
      else
         machine=1
      fi
   else
      [ $human -eq 1 ] && echo $prev_ip >> b.out
      [ $machine -eq 1 ] && echo $prev_ip >> c.out
      prev_ip=$IP
      prev_time=$time
      human=0
      machine=0
   fi
done
[ $human -eq 1 ] && echo $prev_ip >> b.out
[ $machine -eq 1 ] && echo $prev_ip >> c.out

bash-3.00# ./code1.out

date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:32:29 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:34:31 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:36:51 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:37:00 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:37:12 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:38:11 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:41:06 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:42:34 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:45:28 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:48:08 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:49:22 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:52:39 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:55:36 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:56:04 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:58:17 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:01:36 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:08:25 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:17:03 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:19:10 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:26:57 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:28:53 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:30:30 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:43:08 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:47:46 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:47:47 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:49:22 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:49:35 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:49:54 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:51:45 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:52:32 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:54:25 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:56:09 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:59:38 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]

bash-3.00# ls

a.out      c.out      code1.out
19

Yes, your date command doesn't support -d option (you need gnudate for the posted script). What OS are you using?

20

bash-3.00# uname -a
SunOS mldap1 5.10 Generic_137138-09 i86pc i386 i86pc

its Solaris 10