cmontr
August 7, 2008, 11:55am
1
Hello everyone,
By any chance, anyone has a script (shell) that I can modify for the following:
I have daily log files and they look like:
Accept aabbcc [02/Jun/2008:10:34:36 -0600] "111.11.111.111 uid=MyID, ou=People, o=helloworld.com " "webagent" [] []
Reject aabbcc [02/Jun/2008:10:34:36 -0600] "111.11.111.111 uid=MyID, ou=People, o=helloworld.com " "webagent" [] []
Validate ddeeff [02/Jun/2008:10:34:36 -0600] "111.11.111.111 uid=MyID, ou=People, o=helloworld.com " "webagent" [] []
The need is to run a script against it with some inputs, such as:
./script.sh uid start_date end_date all_dates
above may not contain the end_date; but eventually there should be sometimes.
The output should be something like this:
# Total Activities based on uid
uid=aabbcc
Accept: #
Reject: #
Validate: #
uid=ddeeff
Accept: #
Reject: #
Validate: #
so on... If anyone can help, greatly appreciated.
If you get errors use nawk, gawk or /usr/xpg4/bin/awk on Solaris.
awk '
!($1 in a){a[++i]=$1;a[$1]}
!($2 in b){b[++j]=$2;b[$2]}
END{
for(k=1;k<=j;k++){
print "uid=" b[k]
for(l=1;l<=i;l++){
print a[l] ": #"
}
print ""
}
}' file
Regards
cmontr
August 7, 2008, 3:22pm
3
franklin52:
If you get errors use nawk, gawk or /usr/xpg4/bin/awk on Solaris.
awk '
!($1 in a){a[++i]=$1;a[$1]}
!($2 in b){b[++j]=$2;b[$2]}
END{
for(k=1;k<=j;k++){
print "uid=" b[k]
for(l=1;l<=i;l++){
print a[l] ": #"
}
print ""
}
}' file
Regards
I appreciate for the response. I tried as you suggested on solaris but getting the error:
i used it as
awk -f a.awk a.txt > a.out
awk: syntax error near line 1
awk: bailing out near line 1
any suggestion please?
thnx much again
As mentioned, on Solaris:
/usr/xpg4/bin/awk '
!($1 in a){a[++i]=$1;a[$1]}
!($2 in b){b[++j]=$2;b[$2]}
END{
for(k=1;k<=j;k++){
print "uid=" b[k]
for(l=1;l<=i;l++){
print a[l] ": #"
}
print ""
}
}' file
Regards
cmontr
August 7, 2008, 3:32pm
5
franklin52:
As mentioned, on Solaris:
/usr/xpg4/bin/awk '
!($1 in a){a[++i]=$1;a[$1]}
!($2 in b){b[++j]=$2;b[$2]}
END{
for(k=1;k<=j;k++){
print "uid=" b[k]
for(l=1;l<=i;l++){
print a[l] ": #"
}
print ""
}
}' file
Regards
I appreciate for very fast response; I had tried the same as code above but it was the same error...I am unsure why it is bailing out at the 1st line...and syntax error..
awk: syntax error near line 1
awk: bailing out near line 1
It works well with the newer versions of awk, I made some changes, give this a try:
/usr/xpg4/bin/awk '
{if(!($1 in a)){a[++i]=$1;a[$1]}}
{if(!($2 in b)){b[++j]=$2;b[$2]}}
END{
for(k=1;k<=j;k++){
print "uid=" b[k]
for(l=1;l<=i;l++){
print a[l] ": #"
}
print ""
}
}' file
cmontr
August 7, 2008, 3:56pm
7
franklin52:
It works well with the newer versions of awk, I made some changes, give this a try:
/usr/xpg4/bin/awk '
{if(!($1 in a)){a[++i]=$1;a[$1]}}
{if(!($2 in b)){b[++j]=$2;b[$2]}}
END{
for(k=1;k<=j;k++){
print "uid=" b[k]
for(l=1;l<=i;l++){
print a[l] ": #"
}
print ""
}
}' file
Hi again, I added the following it works partially what I was looking for:
#!/usr/bin/awk -f
#!/usr/bin/nawk -f
#!/usr/bin/mawk -f
#!/usr/bin/gawk -f
output was:
uid=aabbcc
Accept: #
Reject: #
Validate: #
Is there a way to replace # with number of occurance and is there also a way to print the date stamp in the file below it ?
thanks much once again
You can use variables, assuming you have gawk:
#!/bin/sh
dat=`date`
number="5"
/usr/bin/gawk -v num="$number" -v d="$dat" '
{if(!($1 in a)){a[++i]=$1;a[$1]}}
{if(!($2 in b)){b[++j]=$2;b[$2]}}
END{
for(k=1;k<=j;k++){
print "uid=" b[k]
for(l=1;l<=i;l++){
print a[l] ": " num
}
print ""
}
print d
}' file
Save this in a file, make it executable and run it as a shell script.
cmontr
August 7, 2008, 4:27pm
9
franklin52:
You can use variables, assuming you have gawk:
#!/bin/sh
dat=`date`
number="5"
/usr/bin/gawk -v num="$number" -v d="$dat" '
{if(!($1 in a)){a[++i]=$1;a[$1]}}
{if(!($2 in b)){b[++j]=$2;b[$2]}}
END{
for(k=1;k<=j;k++){
print "uid=" b[k]
for(l=1;l<=i;l++){
print a[l] ": " num
}
print ""
}
print d
}' file
Save this in a file, make it executable and run it as a shell script.
I am sorry to keep asking questions ... I have put that in a script and ran. One thing is number of occurances did not print. and date was not working...i removed that part. Is there a way to print " [02/Jun/2008:10:34:36 -0600] " as an example for the corresponding uid in the same line of the input file ?
such as:
cat a.out
uid=aabbcc
Accept: 1
Reject: 1
Validate: 1
TimeStamp: [02/Jun/2008:10:34:36 -0600]
thanks much again
It should work with gawk or nawk. An example if you have the timestamp in a variable:
#!/bin/sh
number="5"
ts="TimeStamp: [02/Jun/2008:10:34:36 -0600]"
/usr/bin/gawk -v num="$number" -v d="$dat" '
{if(!($1 in a)){a[++i]=$1;a[$1]}}
{if(!($2 in b)){b[++j]=$2;b[$2]}}
END{
for(k=1;k<=j;k++){
print "uid=" b[k]
for(l=1;l<=i;l++){
print a[l] ": " num
}
print ts
}
}' file
cmontr
August 8, 2008, 12:50pm
11
franklin52:
It should work with gawk or nawk. An example if you have the timestamp in a variable:
#!/bin/sh
number="5"
ts="TimeStamp: [02/Jun/2008:10:34:36 -0600]"
/usr/bin/gawk -v num="$number" -v d="$dat" '
{if(!($1 in a)){a[++i]=$1;a[$1]}}
{if(!($2 in b)){b[++j]=$2;b[$2]}}
END{
for(k=1;k<=j;k++){
print "uid=" b[k]
for(l=1;l<=i;l++){
print a[l] ": " num
}
print ts
}
}' file
Hi Again,
I used nawk and got it working...I think gawk is not installed in my server. However, counts per variables like Accept, Reject did not work. I was also not bale to print the date stamp ...not sure ...tried modifying but getting the same results...any idea is appreciated. Thanks once again.
Sorry, I forgot to replace the variable in the code and to add a counter:
#!/bin/sh
ts="TimeStamp: [02/Jun/2008:10:34:36 -0600]"
/usr/bin/gawk -v num="$number" -v timestamp="$ts" '
{if(!($1 in a)){a[++i]=$1;a[$1]}}
{if(!($2 in b)){b[++j]=$2;b[$2]}}
END{
for(k=1;k<=j;k++){
c++
print "uid=" b[k]
for(l=1;l<=i;l++){
print a[l] ": " c
}
print timestamp
}
}' file