Im sure this is an easy question, but Ive tried and tried to get this to print all on one line and cant figure out why its not, so maybe someone can help
awk '/AP/{sub(/:80/, "", $4);printf $4"\t"} /User-Agent/{sub(/^[^:][^:]*:/,"");print};sub(/\.80/,"", $4);/Host/{sub(/^[^:][^:]*:/,""); print}'
What this prints is "AP" then "User-Agent" on one line(good) but then prints "HOST" on a separate line? How can I get all three to print on a single line?
Thanks!
Without knowing your input:
awk '/AP/{sub(/:80/, "", $4);printf $4"\t"} /User-Agent/{sub(/^[^:][^:]*:/,"");printf $0}/Host/{sub(/^[^:][^:]*:/,"");print}'
That did it!!! I dont know why I never tried that! Ugh banging my head!!
Each call to print produces a <newline> character in the output. You could try something like:
awk '/AP/{sub(/:80/, "", $4);printf "%s\t", $4} /User-Agent/{sub(/^[^:][^:]*:/,"");printf "%s\t", $0}/Host/{sub(/^[^:][^:]*:/,"");print}'
Note that I changed the format string for your printf call. If $4 happens to contain any % or \ characters, you could end up with output significantly different from what you might expect; using the format %s avoids that possibility.
Thanks! So I tried the last one and man the output format is kind of jacked up 
8.254.53.254 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0). mi.adinterax.com.
98.139.200.238 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5 68.142.250.161 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0). ads.yimg.com.
Sometimes it appends other lines and sometimes it throws the the first AP in at the end or middle?
This is what it looks like if I go back to my original awk commands.
63.241.108.124 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0).
bs.serving-sys.com.
8.254.53.254 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0).
mi.adinterax.com.
I will play with this and see what I can figure out...it seems like its combining or adding the User-Agent field. strange output. Thanks again for your help!
---------- Post updated at 03:50 PM ---------- Previous update was at 01:25 AM ----------
Is there anyway to force this to always print the "/AP/" field first?
awk '/AP/{sub(/:80/, "", $4);printf "%s\t", $4} /User-Agent/{sub(/^[^:][^:]*:/,"");printf "%s\t", $0}/Host/{sub(/^[^:][^:]*:/,"");sub(/\.80/,"", $4);print}'
Right now it does maybe 50% sometimes it ends up after the User-Agent. There must be something causing this but I cannot figure it out?
Thanks!!
What do you mean you cannot figure it out? It is obvious! With the code suggested for handling your problem, it is simple: when the string AP appears first in your input file, it will appear first in your output file.
Is there something in your input file that can be used to indicate that it is the first line of a set or the last line of a set?
Will it always be true that there is exactly one line containing the string AP , one line containing the string User-Agent , and one line containing the string Host in a group of lines to be treated as a set?
You know what your input looks like, you have left the rest of us guessing.
Found that User-Agent and Host are switching spots when using different browsers!?!!?!?!?!?!! See below and let me know if there is a solution to these variations.
When IE does the request:
T 12.237.222.221:57578 -> 70.37.131.11:80 [AP]
GET /c.gif?clid=3A8ACC9C01566FAB3DA8C8E105566FE0%26TUID%3D1&rid=CDA92F497C2347BEBCAD28088CA12CB7&cts=1362349841890&evt=unload HTTP/1.1.
Accept: image/png, image/svg+xml, image/*;q=0.8, */*;q=0.5.
Referer: http://photos.msn.com/browse/places.
Accept-Language: en-US.
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0).
Accept-Encoding: gzip, deflate.
Host: udc.msn.com.
DNT: 1.
Connection: Keep-Alive.
Cookie:
Chrome and Safari/Iphone/Android
T 12.237.222.221:57991 -> 4.59.125.171:80 [AP]
GET /unix-dummies-questions-answers/217117-basic-awk-help.html HTTP/1.1.
Host: www.unix.com.
Connection: keep-alive.
Cache-Control: max-age=0.
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.97 Safari/537.22.
Referer: http://www.unix.com/unix-dummies-questions-answers/217117-basic-awk-help.html
Accept-Encoding: gzip,deflate,sdch.
Accept-Language: en-GB,en-US;q=0.8,en;q=0.6.
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3.
Cookie:
Is there any way to be strict about always printing HOST first even if its after User-Agent? or vice-versa
This seems to do what you want:
awk -F ': *' '
function p(){
if(host=="")return
printf("%s\t%s\t%s\n",node,host,agent)
host=agent=""
}
/^T /{p();node=$2;sub(/.* /,"",node)}
$1=="Host"{host=$2}
$1=="User-Agent"{agent=$2}
END{p()}'
You can switch the order of the operands to the printf call in function p if you want the output fields in a different order.
Given the two data samples from your last message, the output produced is:
70.37.131.11 udc.msn.com. Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0).
4.59.125.171 www.unix.com. Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.97 Safari/537.22.
Well the problem with this, and sorry I left it out, Im piping live data into AWK then printing to a file. I appreciate the attempt, but unfortunately I need to pipe it in and pipe out live.
I don't understand the problem. Don't:
program_to_produce_input | awk -F ': *' '
function p(){
if(host=="")return
printf("%s\t%s\t%s\n",node,host,agent)
host=agent=""
}
/^T /{p();node=$2;sub(/.* /,"",node)}
$1=="Host"{host=$2}
$1=="User-Agent"{agent=$2}
END{p()}' > output_file
and
program_to_produce_input | awk -F ': *' '
function p(){
if(host=="")return
printf("%s\t%s\t%s\n",node,host,agent)
host=agent=""
}
/^T /{p();node=$2;sub(/.* /,"",node)}
$1=="Host"{host=$2}
$1=="User-Agent"{agent=$2}
END{p()}' | program_to_process_output
do what you want?
That did work!!!! A million thanks!! I was assuming I would have to run that in a shell script against a file. I didnt think I could run it like that! Thank you!