Argument list too long for date command

Shell Programming and Scripting
1

Dear Friends,

The following script processes a 14508 lines log file.

#!/bin/sh
while read line
do
     d=`sed 's/[0-9|.]* - [-|a-zA-Z]* \[\([0-9]*\/[a-zA-Z]*\/[0-9:]* [0-9+]*\)\] .*/\1/' | tr '/' ' ' | sed 's/\([.]*\):\([.]*\)/\1 \2/'`
     y=`date -d "${d}" "+%Y%m%d%H%M%S"`
     echo "${y}"
done

While running the above script, I am getting argument list long error. Can anyone help me to come out this error. I don't this issue if I split two processes(sed and date) in to different files.

Thanks,
Tamil Pamaran

2

Please provide a small sample of your log file with one or more date strings in it.

3

fpmurphy,

Thanks for you response, part of the log file is given below,

213.64.56.208 - - [01/Jan/2003:08:18:42 +0100] "GET /scripts/..%%35c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:08:18:44 +0100] "GET /scripts/..%25%35%63../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:08:18:56 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:09:46:06 +0100] "GET /scripts/root.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:34 +0100] "GET /scripts/root.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:36 +0100] "GET /MSADC/root.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:39 +0100] "GET /c/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:42 +0100] "GET /d/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:47 +0100] "GET /scripts/..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:49 +0100] "GET /_vti_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:52 +0100] "GET /_mem_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:57 +0100] "GET /msadc/..%255c../..%255c../..%255c/..%c1%1c../..%c1%1c../..%c1%1c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.67.145.223 - - [01/Jan/2003:11:00:06 +0100] "HEAD / HTTP/1.0" 401 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:15 +0100] "GET /scripts/root.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:15 +0100] "GET /MSADC/root.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET /c/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET /d/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET /scripts/..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:17 +0100] "GET /_vti_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:17 +0100] "GET /_mem_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:18 +0100] "GET /msadc/..%255c../..%255c../..%255c/..%c1%1c../..%c1%1c../..%c1%1c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:18 +0100] "GET /scripts/..%c1%1c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:18 +0100] "GET /scripts/..%c0%2f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""

I have modified the code in the following manner and its works fine now,

while read data
do
     datestring=`echo "${data}" | sed 's/[0-9|.]* - [-|a-zA-Z]* \[\([0-9]*\/[a-zA-Z]*\/[0-9:]* [0-9+]*\)\] .*/\1/' | tr '/' ' ' | sed 's/\([.]*\):\([.]*\)/\1 \2/'`
     date -d "${datestring}" "+%Y%m%d%H%M%S" > /dev/null
     done

Appreciate if you can explain what was the issue.:slight_smile:

-Tamil Pamaran

4

Your second version processes each line of input one by one, but your first does not !

Because of this line ..

d=`sed 's/[0-9|.]* - [-|a-zA-Z]* \[\([0-9]*\/[a-zA-Z]*\/[0-9:]* [0-9+]*\)\] .*/\1/' | tr '/' ' ' | sed 's/\([.]*\):\([.]*\)/\1 \2/'`

Where is the input for the 1st sed command executed in the above line ??

There is no so it tries reading from STDIN.. Guess you have redirected the input so the remaining the sed reads... when all input are read and send to date ... blah blah blah !

1 Like
5

thegeek,

Your guess is right. Thanks for your explaination.

Cheers,
Tamil Pamaran