Converting date string to different formats

Shell Programming and Scripting
1

Sucks to be a noob :o

See my last post for solution

I have 3 different log formats and the filenames contain a date. I am trying to write a script to grep these files between two date ranges.

I am now stuck at the date conversion bit. I let the user entered a date string in the format "18.05.2009" which I need to convert to these 3 different formats:

example filenames are.

WW Logs format  "access1004211716_" //the last 4 digits are the time
BC log format "HTTP_20100421_"
NC Log format "http_log.100.2010.04.28"

I know I can use awk but having read an hour up on material I still have no clue how to archive what I want in my bash script.

The next step will be even harder as I need to search between 2 date ranges :frowning:

any help is apperciated.

2

Hello,
maybe if the start date and end date are near you can grep for each date in the interval and for each date format.
If the difference between dates is a year...this "solution" is not very nice.

Sorry, I read again your message and is it possible you want to create different date formats from one enter by a user?

So, you can do this:
FECHA=date entered by the user y dd.mm.yyyy format
$day=$(echo $FECHA | cut -d . -f 1)
$mont=$(echo $FECHA | cut -d . -f 2)
$year=$(echo $FECHA | cut -d . -f 3)

And now, you only have to echo in the desired order..

---------- Post updated at 06:09 PM ---------- Previous update was at 05:38 PM ----------

fecha_desde=$1
fecha_hasta=$2

dia1=$(echo $fecha_desde | cut -d . -f 1)
mes1=$(echo $fecha_desde | cut -d . -f 2)
ano1=$(echo $fecha_desde | cut -d . -f 3)
dia2=$(echo $fecha_hasta | cut -d . -f 1)
mes2=$(echo $fecha_hasta | cut -d . -f 2)
ano2=$(echo $fecha_hasta | cut -d . -f 3)

fecha1=$ano1$mes1$dia1
fecha2=$ano2$mes2$dia2

# format accessDATE...
for fich in $(ls -1 access*)
do
   valor=20${fich:6:6}
   if [ $valor -ge $fecha1 ] && [ $valor -le $fecha2 ]
   then
       echo $fich " Ok!"
   fi
done

# format HTTP_DATE...

 "similar"
3

ahhh I see what you did :cool:

will try the code this weekend.

The date range could be for several month... but lets see if I can figure that one out myself with the help of google :cool:

4

Something like this. Format date yyyymmdd, then you can use numeric comparing.
You need fix the awk substr lines. Look values from real loglines.

#!/bin/ksh or bash or ...
datestr()
{
input=$1
oifs="$IFS"
IFS="."
values=($input)
IFS="$oifs"
d=${values[0]}
m=${values[1]}
y=${values[2]}
datestr="$y$m$d"
}

date1=$(datestr 12.10.2009)
date2=$(datestr 12.10.2010)

awk -v d1=$date1 -v d2=$date2 '
/access[0-9][0-9]/ {
    datestr=substr($0:7:6}
    datelong=sprintf("20%s",datestr)
    if (datelong < d1  || datelong > d2) { next }
    print $0
    next
    }
/HTTP_[0-9][0-9]/ {
    datelong=substr($0:6:6}
    if (datelong < d1  || datelong > d2) { next }
    print $0
    next
    }
/http_log.*[0-9][0-9]/ {
    y=substr($0:14:4}
    m=substr($0:19:2}
    d=substr($0:22:2}
    datelong=sprintf("%s%s%s",y,m,d)
    if (datelong < d1  || datelong > d2) { next }
    print $0
    next
    }
' logfile
5

mhh I can not get it to work. The last example I do not even understand?

For example lets just do files which are called access, e.g.

access1004202232.merged-00.21.9b.94.43.a1.log.gz

and I want to find all all entries between date1 and date2 I can not get it to work.

#!/bin/bash

date1="100501"
date2="100502"


for i in `find /APAC/ -name *.gz* -name access*`; do awk -v d1=$date1 -v d2=$date2 ' ;done

moreover if I just try this in bash I get weird results?

 find /APAC/ -name *.gz* -name access* | awk -v /100501/,/100502/

I also get files that are 100503 and 04 etc etc.

6

Maybe this ?

$ 
$ # show all files whose names start with "access"
$ ls -1 access*
access1004202232.merged-00.21.9b.94.43.a1.log.gz
access1005010932.merged-00.21.9b.94.43.a1.log.gz
access1005012357.merged-00.21.9b.94.43.a1.log.gz
access1005021349.merged-00.21.9b.94.43.a1.log.gz
access1005022359.merged-00.21.9b.94.43.a1.log.gz
access1005030058.merged-00.21.9b.94.43.a1.log.gz
$ 
$ # pick up files created between "100501" and "100502"
$ perl -le 'print foreach (grep /^access10050[12]/, (glob "access*"))'
access1005010932.merged-00.21.9b.94.43.a1.log.gz
access1005012357.merged-00.21.9b.94.43.a1.log.gz
access1005021349.merged-00.21.9b.94.43.a1.log.gz
access1005022359.merged-00.21.9b.94.43.a1.log.gz
$ 
$ # or using just the shell
$ ls -1 access10050[12]*
access1005010932.merged-00.21.9b.94.43.a1.log.gz
access1005012357.merged-00.21.9b.94.43.a1.log.gz
access1005021349.merged-00.21.9b.94.43.a1.log.gz
access1005022359.merged-00.21.9b.94.43.a1.log.gz
$ 
$

tyler_durden

7

So after spending hours being stuck at this I found out that the best way to do this is to convert the filename date into unix time and then simple do a bigger then smaller then if statement with the user entered dates.
Converting to unix time was another issue as I am on solaris not GNU.

Its a blast now analyzing logs between two dates out of 3000 or more files in 1 year

Here is my solution:

User entered dates:

choosedates()
	{
		echo "----------------------------------------------------------------"
		echo "Which date range would you like to search for in region $REGION"? 
		echo "Choose your start (from) date with the below format:"
		echo "dd.mm.yyyy e.g. 08.05.2009"
		until (test "$CHECK" = "Y" || test "$CHECK" = "y")
			do 
				read STARTDATE
				echo "You have choose $STARTDATE, is this ok? [y/n]"
				read CHECK
		done
		echo "Choose your end date with the below format:"
		echo "dd.mm.yyyy e.g. 02.11.2010"
		until (test "$CHECK2" = "Y" || test "$CHECK2" = "y")
			do 
				read ENDDATE
				echo "You have choose $ENDDATE, is this ok? [y/n]"
				read CHECK2
		done
		
	CHECK=""
	CHECK2=""
	#Converting the user dates and converting them to unix epoche time	
	dayS=$(echo $STARTDATE | cut -d . -f 1)
	monS=$(echo $STARTDATE | cut -d . -f 2)
	yeaS=$(echo $STARTDATE | cut -d . -f 3)
	dayE=$(echo $ENDDATE | cut -d . -f 1)
	monE=$(echo $ENDDATE | cut -d . -f 2)
	yeaE=$(echo $ENDDATE | cut -d . -f 3)
		
	STARTDATE=`/opt/bin/perl /home/user/scripts/perldateconverter.pl $yeaS$monS$dayS`
	ENDDATE=`/opt/bin/perl /home/user/scripts/perldateconverter.pl $yeaE$monE$dayE`
	}

Reading log files and converting log file names into a timestamp then converting to unix epoch

for logfile in `ls -R1 $REGION/*/*/access*.gz` ;
			do 
			#Convert date in filename to unix timestamp
			logfiledate=$(echo $logfile | sed -e 's/.*access\(......\).*/\1/')
			logfiledate=`/opt/bin/perl /home/user/scripts/perldateconverter.pl 20$logfiledate`  ##this particular logfile has dates as 09 or 01 not 2009 or 2010
			#Analyse log files
			if [ $logfiledate -ge $STARTDATE ] && [ $logfiledate  -le  $ENDDATE ];then
				echo "Analysing file: $logfile"
				if [ -z "$GREPCOMMAND" ];then
					/usr/bin/gzcat $logfile >> ~/$FILETOWRITEWW
					else
					/usr/bin/gzcat $logfile |  (eval $GREPCOMMAND) >> ~/$FILETOWRITEWW
				fi
			fi
		done

perl date converter

use strict;
use warnings;
use DateTime::Format::Strptime;

my $str = shift;
my $parser =
  DateTime::Format::Strptime->new( pattern => '%Y%m%d' );
my $dt = $parser->parse_datetime( $str );
print $dt->epoch;