fetch data between two timestamp using script

Shell Programming and Scripting
1

Hi Guys,

I have the data in below format.

25 Dec 2011 03:00:01 : aaaaaaaaaaaaaaa
25 Dec 2011 04:23:23 : bbbbbbbbbbbbbbb
25 Dec 2011 16:12:45 : ccccccccccccccc
26 Dec 2011 04:45:34 : ddddddddddddddd
26 Dec 2011 17:01:22 : eeeeeeeeeeeeeee
27 Dec 2011 12:33:45 : ffffffffffffffffffffffff
28 Dec 2011 14:23:23 : ggggggggggggggg

Now i want to fetch the complete data between the two dates
25 Dec 2011 03:00:01 and 27 Dec 2011 12:33:45.

Which command to use.
I tried sed but that is very case sensitive and not giving the output properly.

Please help.
Thanks in advance.

2

Welcome to the forum.

Try,

awk '/25 Dec 2011 03:00:01/,/27 Dec 2011 12:33:45/ {print $0}' file

Even shorter,

awk '/25 Dec 2011 03:00:01/,/27 Dec 2011 12:33:45/' file
3

Hey Anchal,
Thanks for the command, but there seems to be some problem.

If i try a

awk '/25 Dec 2011 02:59:59/,/27 Dec 2011 12:33:45/' file

instead of

awk '/25 Dec 2011 03:00:01/,/27 Dec 2011 12:33:45/' file, it doesn't work.

I want to fetch data between two particular date with timestamp.

4

[highlight=perl]#! /usr/bin/perl -w
use strict;

my ($start_time, $end_time, $dt);

(@ARGV != 2) && die "Invalid parameters. Exiting";

$start_time = parse_date ($ARGV[0]);
$end_time = parse_date ($ARGV[1]);

open I, "< inputfile.txt";
for (<I>) {
$dt = parse_date (substr ($_, 0, 20));
($dt >= $start_time && $dt <= $end_time) && print;
}
close I;

sub parse_date {
my $t = shift;
my @d = split /\s+/, $t;
my %mnths = ( "Jan" => "01", "Feb" => "02", "Mar" => "03", "Apr" => "04", "May" => "05", "Jun" => "06",
"Jul" => "07", "Aug" => "08", "Sep" => "09", "Oct" => "10", "Nov" => "11", "Dec" => "12" );
for (keys %mnths) { if ($d[1] eq $) { $d[1] = $mnths{$}; last } }
$d[3] =~ s/://g;
return "$d[2]$d[1]$d[0]$d[3]";
}[/highlight]

Run this script with two arguments.
scriptname "<start_time>" "<end_time>"
For e.g. this will be the output for input provided in post #1

$ ./test.pl "25 Dec 2011 03:30:59" "27 Dec 2011 12:45:00"
25 Dec 2011 04:23:23 : bbbbbbbbbbbbbbb
25 Dec 2011 16:12:45 : ccccccccccccccc
26 Dec 2011 04:45:34 : ddddddddddddddd
26 Dec 2011 17:01:22 : eeeeeeeeeeeeeee
27 Dec 2011 12:33:45 : ffffffffffffffffffffffff
5

Thanks balajesuri,
but is there any other way except using perl..

Using any command in unix whether awk/sed,etc????

---------- Post updated at 11:47 PM ---------- Previous update was at 10:20 PM ----------

Guys... i need help on this..!!!

6

[highlight=bash]#! /bin/bash
[ $# -ne 2 ] && echo "Invalid Parameters. Exiting." && exit

st_date=`date -d "$1" +%Y%m%d%H%M%S`
nd_date=`date -d "$2" +%Y%m%d%H%M%S`

while read x
do
dt=$(date -d "`echo $x | cut -c1-20`" +%Y%m%d%H%M%S)
[ $dt -ge $st_date -a $dt -le $nd_date ] && echo $x
done < inputfile.txt[/highlight]

$ ./test.sh "25 Dec 2011 03:30:30" "27 Dec 2011 12:45:00"
25 Dec 2011 04:23:23 : bbbbbbbbbbbbbbb
25 Dec 2011 16:12:45 : ccccccccccccccc
26 Dec 2011 04:45:34 : ddddddddddddddd
26 Dec 2011 17:01:22 : eeeeeeeeeeeeeee
27 Dec 2011 12:33:45 : ffffffffffffffffffffffff
7

What the Invalid Paramter existing???

Please explain the script if you can.
Thanks..

8

Line 2: When you run the script from command line, you've to provide exactly 2 parameters. Otherwise, it'll exit.
Line 4,5: The start and end timestamp that you enter as 1st and 2nd parameters while invoking the script are converted to YYYYMMDDHHMMSS format (a single 14 digit number).
Line 7: Input file is read line by line in a while loop
Line 9: The timestamp at the beginning of each line (first 20 characters) in input file is converted to YYYYMMDDHHMMSS format.
Line 10: Date obtained in line 9 is checked with start time stamp and end time stamp found in line 4 & 5. If >= start time and <= end time, then print the line.

9 
[ $# -ne 2 ] && echo "Invalid Parameters. Exiting." && exit

what does the above line mean???
and why the invalid parameters??

10

$# --> Refers to the number of parameters provided at command line alongside scriptname.
[ $# -ne 2 ] --> Number of parameters ($#) is not equal (-ne) to two (2)
&& --> Logical 'and' operator.
"Invalid Parameters. Exiting" --> This will be printed on screen if you don't provide exactly two parameters. Ok, my mistake.. I should've mentioned "Invalid number of parameters. Exiting."
&& --> Another logical 'and' operator
exit --> Script will exit the moment this keyword in encountered.

So, summing it up, if [ $# -ne 2 ] is true, then do echo "Invalid Parameters. Exiting." . And if that is true then do exit

This whole line is just a check to ensure that user is entering exactly two parameters alongside script name while invoking it at command line. This line is not necessary, if the user knows what exactly to enter and how much to enter.

1 Like
11 
JAI@JAI-PC ~/hello/Recycle/check/pearl
$ ./test1.sh "26 Dec 2011 11:39:43,070" "28 Dec 2011 09:15:17"
./test1.sh: line 1: $'\r': command not found
./test1.sh: line 4: $'\r': command not found
./test1.sh: line 7: $'\r': command not found
./test1.sh: line 12: syntax error near unexpected token `done'
./test1.sh: line 12: `done < New_File.txt'

I am getting the below errors while running this script..

12

In the first parameter why did you give milliseconds? -- 26 Dec 2011 11:39:43,070
That was not your original requirement right? Anyway, that shouldn't matter.

Post the output of the following command:

cat -A test1.sh
1 Like
13

I recently tried the below command.

sed -n '/28 Dec 2011 10:17:53/,/28 Dec 2011 18:43:07/p' filename

But it shows some problem..

---------- Post updated at 12:45 AM ---------- Previous update was at 12:42 AM ----------

THats correct.

But still it is throwing me an error.

$ ./test1.sh "26 Dec 2011 11:39:43" "28 Dec 2011 09:15:17"
./test1.sh: line 1: $'\r': command not found
./test1.sh: line 4: $'\r': command not found
./test1.sh: line 7: $'\r': command not found
./test1.sh: line 12: syntax error near unexpected token `done'
./test1.sh: line 12: `done < New_BPPError.log'

I need to fetch data between two dates with timestamp.
It may occur that the starting date is not present in the file with the same timestamp.
The timestamp can vary. I need to apply the between condition.

Please help.!!

---------- Post updated at 12:54 AM ---------- Previous update was at 12:45 AM ----------

Hey..

Its working fine..
Thanks a ton..
You have been great help Mr.balajesuri..! :slight_smile:

---------- Post updated at 01:02 AM ---------- Previous update was at 12:54 AM ----------

Can you explain me the below part of the script.

    dt=$(date -d "`echo $x | cut -c1-20`" +%Y%m%d%H%M%S)
    [ $dt -ge $st_date -a $dt -le $nd_date ] && echo $x
14 
dt=$(date -d "`echo $x | cut -c1-20`" +%Y%m%d%H%M%S)

--> Break this into pieces. echo $x | cut -c1-20 fetches the first 20 characters of each line of log file. So, this refers to the time stamp.
--> -d switch of date command shows date as specified by the following string. Think of this as an intelligent switch of date command. For e.g., if you say date -d "1 day ago" , it would display the previous days' date.
--> +%Y%m%d%H%M%S is the format in which date should be printed. Check man pages of date command.
So date -d "25 Dec 2011 10:10:10" +%Y%m%d%H%M%S would show 20111225101010. Check it out.

[ $dt -ge $st_date -a $dt -le $nd_date ] && echo $x

$dt -ge $st_date --> if value in variable dt ($dt) is greater than equal to (-ge) value in st_date ($st_date)
-a --> logical 'and' operator
$dt -le $nd_date --> if value in variable dt ($dt) is less than equal to (-le) value in nd_date ($nd_date)
&& --> another logical 'and' operator
echo $x --> print line ($x) on screen
So summing it up, if [ $dt -ge $st_date -a $dt -le $nd_date ] is true then do echo $x