Remove duplicate text

Shell Programming and Scripting
1

Hello,

I have a log file which is generated by a script which looks like this:

userid: 7
starttime: Sat May 24 23:24:13 CEST 2008
endtime: Sat May 24 23:26:57 CEST 2008
total time spent: 2.73072 minutes / 163.843 seconds
date: Sat Jun 7 16:09:03 CEST 2008

userid: 8
starttime: Sun May 25 00:14:30 CEST 2008
endtime: Sun May 25 00:14:32 CEST 2008
total time spent: 0.0304667 minutes / 1.828 seconds
date: Sat Jun 7 16:10:02 CEST 2008

userid: 9
starttime: Sun May 25 00:14:30 CEST 2008
endtime: Sun May 25 00:14:32 CEST 2008
total time spent: 0.0304667 minutes / 1.828 seconds
date: Sat Jun 7 16:11:01 CEST 2008

Everytime when I run the script, it will increase the userid by one and adds information(start time, end time etc).
Anyone knows if there is an efficient way to remove the whole last block of text when its starttime or endtime or both is duplicate of the previous block?

Thanks in advance.

2

Try this:

awk 'BEGIN{RS="";FS="\n"}
t!=$2$3{t=$2$3;print $0,"\n"}
' "file"

Regards

3

Doesn't work :frowning: , looks like there's no change when I execute the script.
Can you also explain what the code does? I'm new at using awk.

4

Does your file has a different format as you posted? Is every block seperated by an empty line?
I've copy and paste your example and it works fine for me.

This changes the default record and field seperator so you can treat every line as a field and every block as a record. The 2nd field is the start time end the 3th the end time:

awk 'BEGIN{RS="";FS="\n"}

Variable t is a reminder, print only records if the start and end time is different from the previous block:

t!=$2$3{t=$2$3;print $0,"\n"}

Regards

5

The original file format looks like this:

UserID: 7
Start Time: Sat May 24 23:24:13 CEST 2008
End Time: Sat May 24 23:26:57 CEST 2008
Total time spent: 2.73072 minutes / 163.843 seconds
Date: Sat Jun 7 16:09:03 CEST 2008

UserID: 8
Start Time: Sun May 25 00:14:30 CEST 2008
End Time: Sun May 25 00:14:32 CEST 2008
Total time spent: 0.0304667 minutes / 1.828 seconds
Date: Sat Jun 7 16:10:02 CEST 2008

UserID: 9
Start Time: Sun May 25 00:14:30 CEST 2008
End Time: Sun May 25 00:14:32 CEST 2008
Total time spent: 0.0304667 minutes / 1.828 seconds
Date: Sat Jun 7 16:11:01 CEST 2008

I forgot to mention that there's an extra line at the beginning of the file followed by an empty line. Which says something about my log file. Thought it would be possible to get awk to read from the end of the file instead of from the top.

[edit2]

My fault, it's also in html format, not plain text in a log file, so when awk reads the file, it reads multiple lines in the .html file instead of what you see in the output of the log in a browser. There are a lot of lines in the file, should I just paste them here or can I send you the file?

6

You should strip the html tags before processing the log file. Different solutions here:

  • PHP function strip_tags()
  • lynx text browser with option -dump
  • html2text utility.
  • using sed to get rid of the tags

Can you post an extract of your html file. Or attach it to your reply.

7

I only stripped the last two html tags with a bit of code Franklin52 wrote for me in another post. So my script is able to add new log information and then put back the last two html tags after the log information over and over again. I'd rather not strip the html tags before processing the log, cause it's gonna be a hassle to put them back in the exact place where they were.

I've put the log up on pastebin:
http://pastebin.com/m2ba76b50

8

Just found a more efficient way to make the script do what it needs to do.

Thanks.