To remove date and duplicate rows from a log file using unix commands

Shell Programming and Scripting
1

Hi,

I have a log file having size of 48mb.
For such a large log file. I want to get the message in a particular format which includes only unique error and exception messages.

The following things to be done :

1) To remove all the date and time from the log file
2) To remove all the duplicate rows from the same.
3) To remove all the lines that shows the message like

"at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)"

For e.g if my log file is like :

29 Jul 2009 04:36:53,915 [ajp-0.0.0.0-8310-415] ERROR impl.ProfileSearchManagerImpl - /JProduction/JHomeJuly17/search/user/Ixo0PRS9/_2ox.cfs (No such file or directory)
java.io.FileNotFoundException: /JProduction/JHomeJuly17/search/user/Ixo0PRS9/_2ox.cfs (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:506)
29 Jul 2009 04:36:53,956 [ajp-0.0.0.0-8310-140] ERROR com.Jsoftware - com.Jsoftware.community.ForumThreadNotFoundException: Thread 5174 could not be loaded from the database.
29 Jul 2009 04:36:58,335 [ajp-0.0.0.0-8310-239] ERROR interceptor.ExceptionMappingInterceptor - 
java.lang.NullPointerException
at com.Jsoftware.community.action.RSSPolls.execute(RSSPolls.java:95)
at sun.reflect.GeneratedMethodAccessor3418.invoke(Unknown Source)29 Jul 2009 04:36:53,956 [ajp-0.0.0.0-8310-140] ERROR com.Jsoftware - com.Jsoftware.community.ForumThreadNotFoundException: Thread 5174 could not be loaded from the database.
29 Jul 2009 04:36:58,335 [ajp-0.0.0.0-8310-239] ERROR interceptor.ExceptionMappingInterceptor - 
java.lang.NullPointerException
at com.Jsoftware.community.action.RSSPolls.execute(RSSPolls.java:95)
at sun.reflect.GeneratedMethodAccessor3418.invoke(Unknown Source)

I want the ouput message as :

ERROR impl.ProfileSearchManagerImpl - /JProduction/JHomeJuly17/search/user/Ixo0PRS9/_2ox.cfs (No such file or directory)
java.io.FileNotFoundException: /JProduction/JHomeJuly17/search/user/Ixo0PRS9/_2ox.cfs (No such file or directory)
ERROR interceptor.ExceptionMappingInterceptor - 
java.lang.NullPointerException

Kindly let me know how to do the same.

Regards,
pank

Next time use CODE-tags when posting code, data or logs to enhance readability and to preserve formatting like indention etc., ty.

2

Something like that :

awk '
/^\"*at[[:space:]]+/ {next}
NF>5 && /^[0-9]/ { sub(/[^]]*][[:space:]]*/,""); print }
' log.txt | sort -u

Output:

ERROR com.Jsoftware - com.Jsoftware.community.ForumThreadNotFoundException: Thread 5174 could not be loaded from the database.
ERROR impl.ProfileSearchManagerImpl - /JProduction/JHomeJuly17/search/user/Ixo0PRS9/_2ox.cfs (No such file or directory)
ERROR interceptor.ExceptionMappingInterceptor -

Jean-Pierre.