Search for a specific data in a file based on a date range

svajhala · September 27, 2011, 10:45am

Hi,

Currently I am working on a script to automate the process of converting the log file from binary into text format. To achieve this, partly I am depending on my application�s utility for this conversion and the rest I am relying on shell commands to search for directory, locate the file and search for specific data in the converted file and email that to the user.

Here is the script that will do the work.

#!/bin/ksh
 
. /home/.env
 
# Go to the location where log file is stored.
 
cd $lOg_Path
 
cd `find . -type d |sort | tail -1`
 
LogName=`ls -ltr | grep -v 'resend' | grep -E '\.dat$'|tail -1 | awk  '{ print $9 }'`
 
NewLogName=`echo $LogName | cut -f1 -d .`.txt
 
infacmd.sh isp ConvertLogFile -in $LogName -lo $NewLogName
 
 
tail -4 ${Log_File} > tmp.txt
 
#echo "=============================================================================================================================">>tmp.txt
 
#echo "Please find attached Log below">>tmp.txt
 
EndTime=`date +%H-%M-%S`
 
#(  cat tmp.txt; uuencode $NewLogName $NewLogName) | mailx -s "Subject" `echo ${Alertlist_All}`

But the problem is that the converted output file wrapped for every 80 characters and rest of the information is printed on the next line. Due to this, I am having difficulty in searching for a specific data in the log file based on a date range along with timestamp and print the output.

Here is the file output

2011-09-26 INF UM_10 UserMana Starting synchronization for security domain
14:37:31       080   gementSe [12345].
                     rvice
2011-09-26 INF LDAP_ UserMana LDAP user [xxxx] in security domain
14:37:31       10002 gementSe [12345] queried with LDAP DN
                     rvice    [CN=sreich,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 INF LDAP_ UserMana LDAP user [yyyyy] in security domain
14:37:31       10002 gementSe [12345] queried with LDAP DN
                     rvice    [CN=yyyyy,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 INF LDAP_ UserMana LDAP user [zzzzz] in security domain [12345]
14:37:31       10002 gementSe queried with LDAP DN
                     rvice    [CN=zzzzz,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10003 gementSe security domain [12345] queried from LDAP.
                     rvice
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                     rvice    contains user [yyyyy] in security domain
                              [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                     rvice    contains user [zzzzz] in security domain
                              [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                     rvice    contains user [sreich] in security domain
                              [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_ADM_LDAP] in
14:37:31       10003 gementSe security domain [12345] queried from LDAP.
                     rvice
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_ADM_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                     rvice    contains user [yyyyy] in security domain
                              [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_ADM_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                     rvice    contains user [zzzzz] in security domain
                              [12345].
2011-09-26 INF LDAP_ UserMana Queried [3] users based on the user search base
14:37:31       10000 gementSe [CN=Users,DC=DOMAIN,DC=COM] with filter
                     rvice    [(&(ObjectClass=user)(|(memberOf=CN=INFA_LDAP_ADM,\
          CN=Users,DC=DOMAIN,DC=COM)(m
                              emberOf=CN=INFA_LDAP,CN=U
                              sers,DC=DOMAIN,DC=COM)))] specified in
                              [12345] security domain definition.
2011-09-26 INF LDAP_ UserMana Queried [2] groups based on the group search base
14:37:31       10001 gementSe [CN=Users,DC=DOMAIN,DC=COM] with filter
                     rvice    [(&(ObjectClass=group)(|(CN=INFA_
                               LDAP_ADM)(CN=INFA_LDAP_DEV)))] 
          specified in [12345] security domain
                               definition.
2011-09-26 INF LDAP_ UserMana Security domain [12345] synchronization with
14:37:32       10007 gementSe LDAP server successful.

To overcome this problem, I am finding the row numbers of starting row and ending row which I am interested in and pass them back to the script and store them in variables. After that, I am printing the data to a output file based on the row numbers.

Here is the code I am thinking of putting into the code.

RowNum=`grep -n '$StartTime' 1_0.txt |head -1|cut -c 1`  
cat 1_0.txt|awk 'NR>$RowNum'

I feel this is a crude way of coding but I am not aware of any better technique to handle this problem.

Sri

Shell_Life · September 27, 2011, 11:14am

To unfold the lines using the date as the key, try this code:

#!/usr/bin/ksh
while read mDate mRest; do
  if [[ ${#mDate} -eq 10 ]]; then
    if [[ "${mOut}" != "" ]]; then
      echo ${mOut}
    fi
    mOut=''
  fi
  mOut=${mOut}' '${mDate}' '${mRest}
done < Inp_File
if [[ "${mOut}" != "" ]]; then
  echo ${mOut}
fi

svajhala · September 27, 2011, 11:52am

Hi Shell_Life,

Thanks for the code snippet. I tried your solution and I was able to unwrap the text but later I identified that the data is actually wrapped on a column by column basis but not like as mentioned before. In other words, the data in each column is wrapped after certain length and continuing in the next line.

Your script converts the data like this.

2011-09-26 INF UM_10 UserMana Starting synchronization for security domain 08:45:10 080 gementSe [12345]. rvice

Is there a way to unfold the data for each column.

This is what I am looking for.

2011-09-26 14:37:31 INF UM_10080 UserManagementService Starting synchronization for security domain [12345].

Thanks in advance.

durden_tyler · September 27, 2011, 12:39pm

Here's one way to do it with Perl:

$
$
$ cat data.txt
2011-09-26 INF UM_10 UserMana Starting synchronization for security domain
14:37:31       080   gementSe [12345].
                    rvice
2011-09-26 INF LDAP_ UserMana LDAP user [xxxx] in security domain
14:37:31       10002 gementSe [12345] queried with LDAP DN
                    rvice    [CN=sreich,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 INF LDAP_ UserMana LDAP user [yyyyy] in security domain
14:37:31       10002 gementSe [12345] queried with LDAP DN
                    rvice    [CN=yyyyy,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 INF LDAP_ UserMana LDAP user [zzzzz] in security domain [12345]
14:37:31       10002 gementSe queried with LDAP DN
                    rvice    [CN=zzzzz,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10003 gementSe security domain [12345] queried from LDAP.
                    rvice
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                    rvice    contains user [yyyyy] in security domain
                          [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                    rvice    contains user [zzzzz] in security domain
                          [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                    rvice    contains user [sreich] in security domain
                          [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_ADM_LDAP] in
14:37:31       10003 gementSe security domain [12345] queried from LDAP.
                    rvice
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_ADM_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                    rvice    contains user [yyyyy] in security domain
                          [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_ADM_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                    rvice    contains user [zzzzz] in security domain
                          [12345].
2011-09-26 INF LDAP_ UserMana Queried [3] users based on the user search base
14:37:31       10000 gementSe [CN=Users,DC=DOMAIN,DC=COM] with filter
                    rvice    [(&(ObjectClass=user)(|(memberOf=CN=INFA_LDAP_ADM,\
                          CN=Users,DC=DOMAIN,DC=COM)(m
                          emberOf=CN=INFA_LDAP,CN=U
                          sers,DC=DOMAIN,DC=COM)))] specified in
                          [12345] security domain definition.
2011-09-26 INF LDAP_ UserMana Queried [2] groups based on the group search base
14:37:31       10001 gementSe [CN=Users,DC=DOMAIN,DC=COM] with filter
                    rvice    [(&(ObjectClass=group)(|(CN=INFA_
                           LDAP_ADM)(CN=INFA_LDAP_DEV)))]
                          specified in [12345] security domain
                           definition.
2011-09-26 INF LDAP_ UserMana Security domain [12345] synchronization with
14:37:32       10007 gementSe LDAP server successful.
$
$
$ perl -ne 'BEGIN {$fmt = "%-19s %3s %-10s %-22s %s\n"}
        chomp;
        if (/^\d{4}-\d\d-\d\d/) {
          $items[0] =~ s/^(.{10})/$1 /;
          printf($fmt, @items);
          @items = unpack ("A10 x A3 x A5 x A8 x A*");
        } else {
          if (length $_ == 26) {@y = unpack ("A10 x A3 x A5 x A*")}
          else {@y = unpack ("A10 x A3 x A5 x A8 x A*")}
          for ($i=0; $i<=$#y; $i++) {$items[$i] .= $y[$i]}
        }
        END {$items[0] =~ s/^(.{10})/$1 /; printf($fmt, @items)}
       ' data.txt
 
2011-09-26 14:37:31 INF UM_10080   UserManagementService  Starting synchronization for security domain[12345].
2011-09-26 14:37:31 INF LDAP_10002 UserManagementService  LDAP user [xxxx] in security domain[12345] queried with LDAP DN[CN=sreich,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 14:37:31 INF LDAP_10002 UserManagementService  LDAP user [yyyyy] in security domain[12345] queried with LDAP DN[CN=yyyyy,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 14:37:31 INF LDAP_10002 UserManagementService  LDAP user [zzzzz] in security domain [12345]queried with LDAP DN[CN=zzzzz,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 14:37:31 INF LDAP_10003 UserManagementService  LDAP group [INFA_LDAP] insecurity domain [12345] queried from LDAP.
2011-09-26 14:37:31 INF LDAP_10004 UserManagementService  LDAP group [INFA_LDAP] insecurity domain [12345] queried from LDAPcontains user [yyyyy] in security domain[12345].
2011-09-26 14:37:31 INF LDAP_10004 UserManagementService  LDAP group [INFA_LDAP] insecurity domain [12345] queried from LDAPcontains user [zzzzz] in security domain[12345].
2011-09-26 14:37:31 INF LDAP_10004 UserManagementService  LDAP group [INFA_LDAP] insecurity domain [12345] queried from LDAPcontains user [sreich] in security domain[12345].
2011-09-26 14:37:31 INF LDAP_10003 UserManagementService  LDAP group [INFA_ADM_LDAP] insecurity domain [12345] queried from LDAP.
2011-09-26 14:37:31 INF LDAP_10004 UserManagementService  LDAP group [INFA_ADM_LDAP] insecurity domain [12345] queried from LDAPcontains user [yyyyy] in security domain[12345].
2011-09-26 14:37:31 INF LDAP_10004 UserManagementService  LDAP group [INFA_ADM_LDAP] insecurity domain [12345] queried from LDAPcontains user [zzzzz] in security domain[12345].
2011-09-26 14:37:31 INF LDAP_10000 UserManagementService  Queried [3] users based on the user search base[CN=Users,DC=DOMAIN,DC=COM] with filter[(&(ObjectClass=user)(|(memberOf=CN=INFA_LDAP_ADM,\CN=Users,DC=DOMAIN,DC=COM)(memberOf=CN=INFA_LDAP,CN=Users,DC=DOMAIN,DC=COM)))] specified in[12345] security domain definition.
2011-09-26 14:37:31 INF LDAP_10001 UserManagementService  Queried [2] groups based on the group search base[CN=Users,DC=DOMAIN,DC=COM] with filter[(&(ObjectClass=group)(|(CN=INFA_ LDAP_ADM)(CN=INFA_LDAP_DEV)))]specified in [12345] security domain definition.
2011-09-26 14:37:32 INF LDAP_10007 UserManagementSe       Security domain [12345] synchronization withLDAP server successful.
$
$
$

tyler_durden

svajhala · September 27, 2011, 1:00pm

Hi Tyler,

Thanks for your wonderful solution. It looks great. However, I do not want to do it perl since most of the script is already written in kourne shell except this data formatting piece.

Would you help me integrating this piece using shell?

Thanks

durden_tyler · September 27, 2011, 1:17pm

Nope, sorry.

tyler_durden