# Sed to parse log file

**URL:** <https://community.unix.com/t/sed-to-parse-log-file/309413>\
**Category:** Shell Programming and Scripting\
**Created:** [May 1, 2012, 11:19am UTC](https://community.unix.com/t/sed-to-parse-log-file/309413 "2012-05-01T11:19:11Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![chipperuga](https://community.unix.com/letter_avatar/chipperuga/32/5_5575768a8748004e209b776fc1b2916d.png) [@chipperuga](https://community.unix.com/u/chipperuga)\
**Post date:** [May 1, 2012, 11:19am UTC](https://community.unix.com/t/sed-to-parse-log-file/309413/1 "2012-05-01T11:19:11Z")

</div>

Hi all, thanks for reading the post.

I'm trying to parse hundreds of log files in a directory. One log file looks similar to below:

```nohighlight
Investigator : Jim_Foo
Custodian : Jim_Foo-HDD1-FOO-1234
Export Path : N:\FOO-1234\Foo_Foo
Compute MD5 : No
File List Only: No
Extensions Selected:
     DOC
     DOCX
     EML
     MSG
     OST
     PDF
     PPT
     PPTX
     PST
     XLS
     XLSX
     XLSM
     ZIP
     REP

Matched File : foo_file1.pdf
Match Type : File Extension Match
File Number : 0
File Size : 640
MD5 Hash : 'Compute MD5 Hash' not selected
MAC Times : M=09/27/10 02:05:26PM A=02/01/12 09:59:49AM C=09/27/10 02:05:26PM  
Original Path: foopath\foopath\foopath\foobar\foo_file.pdf

Matched File : foo_file2.pdf
Match Type : File Extension Match
File Number : 0
File Size : 123
MD5 Hash : 'Compute MD5 Hash' not selected
MAC Times : M=09/27/10 02:05:26PM A=02/01/12 09:59:49AM C=09/27/10 02:05:26PM  
Original Path: foopath\foopath\foopath\foobar\foo_file.pdf

```

I would like to search for strings with sed and put the results into a csv. I can search for the right patterns, but I'm having trouble printing it correctly.

```nohighlight
$ sed -e '/Custodian/b' -e '/Matched File/b' -e '/File Size/b' -e '/Original Path/b' -e d *

```

Custodian : Jim\_Foo-HDD1-FOO-1234  
Export Path : N:\FOO-1234\Foo\_Foo  
Matched File : foo\_file1.pdf  
File Size : 640  
Original Path: foopath\foopath\foopath\foobar\foo\_file1.pdf  
Matched File : foo\_file2.pdf  
File Size : 123  
Original Path: foopath\foopath\foopath\foobar\foo\_file.pdf[/CODE]

CSV needs to look something like this:

```nohighlight
Custodian,Export Path,Matched File,File Size,Original Path
Custodian,Export Path,Matched File,File Size,Original Path

```

```nohighlight
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file1.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"

```

---

<div class="post-metadata">

**Author:** ![Corona688](https://community.unix.com/letter_avatar/corona688/32/5_5575768a8748004e209b776fc1b2916d.png) [@Corona688](https://community.unix.com/u/Corona688)\
**Post date:** [May 1, 2012, 11:53am UTC](https://community.unix.com/t/sed-to-parse-log-file/309413/2 "2012-05-01T11:53:23Z")

</div>

I'd use awk to do this, since it has much easier recall with variables and sparse arrays. Workign on it.

---

<div class="post-metadata">

**Author:** ![itkamaraj](https://community.unix.com/user_avatar/community.unix.com/itkamaraj/32/1880_2.png) [@itkamaraj](https://community.unix.com/u/itkamaraj)\
**Post date:** [May 1, 2012, 11:55am UTC](https://community.unix.com/t/sed-to-parse-log-file/309413/3 "2012-05-01T11:55:19Z")

</div>

```nohighlight
$ awk -F": " '/Custodian/{c=$2}/Export Path/{p=$2}/Matched File/{m=$2}/File Size/{s=$2}/Original Path/{o=$2} {if(c && p && m && o && s){printf("\"%s\",\"%s\",\"%s\",\"%s\",\"%s\"\n",c,p,m,s,o);m=s=o=0;}}' test.txt
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"

```

---

<div class="post-metadata">

**Author:** ![Corona688](https://community.unix.com/letter_avatar/corona688/32/5_5575768a8748004e209b776fc1b2916d.png) [@Corona688](https://community.unix.com/u/Corona688)\
**Post date:** [May 1, 2012, 12:03pm UTC](https://community.unix.com/t/sed-to-parse-log-file/309413/4 "2012-05-01T12:03:51Z")

</div>

perhaps more easily configurable:

```nohighlight
$ cat myscript.awk

BEGIN { OFS="\",\"" }

FNR==1 { for(X in A) delete A[X]; }

match($0, /:/) {
        N=substr($0, 1, RSTART-1);
        V=substr($0, RSTART+2);
        sub(/ *$/, "", N);
        A[N]=V;
}

/^Original Path/ {
        print "\"" A["Custodian"], A["Export Path"], A["Matched File"], A["File Size"], A["Original Path"] "\""
}

$ awk -f myscript.awk datafile datafile datafile

"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"

$

```

---

<div class="post-metadata">

**Author:** ![chipperuga](https://community.unix.com/letter_avatar/chipperuga/32/5_5575768a8748004e209b776fc1b2916d.png) [@chipperuga](https://community.unix.com/u/chipperuga)\
**Post date:** [May 1, 2012, 12:17pm UTC](https://community.unix.com/t/sed-to-parse-log-file/309413/5 "2012-05-01T12:17:45Z")

</div>

Made some progress, but still need help formatting the output:

```plaintext
$sed -e '/Custodian/b' -e '/Matched File/b' -e '/File Size/b' -e '/Original Path/b' -e d * | sed -e 's/.*= \(.*\) \(.*\)/"\2, \1",/' -e 's/.*= \(.*\)/"\1",/' -e 's/.*: \(.*\)/"\1",/' -e 's/"\(.*\) ..:..:.*/"\1"/'

"Jim_Foo-HDD1-FOO-1234",
"foo_file1.pdf",
"640",
"foopath\foopath\foopath\foobar\foo_file.pdf",
"foo_file2.pdf",
"123",
"foopath\foopath\foopath\foobar\foo_file.pdf",

```

It should print as:

```plaintext
"Jim_Foo-HDD1-FOO-1234","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"

```

Need to:

1. Repeat "Custodian" before each "Matched File"
2. Remove some of the extra commas

---------- Post updated at 11:17 AM ---------- Previous update was at 11:04 AM ----------

This is perfect, thank you sir.

> [@corona688](#):
>
> perhaps more easily configurable:
> 
> ```plaintext
> $ cat myscript.awk
> 
> BEGIN { OFS="\",\"" }
> 
> FNR==1 { for(X in A) delete A[X]; }
> 
> match($0, /:/) {
> N=substr($0, 1, RSTART-1);
> V=substr($0, RSTART+2);
> sub(/ *$/, "", N);
> A[N]=V;
> }
> 
> /^Original Path/ {
> print "\"" A["Custodian"], A["Export Path"], A["Matched File"], A["File Size"], A["Original Path"] "\""
> }
> 
> $ awk -f myscript.awk datafile datafile datafile
> 
> "Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
> "Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"
> "Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
> "Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"
> "Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
> "Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"
> 
> $
> 
> ```