parsing log files, removing spaces and replace with commas

Shell Programming and Scripting
1

Hello all

i am working on a database to import log files from my systems, but i cannot seem to find the answer. I searched here for a good bit and couldnt peice together what i was looking for.

I know you could do this with awk, i just dont know how. Any help would be greatly appreciated.

ok, so take for instance the /var/adm/messages file:
Jun 5 08:40:02 SunNiprS1 usba: [ID 349649 kern.info] PLEXTOR Corp. USB Storage Adapter 004400947B
Jun 5 08:40:02 SunNiprS1 genunix: [ID 936769 kern.info] scsa2usb0 is /pci@1e,600000/usb@a/cdrom@2
Jun 5 08:40:02 SunNiprS1 genunix: [ID 408114 kern.info] /pci@1e,600000/usb@a/cdrom@2 (scsa2usb0) online
Jun 5 08:41:09 SunNiprS1 scsi: [ID 193665 kern.info] sd16 at scsa2usb0: target 0 lun 0
Jun 5 08:41:09 SunNiprS1 genunix: [ID 936769 kern.info] sd16 is /pci@1e,600000/usb@a/cdrom@2/disk@0,0
Jun 5 08:41:35 SunNiprS1 genunix: [ID 408114 kern.info] /pci@1e,600000/usb@a/cdrom@2/disk@0,0 (sd16) online

all i want to do is seperate each field by a comma, i thought it would be pretty easy...but i am having a difficult time with it. Some sample output would look like this:

Jun 5, 08:40:02, SystemName, usba: [ID 349649 kern.info], PLEXTOR Corp. USB Storage Adapter 004400947B
Jun 5, 08:40:02, SystemName, genunix: [ID 936769 kern.info], scsa2usb0 is /pci@1e,600000/usb@a/cdrom@2

thank you

2

This would doi it:

$ sed -e s'/ /, /2' -e 's/ /, /3'  -e 's/ /, /4' -e 's/ /, /8' < /var/adm/messages > /outputlogfile

e.g.:

$ sed -e s'/ /, /2' -e 's/ /, /3'  -e 's/ /, /4' -e 's/ /, /8' logfile
Jun 5, 08:40:02, SunNiprS1, usba: [ID 349649 kern.info], PLEXTOR Corp. USB Storage Adapter 004400947B
Jun 5, 08:40:02, SunNiprS1, genunix: [ID 936769 kern.info], scsa2usb0 is /pci@1e,600000/usb@a/cdrom@2
Jun 5, 08:40:02, SunNiprS1, genunix: [ID 408114 kern.info], /pci@1e,600000/usb@a/cdrom@2 (scsa2usb0) online
Jun 5, 08:41:09, SunNiprS1, scsi: [ID 193665 kern.info], sd16 at scsa2usb0: target 0 lun 0
Jun 5, 08:41:09, SunNiprS1, genunix: [ID 936769 kern.info], sd16 is /pci@1e,600000/usb@a/cdrom@2/disk@0,0
Jun 5, 08:41:35, SunNiprS1, genunix: [ID 408114 kern.info], /pci@1e,600000/usb@a/cdrom@2/disk@0,0 (sd16) online

Note the device paths have commas in already which you may want to substitute for a different character first.

If you don't want spaces after the commas then add a "-e 's/, /,/g'" to the end.

3

if you have Python

#!/usr/bin/env python
for line in open("file"):
    line=line.strip()
    st = line.index("[")
    line=line.replace("]","],")
    o = line[:st].split()
    print ' '.join(o[:2]),','.join(o[2:]),line[st:]

output

# ./test.py
Jun 5 08:40:02,SunNiprS1,usba: [ID 349649 kern.info], PLEXTOR Corp. USB Storage Adapter 004400947B
Jun 5 08:40:02,SunNiprS1,genunix: [ID 936769 kern.info], scsa2usb0 is /pci@1e,600000/usb@a/cdrom@2
Jun 5 08:40:02,SunNiprS1,genunix: [ID 408114 kern.info], /pci@1e,600000/usb@a/cdrom@2 (scsa2usb0) online
Jun 5 08:41:09,SunNiprS1,scsi: [ID 193665 kern.info], sd16 at scsa2usb0: target 0 lun 0
Jun 5 08:41:09,SunNiprS1,genunix: [ID 936769 kern.info], sd16 is /pci@1e,600000/usb@a/cdrom@2/disk@0,0
Jun 5 08:41:35,SunNiprS1,genunix: [ID 408114 kern.info], /pci@1e,600000/usb@a/cdrom@2/disk@0,0 (sd16) online
4

ghostdog74, can python be made to not put a comma between the month and date, not after the ":" and put a comma in after the "]"?

5

You can make use of tr filter here:

echo "a:c:d" | tr ":" ","

This will replace : with ,

6

of course, see my edits.

-----Post Update-----

see the requirement again. don't think OP wants to change the date's ":" to ,

7

wow, thanks to you all!

this should work perfectly, i will update if i had to change anything...

i see where i was messing up, i wasnt using the -e option...