Converting data for text file to csv

Shell Programming and Scripting
1

Gents

Using the script attached (raw2csv). i use to create the file .csv.. The input file is called 201.raw.

Kindly can you check if there is easy way to do it. The script works fine but takes a lot time to process

Thanks for your help

2

Do you really need to use csh?
See some reasons to avoid it in sticky thread at top of this forum.

Below is a ksh or bash solution:

date
filename=${1%.*}
sed 's/# ===== (.*) =====/#-#-#-#/g' $1 | \
awk '
NR>1 {
    gsub(/[^M\"\]]/,"")
    gsub(/[:\t]/," ")
    gsub(" *\n", "\n")
    gsub("\n *", "\n")
    gsub("\n#^[\n]*\n", "\n")
    gsub("\n\n+", "\n")
    printf "%s", $0
}' RS="#-#-#-#" | egrep -v '(====|Report|^[0-9]|^Live_Seis_Channels)' | \
awk -F"\n" -vRS="" 'NR>1{
  for(i=1;i<=NF;i++) {
     H=$i
     gsub(" .*","",H)
     gsub(H" *","",$i)
     gsub("# [(](ms|msec)[)]","",$i)
     gsub("#","",$i)
     V[NR]=V[NR]"  ,"$i
     if(NR==2)HD=HD" ,"H
  }
}
END{ print substr(HD,3)
  for(i=1;i<=NR;i++) print substr(V[NR], 4)
}' > "${filename}.csv"
date

Note that in above ^M is a CTRL-M character to enter this in vi use CTRL-V CTRL-M

1 Like
3

XL,

I don't get any output and the script take a lot time to finish?..

4

It will not produce any output in the file until it has finished (it reads the whole file into memory).

How large is your input file?
What OS are you on?
Are you sure you got the CTRL-M value correctly, as described above?
Can you try it with the demo file you posted, any output from that?

5

Hello XL
The original file has 206.700 lines
I am using redhat version 5
I have tyr with the short file I have posted and it takes long time to finish but nothing output empty file.
Sorry I dont understand well what i need to do to get the CTRL-Value?? Please help me.
Thanks

6

This version should avoid the CTRL-Value issue:

filename=${1%.*}
sed 's/# ===== (.*) =====/#-#-#-#/g' $1 | \
awk '
NR>1 {
    gsub(/[\r\"\]]/,"")
    gsub(/[:\t]/," ")
    gsub(" *\n", "\n")
    gsub("\n *", "\n")
    gsub("\n#^[\n]*\n", "\n")
    gsub("\n\n+", "\n")
    printf "%s", $0
}' RS="#-#-#-#" | egrep -v '(====|Report|^[0-9]|^Live_Seis_Channels)' | \
awk -F"\n" -vRS="" 'NR>1{
  for(i=1;i<=NF;i++) {
     H=$i
     gsub(" .*","",H)
     gsub(H" *","",$i)
     gsub("# [(](ms|msec)[)]","",$i)
     gsub("#","",$i)
     V[NR]=V[NR]"  ,"$i
     if(NR==2)HD=HD" ,"H
  }
}
END{ print substr(HD,3)
  for(i=1;i<=NR;i++) print substr(V[NR], 4)
}' > "${filename}.csv"

Be sure to call the script like this:

$ ./new_raw2csv 201.raw
1 Like
7

Dear XL

Thanks a lot for your support. Yes the script works fine now.. But there is a problem it repeats the last entrance 5 times each loop same values.. In the attached files I send a example of the output with the old script ( i have )and the new one you did.

Please help me to check this

8

Replace this:

  for(i=1;i<=NR;i++) print substr(V[NR], 4)

with this:

  for(i=1;i<=NR;i++) print substr(V[i], 4)
1 Like
9

Thanks XL
it works :slight_smile: thanks for your help and support