sed multilines + separator confusion !!

Shell Programming and Scripting
1

Hi Seders,

i am new to this forum, but i think it's quite the best place to post.
So, here is my pb :

I have a csv file, with comma separator and text enclosed by ".

First pb is with text in " ......... ", wich sometimes includes lines break, and commas
And to complicate a little more, there can have many text fields on one line.

Which make it difficult to pushup lines and keep the good field number.
Which also make it impossible to use /,{87}/ s/....// to filter complete lines.

i spend quite a big time trying to reformat the lines using sed , but without success.

The file comes from gmail contacts, and is 88 fields ( so 87 separators ) .

You'll find an example file in attachment

I feel quite lost on this one, so any help highly welcome,
thanks in advance
Yogeek

2

There have been presented umpteen solutions to similar problems in these forums. In this case, and for exactly your sample (which BTW has 89 fields per line), try

sed ':A;/,$/!N; s/\n//;tA'  /tmp/sed_example.txt
3

Ok, RudiC, thanks for your answer, i tryied this one but it does not the trick.

This case can happen :

,,,,,,, " ..., some text \r\n
,,,,, \r\n
,"

Here you push every line up, if the pattern space line doesn't end by , until the line ends by a comma .
Which doesn't mean the line is well formed, see this case :

,,,, " text text text text \r\n
,, some text, \r\n
,,, blabla "

here the second line get up, ends by a comma and the script stop and work with next line ...

hence this solution work with well endeing lines, but not with broken ones

i'd like to do it the sed way, but for the moment i can't figure out a proper solution.

Thanks again

4

hi yogeek,
i can try to summarize for commas as a an idea.
so its like below.
your example file:

# cat testfile
0153400301,,,,,,,,,,,,,,,,,,,,0153400301,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Normal,,,Mes contacts;Importation du 25/10/2014,
0240153346,,,,,,,,,,,,,,,,,,,,0240153346,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Normal,,Mes contacts;Importation du 25/10/2014,
0687449343,,,,,,,,,,,,,,,,,,,,0687449343,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Normal,,Mes contacts;Importation du 25/10/2014,
1&1,,,,,,,,,,,,,"Client : 27004633
num contrat : 30081506
pw : kiaybi1
Client : 27004633
num contrat : 30081506
pw : kiaybi1",,,,,,,+33240151245,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Normal,,Mes contacts;Importation du 25/10/2014,
2,42,25,,,,,,,,,,,,,,,,,,24225,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Normal,,Mes contacts;Importation du 25/10/2014,
3G Bouygues Telecom,,Adrien Corre,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Normal,,Mes contacts;Importation du 25/10/2014,
3iX,,,,,,,,,,,,,"Client area :
Log : mail
pw : classic
cpanel : url:2082
afflink : http://www.3ix.org/sys/aff.php?aff=10157

ftp / cpanel / ssh :
chemises
k75J1FioOv",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Normal,,Mes contacts;Importation du 25/10/2014,
660,,,,,,,,,,,,,,,,,,660,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Normal,,Mes contacts;Importation du 25/10/2014,

and try

# awk -F, -va="x" '{for(i=1;i<=NF;i++){if(!$i&&i!=1){x++;printf "%s",$i} else
> {if(x)printf " %s%d%c%s%s %s","[",(x+1),a,FS,"]",$i;else printf "%s",$i;x=0}
> if(i==NF&&x)printf " %s%d%c%c%s","[",x,a,FS,"]"};x=0;print""}' testfile

0153400301 [20x,] 0153400301 [65x,] Normal [3x,] Mes contacts;Importation du 25/10/2014 [1x,]
0240153346 [20x,] 0240153346 [65x,] Normal [2x,] Mes contacts;Importation du 25/10/2014 [1x,]
0687449343 [20x,] 0687449343 [65x,] Normal [2x,] Mes contacts;Importation du 25/10/2014 [1x,]
1&1 [13x,] "Client : 27004633
num contrat : 30081506
pw : kiaybi1
Client : 27004633
num contrat : 30081506
pw : kiaybi1" [7x,] +33240151245 [65x,] Normal [2x,] Mes contacts;Importation du 25/10/2014 [1x,]
24225 [18x,] 24225 [65x,] Normal [2x,] Mes contacts;Importation du 25/10/2014 [1x,]
3G Bouygues Telecom [2x,] Adrien Corre [83x,] Normal [2x,] Mes contacts;Importation du 25/10/2014 [1x,]
3iX [13x,] "Client area :
Log : mail
pw : classic
cpanel : url:2082
afflink : http://www.3ix.org/sys/aff.php?aff=10157

ftp / cpanel / ssh :
chemises
k75J1FioOv" [72x,] Normal [2x,] Mes contacts;Importation du 25/10/2014 [1x,]
660 [18x,] 660 [67x,] Normal [2x,] Mes contacts;Importation du 25/10/2014 [1x,]

regards
ygemici

5

OK, try

sed ':A;/Importation du 25\/10\/2014,$/!N; s/\n//;tA'  /tmp/sed_example.txt

then.