Help with Creation of Script to Input Separators in Data

Shell Programming and Scripting
1

Hi all,
I have one problem that is preparing datas so I can run a script to extrat informations for my statistic reports.
I receive some datas, that are informations mixed and I need to separate them to analyse. This is an exemple of datas:
#header: 0MEPS000006000000051620090508001200905070010001613200020090508001
#body: 20421730007384620090507144018000001000000000A000000020283139Somewhere 00000001600
#body: 20421730007414520090507144237000001000000000A000000020283148Somewhere 00000001700
#body: .
#body: .
#body: .
#body: .
#body: .
#body: .
#tail: 9000000020000000000020000000000000000000000000000

Note that this #body, header does not include in this data.

So I receive them like this, but I need them to be like this:
0,MEPS,00000600,00000516,20090526001,20090525001,00016,132,000,20090526001,
2,04,2191,00112407,20090525233138,0000200000,00000,A,0000000502,98032,Somewhere ,020000008,0,0,
2,04,2191,00008894,20090525075812,0000010000,00000,A,0000001301,55245,Somewhere ,100000147,0,0,
.
.
.
.
9,00000059,0000000001790000,000000000000,000000000000,

So in this manner I can run my script:

#!/bin/bash
echo "
serial_nb amount date time ATM_nb location"
awk -F, '$1==2 {print$13,$7,$5,$6,$10,$12}' file_name

which produce me the results I want:

serial_nb amount date time ATM_nb location
000000016 0000010000 20090507 144018 0000000202 Somewhere
000000017 0000010000 20090507 144237 0000000202 Somewhere

So, right now I prepare my datas with the aid of excell, but I rather use an awk command to to this.
I need a command that says: input "," after 2 characters, then input another one after 5, etc etc.
Bottom line, a command that inputs separators acording to lenght.

Thanks for all your help.

2

This command should place comma's after 2 then after 5,7 and 10 positions, so you should get columns of 2, 5, 7, .. positions:

awk -v p="2,5,7,10" '
BEGIN { n=split(p,pos,",");b=1 }
{ for(i=1;i<=n;i++) {
    printf("%s%c",substr($0,b,pos),",")
    b+=pos
  }
  print substr($0,b)
}' file
3

Thanks Franklin for the solution, but while this brings me closer to solving everything to me, there are still some lose ends.
This code you provided me with, only applies to the first line, and it doesn�t stop where I want it.
So I made some changes, but i still have this problem with first line:
The code changed by me:

#!/bin/bash
#script to separate the fields in MEPS file
for x in `cat $1`
do awk '$1~/^2/ {print}' $1 |awk -v p="1,2,4,8,8,6,10,5,1,10,5,15,9,1,1" '
BEGIN { n=split(p,pos,",");b=1 }
{ for(i=1;i<=n;i++) {
    printf("%s%c",substr($0,b,pos),",")
    b+=pos
  }
  print substr($0,b)
}'
done

This produces the following result:

2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,

So you see, that it repeats the first line and it produces these series of ,,, ,,, ,,, ,,,,, that I do not want.
I tried to modified the code many times, but I think the problem is that I did not understant all of the code.
So what I�m asking is for you to give me some pointer on how to make this work and some explanation on the original code you provided me.

Thanks in advance.

4

Post the input file of the awk command and the desired output within code tags (select the text and press the # button above the edit box).

5

Corrected the post. Can you check the post again please.
Thanks you

6

It should be easier if you post a part of the input file of the awk command and the desired output.

Regards

7

Hi there,
I�m not sure I understand exactly what you need, but let me give you exactly what I want:
This is my input file:

20421910011240720090525233138000020000000000A000000050298032Boavista       02000000800
20421910000889420090525075812000001000000000A000000130155245Praia          10000014700
20421910002863720090525103836000001000000000A000000070128094Sal            10000014800
20421910003524020090525112959000001000000000A000000010746168Praia          10000014900
20421910003616520090525113544000001000000000A000000020185311Sao Vicente    10000015000
20421910003807620090525114955000001000000000A000000130155277Praia          10000015100

And this is the output I need:

2,04,2191,00112407,20090525233138,0000200000,00000,A,0000000502,98032,Boavista       ,020000008,0,0
2,04,2191,00008894,20090525075812,0000010000,00000,A,0000001301,55245,Praia          ,100000147,0,0
2,04,2191,00028637,20090525103836,0000010000,00000,A,0000000701,28094,Sal            ,100000148,0,0
2,04,2191,00035240,20090525112959,0000010000,00000,A,0000000107,46168,Praia          ,100000149,0,0
2,04,2191,00036165,20090525113544,0000010000,00000,A,0000000201,85311,Sao Vicente    ,100000150,0,0
2,04,2191,00038076,20090525114955,0000010000,00000,A,0000001301,55277,Praia          ,100000151,0,0
2,04,2191,00078149,20090525171505,0000010000,00000,A,0000000401,63329,Santa Catarina ,100000152,0,0

But with the code you provide me (with some changes performed, but the same basic codes):

#!/bin/bash
#script to separate the fields in MEPS file
for x in `cat $1`
do awk '$1~/^2/ {print}' $1 |awk -v p="1,2,4,8,8,6,10,5,1,10,5,15,9,1,1" '
BEGIN { n=split(p,pos,",");b=1 }
{ for(i=1;i<=n;i++) {
    printf("%s%c",substr($0,b,pos),",")
    b+=pos
  }
  print substr($0,b)
}'
done

I receive insted this output:

2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,

So you see, it only applies to the first input line and reapets it self over and over again.

Hope this was clear for you, and if you could help me I would be very grateful.

Best Regards

P.S. this last output (the one that reapets itself) is not from the original file, I present here, but you get the point.

8

Ok, I've adjust the code:

awk -v p="1,2,4,8,14,10,5,1,10,5,15,9,1,1" '
BEGIN { n=split(p,pos,",");b=1 }
{ for(i=1;i<=n;i++) {
    printf("%s%c",substr($0,b,pos),i==n?"\n":",")
    b+=pos
  }
  b=1

}' file

This is what I get:

$ cat file
20421910011240720090525233138000020000000000A000000050298032Boavista       02000000800
20421910000889420090525075812000001000000000A000000130155245Praia          10000014700
20421910002863720090525103836000001000000000A000000070128094Sal            10000014800
20421910003524020090525112959000001000000000A000000010746168Praia          10000014900
20421910003616520090525113544000001000000000A000000020185311Sao Vicente    10000015000
20421910003807620090525114955000001000000000A000000130155277Praia          10000015100
$ awk -v p="1,2,4,8,14,10,5,1,10,5,15,9,1,1" '
BEGIN { n=split(p,pos,",");b=1 }
{ for(i=1;i<=n;i++) {
    printf("%s%c",substr($0,b,pos),i==n?"\n":",")
    b+=pos
  }
  b=1

}' file
2,04,2191,00112407,20090525233138,0000200000,00000,A,0000000502,98032,Boavista       ,020000008,0,0
2,04,2191,00008894,20090525075812,0000010000,00000,A,0000001301,55245,Praia          ,100000147,0,0
2,04,2191,00028637,20090525103836,0000010000,00000,A,0000000701,28094,Sal            ,100000148,0,0
2,04,2191,00035240,20090525112959,0000010000,00000,A,0000000107,46168,Praia          ,100000149,0,0
2,04,2191,00036165,20090525113544,0000010000,00000,A,0000000201,85311,Sao Vicente    ,100000150,0,0
2,04,2191,00038076,20090525114955,0000010000,00000,A,0000001301,55277,Praia          ,100000151,0,0
$
$

Regards

9

Mr. Franklin
I thank you for you for you time an pacience to show me these codes.
It work exactly like I wanted.
This part of the code that was making me confusion:

printf("%s%c",substr($0,b,pos),i==n?"\n":",")

Now I see that because of the i==n?"\n":"," part that makes the code go to the other line in order to execut withouth repeating the first line over and over again.

Once again, thanks for all the help.

Best Regards