Shell script to extract data in a file

UNIX for Beginners Q & A
21

This doesn't match what you posted in post#16.
As I said in post#17, when I do exactly what you describe in post#20, I don't have a problem, a file "output" is created with the expected contents.

What's your OS, shell, and awk versions? Any error messages when you run your approach?

22

I have the following error. My unix version is :SunOS cx01 5.10 Generic_139555-08 sun4v sparc SUNW,SPARC-Enterprise-T5220

awk: syntax error near line 11
awk: bailing out near line 11
23

This is the third time you are pointed to post#11.

24

this has been commented on in this thread here

25

I used nawk instead of awk as advised in your post #11, stil got the error below. In post #11 what do u mean by output is to stdout,PLEASE

>sh gil
nawk: you can only delete array[element] at source line 14
 context is
                                 delete RES >>>
 <<<                         }
nawk: syntax error at source line 15
nawk: illegal statement at source line 15
26

if you're using RudiC's code, change delete RES to split("", RES)

27

@ vgersh99 and RudiC you are the best. It's working fine. It'll take at least 6 hours cause i have about 4 milions lines, but still we made it. Thank very much.

28

it would be interesting to know the exact timing, but 6 hours sounds somewhat/extremely high... How did you arrive at this number? 6 hours?

Could you try the suggestion from post 5 and compare the timings?

29

I'd be very surprised if it really took 6h, except on a very busy host. For a 15MB file with 1101925 lines:

time sh ou.awk 
real    0m4.469s
user    0m4.400s
sys     0m0.040s

EDIT:
For a comparison with vgersh99's proposal in post#5:

time awk -f gil.awk DS2_export100217.ldif
real    0m1.713s
user    0m1.076s
sys     0m0.072s

And, last but not least, Corona688's proposal in post#9

time awk -f mline.awk DS2_export100217.ldif
real    0m0.833s
user    0m0.752s
sys     0m0.072s

Both use ($1 in ARRAY) which is WAY faster than my HDCM ~ "," $1 "," ; a good learning, isn't it?

EDIT2:
Applying our learning:

time ./ou.awk >output                    
real    0m0.678s
user    0m0.660s
sys     0m0.016s

BTW - these timings are far from being scientifically correct, reliable nor objective - other influences like file buffering etc. might heavily modify the results. They're for getting a feeling only...

30

@ RudiC it's really taking such time 4hrs at least. I really have to improve the time. Maybe it's my machine that's also running other stuff. But all the same the script is working perfectly.

@vgersh99 i tried the proposal on post 5, it gave me an error. I'm trying again with a file of the same size, and let you know of the timing and error if there is.

---------- Post updated at 04:23 AM ---------- Previous update was at 04:11 AM ----------

Please guys could you explain to me briefly each line of your code?

31

Try this "optimized" version:

awk -F: '
BEGIN           {HD="MSISDN,IMSI,NAM,TS11,TS21,TS22,TS62,BAIC,BAOC,APNID1,APNID2"       # define the fields to extract, and prepare a header for eventual print
                 for (i=HDCnt=split(HD, HDArr, ","); i>0; i--)  X[HDArr]             # create am "associative" array  for later $1 matching 
                 NXTREC = "dn"                                                          # this is where the next record starts
                }


$1 == NXTREC    {if (PR) for (i=1; i<=HDCnt; i++) printf "%s%s", RES[HDArr], i==HDCnt?ORS:OFS
                                                                                        # print old record collected up to now; don't on the first record (Pr still undefined)
                 split ("", RES)                                                        # remoce RES array
                 PR = 1                                                                 # define the PR logical var so all next records are printed
                }

$1 in X         {RES [$1] = $0                                                          # collect data line if $1 is among the requested fields
                }

END             {for (i=1; i<=HDCnt; i++) printf "%s%s", RES[HDArr], i==HDCnt?ORS:OFS
                }                                                                       # see above for the last record

' FS=":" OFS="," DS2_export100217.ldif

and report back the timing...

32

It's running but the output file is not receiving the output expected. It's taking more time to load the output file.

---------- Post updated 02-19-17 at 07:07 AM ---------- Previous update was 02-18-17 at 11:52 AM ----------

Please another script to extrat the information highlighted in yellow. Can you guys help? i tried with the other script, it's not giving the results. the results should be on the same line. Please!!

dn: EpsStaInfId=EpsStaInf,serv=EPS,mscId=aaaaaa001aaaaaaaa629100100138702,ou=multiSCs,dc=mtncg
structuralObjectClass: EpsStaticInf
objectClass: EpsStaticInf
entryDS: 1
nodeId: 1
createTimestamp: 20170119123400Z
modifyTimestamp: 20170119123400Z
EpsStaInfId: EpsStaInf
EpsProfileId: 29
EpsOdb: 0
EpsRoamAllow: TRUE
CDC: 1
EpsIndSubChargChars: 123

dn: EpsStaInfId=EpsStaInf,serv=EPS,mscId=aaaaaa001aaaaaaaa629100100165619,ou=multiSCs,dc=mtncg
structuralObjectClass: EpsStaticInf
objectClass: EpsStaticInf
entryDS: 1
nodeId: 1
createTimestamp: 20170211115449Z
modifyTimestamp: 20170211115449Z
EpsStaInfId: EpsStaInf
EpsProfileId: 29
EpsOdb: 0
EpsRoamAllow: TRUE
CDC: 1
EpsIndSubChargChars: 123
33

What did you try, esp. in the light of the proposals made for your first problem? Show your attempts including error messages.
Don't miss to post the expected output. Does it go to the same result file as the other fields?

If not: for a new request create a new thread.

34

@RudiC, i tried with the script, it didn't give any results at all. I lunched it, but the output file was empty. So i'm wondering whether the output was still in the buffer or something else..I'm creating a new thread for the new problem right away.