awk not working as expected in script

Shell Programming and Scripting
1

Dear all,
I had script which used to work, but recently it is not working as expected.
I have command line [1] in my shell script to choose the following format [2] from the output_elog [3] and perform some task afterwards on [2]

As you see, I want all numbers in foramt [2] following RED mark except for "Jobs with Wrapper Exit Code : 0".

Thank you,
emily

[1]

jobNo=$(awk '/Jobs with Wrapper/ && $NF != 0{s=1}   /List of jobs/ && s{if(p){p=p","$NF}else{p=$NF};s=""}END{print p}' output_elog)

[2]
Format is to pick up all numbers following the RED coded command_line in [3] but not BLUE coded, here I would expect to have the following output of [1]:

14,35,40,65,94,132,144,159,182,197,211,213,228,236,239,22,24-25,60,70,78,83,126,145,153,171,219,225,231,240,3,29,53,59,66,95,112-113,133,160,164,175,220,57,72,92,99,207,214,247,242,19,49,54,76,86,102,107,11\
0,119-120,181,210,221,234,5,63,97,204,246,71,117,84,215,50,208,10,114,229,87,148,158,166-168,244,9,30,51,61-62,69,127,150,157,165,174,177,238,243

[3] output_elog

crab:  ExitCodes Summary
 >>>>>>>>> 50 Jobs with Wrapper Exit Code : 50664
         List of jobs: 14,35,40,65,94,132,144,159,182,197,211,213,228,236,239,22,24-25,60,70,78,83,126,145,153,171,219,225,231,240,3,29,53,59,66,95,112-113,133,160,164,175,220,57,72,92,99,207,214,247
        See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 21 Jobs with Wrapper Exit Code : 60317
         List of jobs: 242,19,49,54,76,86,102,107,110,119-120,181,210,221,234,5,63,97,204,246,71
        See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 4 Jobs with Wrapper Exit Code : 50662
         List of jobs: 117,84,215,50
        See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 141 Jobs with Wrapper Exit Code : 0
         List of jobs: 1,6-8,13,15-18,20-21,23,26,33,39,42,44-48,55-56,58,64,68,79,81,93,96,98,100-101,106,109,115-116,118,121,123-125,128-131,134-143,146-147,149,151-152,154-156,161-162,170,173,176,178-179\
,183-194,196,198,200-203,205-206,209,216-217,230,233,235,237,241,31-32,36-37,43,82,91,111,122,2,4,11,27,38,52,67,73,75,80,85,103-105,108,169,172,199,222-223,227,232,12,74,88,163,218,226,245,248-249,41,195
        See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 1 Jobs with Wrapper Exit Code : 127
         List of jobs: 208
        See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 1 Jobs with Wrapper Exit Code : 10034
         List of jobs: 10
        See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:   250 Total Jobs
 >>>>>>>>> 9 Jobs Running
        List of jobs Running: 212,28,224,250,89,34,77,90,180
 >>>>>>>>> 210 Jobs Retrieved
        List of jobs Retrieved: 1,6-8,13,15-18,20-21,23,26,33,39,42,44-48,55-56,58,64,68,79,81,93,96,98,100-101,106,109,115-118,121,123-125,128-131,134-143,146-147,149,151-152,154-156,161-162,170,173,176,17\
8-179,183-194,196,198,200-203,205-206,209,216-217,230,233,235,237,241,14,31-32,35-37,40,43,65,82,91,94,111,122,132,144,159,182,197,211,213,228,236,239,242,2,4,11,19,22,24-25,27,49,52,54,60,67,70,73,75-76,78\
,80,83-86,102-105,107-108,110,119-120,126,145,153,169,171-172,181,199,210,215,219,221-223,225,227,231-232,234,240,3,5,12,29,50,53,59,63,66,74,88,95,97,112-113,133,160,163-164,175,204,218,220,226,245-246,248\
-249,10,41,71,195,208
 >>>>>>>>> 23 Jobs Cancelled
        List of jobs Cancelled: 114,229,87,148,158,166-168,244,9,30,51,61-62,69,127,150,157,165,174,177,238,243
 >>>>>>>>> 8 Jobs Done
        Jobs terminated: retrieve them with: crab -getoutput <List of jobs>
        List of jobs: 38,57,72,92,99,207,214,247
2

This:

>>>>>>>>> 23 Jobs Cancelled

is not selected by your awk script. So try changing

/Jobs with Wrapper/

to

/Jobs (with Wrapper|Cancelled)/

and try again

3

Hi, yes you are right..I did not include this cancelled category for now.
But the problem is when I try to print the jobID, its showing blank..just nothing..:frowning:

don't know, what is wrong with [1]..do you see any issues with [1] to perform this task (excluding cancelled category)?

thanks
emily

4

Wrong answer

5

Hello,
your answer is too short for me to understand what modification is needed in my script line [1].

thanks
emily

6

I don't understand. This is your expected output:

14,35,40,65,94,132,144,159,182,197,211,213,228,236,239,22,24-25,60,70,78,83,126,145,153,171,219,225,231,240,3,29,53,59,66,95,112-113,133,160,164,175,220,57,72,92,99,207,214,247,242,19,49,54,76,86,102,107,110,119-120,181,210,221,234,5,63,97,204,246,71,117,84,215,50,208,10,114,229,87,148,158,166-168,244,9,30,51,61-62,69,127,150,157,165,174,177,238,243

The part in red is from the "Cancelled bit". When I execute:

awk '/Jobs (with Wrapper|Cancelled)/ && $NF != 0{s=1}   /List of jobs/ && s{if(p){p=p","$NF}else{p=$NF};s=""}END{print p}' output_elog

I get

14,35,40,65,94,132,144,159,182,197,211,213,228,236,239,22,24-25,60,70,78,83,126,145,153,171,219,225,231,240,3,29,53,59,66,95,112-113,133,160,164,175,220,57,72,92,99,207,214,247,242,19,49,54,76,86,102,107,110,119-120,181,210,221,234,5,63,97,204,246,71,117,84,215,50,208,10,114,229,87,148,158,166-168,244,9,30,51,61-62,69,127,150,157,165,174,177,238,243
7

Hi,
yes you are right, thank you!

I just executed myself and found the expected answer...seems, issue is the way I get that output_elog, cos the script is not able to read it, as it seems!

I am checking that.

thank you,
emily

8

I modified my post, because I understood you incorrectly. Sorry about that.

9

Hi again,
I am still puzzeled as its not working [4], though that script line t is all OK. The output of this [4] is following [5] as I checked. And I checked that the output of the GEN-SIM-DIGI-L1_v2_010813_status.log is [6],. which is all OK..:frowning:

Maybe I am making some stupid mistake,if anybody spot it ...please let me know.

thank in advance,
emily

[4]

crab -getoutput -c $SAMPLE
    crab -status -c $SAMPLE >& $SAMPLE"_status.log" &

    echo $CODEPATH/$SAMPLE"_status.log" "====="
    jobNo=$(awk '/Jobs with Wrapper/ && $NF != 0{s=1}   /List of jobs/ && s{if(p){p=p","$NF}else{p=$NF};s=""}END{print p}' "$SAMPLE"_status.log)
    sleep 50
    echo "The present working directory is `pwd`"
    echo ">>>>>>>>>>>>>>>>>>>>> JObs SUBMITTED ARE FOLLOWING: " $jobNo
    echo "DONE"

[5]

 /nfs/dust/cms/user/emily/scratch/hadron_outer/emulation_HO/CMSSW_7_1_0/src//L1Trigger/L1IntegratedMuonTrigger/test/GEN-SIM-DIGI-L1_v2_010813_status.log =====
The present working directory is /nfs/dust/cms/user/emily/scratch/hadron_outer/emulation_HO/CMSSW_7_1_0/src/L1Trigger/L1IntegratedMuonTrigger/test
>>>>>>>>>>>>>>>>>>>>> JObs SUBMITTED ARE FOLLOWING: 

DONE

[6]

 crab:  ExitCodes Summary
 >>>>>>>>> 2 Jobs with Wrapper Exit Code : 60317
         List of jobs: 207,250
        See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 85 Jobs with Wrapper Exit Code : 0
         List of jobs: 5,36-37,57,88,104,126,131,137-138,141,143,146,148,160,179-180,185,196,214-216,223-224,229,233,237-238,248-249,54,90,97,115-117,129,167,173,176,189,239,245,158,1,19,38,41,43,52-53,67,8\
0,82-84,105,108,110,112,120,124,136,139,142,147,150,170,186,227,230,234,6,9-11,16,45,85,101,106,194,201-202,205
        See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 2 Jobs with Wrapper Exit Code : 50664
         List of jobs: 236,190
        See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:   250 Total Jobs
 >>>>>>>>> 131 Jobs Running
        List of jobs Running: 134,187,206,2-3,8,12-13,15,17-18,20,22,25,27-30,32-33,35,39-40,42,44,46-51,55,58-66,68-76,78,86-87,89,91-96,98-100,102-103,107,109,111,113-114,118-119,121-123,125,127-128,130,1\
32-133,144-145,151,153,155-157,159,161-164,166,168-169,171-172,174-175,177-178,181-183,191-193,195,203-204,208-209,211-213,217,219-222,225-226,228,231-232,235,241,244,246-247
 >>>>>>>>> 1 Jobs Aborted
        You can resubmit them specifying JOB numbers: crab -resubmit <List of jobs>
        List of jobs: 218
 >>>>>>>>> 90 Jobs Retrieved
        List of jobs Retrieved: 5,36-37,57,88,104,126,131,137-138,141,143,146,148,160,179-180,185,196,198,214-216,223-224,229,233,237-238,248-249,54,90,97,115-117,129,167,173,176,189,236,239,245,158,207,1,1\
9,38,41,43,52-53,67,80,82-84,105,108,110,112,120,124,136,139,142,147,150,170,186,227,230,234,250,6,9-11,16,45,85,101,106,194,201-202,190,205
 >>>>>>>>> 28 Jobs Cancelled
        List of jobs Cancelled: 7,14,21,23,34,135,140,149,152,154,165,197,210,240,242,4,24,26,31,56,77,79,81,184,188,243,199-200
10

You did not adjust the code? The same line seems to be used.

Another thing I noticed is that on this line:

crab -status -c $SAMPLE >& $SAMPLE"_status.log" &

The command is executed in the background and is writing to a file.
On the second line after that, line the file is being read however., which should only happen after the background job has finished.

So either insert a wait statement between those two lines, or remove the closing & of the first line and thus run it in the foreground.

2 Likes
11

Hi,
I put sleep 50, should not that be OK?
Even I increase it to sleep 200, it does not help..

emily

12

emily did you ever watch Jerry Maguire?: "Help me help you!!"

Scrutinizer asked: "You did not adjust the code? The same line seems to be used."
You have not addressed that.

Have you tried wait as Scrutinizer mentioned?
Have you tried to remove the process from background?

It is OK to ignore any advise you receive after you asked for, nevertheless, it would be proper to explain why if you want to continue asking.

1 Like
13

Scrutinizer- thanking you so much for your help. And sorry for posting hurriedly without trying your advice.

Greetings,
emily