extracting Number variable and the following digits.

Shell Programming and Scripting
1

HI all,
I have output of something like this:

  
crab:  ExitCodes Summary
 >>>>>>>>> 12 Jobs with Wrapper Exit Code : 50117 
	 List of jobs: 1-12 
	See https:///twiki/something/ for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 5 Jobs with Wrapper Exit Code : 8001 
	 List of jobs: 1-3,7,8 
         See https://twiki/something/ for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 3 Jobs with Wrapper Exit Code : 8021 
	 List of jobs: 140,123,342
	See https://twiki/something/ for Exit Code meaning

I would like to recognize the no 50117 ( 8001,8021). And then then digits following this no: 1-12, [which is actually 1,2,3,4,5,6,7,8,9,10,11,12]
Once I have these digits I have to perform some command on these digits.

The no 50117 ( 8001,8021) is important. Reason is depending on this no, the different commands would be performed on the following digits.

Thanks in advance,
emily

2

Like this?

awk '/Jobs with Wrapper Exit Code/{a=$NF}
/List of jobs/{print a,$NF;a=""}' file
3

hi,
Thanks for replying. I tried these command on the following txt file (file.txt):

 crab:  ExitCodes Summary
 >>>>>>>>> 568 Jobs with Wrapper Exit Code : 0 
  List of jobs: 1-95,97-114,116-121,123,125-129,131-159,161-184,186,188-191,194,199-206,208-230,232-233,235-236,238-239,241,243-246,248-250,252-255,258-264,266-280,282-288,290,292-296,298-300,302-304,306-312,314-316,318-319,321-373,375-380,382-458,460-500,502-511,513-516,519-524,526-531,533-574,576-597,599-613 
  See https://twiki/something

crab:  ExitCodes Summary
 >>>>>>>>> 1 Jobs with Wrapper Exit Code : 8001 
  List of jobs: 381 
  See https://twiki.soomething

and the output is here

   [extra]$ awk '/Jobs with Wrapper Exit Code/{a=$NF}
/List of jobs/{print a,$NF;a=""}' file.txt
0 1-95,97-114,116-121,123,125-129,131-159,161-184,186,188-191,194,199-206,208-230,232-233,235-236,238-239,241,243-246,248-250,252-255,258-264,266-280,282-288,290,292-296,298-300,302-304,306-312,314-316,318-319,321-373,375-380,382-458,460-500,502-511,513-516,519-524,526-531,533-574,576-597,599-613
8001 381

It seems to be working, but I doubt I did not get as how to use it. And also, I wonder if it would paste the NO as well like 8001 along with the digits?
:confused:

thanks

4

Do you need this then?

awk '/Jobs with Wrapper Exit Code/{a=$NF}
/List of jobs/{n=split($NF,t,",")
for(i=1;i<=n;i++)
 if(split(t,t2,"-")==2)
  for(j=t2[1];j<=t2[2];j++)
   b=length(b)?b "," j:j
 else
  b=length(b)?b "," t:t
print a,b;a=b=""
}' file

which produces

0 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,
45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86
,87,88,89,90,91,92,93,94,95,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,116,117,118,119,120,121,123,
125,126,127,128,129,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,1
57,158,159,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,186,188,189,190,19
1,194,199,200,201,202,203,204,205,206,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229
,230,232,233,235,236,238,239,241,243,244,245,246,248,249,250,252,253,254,255,258,259,260,261,262,263,264,266,267,268,269,270,
271,272,273,274,275,276,277,278,279,280,282,283,284,285,286,287,288,290,292,293,294,295,296,298,299,300,302,303,304,306,307,3
08,309,310,311,312,314,315,316,318,319,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,34
2,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373
,375,376,377,378,379,380,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,
407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,4
38,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,460,461,462,463,464,465,466,467,468,469,47
0,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,502
,503,504,505,506,507,508,509,510,511,513,514,515,516,519,520,521,522,523,524,526,527,528,529,530,531,533,534,535,536,537,538,
539,540,541,542,543,544,545,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,565,566,567,568,569,5
70,571,572,573,574,576,577,578,579,580,581,582,583,584,585,586,587,588,589,590,591,592,593,594,595,596,597,599,600,601,602,60
3,604,605,606,607,608,609,610,611,612,613
8001 381

on the mentioned (latest) input file.

5

Hi,

I think, I did not clear my purpose. For the given input file, it is following :

  crab:  ExitCodes Summary
 >>>>>>>>> 568 Jobs with Wrapper Exit Code : 0 
  List of jobs: 1-95,97-114,116-121,123,125-129,131-159,161-184,186,188-191,194,199-206,208-230,232-233,235-236,238-239,241,243-246,248-250,252-255,258-264,266-280,282-288,290,292-296,298-300,302-304,306-312,314-316,318-319,321-373,375-380,382-458,460-500,502-511,513-516,519-524,526-531,533-574,576-597,599-613 
  See https://twiki/something

crab:  ExitCodes Summary
 >>>>>>>>> 1 Jobs with Wrapper Exit Code : 8001 
  List of jobs: 381 
  See https://twiki.soomething

I would like to provide no(=0) to some script and would like it give me back output as

 1-95,97-114,116-121,123,125-129,131-159,161-184,186,188-191,194,199-206,208-230,232-233,235-236,238-239,241,243-246,248-250,252-255,258-264,266-280,282-288,290,292-296,298-300,302-304,306-312,314-316,318-319,321-373,375-380,382-458,460-500,502-511,513-516,519-524,526-531,533-574,576-597,599-613

Then I would provide it another no (=8001) and would seek the output as

 381

Similarly, there would be different no string that I will provide and would seek the output.
And sometimes would seek the output for this case as well.

 pastes all digits following all no(like 8001) except 0

.

I hope it is clear now.

thanks

6

something like this..?
VM=8001 as your input variable...

awk -v VM="8001" '/Jobs with Wrapper/ && $NF == VM{s=1}
  /List of jobs/ && s{print $NF}' file
7

Hi,
I tried it over this file

crab:  ExitCodes Summary
 >>>>>>>>> 568 Jobs with Wrapper Exit Code : 0 
  List of jobs: 1-95,97-114,116-121,123,125-129,131-159,161-184,186,188-191,194,199-206,208-230,232-233,235-236,238-239,241,243-246,248-250,252-255,258-264,266-280,282-288,290,292-296,298-300,302-304,306-312,314-316,318-319,321-373,375-380,382-458,460-500,502-511,513-516,519-524,526-531,533-574,576-597,599-613 
  See https://twiki.somthing for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 1 Jobs with Wrapper Exit Code : 8001 
  List of jobs: 381

Following are the command and the output:

 
 [extra]$ awk -v VM="0" '/Jobs with Wrapper/ && $NF == VM{s=1}
  /List of jobs/ && s{print $NF}' file.txt
1-95,97-114,116-121,123,125-129,131-159,161-184,186,188-191,194,199-206,208-230,232-233,235-236,238-239,241,243-246,248-250,252-255,258-264,266-280,282-288,290,292-296,298-300,302-304,306-312,314-316,318-319,321-373,375-380,382-458,460-500,502-511,513-516,519-524,526-531,533-574,576-597,599-613
381

[extra]$ awk -v VM="8001" '/Jobs with Wrapper/ && $NF == VM{s=1}
  /List of jobs/ && s{print $NF}' file.txt
381

As you can see that with VM=0 it is also passing the digits following 8001..:confused:

8

one correction...
now try..

$awk -v VM="0" '/Jobs with Wrapper/ && $NF == VM{s=1}   /List of jobs/ && s{print $NF;s=""}' file
1-95,97-114,116-121,123,125-129,131-159,161-184,186,188-191,194,199-206,208-230,232-233,235-236,238-239,241,243-246,248-250,252-255,258-264,266-280,282-288,290,292-296,298-300,302-304,306-312,314-316,318-319,321-373,375-380,382-458,460-500,502-511,513-516,519-524,526-531,533-574,576-597,599-613
9

Hi pamu,
Thanks,Yeah it is fine.

Would you tell me if I need to ask that :
Give me the digits for all other NO(ex 8001, 5001, 8221) except "0". How shall I do that?

Ans also explain the command a bit please.

emily

10

for all digits except "0" try this..

awk '/Jobs with Wrapper/ && $NF != 0{s=1}   /List of jobs/ && s{print $NF;s=""}' file

awk '/Jobs with Wrapper/ && $NF != 0{s=1} Here command check for the line which contains Jobs with Wrapper & $NF should be not equal zero. If both the conditions are satisfy then set s=1 .
/List of jobs/ && s{print $NF;s=""}' Here command check for line which contains List of jobs & s presents. If both the conditions satisfy then print $NF and set s=""(null).

Hope this helps you:)

1 Like
11

Hi Pamu,
Thanks it works but there is litl issue.
I tested it over this file

 crab:  ExitCodes Summary
 >>>>>>>>> 568 Jobs with Wrapper Exit Code : 0 
  List of jobs: 1-95,97-114,116-121,123,125-129,131-159,161-184,186,188-191,194,199-206,208-230,232-233,235-236,238-239,241,243-246,248-250,252-255,258-264,266-280,282-288,290,292-296,298-300,302-304,306-312,314-316,318-319,321-373,375-380,382-458,460-500,502-511,513-516,519-524,526-531,533-574,576-597,599-613 
  See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 1 Jobs with Wrapper Exit Code : 8001 
  List of jobs: 381 
  See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 1 Jobs with Wrapper Exit Code : 5001
  List of jobs: 381,45,6,78,9,0,6,5,3-4
  See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

following is the output"

  $ awk '/Jobs with Wrapper/ && $NF != 0{s=1}   /List of jobs/ && s{print $NF;s=""}' file.txt        
381
381,45,6,78,9,0,6,5,3-4

There is litl problem, I need no in the following orders to execute other command on them:

381,45,6,78,9,0,6,5,3-4

emily

12

like this..?

$ awk '/Jobs with Wrapper/ && $NF != 0{s=1}   /List of jobs/ && s{if(p){p=p","$NF}else{p=$NF};s=""}END{print p}' file
381,381,45,6,78,9,0,6,5,3-4

or like this..?

$awk '/Jobs with Wrapper/ && $NF != 0{s=1}   /List of jobs/ && s{if(p){p=p","$NF}else{p=$NF};s=""}END{
num=split(p,arr,",");
for(i=1;i<=num;i++){
if(!x[arr]++){if(FV){FV=FV","arr}else{FV=arr}}
}}END{print FV}' file
381,45,6,78,9,0,5,3-4

and as per your specified output.(but not clear what is your idea behind this)

$ awk '/Jobs with Wrapper/ && $NF != 0{s=1}   /List of jobs/ && s{if(p && $NF !~ p){p=p","$NF}else{p=$NF};s=""}END{print p}' file
381,45,6,78,9,0,6,5,3-4
13

Would this provide what you need:

awk '/Wrapper/ {tmp=$NF} /List/ && tmp {print tmp, $4}' file
8001 381
5001 381,45,6,78,9,0,6,5,3-4

If not, pls. provide an exact output sample derived from your above input.

Alternatively, if the "List of jobs" always is the next line after "Wrapper" line:

awk '/Wrapper/ && $NF {printf "%s: ",$NF; getline; print $4}' file
14

Hi RudiC,
I needed the output from file.txt

 crab:  ExitCodes Summary
 >>>>>>>>> 568 Jobs with Wrapper Exit Code : 0 
  List of jobs: 1-95,97-114,116-121,123,125-129,131-159,161-184,186,188-191,194,199-206,208-230,232-233,235-236,238-239,241,243-246,248-250,252-255,258-264,266-280,282-288,290,292-296,298-300,302-304,306-312,314-316,318-319,321-373,375-380,382-458,460-500,502-511,513-516,519-524,526-531,533-574,576-597,599-613 
  See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 1 Jobs with Wrapper Exit Code : 8001 
  List of jobs: 381 
  See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

crab:  ExitCodes Summary
 >>>>>>>>> 1 Jobs with Wrapper Exit Code : 5001
  List of jobs: 380,45,6,78,9,0,6,5,3-4
  See https://twiki.cern.ch/twiki/bin/view/CMS/JobExitCodes for Exit Code meaning

as following:

 381,380,45,6,78,9,0,6,5,3-4

i.e every followed digits except the digits following "0" in 'with Wrapper Exit Code : 0'

thanks

---------- Post updated at 04:09 AM ---------- Previous update was at 04:09 AM ----------

Hi Pamu:)
Second is what i need.

Thanks
pooja

---------- Post updated at 04:28 AM ---------- Previous update was at 04:09 AM ----------

Cant we assign it to some variable? so that i can use that later.
for example, mycomplee task is fllowing:

 
geThose no, which your code pass
381,381,45,6,78,9,0,6,5,3-4

then perform command like this,

crab -resubmit 381,381,45,6,78,9,0,6,5,3-4

/i have to pass those no to this command. :frowning:

15

Yes you can. Just do this..

Var_no=$(awk '/Jobs with Wrapper/ && $NF != 0{s=1}   /List of jobs/ && s{if(p){p=p","$NF}else{p=$NF};s=""}END{print p}' file)

#and then use in  your commands..

crab -resubmit $Var_no
16

If you just want to resubmit your jobs, try

$(awk 'BEGIN {printf "crab -resubmit "} /Wrapper/ && $NF {getline; printf "%s%s", cm, $4; cm=","} END {print""}' file)

This will collect your job no.s and run crab to resubmit them.

BTW, would have been MUCH easier had you specified your problem correctly and entirely from the start...

17

Thanks pamu and RudiC
I am sorry for the confusion.

But wondering wont it work like this:

 crab ntuplize_crab -status -c $SAMPLE >& $SAMPLE"_status.log" &
 jobNo=$(awk '/Jobs with Wrapper/ && $NF != 0{s=1}   /List of jobs/ && s{if(p){p=p","$NF}else{p=$NF};s=""}END{print p}' $SAMPLE"_status.log")
 echo ' crab ntuplize_crab -resubmit' $jobNo

Sams if for your script RudiC. It is not accepting this '$SAMPLE"_status.log'...????

18

Use

 crab ntuplize_crab -status -c $SAMPLE >& $SAMPLE"_status.log" &

As this is submitted in background, wait for it to finish. Then launch

 $(awk 'BEGIN {printf "crab -resubmit "} /Wrapper/ && $NF {getline; printf "%s%s", cm, $4; cm=","} END {print""}' $SAMPLE"_status.log" )

Please be aware that if your file $SAMPLE"_status.log" does NOT conform to what you specified before, all of this won't work!

19

ohh, yes that was the issue..the earlier command was running background.
Cant I ask that when that $SAMPLE"_status.log" is done, then only execute the "crab -submit command "?

Else how much sleep time should I give..?? there is not fix time..

20

How do you tell that log file is done?
You could wait for your bg process. If that's not satisfactory, you'll need a loop anyway to check for the log file, the PID, or what have you. Search this forum for posts regarding sth. like "sync with background processes"