Assigning variable using script

emily · September 30, 2012, 8:39am

Hi all,
I have to write script to make my usual job easy. Basically it involve reading a output of following pattern:

crab:  ExitCodes Summary
 >>>>>>>>> 45 Jobs with Wrapper Exit Code : 0 
	 List of jobs: 607-608,616,619,628,630,632,638-640,646-647,658,666,673-674,680,687,692,699-700,702,708,720,725,731,735,737,740,743,748,750,752,754,764,769,771-772,778,781,786,790,796,799,1114 
	See https:// for Exit Code meaning

crab:   1161 Total Jobs 
 >>>>>>>>> 41 Jobs Submitted 
	List of jobs Submitted: 5,101,106,255,280,288,310,313-314,401-403,405,718,723,766,776-777,785,797,986-987,991,997,999,1053-1054,1057,1065,1069,1076,1083,1092,1095,1106,1115,1119-1120,1127-1128,1137 
 >>>>>>>>> 520 Jobs Scheduled 
	List of jobs Scheduled: 1-2,7-8,10,13,15-18,22-23,26,31-32,34-35,39-40,42,44-47,51,54-58,61,63,66-69,72,74,76,78-80,86-88,92-97,100,103,114-115,117,119,125,130-131,138,140,144,147,149-150,158,161-163,165,169-170,172-173,176,178-180,184-187,189,192,196,198,204,208,218,221,225,228,231,236,239,243-244,246,248,250-251,253-254,256,258,261,264-265,267,270,276-279,281-282,284,287,299-300,302-303,309,311,316-321,323-325,328,332,336,338,341,345,349,352-355,358,360,363-365,371-372,376,378-380,382,385,388-390,395-398,404,406,409,411,413,415-416,418-419,422-423,426-427,431,438-439,441-442,444-446,448,451,453,455-460,462,468,476,478-490,494-495,498,502,504-505,507,513,517,519-520,522,524,526,531,533,537-538,542-543,545-547,550,555,557-558,561-562,564-566,568,573,575-576,583,589,592-593,595,598-599,601-606,609,611,613,615,617-618,620-623,625-627,629,631,634-636,641-643,645,648-657,659-661,663-665,667-672,676-679,681-686,688-689,691,693-698,701,703-707,709-710,712-717,719,721-722,724,726-730,732-734,736,738-739,741-742,744-747,749,751,753,755-760,762-763,765,767-768,773-775,779-780,782-784,787-789,791-795,798,800,803-804,806,808-809,816,818-819,824,834-835,842-843,845,849,851,855,859,861,863-864,867,873,877-879,885,888-889,893,896,898,901,906,917-920,923-924,929,931-934,937,941,945,950-951,953,958-961,963,966,968-969,972,974-975,977,980,982,984-985,988-989,996,998,1005-1007,1009-1010,1013,1018,1022,1024-1025,1027,1030,1032,1035,1039-1040,1043,1045-1046,1048,1052,1055-1056,1059,1061-1062,1070,1072-1075,1077-1078,1081,1084-1085,1087-1090,1094,1096-1097,1099,1103-1105,1109-1110,1112,1116-1118,1121,1129,1131,1140,1152-1153 
 >>>>>>>>> 555 Jobs Aborted
	You can resubmit them specifying JOB numbers: crab -resubmit <List of jobs>
	List of jobs: 3-4,6,9,11-12,14,19-21,24-25,27-30,33,36-38,41,43,48-50,52-53,59-60,62,64-65,70-71,73,75,77,81-85,89-91,98-99,102,104-105,107-113,116,118,120-124,126-129,132-137,139,141-143,145-146,148,151-157,159-160,164,166-168,171,174-175,177,181-183,188,190-191,193-195,197,199-203,205-207,209-217,219-220,222-224,226-227,229-230,232-235,237-238,240-242,245,247,249,252,257,259-260,262-263,266,268-269,271-275,283,285-286,289-298,301,304-308,312,315,322,326-327,329-331,333-335,337,339-340,342-344,346-348,350-351,356-357,359,361-362,366-370,373-375,377,381,383-384,386-387,391-394,399-400,407-408,410,412,414,417,420-421,424-425,428-430,432-437,440,443,447,449-450,452,454,461,463-467,469-475,477,491-493,496-497,499-501,503,506,508-512,514-516,518,521,523,525,527-530,532,534-536,539-541,544,548-549,551-554,556,559-560,563,567,569-572,574,577-582,584-588,590-591,594,596-597,600,610,612,614,624,633,637,644,662,675,690,711,761,770,801-802,805,807,810-815,817,820-823,825-833,836-841,844,846-848,850,852-854,856-858,860,862,865-866,868-872,874-876,880-884,886-887,890-892,894-895,897,899-900,902-905,907-916,921-922,925-928,930,935-936,938-940,942-944,946-949,952,954-957,962,964-965,967,970-971,973,976,978-979,981,983,990,992-995,1000-1004,1008,1011-1012,1014-1017,1019-1021,1023,1026,1028-1029,1031,1033-1034,1036-1038,1041-1042,1044,1047,1049-1051,1058,1060,1063-1064,1066-1068,1071,1079-1080,1082,1086,1091,1093,1098,1100-1102,1107-1108,1111,1113,1122-1126,1130,1132-1136,1138-1139,1141-1151,1154-1161 
 >>>>>>>>> 45 Jobs Cleared
	List of jobs: 607-608,616,619,628,630,632,638-640,646-647,658,666,673-674,680,687,692,699-700,702,708,720,725,731,735,737,740,743,748,750,752,754,764,769,771-772,778,781,786,790,796,799,1114 

crab:  You can also follow the status of this task on :
	CMS Dashboard: http://dashb-cms-

I would like to write a script which look for the common text like "Aborted" , ("Scheduled", "Submitted") and some VARIABLE can be assigned to the rest of the no appearing below so that I could be able to perform some command on those no.

Thanks in advance,
emily

bartus11 · September 30, 2012, 9:00am

Try:

grep "jobs Scheduled" file | sed 's/.*: //;s/,/\n/g' | while read job; do 
  echo $job
done

Change "echo $job" to whatever command you need. To get Aborted and Submitted jobs, just change grep argument.

emily · September 30, 2012, 10:05am

Hi,
Thanks for the promt reply.
It work fine for "Scheduled", but for "Aborted" it did not work.

However, for the output, I will need the no in the same pattern as they are in the Infile like, 1-2,3-4,5, 8,

 grep "jobs Scheduled" dataB_status.log  | sed 's/.*: //' | tr ',' '\n' | while read job; do echo $job; done
1-2
7-8
10
13
15-18
22-23
26
31-32
34-35
39-40
42

Also When I tried using string "Aborted" as following, it did not give the required output

  
$grep "jobs Aborted" dataB_status.log  | sed 's/.*: //' | tr ',' '\n' | while read job; do echo $job; done
$ grep "Jobs Aborted" dataB_status.log  | sed 's/.*: //' | tr ',' '\n' | while read job; do echo $job; done
>>>>>>>>> 47 Jobs Aborted

Infile

  >>>>>>>>> 47 Jobs Aborted
	You can resubmit them specifying JOB numbers: crab -resubmit <List of jobs>
	List of jobs: 188,190,195,235,237-238,242,247,262-263,268,329,333,357,362,366,373,381,386,391,408,412,414,425,433,435,440,449,463,466,475,493,503,511,515,525,529,535-536,560,563,570,572,574,580,584,633

the_gripmaster · September 30, 2012, 10:30am

-A 2 tell grep to get the +2 line from the string "Jobs Aborted" because that is the line containing the numbers.

grep -A 2 "Jobs Aborted" file | sed 's/.*: //;s/,/\n/g' | while read job; do    
  echo $job
done

emily · September 30, 2012, 10:35am

Hi,
Thanks for the reply. It did work.
But again the format of the output is not the same as I wish.
I would like it to be in a row with no space and with "," to seperate:

1,2-3,6,7,500-600

Also, please let me know how does it command work. That will be learning part for me

Thanks again,

the_gripmaster · September 30, 2012, 10:48am

emily:

Hi,
Thanks for the reply. It did work.
But again the format of the output is not the same as I wish.
I would like it to be in a row with no space and with "," to seperate:
1,2-3,6,7,500-600
Also, please let me know how does it command work. That will be learning part for me

Thanks again,

I could explain but the sed is very difficult to explain. You need to learn regular expressions if you don't know that already to understand the sed commands.

I guess this command will give you the output you are looking for

grep -A 2 "Jobs Aborted" file | sed 's/.*: //' | grep -E '^[0-9]+'

emily · September 30, 2012, 11:14am

Hi,
It did work fine.
I have used sed for the simple task like renaming some common text in the file.
But whenever it comes to non-regular expression use, I am complete naive for it.

sed -e "s/oldname/newname/" file > file1

Also what does this command does

 grep -E '^[0-9]+'

?
It seems to aligned them, but how it is suppose to be used?

thanks again,

the_gripmaster · September 30, 2012, 11:40am

emily:

Hi,
It did work fine.
I have used sed for the simple task like renaming some common text in the file.
But whenever it comes to non-regular expression use, I am complete naive for it.
sed -e "s/oldname/newname/" file > file1
Also what does this command does
 grep -E '^[0-9]+' 
?
It seems to aligned them, but how it is suppose to be used?

thanks again,

 grep -E '^[0-9]+'

?

finds the line starting(^) with any digit ([0-9])

Did my previous command serve your purpose? Outputs like 222,22-33,44,..... in one line.

emily · September 30, 2012, 11:57am

Hi,
This command

 grep -A 2 "Jobs Aborted" file | sed 's/.*: //' | grep -E '^[0-9]+'

serve my purpose.

Thanks,

---------- Post updated at 10:57 AM ---------- Previous update was at 10:44 AM ----------

Hi, in the bash script below

DATA=/11/store/data/

if [ "$2" = "A1" ]; then
FINALPATH=$DATA/Run2012A/DoubleElectron/AOD/PromptReco-v1
elif [ "$2" = "A2" ]; then
FINALPATH=$DATA/Run2012A/DoubleElectron/AOD/PromptReco-v2
elif [ "$2" = "A4" ]; then
FINALPATH=$DATA/Run2012A/DoubleElectron/AOD/PromptReco-v4
elif [ "$2" = "A5" ]; then
fi

GetFileName() {
    echo '>>>>Check, DataPATH is: '$FINALPATH
    pause

    find $FINALPATH -name "*root" > $OUTPUTFILE
    echo  $2"_dataFile.list"
    cp $OUTPUTFILE $ANALYZERPATH/File
    echo '>>> Copied OutPut : '$ANALYZERPATH/File/$OUTPUTFILE
    echo '===========================DONE======================== '
    echo '                                            '
}

elif  [ "$1" = "find" ]; then
        GetFileName
fi

I would call it like

 ./script find A1 A1_datafile.list

I want $OUTPUTFILE to be defined in the function as like,
$2"_datafile.list" . But it failed to work.

Any hint?

thanks

the_gripmaster · September 30, 2012, 12:32pm

I am a bit confused with the code. elif instead of if in the ending section?

DATA=/11/store/data/

if [ "$2" = "A1" ]; then
  FINALPATH=$DATA/Run2012A/DoubleElectron/AOD/PromptReco-v1
elif [ "$2" = "A2" ]; then
  FINALPATH=$DATA/Run2012A/DoubleElectron/AOD/PromptReco-v2
elif [ "$2" = "A4" ]; then
  FINALPATH=$DATA/Run2012A/DoubleElectron/AOD/PromptReco-v4
elif [ "$2" = "A5" ]; then
fi

GetFileName() {
    echo '>>>>Check, DataPATH is: '$FINALPATH
    pause

    OUTPUTFILE=$(echo $2"_datafile.list")   # is this what you need?
    
    find $FINALPATH -name "*root" > $OUTPUTFILE
    echo  $2"_dataFile.list"
    cp $OUTPUTFILE $ANALYZERPATH/File
    echo '>>> Copied OutPut : '$ANALYZERPATH/File/$OUTPUTFILE
    echo '===========================DONE======================== '
    echo '                                            '
}

if  [ "$1" = "find" ]; then  # correction
        GetFileName
fi

emily · September 30, 2012, 3:26pm

opps sorry./.its copy paste error..

but the original code is all fine..

---------- Post updated at 01:15 PM ---------- Previous update was at 11:35 AM ----------

No Reply..??? should not be difficult..

---------- Post updated at 01:53 PM ---------- Previous update was at 01:15 PM ----------

the_gripmaster:

-A 2 tell grep to get the +2 line from the string "Jobs Aborted" because that is the line containing the numbers.
grep -A 2 "Jobs Aborted" file | sed 's/.*: //;s/,/\n/g' | while read job; do    
  echo $job
done

Hi again
I would like to modify it little.
For example, I am suppose to peform same task for some of the GREP:
1.) jobs Aborted
2.) Jobs terminated
3.) 271 Jobs with Wrapper Exit Code : 0

I wonder if some array can be formed with these strings and I could use that?

---------- Post updated at 02:26 PM ---------- Previous update was at 01:53 PM ----------

Hi,
For the given string

crab:  ExitCodes Summary
 >>>>>>>>> 271 Jobs with Wrapper Exit Code : 0
         List of jobs: 3,12,14,20-21,24-25,27,29-30,33,36,38,41,52-53,59-60,62,64,71,73,75,77,82-84,89-91,98-99,102,104,107-110,112,116,120-123,126-1\
29,132-134,136,139,141-143,145-146,148,151-155,159-160,164,168,181-182,191,197,201-202,205,209-217,219-220,222,224,226-227,229-230,232-234,252,257,25\
9-260,266,269,271-275,283,285-286,289,292-293,295,297-298,304-307,315,326-327,330-331,335,340,343-344,346-348,350-351,356,359,361,367-370,374-375,377\
,383-384,392,400,410,421,424,429-430,436,443,447,450,452,454,465,467,469,471,473,492,499-500,508,512,516,521,528,548-549,553,556,559,567,569,571,582,\
586-588,590,594,596-597,600,607-608,612,614,616,619,624,628,630,632,637-640,644,646-647,658,662,666,673-675,680,687,692,699-700,702,708,720,725,731,7\
35,737,740,743,748,750,752,754,764,769-772,778,781,786,790,796,799,805,807,810,813,821,823,825,827,829,837-841,846-848,850,852-854,856-858,870-871,87\
4-876,880,1114

I GREP the following "Wrapper Exit Code : 0". But while processing some other commands on the no's, it added the extra '0' as well..????

Thanks,