Find the nth value in a list

Shell Programming and Scripting
1

Hello,

I have some code that searches file names, sorts them on a field in the name, and returns the top value.

# list the contents, strip off the path, and sort on field 3 as a number, returns top value
TOP_OUTCOME=$(ls  './'$SET_F'/'$FOLD'/'$FOLD'_anneal/'$C_PARAMS'/'$A_SET'/'*'out.txt' | \
              awk 'BEGIN {FS="/"} {print $7}' | \
              sort -t_ -k 3 -n | \
              head -n 1)

The file names look like,

V_se_69.36_E1_f5_r09_1-ON-0.25_S3C_v1_36.35.1.out.txt
V_se_69.62_E195_f5_r17_195-ON-0.25_S3C_v1_36.35.1.out.txt
V_se_70.14_E254_f5_r06_254-ON-0.25_S3C_v1_36.35.1.out.txt
V_se_70.41_E230_f5_r08_230-ON-0.25_S3C_v1_36.35.1.out.txt
V_se_71.00_E150_f5_r42_150-ON-0.25_S3C_v1_36.35.1.out.txt

The code sorts on the real in the third underscore delimited field. There are circumstances where I may not actually want the top value, but I need to read the file name to find out. In this case, since the value of the 4th field is E1 (the 1 is the important thing), I would want to look at the next value.

I guess I could collect the top 10 or so and loop though the file names, but I was also considering something like grabbing the top value as above, testing a filed (I will want the top value 99% of the time so most will pass the test), and then grabbing the 2nd value in the event of a fail, testing that name, etc.

It seems like grabbing a bunch,

TOP_OUTCOME=$(ls  './'$SET_F'/'$FOLD'/'$FOLD'_anneal/'$C_PARAMS'/'$A_SET'/'*'out.txt' | \
              awk 'BEGIN {FS="/"} {print $7}' | \
              sort -t_ -k 3 -n | \
              head -n 10)

and testing fields in a while loop would be the most efficient method, but I thought I would check. There are times where I would need to use tail instead of head with the same code, so that complicates things some I guess.

LMHmedchem

2

How about

TOP_OUTCOME=$(cd './'$SET_F'/'$FOLD'/'$FOLD'_anneal/'$C_PARAMS'/'$A_SET'/'; ls *'out.txt' |sort -t_ -k3,3n | awk -F_ '$4 == "E1" {next} {print; exit}'

?

3

If you get the list in a file, you can use sed to get a particular line:-

line=10
sed -n ${line}p file

I'm not sure if this is the best way or even truly supported, but it works for me. The curly brackets (are they braces?) are critical to keep the variable name clear from the required trailing p

I hope that this helps,

Regards,
Robin

4

Is this iterative, meaning if my files names look like,

S_se_41.30_E1_f5_r34_1-ON-0.25_S3B_v1_36.35.1.out.txt
S_se_59.22_E1_f5_r05_1-ON-0.25_S3B_v1_36.35.1.out.txt
S_se_61.42_E1375_f5_r47_1375-ON-0.25_S3B_v1_36.35.1.out.txt

is this going to find the third file since the first two are E1?

In the end, I will probably need to do something like <= E5, meaning I will have to separate the E from the integer so I can evaluate it numerically.

This code works,

MIN_TO_ACCEPT='5'
SORT_ORDER='head'
NUMBER_TO_PROCESS='10'

# get list of $NUMBER_TO_PROCESS best outcomes
TOP_OUTCOMES=($(ls  './'$SET_F'/'$FOLD'/'$FOLD'_anneal/'$C_PARAMS_F'/'$A_SET'/'*'out.txt' | \
                           awk 'BEGIN {FS="/"} {print $7}' | \
                           sort -t_ -k 3 -n | \
                           $SORT_ORDER -n $NUMBER_TO_PROCESS))

# reset (code is in a loop)
unset OUTCOME_FIELDS
OPTIMUM_EPOCH=0
COUNTER=0

# keep looping until a value gt $MIN_TO_ACCEPT is found
while [ "$OPTIMUM_EPOCH" -lt "$MIN_TO_ACCEPT" ]
do
   # store file name
   TOP_OUTCOME=${TOP_OUTCOMES[$COUNTER]}
   # parse OUTCOME_FILE name to find optimum at $3
   IFS='_' read -a  OUTCOME_FIELDS <<< "$TOP_OUTCOME"
   OPTIMUM=${OUTCOME_FIELDS[3]}
   #remove leading E
   OPTIMUM=${OPTIMUM_EPOCH#E}
   RAND_SET=${OUTCOME_FIELDS[5]}
   echo "RAND_SET" $RAND_SET "OPTIMUM" $OPTIMUM
   (( COUNTER++ ))
   # if counter gets past number to process, quit with error
   if [ "$COUNTER" -eq "$NUMBER_TO_PROCESS" ]; then
      echo "no values greater than" $MIN_EPOCHS "were found for" $SET $ANNEALING_SET $FOLD
      exit
   fi
done

echo "SET" $SET "FOLD" $FOLD "TOP_OUTCOME"
echo $TOP_OUTCOME

This is relatively long hand and can be cleaned up somewhat but works fine for head sorting. Since most of the time I will just want the first file, it does not seem like an efficient solution.

Also, when I use tail instead of head, it seems like I get the top 10 files, but they are in reverse order, meaning that the 10th best outcome is processed first. Am I not using tail correctly?

LMHmedchem

5

Yes . Why don't you just try it?

make the awk part read

awk -F_ '0+substr($4,2) <= 5 {next} {print; exit}'