parsing a file by line

Shell Programming and Scripting
1

I'm trying to make a script that will read variables line by line from a flatfile

i.e.

$ cat testfile
dbfoo sfoo prifoo poofoo bfoo osfoo
dbfoo2 sfoo2 prifoo2 poofoo2 bfoo2 osfoo2
$

The first pass of the script through the flatfile I want:

$1=dbfoo $2=sfoo $3=prifoo $4=poofoo $5=bfoo $6=osfoo

next pass I want:

$1=dbfoo2 $2=sfoo2 ... etc

I've tried several ideas:

doing a 'for i in `cat testfile`' runs a loop for each tab separated item

trying to use awk to get a line reads my variable literally...#PASSNFO=`cat $1 | awk 'NR=="$PASSNO"'` come out on the command line as #PASSNFO=`cat $1 | awk 'NR==$PASSNO'`

and I tried using shift to push to the next 6, but that seems to be used for interpreting command line arguments and not a file.

EDIT: trying the below from ksh seems to do nothing either, it won't even echo the output of $LINE
function while_read_LINE
{
cat $1 | while read LINE
do

2 
awk '{ for (i=1; i <= NF ; i++) {printf("$%d=%s ", i, $i) }
         printf("\n")  } ' inputfilename

The default names for fields in awk are:
$0 - whole line
$1 - field 1
$2 - field 2
.........
NF is the number of fields on the line.
You can substitute a variable like "i" for numbers of fields: "$i" is the field
that when i is evaluated, that field number is referenced.

3

Thanks for the reply Jim, I apologize in advance for my newbishness

so the result I get from the above is:

# ./awktest.sh testfile
$1=dbfoo $2=sfoo $3=prifoo $4=poofoo $5=bfoo $6=osfoo
$1=dbfoo2 $2=sfoo2 $3=prifoo2 $4=poofoo2 $5=bfoo2 $6=osfoo2
#

but I'm not sure how I would reference (for example) $4 on line 2 without getting $4 on line 1

4

I'm not sure I'm entirely clear on what you want to accomplish, so please feel free to reply so we can tweak the commands as needed. It might help to know one more standard awk variable, FNR. This is the line number. Here are a couple of examples of its use, with the data file being piped in to simplify the syntax a hair. In general, though, I'd suggest calling the data file as part of the command.

> cat testfile | awk 'FNR == "1"'
dbfoo sfoo prifoo poofoo bfoo osfoo

> cat testfile | awk 'FNR == "2"'
dbfoo2 sfoo2 prifoo2 poofoo2 bfoo2 osfoo2

> cat testfile | awk 'FNR == "2" || FNR == "4"'
dbfoo2 sfoo2 prifoo2 poofoo2 bfoo2 osfoo2
dbfoo4 sfoo4 prifoo4 poofoo4 bfoo4 osfoo4

The above shows how to print a specific line(s), identified by line number(s). Oh, BTW, I extended your original testfile to 5 lines.

Now, suppose that you want to print only field 4 of line 1. You can use:

> cat testfile | awk 'FNR == "2" {print $4}'
poofoo2

I hope that's a start. Reply as needed... I'm sure we can scrape together something that'll get the job done.

5

You should not use $1 as a variable name in shell - it is already reserved
That said you can create an array:

#/bin/ksh

while read line
do
     set -A array $line
     # at this point you have array elements that you reference with a number
     # ${#array[*]}   number of fields or elements in the array.
     let i=0
     while [[ $i -lt  ${#array[*]} ]]
     do
           echo " field $i = ${array} \c"
           i=$(( i + 1 ))
     done
     echo " "
done < inputfile

arrays count from zero, and go from zero to one less than the number of elements.
echo "...\c" supresses a new line, so all of the output is one one line.

6

Thanks for the post Tree,

allrighty, the end goal is is to create policies in netbackup, I'm going to be creating a LOT of policies very soon to where a script will make life easier.

Below is what I have as it's been edited repeatedly (so there's lots of stuff I commented out rather than delete until I figure out what works, possibly not considered the best method of hammering things out) All the commands are just echo'd right now as I don't want it to do anything until I'm sure it's submitting correct information.

The stuff in the while statement is what needs to happen for each line. Since which line needs to be read from the file advances with each pass I can't hard code an awk '[NR|FNR]==X' and putting a $PASSNO in place of 'X' didn't seem to work.

$ cat RMAN_policy.sh
#! /bin/ksh
#
# set command line variables
#
PATH=/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/netbackup/bin/goodies/support:/usr/openv/volmgr/bin:/usr/openv/volmgr/bin/goodies:/opt/VRTSvxfs/sbin:$PATH
#USAGE="USAGE: RMAN_policy.sh <filename> "

#shift `expr $OPTIND - 1`

#INSTANCE=$1
#SERVER=$2
#PRIORITY=$3
#VPOOL=$4
#BPATH=$5
#OSV=$6

#PASSES=`wc -l $1 | awk '{ print $1 }'`
#PASSNO=0

function while_read_LINE
{
cat $1 | while read LINE
do

echo "$LINE"

INSTANCE=`echo "$LINE" | awk '{ print $1 }'`
SERVER=`echo "$LINE" | awk '{ print $2 }'`
PRIORITY=`echo "$LINE" | awk '{ print $3 }'`
VPOOL=`echo "$LINE" | awk '{ print $4 }'`
BPATH=`echo "$LINE" | awk '{ print $5 }'`
OSV=`echo "$LINE" | awk '{ print $6 }'`

echo "$INSTANCE"
echo "$SERVER"
echo "$PRIORITY"
echo "$VPOOL"
echo "$BPATH"
echo "$OSV"

echo bppolicynew ora_"$INSTANCE"
echo bpplinfo ora_"$INSTANCE" -set -ut -active -blkincr 0 -collect_tir_info 0 -compress 0 -crossmp 1 -disaster 0 -encrypt 0 -follownfs 0 -multiple_streams 1 -policyjobs 0 -pool "$VPOOL" -priority "$PRIORITY" -pt Standard -residence foo -chkpt 1 -chkpt_intrvl 30
echo bpplsched ora_"$INSTANCE" -add Full
echo bpplschedrep ora_"$INSTANCE" Full -cal 0 -freq 86400 -mpxmax 1 -rl 5 -st FULL
echo bpplinclude ora_"$INSTANCE" -add "$BPATH"/*
echo bpplclients ora_"$INSTANCE" -add "$SERVER" Solaris "$OSV"

#PASSNO=`expr $PASSNO + 1`

#echo END PASS "$PASSNO"

#INSTANCE=$INSTANCE; shift 6
#SERVER=$SERVER; shift 6
#PRIORITY=$PRIORITY; shift 6
#VPOOL=$VPOOL; shift 6
#BPATH=$BPATH; shift 6
#OSV=$OSV; shift 6
done
}
$

7

Thanks for the help guys! I managed to get something working, though it was a little hectic and there's probably more graceful way of dealing with a little problem I had involving the script running once for each "word" in the flatfile instead of once for each line.

Here's the practical example should someone be interested in the future.

$ cat array.sh
#/bin/ksh

TESTFILE=$1

while read line
do
set -A array $line
# at this point you have array elements that you reference with a number
# ${#array
[]} number of fields or elements in the array.
let i=0
while [[ $i -lt ${#array
[]} ]]
do

INSTANCE=${array[0]}
i=$(( i + 1 ))
SERVER=${array[1]}
i=$(( i + 1 ))
PRIORITY=${array[2]}
i=$(( i + 1 ))
VPOOL=${array[3]}
i=$(( i + 1 ))
BPATH=${array[4]}
i=$(( i + 1 ))
OSV=${array[5]}
i=$(( i + 1 ))

#echo $INSTANCE
#echo $SERVER
#echo $PRIORITY
#echo $VPOOL
#echo $BPATH
#echo $OSV

echo bppolicynew ora_"$INSTANCE"
echo bpplinfo ora_"$INSTANCE" -set -ut -active -blkincr 0 -collect_tir_info 0 -compress 0 -crossmp 1 -disaster 0 -encrypt 0 -follownfs 0 -multiple_streams 1 -policyjobs 0 -pool "$VPOOL" -priority "$PRIORITY" -pt Standard -residence \"foo_bar\" -chkpt 1 -chkpt_intrvl 30
echo bpplsched ora_"$INSTANCE" -add Full
echo bpplschedrep ora_"$INSTANCE" Full -cal 0 -freq 86400 -mpxmax 1 -rl 5 -st FULL
echo bpplinclude ora_"$INSTANCE" -add "$BPATH"/*
echo bpplclients ora_"$INSTANCE" -add "$SERVER" Solaris "$OSV"

#echo ENDPASS

 done

done < $TESTFILE
$