Problem with reading file line-by-line, and outputting to a new file

Shell Programming and Scripting
1

Hi everyone. I realise this is probably a bit of a noob question, but I'm actually a C# developer working on a legacy system, and can't remember much unix.

I want to read from a pipe-delimeted file like formatted thusly:

idno|PRODUCT|Name|street town postcode|etc|etc|etc|etc
idno|PRODUCT|Name|street town postcode|etc|etc|etc|etc
idno|PRODUCT|Name|street town postcode|etc|etc|etc|etc

I want to copy each line to another file, dependant upon the PRODUCT existing in a string variable I'm holding in memory: "prod1|prod2|prod3" ($products in example below)

Therefore I was trying:

for line in `cat $temp_dir/$Filename | awk '{ print $0 } '`
do
prod=`echo "$line" | awk ' BEGIN {FS="|"} {print $2} '`
valid_prod=`echo $products | grep "$prod"`
if [ -n "$valid_prod" ] ; then
echo "$line" >> $new_file.txt
fi
done

however this line variable seems to separate based on spaces, creating the new file as:

idno|PRODUCT|Name|street
town
postcode|etc|etc|etc|etc

Is there anyway I can pass each line through as a single parameter, regardless of spaces?

Any help is greatfully appreciated,
Dave

2 
while read zf
   do
   prod=$(echo "$zf" | cut -d"|" -f2)
   valid_prod=$(echo "$products" | grep "$prod")
   if [ -n "$valid_prod" ]
      then
      echo "$zf" >>$new_file
   fi
done <$filename

Sometimes I get so confused using the ' and ` thus I prefer the above approach.

3

you can manipulate the IFS (internal field separator) - as an example - given that go.dat contains the lines with spaces

# create the array, without manipulating IFS, spaces will break the array
#
set -A lines $(cat go.dat)
for line in "${lines[@]}"; do
echo "'$line'"
done

# create the array, but first change the IFS (set it back - very important)
# after the array is set
#
oIFS=$IFS
IFS=${IFS##?}
set -A lines $(cat go.dat)
IFS=$oIFS

for line in "${lines[@]}"; do
echo "'$line'"
done

### go.dat ###
idno a|PRODUCT|Name|street town postcode|etc|etc|etc|etc
idno b|PRODUCT|Name|street town postcode|etc|etc|etc|etc
idno c|PRODUCT|Name|street town postcode|etc|etc|etc|etc

### end go.dat ###

i realize that does not help your problem (darkness fish already answered) so i thought i would throw my .02 in about IFS - since i have been bitten by that in the past - and it comes in handy quite often

4

Thanks very much for your help, both of you, it's greatly appreciated.

I was getting lost in a world of random unintelligible awk commands found via google.

5

As you read through this forum, you should get a flavor of the different ways to solve a problem. There are issues of
awk vs grep or cut
useless cat's
and many others. Some solutions may be more robust, while others just look good.

So, don't belittle poor little awk - it serves great purposes for certain database problems.