Problem passing a search pattern to AWK inside a script loop

Shell Programming and Scripting
1

Learning, stumbling! My progress in shell scripting is slow. Now I have this doubt:

I have the following file (users.txt):

AU0909,on
AU0309,off
AU0209,on
AU0109,off

And this file (userson.txt)

AU0909
AU0209
AU0109
AU0309

I just want to set those users on userson.txt to "off" in the users.txt file. String substitution.
This is an exercise with AWK, I already know other methods to change it but I'm interested in AWK.

Now, I tried this in a script:

Script1.sh

 
AuxLine="AU0909"
awk -v idusr=${AuxLine} 'BEGIN { FS = "," } ; $0 ~ idusr { print $2}' users.txt

That works for that given string, but I for hours I can't see why this does not work:

while read AuxLine
do
  awk -v idusr=${AuxLine} 'BEGIN { FS = "," } ; $0 ~ idusr { print $2}' users.txt
done < userson.txt

The variable is passed correctly (read from usersons.txt) as I already tried printing it with no search pattern and it did print. I just can't see how the loop is preventing this simple command to work.

Any clues?

2

First something really basic. In the second part you are using a file with a different name? Does it exist?

3

Which file is different?

I was typing myself the names of the files on the post so if there is any mistake with it I am sure the files do exist (they're being read properly)

4

I mean users.txt and UserUpdateList . The two examples operate on different files.

5

Oh yes, users.txt = UserUpdateList.

I changed the name in the post just to make it less confusing. I'll edit now.

But to add somethig: I just discovered this:

while read AuxLine
do
   echo $AuxLine "is it this?: AU0922"
   awk -v idusr=${Auxa} 'BEGIN { FS = "," } ; $0 ~ idusr { print $4}' users.txt
done < userson.txt

That echo DOES NOT show the value of $AuxLine on the result of the script. the output is:

 is it this?: AU0922
 is it this?: AU0922
 is it this?: AU0922
 is it this?: AU0922
         ...

However, if I echo the value alone (echo $AuxLine) it does show.

Maybe this is related to my issue?

6

This is strange. What OS are you using and what shell?
Is userson.txt exactly like in you sample?
If you are on Solaris, use /usr/xpg4/bin/sh and /usr/xpg4/bin/awk instead of /bin/sh and awk

7

I am using ubuntu 10.10: #!/bin/bash

Scrutinizer , you are leading me closer to the answer :slight_smile:
The file userson.txt "looked" exactly like in the code I wrote, but it was generated like this:

grep "0 AU.*" usersin.txt | cut -d' ' -f2 >> userson.txt

The file usersin.txt :

 Issued: message 1
1 AU0922
1 AU0905
1 AU0912
1 AU0923
0 AU0922
0 AU0900
0 AU0901
0 AU0902

I just tried the previous codes with a manually generated userson.txt and IT DID work..

Any idea what is happening after that grep? My guess is some encoding issue..

8

Can you cat part of the generated userson.txt and pipe it through od -c ?

sed -n 1,9p userson.txt | od -c
9

Ok, removing CR made it work:

while read AuxLine
do
  user_search=`echo $AuxLine | tr -d "\r"`
  awk -v idusr=${user_search} 'BEGIN { FS = "," } ; $0 ~ idusr { print $4}' users.txt
done < userson.txt

I still don't understand why a CR was bothering nor why a file I create using VI does not have it while the same file created with grep does have.

Looks like there's much more to learn the basics :slight_smile:

10

That is because the original file came from a windows platform and was not transferred in a correct way to your unix platform. The testfiles you created correctly on your unix platform, so they do not contain the CR's.

11

Hey Scrutinizer, I didn't read your message when I last posted. Thanks for your help

The output of the cat was:

0000000   A   U   0   9   2   2  \r  \n   A   U   0   9   0   1  \r  \n
0000020   A   U   0   9   1   3  \r  \n   A   U   0   9   1   0  \r  \n
0000040   A   U   0   9   0   3  \r  \n
0000050
12

OK, there are the CR's.. Do you need to use a while loop by the way? You could also use only awk, for example, like so:

awk 'NR==FNR{A[$1];next}$1 in A{$2="off"}1' FS=, OFS=, userson.txt users.txt

once you converted userson.txt to unix format..

1 Like
13

Oh .. that's what I was looking for all along: I do not need the loop it's just that I'm at level 0 in shell scripting :wink: thanks again.

And BTW, the file was created in linux, but with PHP and formatted for html. Maybe the CR came from there?

In conclusion: when reading lines I should always get rid of CR?

14

OK, indeed, if the PHP is coded to produce DOS type files for use on Windows platforms, the CR's will be there too.. Normally you would not need to check or get rid of CR's on normal unix type files...

15

use this

awk -F"," '$2~/off/{print $1,$2}' filename