Read line and save fields as variables

Hej guys,
I am trying to read a csv file line by line, save it's fields as variables per line so I can use them as parameters and execute stuff.
I am new to shell scripting and was just strictly following a tutorial. Somehow my version seems to ignore the loop. Any help?

TY! :slight_smile:

#!/bin/bash
echo " I AM OUTSIDE THE LOOP"

infile="./calls.txt"

while IFS=',' read -r f1 f2
do
        echo " I AM INSIDE THE LOOP"

        foo=$f1
        bar=$f2

        #many program calls here

done < "$infile"

Execution is:

I AM OUTSIDE THE LOOP

What's inside ./calls.txt ? An empty file would just skip the loop.

does 'calls.txt' reside in the SAME directory your script is at? [hint: ./calls.txt]
How do you cal you script?
What's the content of 'calls.txt'? Could you share a snippet using code tags, please.

calls.txt is CSV file like:

foo1,bar1
foo2,bar2
foo3,bar3

where foo/bar are integers, strings or paths/to/things

I call the .sh with:

./HiCoPi.sh

And yes, calls.txt and the HiCoPi.sh are in the same folder

Strange, I get the desired out:

 I AM OUTSIDE THE LOOP
 I AM INSIDE THE LOOP
 I AM INSIDE THE LOOP
 I AM INSIDE THE LOOP

Could you post the output of cat -vet ./calls.txt using the code tags.

Maybe one of the commands in the loop tries to read from stdin? Then it reads from the loop's input; for example ssh does it; workaround is ssh -n ...
An extra descriptor can harden the while loop

while IFS=',' read -r f1 f2 <&3
do
  ...
done 3< "$infile"
1 Like

thanks so far for your replies. I kind of have the feeling too that it is somehow related to the calls.txt
I thought the minimal example would do the job, but let's give you a more real copy of what I have done:

my.sh :

#!/bin/bash
echo " I AM OUTSIDE THE LOOP"

infile="calls.txt"

while IFS=',' read -r f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12 f13 f14 f15 f16 f17 f18 f19
do
        echo " I AM INSIDE THE LOOP"

        my_file1=$f1
        species1=$f2
        chrom1=$f3
        my_file2=$f4
        species2=$f5
        chrom2=$f6
        binSize=$f7
        foo1=$f8
        foo2=$f9
        foo3=$f10
        foo4=$f11
        variance=$f12
        bar1=$f13
        bar2=$f14
        java_jar=$f15
        bar3=$f16
        bar4=$f17
        unit=$f18
        bar5=$f19

        # program calls here

done < "$infile"

my calls.txt is just a one liner right now, but later many lines according to the same format shall follow:

/path/to/my/file.txt,string1,string2,/path/to/my/other/file.txt,string3,string4,int1,string5,string6,int2,int3,int4,/path/to/a/third/file.txt,path/to/a/fourth/file.txt,/path/to/a/java.jar,anotherString,anotherString,anotherString,aFloatNumber

so this is exactly a renamed representation of the calls.txt. Sorry for the censorship, but in fact I am currently developing a software tool I am ofc not allowed to share everything in detail :confused:

I came up with the thoughts of maybe the paths messing it up? I 'll retry with quoting the paths. Other ideas?

EDIT: quoting didn't help :confused:

If you want to read 19 values from a file I have no idea why you don't read those values from the file instead of using unneeded intermediate variables:

#!/bin/bash
echo " I AM OUTSIDE THE LOOP"

infile="calls.txt"

while IFS=',' read -r my_file1 species1 chrom1 my_file2 species2 chrom2 binSize foo1 foo2 foo3 \
    foo4 variance bar1 bar2 java_jar bar3 bar4 unit bar5
do
        echo " I AM INSIDE THE LOOP"

        # program calls here

done < "$infile"

But, of course, neither the way you were doing it nor what I have shown above does anything that shows the values of any of the 19 variables read from line in calls.txt .

What makes you think your code or the code above is not working? Do you actually have any code in place of:

        # program calls here

that actually tries to print those values? The only thing you have shown us in this script that produces (or should produce) any output is the two echo statements (which shows us that despite you saying that calls.txt contains one line, it appears that your script is reading three lines from it).

the point is, I can't even echo variables.

I get

> bash HiCoPi.sh 
I AM OUTSIDE THE LOOP

It seems he never enters the while loop. this is why the stuff in the "program calls"
section never runs.

once again, please show the output of cat -vet ./calls.txt using code tags.

calls.txt:

/path/to/my/file.txt,string1,string2,/path/to/my/other/file.txt,string3,string4,int1,string5,string6,int2,int3,int4,/path/to/a/third/file.txt,path/to/a/fourth/file.txt,/path/to/a/java.jar,anotherString,anotherString,anotherString,aFloatNumber

If the output you showed us in post #11 was the output you get from the command:

cat -vet calls.txt

then the problem is that there is no <newline> character at the end of the file. The read command reads lines of input until it hits EOF, and without a <newline> character there is no line.

By definition, non-empty text files end with a <newline> character. By convention, a file with the .txt filename extension should be a text file.

If you are using an editor that creates files in DOS text file format (with <carriage-return><newline> two character line separators and no last line terminator), it is going to cause you headaches whenever you try to use those files on UNIX-like operating systems. Ffix your input file(s) so that they use UNIX <newline> single character line terminators. Then try our suggestions again.

NOTE that the text files containing the code we have suggested MUST also be UNIX format text files; not DOS format text files.

1 Like

Don Cragun Sir, you deserve a medal! That's it! I had no end of line character!

I used sublime text, like I always do for coding. Maybe the editor has some formatting default with the .txt ending, I don't know.
My solution: I put the same line from the former calls.txt into a file just called "calls" (no ending), created with nano.
Now I have a "$" as line ending character, somehow not intuitive for me, I was expecting "\n" but it works anyway. My <#program calls here> section gets executed. Fantastic!

:b::slight_smile: