While read line from text file

Good Morning. I have a script that needs to execute a group of commands based on the value taken from a text file. I am using the “while read line” method.

1. How is the value from the line read from the text file communicated to the commands after the “do” command?
2. In an attempt to execute the script prior to this post , I got the error that the targetdir.txt file could not be opened. That file is in the directory from where the script is executed. So it is finding the text file, but it can’t be opened.

Thanks for any help you can provide.

targetdir.txt content, which is in a Unix directory

DATA1
PRNT1

while read line
do
     commands with the value read from the text file
done<targetdir.txt

Hello,

You've certainly got the beginnings of a functioning loop there, it would appear. So to phrase your code fragment in human language, the loop would essentially work like this:

"While there is still a line of data to be read from the file targetdir.txt, read that line of data from the file, and assign its contents to the variable $line"

So you would then refer to the currently-read line via the variable $line in the part of your script inside the do...done interior of the while loop. And for each pass through the loop (which would correspond to each line of your file, in this case), the contents of $line would change to the line currently under consideration.

When there are finally no more lines to be read from the file targetdir.txt, your script would move past the while loop, and proceed with whatever other code there was beyond that point, if any.

Lastly, if you were receiving an error indicating that targetdir.txt could not be opened, then that most likely means one of two things: either the file is not in the current working directory of the script at the point in time of the while loop executing (which you appear to be confident is not the case, but bear in mind if your script uses cd commands anywhere in it, the current working directory of the script may differ from your own current working directory); or you do not have permissions to read the file.

Hope this helps ! Oh, and also, when posting any code examples in the future, if you could do your best to please use markdown or enclose them in the appropriate tags (such as [CODE] or ```), that would be very helpful for others, as it makes your code easier to read, and easier to distinguish at-a-glance from the rest of your post.

Thank you for that information. It is exactly what I needed to know. I can tell you I searched high and low on the internet to find the answers before I sent this post. Apparently I did not search high enough or low enough. So thanks again for the assist.

How, and why? Please show your code and the exact error message.

If the file didn't exist, the message would be e.g.

-bash: txtfile: No such file or directory

With insufficient permissions, you'd read

bash: txtfile: Permission denied

What did you encounter?

Thanks for the feedback. I finally got past that issue about cannot open the text file. Just had the text file in the wrong directory. Now I have issue where only the first line being read from the text file is processed in the ‘while read line’ loop when there are actually 5 lines in that text file. I’m researching that now, but if you have any insight, I would appreciate hearing it.

Hello,

I think ideally to figure out what may be going on here we'd need to see the actual code in question (as per @RudiC's request in his reply to you), and also preferably the input file as well. If we can see the code that is being executed, and the real input that it is being asked to operate on, then we may be able to suggest what could be happening here.

The normal situation (when data "disappears" inside a while read loop) is that one of the commands executed inside the loop body also reads stdin, because it is inherited from the redirection after done.

If the commands should have redirections of their own, that works. You can prevent them reading stdin by redirecting each command inside the loop with < /dev/null.

If those commands need to read stdin itself, you need to consider how to manage the multiple data sets that may be implied (e.g. how to give appropriate EOF to each of several commands).

It is possible to redirect the while read from a different file descriptor (e.g. 3<targetdir.txt) and give this to the loop like read -u 3. Then the commands in the loop can use stdin without depriving the while read.

Here is the situation, including the script code. Please keep in mind that my exposure to UNIX scripting is only about 2 and a half weeks.

The targetdir.txt file contains:

DATA1
MERG1
PCL1
PMGT1
PRNT1

The issue is this:

1. Only the first line in the targetdir.txt file is read. Additionally, there appears to be a character at the end of each line, as seen in the 1st image below. I don't know what that character is, but it may have something to do with this issue.

2. The result of the script execution is that the DATA1 directory does NOT exist on BTSUR1 as seen in the 1st image below. THIS HAS BEEN RESOLVED.

targetdir.txt

rel_uq1_2_ur1.sh:

image844×894 22.3 KB

It is like Paul suspected: the ssh command reads from stdin i.e. from the file.
Remediations:
ssh -n ... (tell it to not read from stdin)
</dev/null ssh ... (redirect its stdin)

Hi,

Ah, this is an interesting example of how sometimes things that look obvious don't quite behave as you might intuitively expect. This is happening due to the use of ssh within your loop, as per the earlier replies you received. To understand why it's happening, however, let's look at some examples that demonstrate what's going on.

So, let's imagine we have a file that contains the numbers 1 through 10, one per line, and we have a loop that reads through them, prints out the loop number, and then runs an SSH command within that loop. Here's what we get, in that circumstance:

$ cat sshtest.txt 
1
2
3
4
5
6
7
8
9
10
$ while read number; do echo Loop pass: "$number"; ssh localhost "sleep 1"; done < sshtest.txt 
Loop pass: 1
$ 

So, as you can see, we get exactly what you describe - rather than seeing 10 passes through this loop, we only see the first one, despite our SSH command being about as simple as it gets (log in and run sleep 1 to wait one second).

Now, why is that happening ? It's happening because the SSH client is reading in standard input from the loop, and so ends up effectively "stealing" the rest of your read data. We can demonstrate that as follows:

$ while read number; do echo Loop pass: "$number"; ssh localhost "echo The folowing output comes from the SSH connection: ; cat -"; done < sshtest.txt 
Loop pass: 1
The folowing output comes from the SSH connection:
2
3
4
5
6
7
8
9
10
$ 

Here, we can see what happened to numbers 2 through 10 - they were passed to ssh as its input, and if we then print that output with a cat - command (which basically will print out whatever has been passed to it as its standard output), there's the rest of the numbers we read in.

Now, how to prevent this ? Well, there's a few ways, but one method would be to read all our numbers in ahead of time using a for loop, thusly:

$ for number in $(cat sshtest.txt); do echo Loop pass: "$number"; ssh localhost "echo Here is the output from SSH"; done
Loop pass: 1
Here is the output from SSH
Loop pass: 2
Here is the output from SSH
Loop pass: 3
Here is the output from SSH
Loop pass: 4
Here is the output from SSH
Loop pass: 5
Here is the output from SSH
Loop pass: 6
Here is the output from SSH
Loop pass: 7
Here is the output from SSH
Loop pass: 8
Here is the output from SSH
Loop pass: 9
Here is the output from SSH
Loop pass: 10
Here is the output from SSH
$ 

Here, we use command substitution (the meaning of that $(...) syntax) to run the command cat sshtest.txt, capture its output, and assign it to the variable $number. We then have all our input read in and ready for use before we do anything else, and that allows us to use SSH in a loop in the manner that is likely expected.

Oh, and lastly: in the future, could you please post code excerpts and output as actual raw text appropriately bounded with markdown or tags, rather than posting screenshots ? Again, this makes it easier for others to read what you have posted, and avoids needlessly using attachments when you don't have to. It's also more accessibility-friendly, as it would mean that users using screen reader software would be able to hear what you posted, whereas a JPEG or PNG isn't something they'd have any straightforward way of obtaining the content of.

Hope this helps !

Thanks. Given my lack of experience on this, on which ssh command do I put the -n option?

I put the -n option on any use of an ssh command, and that fixed the problem. Thanks.

By the way, the ssh is not being malicious or in error, to read from stdin.

It is intended to be an optimisation: most remote shell commands want to read data from the local client, so ssh pre-reads stdin (I think about 2048 bytes), and sends it down with the initial command. That saves a complete request/receive cycle, so the remote end gets to work immediately. If the stdin contains only a few config details, they may all fit in that one transfer.

The -n option disables that initial transfer. Older OS (IIRC, Sun Solaris) did not have the option, and required a redirection ssh < /dev/null instead. Being a belt-and-braces kind of dude, I usually do both.

You don't need multiple ssh invocations in your above code example (BTW, don't post pictures but copy text over so people can copy it over for test and improvement purposes).
Why open a resource hungry and costly ssh remote session to just determine if a directory exists, and then another to create it? You can, in one session, execute any sequence of commands remotely, or compose a script to be run.