Bash - proper way to append variable to stderr

Shell Programming and Scripting
1

Hello,

Can you please if the bellow is the proper way of appending a variable to the stderr:

The easiest way to test this,I was able to imagine, was by touching 5 files and afterwards looping trough to the results:

-rw-r--r--  1 ab  owner  0 Sep 14 13:45 file1
-rw-r--r--  1 ab  owner  0 Sep 14 13:45 file3
-rw-r--r--  1 ab  owner  0 Sep 14 13:45 file5
-rw-r--r--  1 ab  owner  0 Sep 14 13:45 file6
-rw-r--r--  1 ab  owner  0 Sep 14 13:45 file8

In order to get the missing files logs i run:

for i in {1..8} ; do ls -al file$i 2>> error.log ; done

But if I want to append a variable let's say the date, I was able to came with:

for i in {1..8} ; do ls -al file$i 2> >( while read line; do echo "$(date): ${line}"; done >> error.log ) ; done

The question I have if this the best way to append a variable to stderr?

2

If you have single lines in a loop you do not need another loop within

for i in {1..8} ; do ls -al file$i 2> >( read line; echo "$(date): ${line}" >> error.log ) ; done

It is more efficient to have two loops indeed, but one after the other

for i in {1..8} ; do ls -al file$i; done 2> >( while read line; do echo "$(date): ${line}"; done >> error.log )

where error.log is written in one stream, so you have the option > error.log
And in the case of date , if its output does not change much, one can store it in a variable first, and several time refer to the variable.

date=$(date); for i in {1..8} ; do ls -al file$i; done 2> >( while read line; do echo "${date}: ${line}"; done >> error.log )

for i in {1..8} ; do ls -al file$i; done 2> >( date=$(date); while read line; do echo "${date}: ${line}"; done >> error.log )
1 Like
3

Hello, Thank you very much for taking time to answer and for the useful examples.

4

I am not sure what exactly you want to do, but in this case you can simply test for the existence of the file(s) and then construct your own error message:

for i in 1 2 3 4 5 6 7 8 ; do
     if [ ! -e "/path/to/file$i" ] ; then
          echo "File file$i is missing, date is: $(date)" >>/your/log/file
     fi
done

If you do this in ksh you have an even better and cleanlier way to do this:

#! /bin/ksh

exec 3>>/path/to/your/logfile        # open logfile for appending and use IOD 3 to address it.

print -u3 - "----- start of execution ( $(date) ) -----" # -u3 : send it to IOD 3
for i in 1 2 3 4 5 6 7 8 ; do
     if [ ! -e "/path/to/file$i" ] ; then
          print -u3 - "File file$i is missing, date is: $(date)"
     fi
done
print -u3 - "----- end of execution ( $(date) ) -----"

exec 3>&-     # close IO descriptor 3 again

This way you can open (and close) the file once and do not have to address it anew for every message you want to append. You also don't have to worry about one misplaced ">" instead of a ">>" wiping out the whole logfile. And you can use several such files at once by using IOD 4, 5, 6, ... too.

I hope this helps.

bakunin

1 Like
5

Hi, Thank you for your post.

I was trying to figure out what will be the best way to append a variable to the stderr.

6

Actually the way you presented your problem you were writing into a a file - "error.log". You just happened to redirect stderr to point to that file.

You might picture UNIX processes (running commands) like garden hoses - you put something in on top (this is stdin) then inside something is done with it and finally something pours out at bottom (stdout). You can point both - stdin and stdout - to some file or device. In case of stdin from this device or file is being read, in case of stdout output gets written there. You can even connect stdout of one process to stdin of another process - this is called "pipeline" and looks like this:

command1 | command2

So far, so basic. Now, the garden hoses of UNIX processes are in fact Y-shaped: the normal flow is like explained above, but for diagnostic messages there is a small outlet in the middle (to stay in the hose picture) where errors, diagnostic messages, etc. are shown. This is good, because if the diagnostic output is separated from the normal output it is possible to process it via different processes.

So, this is what "stderr" is normally used for, but in fact it is just a so-called "I/O descriptor" - a possible outlet where a process can output something and in this respect not different from "stdin" or "stdout"*). You can use stderr in the same way you use stdout and if you write:

echo "hello world" >&2

you will see "hello world" appear on stderr. If this is inside a script and you redirect the stderr output of it to some file:

./script 2>/some/logfile

you will find "hello world" in this file.

Per default "stdout" is I/O-descriptor 1 and "stderr" is I/O-descriptor 2 (this is why it is "1>" and "2>" in redirections) but you can use other I/O-descriptors too - you just have to open and close them, just like i did for I/O-descriptor 3 in my sample above.

But the question "include variable in stderr" makes no sense: you can output everything to stderr the same way you output something to stdout or anywhere else - it is just a different part of the y-shaped hose you use.

I hope this helps.

bakunin
_________
*) in fact there are differences: i.e. stdout is buffered, stderr is not, but this is not relevant here

1 Like
7

Thanks for the detailed explanations, for sure will help.

8

I think the question was about post-processing stderr.
This is not trivial.
The command 2> >( post-command > error.log ) is one method.
There is also the classic { command 2>&1 >&3 | post-command > error.log; } 3>&1 method.

1 Like