Weird redirection behaviour

Linux. Bash 4.2.10.

Let's say that i want the stderr of a command redirected to one file (err), and both stdout and stderr redirected to another file (outanderr).

Easy. Many ways to do it. But may times the file outanderr will be messed up (with the output and the error mixed toghether).

See:

lem@biggy:/tmp$ touch imafile
lem@biggy:/tmp$ (ls imafile idontexist 2>&1 >&3 | tee err >&3) 3>outanderr
lem@biggy:/tmp$ cat outanderr
ls: impossibile accedere a idontexist: File o directory non esistente
imafile

It looks nice (sorry for the Italian, but this is so like English that I'm sure it's easily understandable).
But let's retry. After a few times:

lem@biggy:/tmp$ (ls imafile idontexist 2>&1 >&3 | tee err >&3) 3>outanderr
lem@biggy:/tmp$ cat outanderr
ls: impossibile accedere a idontexistimafile
: File o directory non esistente

Here above the output (imafile) has gone in the middle of the error!

Another solution:

lem@biggy:/tmp$ ((ls imafile idontexist >&3) 2> >(tee err >&4)) 3>outanderr 4>&3
lem@biggy:/tmp$ cat outanderr
ls: impossibile accedere a idontexist: File o directory non esistente
imafile

Nice. Let's retry. After a few times:

lem@biggy:/tmp$ ((ls imafile idontexist >&3) 2> >(tee err >&4)) 3>outanderr 4>&3
lem@biggy:/tmp$ cat outanderr
ls: impossibile accedere a idontexistimafile
 directory non esistente

Again: output in the middle of the (semi-cutted) error.

The same with things like:

$ ls imafile idontexist >outanderr 2> >(tee err >>outanderr)

The only one I've found to be safe in my shell seems to be:

$ ls imafile idontexist 2> >(tee err >outanderr) | tee -a outanderr >/dev/null

So, my questions are:
a) what's going on?
b) since all redirections are set before executing the commands, where's the difference with a simple thing as $ ls imafile idontexist >outanderr 2>&1 , which never ever fails?
c) the last solution is really safe? If so, why?

Thanks for everything.
--
Bye

Bash - 3.2.25

I am getting only one error ...:slight_smile:

ls: idontexist: No such file or directory

I've been told that also

$ ls imafile idontexist 2> >(tee err >outanderr) | tee -a outanderr >/dev/null

has created problems to someone, so it isn't safe.

Thinking it were safe confused me.

Of course when there is only one command, as in

command >file 2>&1

the writes are sequential, so there cannot be problems.

When there are many (sub)processes, created by pipes or command substitution or whatever else, all concurrently writing on the same file, I cannot see a clear way to be sure they won't overlap - using only the shell tools: /var/log/syslog works flawlessly, but there's a daemon in control, Which I think has its own importance. :wink:
Depending on the scheduler, on the number of CPUs and maybe on many other things, some solutions may seem to work here for a while, but don't work there.

Beside some awful workarounds with a very tiny field of application, like something that:

( ls imafile idontexist 2>&1 >&3 | { sleep 1; tee err >&3; } ) 3>outanderr

I can't think of any real solution. But since I know very little at all about these black boxes we call "computers", I wouldn't be surprised to be totally wrong.

Thanks to anyone reading all this awkward stuff.
--
Bye

Multiple processes are writing to the same file. Some of the streams are unbuffered (stderr) and some are buffered (stdout). You are at the mercies of the c library's stream buffering and the system scheduler.

(ls imafile idontexist 2>&1 >&3 | tee err >&3) 3>outanderr :

  1. ls writes 'imafile' to its stdout stream (outanderr file). This stream is fully-buffered so the word just sits in a buffer.
  2. ls writes the "idontexist" error message to the stderr stream (the pipe). This is an unbuffered stream, so the c library executes the system call to write the error message to the pipe.
  3. ls is done, so all of its buffers are flushed and it exits. This prints the word "imafile" which had been buffered.

Initially, tee is sleeping, since there's nothing in the pipe. Once ls writes that error message, the system knows that it can wake tee. In your examples, tee is always woken before ls can flush "imafile" (between steps 2 and 3 above). The mixing of the streams occurs when tee isn't able to finish writing the entire message to its stdout before ls resumes at step 3.

There is only one process involved.

No. ls imafile idontexist 2> >(tee err >outanderr) | tee -a outanderr >/dev/null is not immune.

Regards,
Alister

2 Likes