Capture stdout from multiple processes

I have a number of binaries which I currenlty have no control over. They alright data to stdout. I would like to kick off any number of these binaries and capture and process their stdout. Doing this for one process is straight forward for example - comments and error checking removed for simplicity

pid_t pid;
int	commpipe[2];
char buf[BUFSIZE];

pipe(commpipe)
 
pid=fork();

 if(pid){
    dup2(commpipe[0],0);
    close(commpipe[0]);
    while ( fgets ( buf, BUFSIZE, stdin ) )
    { processBuffer ( buf );
 } else{
    dup2(commpipe[1],1); 
   close(commpipe[0]);
    setvbuf(stdout,(char*)NULL,_IONBF,0); 
   execl("child","child",NULL) 	
}

I would run the above from a separate thread. This works fine for one process, if I attempt to repeat the above for a secoind process. the output of the 2 process is intermingled. I can solve the stdin side by not removing the dup2 call in the parent and reading directly from the read side of the pipe i.e. fgets( buf, BUFSIZE, stdin). But the data is still intermingled since it is the same stdout.
I can close stdout before forking but then I only get the output from one process.

The goal is to have a separate thread read the stdout of each process Any help will be appreciated

Thanks in advance

I'm sure there is away to do it

You didn't say whether or not you need to write to the child's stdin. If not, then use popen() and you'll be able to get what you want. Here's a sample:

#include <unistd.h>
#include <stdio.h>
#include <errno.h>

int main( int argc, char **argv )
{
    FILE* pipe1;
    FILE* pipe2;
    char    buf[4096];

    pipe1 = popen( "df -h", "r" );   /* execute commands in parallel */
    pipe2 = popen( "ls -al", "r" );

    printf( "output from command 2:\n" );     /* read and print all data from one command */
    while( fgets( buf, sizeof( buf ), pipe2 ) )
        printf( "%s", buf );

    pclose( pipe2 );

    printf( "output from command 1:\n" );      /* read and print from other command */
    while( fgets( buf, sizeof( buf ), pipe1 ) )
        printf( "%s", buf );

     pclose( pipe1 );


    return 0; 
 }
  

The popen() system call creates a one-way pipe, but if you need to write on stdin all may not be lost. If the writes do not depend on reading from stdout and responding to the data, then you can write your input to a file, and run the command via popen() with redirection in from the file you created in your programme. Ensure you close the file before invoking popen().

Hope this helps a bit.

Agama

Thanks for the quick reply. No I do not need to write to the child stdin. I will give popen a try. I assumed it would behave the same way. You code executes the children serially I will run them in parallel would that make a difference. In any case I will give this a shot I probably will not get to it until Monday, I did not expect such a quick response.

Thanks
Dave.

You are most welcome.

The sample actually executes the commands in parallel, but reads the output serially to demonstrate that the output was not intermixed.

Here's a small example of a main that starts a thread for each filename (up to 10) given on the command line. Each thread uses a popen() to cat the file (please no flames about useless uses of cat!!) and reads the data counting the number of times the character 'e' appears in the file. It's a silly (um, useless) programme, because the thread could open and read the file directly, but rather than invoking a ps or netstat command, I wanted something that I could repeat to ensure that the proper number of e's from each file were being counted (to prove no mixing of output from the commands).

The function invoked as the thread sleeps to demonstrate that the threads are indeed created in parallel -- all "create" messages from the main should print before any thread actually announces that it's running the command.

Hope this helps to get you going

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include <pthread.h>
#include <string.h>



/*
    started as a thread. creates a child process via popen() and reads
    the child's output counting the number of target characters (global)
    that were in the output. When done, the count is printed on stdout.
    A delay of 2s is imposed to demonstrate that the main has started
    all threads in parallel, otherwise it is unnecessary.
*/
void *reader( void *data )
{
    char    target = 'e';       /* count these from the command's output */
    char    *cmd;               /* command to execute */
    FILE    *rpipe;
    char    buf[4096];          /* read bufffer */
    int     tcount = 0;         /* count of target */
    char    *cp;                /* pointer as we walk the buffer looking for target */

    cmd = (char *) data;

    sleep( 2);
    fprintf( stderr, "starting the command: %s\n", cmd );

    rpipe = popen( cmd, "r" );                      /* start the child */
    while( fgets( buf, sizeof( buf ), rpipe ) )     /* read all of the child's stdout */
    {
        cp = buf;
        while( (cp = strchr( cp, target )) )        /* count number of target characters in this buffer */
        {
            tcount++;
            cp++;
        }
    }

    pclose( rpipe );

    printf( "output from `%s` contained %d '%c' characters\n", cmd, tcount, target );

    free( data );
    pthread_exit( 0 );

    return NULL;            /* shouldn't matter, but keeps compilers happy */
}

int main( int argc, char **argv )
{
    pthread_t tids[10];
    char    cbuf[1024];
    int     i;

    if( argc < 2 || argc > 11 )
    {
        fprintf( stderr, "usage: %s filename1 [filename2... filename10]\n", argv[0] );
        exit( 1 );
    }


    for( i = 1; i < argc; i++ )         /* start all threads to process all files in parallel */
    {
        snprintf( cbuf, sizeof( cbuf ), "cat %s", argv );
        fprintf( stderr, "starting thread: %s\n", cbuf );
        pthread_create( &tids[i-1], NULL, reader, (void *) strdup( cbuf ) );        
                                                  /* thread must free dup to avoid leak */
    }

    fprintf( stderr, "all threads started, main thread waits....\n" );

    for( i = 0; i < argc-1; i++ )                       /* wait for all threads to finish */
        pthread_join( tids, NULL );

    fprintf( stderr, "all threads finished\n" );

    return 0;
 }

Thanks again,

Yes is exactly what I'm looking for. I'll give it a shot Monday. If I can do it sooner I will.

Thanks agin

Agama

I gave it a try it seems to be working as you said it would. I tested it with two binaries and none of the data was intermingled. I will try with more once I get the remaining binaries.

Thanks again

Your example is in C. If that is not a requirement, have a look at GNU Parallel which is made for running programs in parallel without having the output mix.

Watch the intro video: Part 1: GNU Parallel script processing and execution - YouTube