Ideas: 'select'ing file descriptors

Hey everyone,

I'm writing a threaded database server application and need to communicate the results of several worker threads to a single multiplexed communication output thread. Basically, the communication output thread sits in a select statement waiting for its client sockets to dole out requests, connect, disconnect, and become ready for sending data. This thread is, therefore, using a select statement, and a round robin scheduling policy, to control its scheduling.

Now I have several other threads whos sole purpose is to take the longer, possibly blocking, tasks and construct a deliverable for the client. Of course, when they are through they need to send that data to the communication output thread who is sitting in the select waiting for socket events.

Well, unless I'm missing something there is no select call that will allow me to use typical thread synchronization with mutexes and condition variables and mix this with my other file descriptors. So I'm guessing my only option is to be able to interrupt the select. This means that either I need to signal the thread to break out of the select. This seems unreliable in my opinion given that the thread that handles said signal is randomly selected from a set of threads not having that signal blocked and therefore there exists a small window where a new thread is being created in which it can obtain a signal it is just about to tell the scheduler it wants to block.

Therefore, it seems I have no choice but to use a file descriptor and set its read, write, or execute condition such that when there is information waiting the select will be capable of seeing it and when there is no more data waiting I can remove said condition. This all, of course, needs to be mutex protected, but only when I bounce out of the thread do I need to attempt to attain said mutex.

Basically, I decided to use a pipe and following this procedure:

select for readability on read fd of pipe, when readable:

lock mutex
pull data off mutex protected in memory queue
if end of queue read char off pipe
unlock mutex
handle data

Then to signal this:

lock mutex
push data until mutex protected in memory queue
if first item put char on pipe
unlock mutex

That should work...but it seems wasteful to use up two file descriptors and an OS pipe buffer for a single character which is nothing more than an on/off switch. What I'm wondering is, for those who have made it this far, is there anything more lightweight I can use that could acheive the same behavior, or is how I'm going about this completely rediculous?

I'm looking for ideas...so any would be appreciated...if not it looks like I'm going to use the above method...I just wonder if there is something more standard that I could do or something better.

Thanks everyone,

Brian Gregg
Computer Sciences Corp.

I didn't really follow all of that, but I would try to go in one of two directions...

1) Lose that memory queue
I would not put one byte on the pipe. I would put the whole shebang. Then the main thread would know to read, say, a 30 byte structure rather than a flag byte. This structure would be everything needed to process the request and there would be no other communication between the main thread and the requesting thread.

2) Lose that pipe
The select system call can be given a timeout so it return when a timer expires or i/o is possible. I would set the timer for, say, 1/10 of a second. Then when the select returns, the main thread would first service any fd's, then check the memory queue for any of those tasks.

And beware: you are dancing very close to a race condition. Between the time that a requester makes a request and the time that the main thread notices the request is a window of time. During that window other requests can arrive. So you must loop and service all outstanding requests.

1) Well, the memory queue is a requirement because the pipe has a byte limit and given the threaded nature and the need for mutex protection it is possible that I run up against a deadlock because the read thread is trying to lock a structure to handle a request and the write thread has it locked but can put it on the pipe because of the byte limit.

I found that out the hard way...and that's why the single byte on and off to prevent this.

2) That's so close to polling it might as well be polling and would decrease the overall responsiveness of the system because I'd have to wait for socket events, switch to wait for response events, and then back. This also wastes CPU cycles as most polling type activity does...and that's bad practice IMO.

As for the race condition...there really is not worry there. First the mutex will protect the memory queue. So a thread putting a request on it has a consistent view of whether it is the first to put a request on or not. The thread taking the data off then (after it locks the mutex) has a consistent view of whether that item is the "last" item. If another is to be added the mutex has to be aquired, which it can not be until the read thread either removes the byte and the request (or the write thread places the byte and the "first" request).

What I'm really looking for is a way to, maybe with fcntl, make a file (rather than a pipe...maybe one open to /dev/null) readable and then non-readable so that a select can see the readable flag set, and then I can clear it. This is acheived with the single byte outstanding on the pipe...but I'm looking for a cleaner way.

Thanks for the suggestions, if you have any more ideas I'd love to hear them.