behaviour of read() and write() after a select()

Hello, I have problems finding clear information about the use of select(). I want to write an application, and for that I need to be sure about the functions behaviour. So to ensure, that I understood it all correctly I make some statements or questions. So a qualified answer could just be: yes, you are right in all points.

The manpage says:

"Three independent sets of descriptors are watched. Those listed in readfds will be watched to see if characters become available for reading (more precisely, to see if a read will not block - in particular, a file descriptor is also ready on end-of-file), those in writefds will be watched to see if a write will not block, and those in exceptfds will be watched for exceptions. On exit, the sets are modified in place to indicate which descriptors actually changed status."

So if I read() from a file after select() says it would not block, it will under no circumstances block, right? When I try to read 1 MB, which might not be available without blocking, read() would then just return the data that is available (it has the liberty to read less bytes then I told it to), right? A second read() after that, on the other hand could block, because read has not the liberty to return 0 bytes, because that would mean EOF.

The same thing with write(): When select() says a file is writable, a write() to that file would under no circumstances block. But write might have a size limit, that is unknown to me. So when I write 1MB, it uses the data to fill some internal kernel buffer, and might return without actually writing all the data, but only part of it. It would take only that much data, that is possible to take without blocking. So the first write does under no circumstances block, but the second write could block.

Of course a read() or write() could block, if someone else (another thread/process) does the first read/write after the select, but thats not what I'm interested in, because I don't need to share files with someone else.

Does all this still hold completely true with sockets?

Does all this still hold completely true without using non blocking I/O?

Is it always possible to safely assume, that the first write() after opening a socket/file will not block even without checking this with select()?

The reason why I am not 100% sure about all that is that I did some googling, and some people in some forums says that one can never be 100% sure, if a read/write won't block, even with select, when not using O_NONBLOCK. Some say, that a write does block, if the size is bigger then some magic value. Also most texts about select don't really cover that write case.

Of course I could just use nonblocking I/O to be sure, but I want to make my app as simple as possible, and I don't want to have a "double net" if it is not required.

"Under no circumstances" does not apply to slow read & write devices. There are no guarantees there, as far as I know.

Why are you not using aio? aio_write(), etc. Instead of worrying about all this. aio_read & aio_write calls do not block. They complete and then fail later if there was a serious system problem, or they terminate early if you call aio_cancel. But they do not block. This applies to POSIX-conformant OSes only.

See the opengroup on aio_write:
aio_write

The aio calls were designed so that you call them whenever, then come back later to get the results. This design precludes blocking. They are asynchronous in the best sense.

hmm... If this doesn't apply to slow devices, then what is select() for anyway? If I am working with a fast device, I wouldn't need it in the first place.

I know I could use async. or nonblocking I/O instead. But I thought that select() was there as a "third choice" to avoid blocking.

select works fine, it is the slow devices that are the problem. select or maybe poll are defacto choices for sockets.

Are you going to use sockets with tape drives?

I don't want to use tape drives. But the term "slow device" is relative. Even with a fast device, a read can block, even if it is only about a few milliseconds. In that case I can use select to watch multiple file descriptors, so that one read/write operation doesn't have to wait on another.

I want to use select for sockets and for ordinary hard disk files. I want to have my whole application centered around a select call, that should be the only blocking call in the whole app. Like the event loop in an event-driven application. And the way I read the select man page, this should be possible even with slow devices under all circumstances, even without any nonblocking or async call, and also without multithreading. Because even with a slow device, select should report it "ready" only, when there is something to read/write. So with a slow device, this would happen later, but when the time comes, it should not block.

Is this right? If not, where am I wrong?

If you're using Linux look at select_tut(2)...

Cheers,Steve

The select man page says: "A descriptor is considered ready for writing when a call to an output function with O_NONBLOCK clear would not block, whether or not the function would transfer data successfully."

If you decide to set O_NONBLOCK, the read call will not return until all of the data you requested is available. If this might take any finite amount of time, even several minutes, the device is called "fast". If it might block for all eternity, the device is called "slow". Slow devices include terminals, raw network devices, and IPC psuedo devices.

If you select on a terminal and I am typing on it, select returns on the first character. If you then do a blocking read for 10 characters, you will block until I type nine more. I really can't imagine any alternative behavior.

I use read(), not fread(). read() could for example only return 1 byte. As all available docs say, that read() *can* return less bytes then requested. So the return value of the read() call in your case would be 1 (in my optinion). Then a program would decide to either

1. Wait for the rest, by calling read() again, this time requesting the 9 bytes missing
2. call select() again, to be notified, when you press your next key

Thank you, didn't know that page. This really helped, since it confirmed my expectations and prove all other posters here wrong (no offence)

EDIT: well, not about the thing with the first write after opening a file. But that is not that important anyway. And nobody disputed that

After some further reading, I found out that all this is not true for regular files. select() behaves as if a file is ready for reading and writing at any time:

"File descriptors associated with regular files shall always select true for ready to read, ready to write, and error conditions."

select

This makes Unix more inconsistent than I thought. Most documents about select don't mention that. I guess it's a "they won't notice" thing. Also, O_NONBLOCK does not work with regular files. I didn't want to do multithreading or multiprocessing to conserve resources. For the same reason I can't use aio_read(), because on Linux it is implemented with threads. The kernel does not support async I/O. This is a suboptimal solution in my opinion, because only the kernel knows, which operations would make sense to do in parallel (because the files might be on different disks)... I'm gonna write my own kernel

But I think I can live with all that. I just need to look out, that my file I/O operations are small, so they don't block for long.

Good luck.

You stated that you don't want to set O_NONBLOCK
which is standard to *nix for alleviating blocking behaviors
and is the basis for fast event interfaces like kqueue and
epoll for sockets, pipes, etc...
You don't want to use threads or processes to allow
for blocking on reads and writes to regular files due to
resource conservation.
Finally, you state that you believe that as long as i/o operations are
'small' you should be okay (as if this somehow alleviates the wait
behavior incurred should an operation block..).
It just seems you are fighting a bad fight against programming with unix.

Here's a little advice:
Optimization (computer science - Wikipedia, the free encyclopedia)

Yes, I realized that given the alternatives I have to "die one death". I think I will make a single threaded app, that multiplexes sockets using select(). File operations will then still block, but if I keep them small, they should not hurt performance much. After all, file I/O should be much faster then network I/O. At least in max. 20 ms every small file read should be finished.

Its just that I realized now, that neither select() nor O_NONBLOCK ("standard to *nix for alleviating blocking behaviors") work for ordinary files (I mean, who would have thought?)

By the way, as long as you don't use ordinary files, O_NONBLOCK is not needed. select() does it's job very well even without O_NONBLOCK. On the other hand, if you use select(), O_NONBLOCK is not really needed anymore. Read man select_tut(2) on this.

So the only alternative would be threading. But this solves only part of the problem, because file operations would be asynchrouos with the select part of my app, but not with each other. Who knows, one file could still be in cache by the time it's needed...

That's why I'm here, right? :rolleyes:

btw, your little advice is neither little nor an advice

You mean I'm trying to optimize too soon? I don't think so, because in this case I try to figure out a good overall structure for the app. This is better done before I write the bulk of code.

Hello,

This is one of the well known *nix gotchas.
Most unix systems programmers have gotten used to it.
Not that it's ideal.

"
Read or write not blocking on a resource after select returns is not entirely reliable at all times. The select_tut example is it's own best argument against using the non-blocking model imho. Also see: http://cr.yp.to/unix/nonblock.html

You are ,imo, optimizing too soon and encountering problems that research may have
ameliorated. Please see if you haven't already: The C10K problem

Hi, this is jack Steven. I have an Interest to answers the questions on JAVA. If any one want to or interest in asking questions, you are welcome.
--------------------------------
Jack Steven