how to prevent deadlock on this...

I am using linux termios structure to configure serial port and read the port by read function. For some reason, if I read the whole buffer, almost every time the buffer does not contain the correct reply message sequence from a device sending reply to my linux PC. So I use "while(!read_start(fd,0x01));" to wait the first legal byte in the reply and read byte by byte. The problem is sometime the user forget to power on the device so it will cause deadlock. I am no expert in linux. Can someone give suggestion about how to: set a timer and if time out, the current program will be killed or exit incorrectly while waiting the first byte?

Use the select() system call to wait for the fd to become readable. You can specify a timeout to select and the return code tells you what happened. Then read the entire expected number of bytes. You will probably get too few. So read the rest. Continue until you have all the bytes you need. Reading a byte at a time is not very efficient. If needed you can use select() in front of each read() so that the program never hangs on a read.

Perderabo, thanks a lot.

After select() and if(FD_ISSET(fd, &rfds)), then I read the buffer. As you mentioned, I only can read 8 bytes one time (my full reply message is 13 bytes) by "result1=read(fd,buffer,13);" (result1=8). After I read these 8 bytes, I have to read the buffer one byte after another to get correct message. If I use "result2=read(fd,buffer1,13)" after first 8 bytes read, result2 always is 0. Is there anyway I can use multiple block reads to get a full message? Or how can I get one read more than 8 bytes in my case in one read?

Post your code for this section.." if(FD_ISSET(fd, &rfds)),' does not sound right

Do a second select before you do the second read. Do a select before each read.

The code:

/////////////////////////////////////////////////////////////////////////////
			        tv.tv_sec = 3;    /*3 second*/
			        tv.tv_usec = 0;
				FD_ZERO(&rfds);
				FD_SET(fd, &rfds);  
/*fd=open("/dev/ttyS0",O_RDWR|O_NDELAY|O_NOCTTY)*/

				retval = select(FD_SETSIZE, &rfds, NULL, NULL, &tv);

				if (retval == -1)
					printf("select error\n");
			        else if (retval){
					if(FD_ISSET(fd, &rfds)){
						int result1=0;
						int result2=0;
						result1=read(fd,buffer,13); /*result1 is always 8*/
/*if I put another read here:result2=read(fd,buffer1,13), result2 is always 0*/
						usleep(100);
						for (i=result1;i<13;i++){   /* have to read the rest bytes, one byte at a time*/
							usleep(100);
							result2=read(fd,cc,1);
							buffer=cc[0];	
						}
/////////////////////////////////////////////////////

You are setting the O_NDELAY flag on the open(). That is a bad idea for several reasons. The newer O_NONBLOCK flag corrects a problem with O_NDELAY. Use O_NONBLOCK for new code. Setting either flag means that the open() call itself will not hang. But subsequent operations on the fd are being affected. Posix says, about O_NONBLOCK, "When opening a block special or character special file that supports non-blocking opens: Subsequent behavior of the device is device-specific." The Linux man page says, "Neither the open nor any subsequent operations on the file descriptor which is returned will cause the calling process to wait".

These flags are affecting read(). The read will return immediately if no characters are available. The old O_NDELAY will return a zero in this case. That is the same thing EOF would return... this is the problem with that flag. If O_NONBLOCK is set, -1 is returned with errno set to EAGAIN.

So my advice: use O_NONBLOCK for the open and then turn it off... something like this:
flags=fcntl(fd, F_GETFL, 0);
flags &= ~O_NONBLOCK;
fcnt(fd, F_SETFL,flags);

Next, you mention that you are setting termios paramters. Take a look at CMIN and CTIME. You can use them to control when the read returns. That means you can get rid of select().

You may still need to do more than one read. Use an overall loop...something like:


nbytes=16;
nreads=5;
ptr=buffer;
for (nreads=5; nbytes && nreads; nreads--)  {
        iret = read(fd, ptr, nbytes);
        if (iret > 0) {
                ptr += iret;
                nbytes -= iret;
        }
}

Of course, you will need to check the sys calls for failure. I'm just sketching in the basic flow, not writing real code.