Odd behavior from GDB while trying to cross-debug an embedded Linux application.

Applications Programming
1

Some background:

The application normally runs on an embedded platform. Currently, for development purposes, I have the rootfs located @ /exports and the target is communicating over NFS. That way I can make a change on my local system, save the application @ /exports, and run the altered version to see if it will work without having to do a full install on the target itself.

Following this guide I was able to get GDB debugging working or at least I thought I was. However, as soon as I altered my code, built my project and copied/replaced the application with the altered application I received the following errors:

Truncated/partial Eclipse Debugger Console output:

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) 0x400007b0 in ?? () from target:/lib/ld-linux.so.3                                                                                                                                           
Reading /usr/lib/libasound.so.2 from remote target...
...
Reading /usr/lib/libgcc_s.so.1 from remote target...
Error while mapping shared library sections:
Remote communication error.  Target disconnected.: Connection reset by peer.
warning: Could not load shared library symbols for 4 libraries, e.g. /usr/lib/libc.so.6.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Error while reading shared library symbols for target:/usr/lib/libasound.so.2:
Can't read symbols from target:/usr/lib/libasound.so.2: Input/output error
...
Error while reading shared library symbols for target:/usr/lib/libm.so.6:
Can't read symbols from target:/usr/lib/libm.so.6: Input/output error

Output from my serial port terminal:

*** glibc detected *** gdbserver: double free or corruption (!prev):
0x000983d8 ****

As I mentioned I believe this gdbserver error may be erroneous and explaining why might be tricky so here goes. As I mentioned GDB seemed to be working. For instance, here is the GDB output I was getting:

Truncated/partial Eclipse Debugger Console output (before compiling and copying application):

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) 0x400007b0 in ?? () from target:/lib/ld-linux.so.3                                                                                                                                           
Cannot access memory at address 0x7ffd8614
...
Cannot access memory at address 0x7ffd8610

Temporary breakpoint 1, main (argc=0, argv=0x0) at main.cpp:413
413       TimeStuff::CStopWatch  dtorDuration;
Cannot access memory at address 0x7ffd8614
...
Cannot access memory at address 0x7ffd8610

So, like expected, the application runs and pauses at main. I can then resume the application and debug whatever I need. In fact, I used it to debug a problem then made a change and tested it. It worked, great! However, as soon as I went to debug the application for a different problem I received the errors you read earlier. So why do I think the gdbserver error is erroneous? Well because both applications are using the same exact version of glibc. In fact, the only differences between the two applications is the changes I made to the code and that code works, I know because I tested it.

That being said I am chooseing, XD, to ignore the error reported in the serial terminal and am concentrating on the errors reported by Eclipse, specifically this jumped out at me:

Remote communication error.  Target disconnected.: Connection reset by peer.

Not sure why there would be a remote communications error, any ideas what might be happening here? Are these glibc errors actually red herrings?

2

I really doubt the crash is erroneous.

Crashes inside libc are almost always programmer-caused. This one is easy to cause:

#include <stdlib.h>

int main(void) {
        void *p=malloc(1000);
        free(p);
        free(p); // crash
}

Never call code good because it "just works", especially when it crashes. What exact changes did you make to what exact code?

It could be the debugger that's crashing, I suppose. Unlikely but plausible.

3

Unfortunately I got pulled off into a bug and had to put this aside for a few days but now I have some more time.

@Corona688

Just to be sure I remounted a brand new copy of the rootfs from a branch and then downloaded the Codebase for that branch and repeated the above exercise and got the same behavior. Another thing I find odd, in the debugger console at the end of the output I showed above it says:

A problem internal to GDB has been detected,
further debugging may prove unreliable.
 Quit this debugging session? (y or n)

but it's actually frozen. Entering y or n does nothing. So I have to manual terminate the debugging session.

I tried running gdb-multiarch from a terminal instead of using it through Eclipse and ended up with similar output except right at the end it says:

Segmentation fault (core dumped)

So, this doesn't seem to be an Eclipse problem. Maybe it is a gdb-multiarch problem?

EDIT: I take that back the output is jumping all over I tried again running gdb-multiarch with sudo and ended up with the same output as before, no mention of a seg fault.

4

What exact changes did you make to the code?

5

@Corona688 I reproduced the error without making any changes to the code to be sure that the changes I was making weren't effecting anything. I now believe the error to be a communications error with gdb-server on the embedded side. Gdb-server was installed onto the embedded target something like 5 or 6 years ago while gdb-multiarch is brand new.

1 Like