*** glibc detected *** : malloc(): memory corruption (fast):

Linux and Unix-Like Emergency UNIX and Linux Support
1

Memory corruption due to erl_interface libraries that are implemented in C

In my application I've an erlang node which periodically communicates with the C-node every 1 second to gather periodic data

The C-node is implemented in such a way that, it consists of two threads, the main thread receives the requests from the erlang node and passes the message to the worker thread. The worker thread serves the queries and replies to erlang node. To reply to the erlang node all the data that is collected in the cnode need to converted to erlang format (in my case to a list of tuples) using the function erl_format .

And I don't get this error repeatedly, sometimes it comes within 15 mins, or the moment I start the application and sometimes it doesn't even occur after running the application for more than 5 hours.

The erl_interface libraries are compiled with thread safe options(such as _REENTRANT)

below mentioned is the complete error

*** glibc detected *** /root/rel-1.0.0.199/galaxy/lib/galaxy-1.9.9/priv/hyphy_cnode: malloc(): memory corruption (fast): 0x1289abf0 ***
======= Backtrace: =========
/lib/libc.so.6[0xfd84610]
/lib/libc.so.6[0xfd864fc]
/lib/libc.so.6(__libc_malloc+0xb4)[0xfd887b8]
/root/rel-1.0.0.199/galaxy/lib/galaxy-1.9.9/priv/hyphy_cnode(erl_eterm_alloc+0xac)[0x1108e71c]
/root/rel-1.0.0.199/galaxy/lib/galaxy-1.9.9/priv/hyphy_cnode(erl_alloc_eterm+0x2c)[0x1108fd68]
/root/rel-1.0.0.199/galaxy/lib/galaxy-1.9.9/priv/hyphy_cnode(erl_cons+0x54)[0x1108c728]
/root/rel-1.0.0.199/galaxy/lib/galaxy-1.9.9/priv/hyphy_cnode(erl_mk_list+0xd4)[0x1108c87c]
/root/rel-1.0.0.199/galaxy/lib/galaxy-1.9.9/priv/hyphy_cnode[0x1108f490]
/root/rel-1.0.0.199/galaxy/lib/galaxy-1.9.9/priv/hyphy_cnode[0x1108f820]
/root/rel-1.0.0.199/galaxy/lib/galaxy-1.9.9/priv/hyphy_cnode[0x1108f408]
/root/rel-1.0.0.199/galaxy/lib/galaxy-1.9.9/priv/hyphy_cnode(erl_format+0x7c)[0x1108f64c]
/root/rel-1.0.0.199/galaxy/lib/galaxy-1.9.9/priv/hyphy_cnode(query_handler+0x7ab8)[0x10078018]
/lib/libpthread.so.0[0xff967f4]
/lib/libc.so.6(clone+0x8c)[0xfde226c]
======= Memory map: ========
00100000-00103000 r-xp 00100000 00:00 0          [vdso]
0fc31000-0fc41000 r-xp 00000000 fd:01 3237       /lib/libresolv-2.5.so
0fc41000-0fc50000 ---p 00010000 fd:01 3237       /lib/libresolv-2.5.so
0fc50000-0fc51000 r--p 0000f000 fd:01 3237       /lib/libresolv-2.5.so
0fc51000-0fc52000 rwxp 00010000 fd:01 3237       /lib/libresolv-2.5.so
0fc52000-0fc54000 rwxp 0fc52000 00:00 0 
0fc64000-0fc68000 r-xp 00000000 fd:01 3238       /lib/libnss_dns-2.5.so
0fc68000-0fc77000 ---p 00004000 fd:01 3238       /lib/libnss_dns-2.5.so
0fc77000-0fc78000 r--p 00003000 fd:01 3238       /lib/libnss_dns-2.5.so
0fc78000-0fc79000 rwxp 00004000 fd:01 3238       /lib/libnss_dns-2.5.so
0fc89000-0fc93000 r-xp 00000000 fd:01 3247       /lib/libnss_nis-2.5.so
0fc93000-0fca2000 ---p 0000a000 fd:01 3247       /lib/libnss_nis-2.5.so
0fca2000-0fca3000 r--p 00009000 fd:01 3247       /lib/libnss_nis-2.5.so
0fca3000-0fca4000 rwxp 0000a000 fd:01 3247       /lib/libnss_nis-2.5.so
0fcb4000-0fcc0000 r-xp 00000000 fd:01 3267       /lib/libnss_nisplus-2.5.so
0fcc0000-0fccf000 ---p 0000c000 fd:01 3267       /lib/libnss_nisplus-2.5.so
0fccf000-0fcd0000 r--p 0000b000 fd:01 3267       /lib/libnss_nisplus-2.5.so
0fcd0000-0fcd1000 rwxp 0000c000 fd:01 3267       /lib/libnss_nisplus-2.5.so
0fce1000-0fceb000 r-xp 00000000 fd:01 3264       /lib/libnss_files-2.5.so
0fceb000-0fcfa000 ---p 0000a000 fd:01 3264       /lib/libnss_files-2.5.so
0fcfa000-0fcfb000 r--p 00009000 fd:01 3264       /lib/libnss_files-2.5.so
0fcfb000-0fcfc000 rwxp 0000a000 fd:01 3264       /lib/libnss_files-2.5.so
0fd0c000-0fe49000 r-xp 00000000 fd:01 3239       /lib/libc-2.5.so
0fe49000-0fe59000 ---p 0013d000 fd:01 3239       /lib/libc-2.5.so
0fe59000-0fe5b000 r--p 0013d000 fd:01 3239       /lib/libc-2.5.so
0fe5b000-0fe5e000 rwxp 0013f000 fd:01 3239       /lib/libc-2.5.so
0fe5e000-0fe61000 rwxp 0fe5e000 00:00 0 
0fe71000-0fe7a000 r-xp 00000000 fd:01 3296       /lib/librt-2.5.so
0fe7a000-0fe89000 ---p 00009000 fd:01 3296       /lib/librt-2.5.so
0fe89000-0fe8a000 r--p 00008000 fd:01 3296       /lib/librt-2.5.so
0fe8a000-0fe8b000 rwxp 00009000 fd:01 3296       /lib/librt-2.5.so
0fe8b000-0fe96000 rwxp 0fe8b000 00:00 0 
0fea6000-0ff49000 r-xp 00000000 fd:01 3235       /lib/libm-2.5.so
0ff49000-0ff58000 ---p 000a3000 fd:01 3235       /lib/libm-2.5.so
0ff58000-0ff59000 r--p 000a2000 fd:01 3235       /lib/libm-2.5.so
0ff59000-0ff5d000 rwxp 000a3000 fd:01 3235       /lib/libm-2.5.so
0ff6d000-0ff70000 r-xp 00000000 fd:01 3226       /lib/libdl-2.5.so
0ff70000-0ff7f000 ---p 00003000 fd:01 3226       /lib/libdl-2.5.so
0ff7f000-0ff80000 r--p 00002000 fd:01 3226       /lib/libdl-2.5.so
0ff80000-0ff81000 rwxp 00003000 fd:01 3226       /lib/libdl-2.5.so
0ff91000-0ffa6000 r-xp 00000000 fd:01 3270       /lib/libpthread-2.5.so
0ffa6000-0ffb5000 ---p 00015000 fd:01 3270       /lib/libpthread-2.5.so
0ffb5000-0ffb6000 r--p 00014000 fd:01 3270       /lib/libpthread-2.5.so
0ffb6000-0ffb7000 rwxp 00015000 fd:01 3270       /lib/libpthread-2.5.so
0ffb7000-0ffb9000 rwxp 0ffb7000 00:00 0 
0ffc9000-0ffdc000 r-xp 00000000 fd:01 3259       /lib/libnsl-2.5.so
0ffdc000-0ffec000 ---p 00013000 fd:01 3259       /lib/libnsl-2.5.so
0ffec000-0ffed000 r--p 00013000 fd:01 3259       /lib/libnsl-2.5.so
0ffed000-0ffee000 rwxp 0
2

@Sirisha , Welcome, bits of a stack dump/backtrace is of limited use in helping you at this point. You have a malloc issue ...

Some debugging (logging) would be helpful (for you for starters - us secondly) - dump message payloads along with packet sizes etc, make sure there's no memory leaks

  • every malloc has a corresponding free (and that previously malloc'd memory is only freed once !) , calls to malloc are checked for success -.... ie !NULL ... - logging activities!

show code (erlang and c) (if 'sensitive' anonymise )
give details of c compiler - including command line compilation, erlang versions , which OS (name and version .... ), amount of memory on the box .....

try running with Valgrid , dbg/erlang/trace etc (presume for erlang debugging ) - been years since I touched any erlang code so - verify this for yourself first.