Hello,

I am not that experienced with Linux, and I am currently facing some issues.

The application I'm working on uses hundreds of threads. To optimize the memory usage, I am putting all my data inside a shared object (so).

The steps for this are as follows:

1. a C file (generated from an XML file) with lots of arrays (const char c1[] ="string" and const char* arr_const[] = {"s1", "s2", ...."sN"}, etc.)

2. a shell script to compile this to a shared object:
   gcc -g -m64 -Wall -fPIC -c bigfile.c
   gcc -m64 -shared -Wl -o <custom_new_name>.so bigfile.o

3. some code (on each thread) to open the shared object and get the refferences to my arrays from the shared object:
   void ** handle;
   handle=dlopen (<full_path_to_so>, RTLD_NOW);
   char* local_var = (char**) dlsym(*handle, "arr_const");

NOTE: the handles don't get unloaded until my application closes.

Using shared objects is mandatory for two reasons:

• optimized memory usage when launchig thousands of threads
• the ability to create new shared objects from new XML files and adding them to the application after it has been installed on client machines (with the help of text config files).

My problem is that the process described above does not work on my x86_64 CentOS 5.8 (dlopen returns NULL). The entire application is built with 64bit settings.
When using this on CentOS 32bit, with gcc using -m32 instead of -m64 for compiling the shared objects, and building my app with 32bit settings, everything is fine.

Any ideas ?
Thank you

I usually see gcc's -fPIC option used in Linux x86_64 architecture, the 64-bit loader may expect relocatable code. Try it.

I already used -fPIC when making the .o object.
I'll redeploy my app on 64bit architecture and add the -fPIC option to the second call to gcc, and let you know how that went.

Try perror after dlopen fails, so you can see why it fails.

There's another way you can share huge gobs of text among programs without wasting memory, by the way -- map it in.

struct stat buf;
int fd=open("/path/to/file", O_RDONLY);
void *mem;

fstat(fd, &buf);

// Read-only file mapping with shared memory
mem=mmap(NULL, buf.st_size, PROT_READ, MAP_SHARED, fd, 0);
if(mem != MAP_FAILED)
{
        printf("Mapped %d bytes of memory at %p\n", buf.st_size, mem);
}

// The contents of the file 'fd' are now available inside the block of memory
// pointed to by 'mem'.

Of course, this doesn't give you handy variables and arrays the way a shared object does, just a big block of raw data, but should be very portable.

Thanks, but we've already looked into this, and it would give us a lot of trouble in case we need debugging, or changing the ridiculous amount of arrays in the shared object.

right now I am looking at 3MB to 6MB of generated C code :wall:

EDIT: besides, I wouldn't want to trash my work for the past 3 weeks (including tons of nifty macros and other code changes I had to do because of switching to shared libraries), and go back to the drawing board with my tale between my legs

Try perror after dlopen fails, so you can see why it fails.

perror or dlerror ?
I tried dlerror so far, but nothing interesting there Didn't think about perror yet, I'll give it a try

EDIT:
Is there anything wrong with the way I build my .so file ?
It seems something crashes when trying to open it.

I just realized the implications of mapping in variables, not just functions. That's not guaranteed to work. There may be funky linker options to force variables to be exported, but I almost never see that done. Usually, you only export functions.

What people usually do is just keep a function around, which returns a pointer to all your relevant variables.

Something like this:

// shared.h
#ifndef __SHARED_H__
#define __SHARED_H__

struct everything
{
        const char **strings_a, **strings_b;
};

struct everything *get_everything(void);

#endif/*__SHARED_H__*/

// shared.c
#include "shared.h"

static const char *strings_a[]={ "string1", "string2", "string3", "string4" };
static const char *strings_b[]={ "string1", "string2", "string3", "string4" };
static const struct everything e={ strings_a, strings_b };

struct everything *get_everything(void) { return(&e); }

Then you load get_everything from your library, and call it to get pointers to all that shared data.

This sort of dynamic loading should be portable across not just linux but some other systems as well...

Thank you for that remark.

However, the reason why my stuff works on CentOS 32bit, and not on CentOS 64bit is way beyond my comprehension.

The code for loading the so and accessing the data insisde using dlsym is exactly the same (no ifdef/else constructs there).

The onyl thing that differs between the two platforms is the way I compile my shared object:

I used gdb on x64. (don't bother with the formatting about the code samples below, they are different in my code)
What I do is I first try to get my hands on a string in the so :

After this, I need to use this to generate the name of the next symbol I need to load from the shared object:

With gdb I realized that the decorator previously retrieved via address from the so, I actualy get some garbage address, and not the one I need. I can't do a print on the decorator from the gdb because it tells me that the address is out of bounds.

Any clue about this ?

EDIT:
Somehow, I found some weird behavior.
Currently, I have a platform independent function for loading a library:

int
util_load_library (const char *lib_file, void **handle)
{
  int err_status = NO_ERROR;
  char err_msg[ERR_MSG_SIZE];

  assert (lib_file != NULL);

  *handle =
#if defined(WINDOWS)
    LoadLibrary (lib_file);
#else
    dlopen (lib_file, RTLD_NOW);
#endif

  if (*handle == NULL)
    {
      err_status = ER_LOC_INIT;
      snprintf (err_msg, sizeof (err_msg) - 1,
		"Error loading library %s", lib_file);
      LOG_LOCALE_ERROR (err_msg, err_status, true);
    }
  return err_status;
}

The code above is in library.c, the declaration for the function is in library.h.
In util.c I include library.h, and open the so using the function above.

Somehow, if I lose the function, and simply dump the dlopen where I need it, it works fine on x64 as well.

Apparently, now I also have to determine the reason for this behavior, since it is not acceptable to duplicate the code (I am calling util_load_library in several places)

Make absolutely sure that this function is declared everywhere you're using it. Including the right header is no longer optional when dealing with 64-bit systems.

In 32-bit x86, you can get away with passing a pointer to undeclared externals because the assumed integer parameters are big enough to hold a pointer.

In 64-bit x86, integers are still 32-bits, but pointers are 64! It will still assume the parameters are 32-bit integers, and mutilate the 64-bit pointer into fitting in the 32-bit variable, causing crashes or weird misbehavior.

Even worse problems are when you have the function declared two ways in different places; for instance having one parameter be a long in one file, and just an int in another. This works in 32-bit because int and long are the same size, but in 64, long is suddenly 64-bits, and the parameters get mangled again. This can be very hard to track down because, unlike undefined externals, it's not a compiler error, it's a linker problem, and the linker has no way to warn you about it.

Well, it is declared in util.h as

int util_load_library (const char *lib_file, void **handle);

Then the function code from the previous post is placed inside util.c, which includes util.h.

After this, all the C files which call util_load_library include util.h

EDIT: could my problems be caused by the fact that some C files are compiled into another so, and loaded from there by the main app ?

Using #include that way makes it possible to put nearly anything in your code so without seeing the code, I really can't tell.

I still think there's linkage problems if it works outside the function but not inside. I'd add trace statements in the function itself like

int
util_load_library (const char *lib_file, void **handle)
{
  int err_status = NO_ERROR;
  char err_msg[ERR_MSG_SIZE];

  fprintf(stderr, "util_load_library('%s', %p)\n", lib_file, handle);
  if(handle) fprintf(stderr, "\thandle points to %p\n", *handle);

  assert (lib_file != NULL);

  *handle =
#if defined(WINDOWS)
    LoadLibrary (lib_file);
#else
    dlopen (lib_file, RTLD_NOW);
#endif

  if (*handle == NULL)
    {
      err_status = ER_LOC_INIT;
      snprintf (err_msg, sizeof (err_msg) - 1,
		"Error loading library %s", lib_file);
      LOG_LOCALE_ERROR (err_msg, err_status, true);
    }
  return err_status;
}

just a side note: why not to use shared memory (shmop,shmctl,shmget)?

1) because SHM is RAM; suddenly corruption is possible, and you have to initialize things.
2) shm is persistent. If you're insufficiently clever in setting things up, you can get junk from before or 'permission denied'.
3) Not as convenient as "give me variable x and let the system handle what's shared or not".

Because I pack a large number of data arrays in the shared object, and I would need to do more work to also implement some kind of memory map on that.
Besides, my shared objects will be patches to my application.

To wrap this thread up, I just want to thank you, Corona688, for your advice.
Although they didn't help me as much as I hoped, you definitely gave me some new things to feed the brain

Now hold on to you shorts, here comes the solution for my problem :wall:

As I said on my first post, on CentOS 32bit release everything was ok, but on 64bit, I got garbage addresses when performing dlsym, hence I assumed it was some kind of compiler issue. After compiling my shared objects with a lot of possible combinations of options for gcc and failing, I got the most retarded idea ever : I just moved util_load_library to another C+h file, and it worked without any complains. The C file where I moved it is one of the files from where I call my loader function.

I assume this had something to do with the fact that the initial C+h file could have been packed in some other so, and maybe this messed up the address space on the 64bit version.

Anyway, just wanted to let you know I solved my issue in the weirdest way possible

Thanks a lot for you support!

As said, not including the right headers is a dire error in 64-bit. It sounds like you needed to include dlfcn.h

dlfcn.h was included in each case in the C file containing the function body.
That was not the issue.
If there was a problem with the headers, I assume the 32bit version wouldn't have worked either.

As I already clarified, there is a big difference between the behavior you get with missing headers on 32-bit, and the behavior you get with missing headers in 64-bit.

Take this program:

int main(void)
{
        printf("Hello World\n");
        return(0);
}

Oops, didn't include stdio.h

On 32-bit, the compiler assumes printf() takes a single 32-bit integer. Which works out okay, since a pointer is also a 32-bit value.

On 64-bit, it still assumes printf takes a single 32-bit integer -- causing it to fold, spindle, and mutilate that 64-bit pointer to fit! If you're lucky, the program crashes. If you're unlucky, you hit valid memory somewhere else and print something unintended.

You get the same problem with things which return pointers.