mprotect fails with ENOMEM in text segment

UNIX AIX
1

Hi guys,

I use AIX version 5 on IBM Power 5+ machine. I am currently trying to experiment with sort of self-modifying code, like this:

ucontext_t ut;
getcontext(&ut);
int iar = ut.uc_mcontext.jmp_context.iar;
int pageSize = getpagesize();
int rest = iar % pageSize;
void *ptr = iar - rest;
mprotect(ptr,pageSize,PROT_READ | PROT_WRITE | PROT_EXEC);

However, mprotect fails with ENOMEM ("Not enough space"). The above code works perfectly on Linux machine.

I also set environment variables MPROTECT_TXT and XPG_SUS_ENV to ON, with no effect.

Is there a way to make it work, or is there any other way to make changes to program's own text segment?

Regards

2

Registers are os architecture specific?

3

Pointers are not integers. On some platforms integers can hold pointers safely but that's just coincidence; here, you're cramming 64-bit values into 32-bit variables and hence truncating them quite a lot. If you include stdint.h I think you can use the pointer-safe ptrdiff_t type instead of int

4

I've been trying to get over this problem for some time now, without success. I use type intptr_t as you advised:

uintptr_t ptr2 = iar;

Then I have a loop with mprotect:

mprotect(ptr2+i,pageSize,PROT_READ | PROT_WRITE | PROT_EXEC);

It failes with "Invalid argument" every time the address is not aligned to page size (as expected) but also fails with "Not enough memory" when the (ptr2-i) is properly aligned. Any more ideas?

5

That is still an unsigned int pointer type, implying you are assigning only 32 bits. IBM says some pages are not mprotect friendly this way: pSeries and AIX Information Center

You might need that env variable. I am familiar with self-modifying code from machine language, what do you want it to do?

6

Thank you for a quick reply.

I don't think that's the problem since the routine works well when performed on non-text segment (for example on page which contains area acquired via malloc).

I have these in my ~/.profile:

MPROTECT_TXT=ON
XPG_SUS_ENV=ON

Is this enough for these env vars to be 'seen'? I tried different values of those, no effect.

I am trying to write code to reload the program text from the original executable file in case an error occurs (e.g. a hardware error which scrambled some memory). Is there another way of doing it, instead of mprotect/memset ?

7

If anyone was wandering, switching to AIX 6.1 did the trick. Documentation for 6.1 states that mprotect for text segment will work with MPROTECT_TXT env variable set to ON and it does. Documentation for 5.3 does not say such thing apparently, I primarily thought that the call in both versions will have the same effect, but apparently it doesn't.

Anyway, thanks for help!