Set uid / grpid exec() turns of LD_LIBRARY_PATH so the program cannot be attacked by a bogus dynamic lib. Either it must be statically linked or compiled with lib path in -R.
Thank you for all the response. The issue was the env variable LD_LIBRARY_PATH.
Quoting from internet
For security reasons, LD_LIBRARY_PATH is ignored at runtime for executables that have their setuid or setgid bit set. This severely limits the usefulness of LD_LIBRARY_PATH.
The exit error code which I was getting : 127 unable to open shared object. When I did an strace I could see that, it was searching for the depedendent libraries on /usr/lib and not the ones present in LD_LIBRARY_PATH. So for testing I created a link from the actual library path to /usr/lib and it worked.