There are quite a lot of free and efficient tools too allowing to track down memory leaks under Solaris, including dbx run time checking.
http://www.unix.com/sun-solaris/73873-how-find-memory-leak-solaris.html\#post302216415
Back to the original question, here is a sample code to monitor a process size an rss:
#include <procfs.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char **argv)
{
char psinfo[32];
struct psinfo p;
int fd;
pid_t pid;
if(argc>1)
pid=atoi(argv[1]);
else
exit(1);
sprintf(psinfo,"/proc/%ld/psinfo",pid);
fd=open(psinfo, O_RDONLY);
if(fd!=-1)
{
read(fd,&p,sizeof(p));
printf("size=%d rss=%d\n", p.pr_size, p.pr_rssize);
close(fd);
}