One of our project has exceeded its assigned max-memory-locked by 3 times .. The said project is using around 9 gigs as described by rss parameter in prstat -J .. and the max-project-memory parameter has been defined as 3gigs .. is it normal or we are monitoring the project memory usage in wrong way ..
prstat -J is used to monitor the CPU usage per project, not the memory.
Check this link and read "Monitoring the RSS of a Project" (from page 104)
Thanks for the info but RSS is not same as the max locked memory .. 
RSS is set with max-rss parameter in project/zone.