How can i Globally set the maximum core memory a single process can take. IE, i want to set that no single process may get more than 11GB.
I am running red hat enterprise unix 4.
Don't know if Red Hat has this command, but see if it has "ulimit"
On solaris, it is the -v parameter
-v maximum size of virtual memory \(in kbytes\)ulimit is specific to bash on red hat linux. and i don't need to limit virtual memory i need to limit actual memory usage by a task. And i also need to limit it globally, not on a task by task basis. Red hat has so far responded with the following:
There is an option to limit the rss usage by a process for a given user using /etc/security/limits.conf, but I found references that it does not work as expected.
I am escalating this to a senior engineer for further investigation, you should get an update soon.
Do you mean physical silicon memory rather than virtual spilled over into swap?
How big is your machine?
The machine is a dual opteron system with 12GB of ram and 20GB of swap configured. I want to prevent a single process from getting more than lets say 90% of ram. (maybe max around 10GB). I have one user running a development progrm which has grown to 14.3Gb on this machine which really killed the machines ability to do stuff.
14.3Gb 
Is it a Java environment that is not running a garbage collector?
We had a similar kind of issue recently
but not 14.3 GB, that was around 3 GB
where there were construction of too many hash maps.
when drilled down to the solving the problem; we diagnosed that after using the hash components we made sure we deleted the contents in the hash one by one .
Well, that's a quite an easy one to address and resolve. 
No these are developers running client circuit testcases thru simulation. They can get quite large. but on this one machine i want it restricted.
Perhaps what you're looking for is as simple as kernel params
I don't know if these will have the desired effect for you or not, but its worth a try. You can set these with the sysctl command on the fly, and they can also be placed in your /etc/sysctl.conf for persistence. See "man sysctl" for more info than a dullard like me can give you.
Parameters for shared memory
kernel.shmmni: maximum number of shared segments, system wide
kernel.shmall: maximum shared memory, system wide
kernel.shmmax: maximum shared memory segment size
Hope this helps.
Currently, there is no possible way to limit rss on a RHEL4/5 system. This is a known issue upstream. The best option would be limit as.
There isn't a good way to implement this, as it opens up for the possiblity of DoS attacks if you set the RSS low enough. Anything that uses glibc will have a good chunk of RSS dedicated to glibc, which is going to hurt them. There is also the question of what to do in NUMA situations.