Nested user groups

Is there a command to nest a group in another group in AIX.

Your question is unclear, please restate giving an example of what it is you want to do and (ideally) a reason for ..

if I understand you correctly than the answer is no - AIX groups can only contain users, not other groups - but any user can be member of multiple groups. As long as you do not use extended permissions (ACLs) anywhere, you should not need nested groups like you have them in windows (local and global groups).
Regards
zxmaus