Secondary groups not working with NFS (+LDAP)

I�m using LDAP for groups and NFS for home dirs. My problem is as follows:
I only have a few groups, so it's not the problem everyone else had. When I've mounted a disk over NFS, I need to have my primary group in order to read in the groups I'm a member of. Secondary groups is not working.

 root@machine:/home/user# smbldap-groupshow secret
...
gidNumber: 1504
displayName: secret
memberUid: user,anotheruser
 root@machine:/home/user# su - user
 user@machine:~$ groups
users secret
 user@machine:~$ ls -ald ../secret/
drwxr-x--- 12 anotheruser secret 4096 2009-07-27 15:39 ../secret/
 user@machine:~$ cd ../secret/
bash: cd: ../secret/: Permission denied
 user@machine:~$ ls ../secret/
ls: cannot open directory ../secret/: Permission denied

But it works if I change the group to primary by hand with newgrp:

user@machine:~$ newgrp secret
user@machine:~$ cd ../secret/
user@machine:/home/secret$ ls
Nice secrets.txt
 

But my users cannot be expected to do this!
It works on the server holding the user files. But not over NFS.

I've tested this on clients: Ubuntu: 9.10 Karmic, 9.04 Jaunty, 8.10 Intrepid
The NFS server is running: Ubuntu 9.04 Jaunty.