I have no idea. Maybe your tape is simply full? Maybe you used the wrong device (with libraries picking the right one from several pseudo-devices can be quite tricky). Maybe something else.
This is correct.The AIX tar will throw error 0511-825 and quit if you attempt to tar a file with 8GB or more because of limitations in the ustar header format. The same is true, btw., for cpio using the ustar header format ("-Hustar").
You could either use GNU-tar, which doesn't adhere to the POSIX-standard (which is in itself not a problem, it just may pose compatibility problems if you work in a heterogenous environment) but allows files to be up to 50 GB in size or you use some other means to back up your data - TSM, whatever, there are quite a few.