Limiting log size in the application is more promising than logrotate, because the application knows its messages best and can rotate at the best point. Can. Maybe your application is buggy.
Logsize 0 usually means "disable". But I don't know your application...
Going for logrotate, copytruncate is most likely to succeed, compared to the default rename(move).
What do you mean by rotating logs inside the application. How'd you do it in glassfish. Note that glassfish doesn't support log compression and archiving.
What do you mean by copytruncate likely to succeed.
Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and optionally creating a new one. It can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. When this option is used, the create option will have no effect, as the old log file stays in place.
That is inside the application glassfish. Should work best. What's wrong with it?
Logrotate does not know the application.
The default is to rename the logfiles, and in postrotate or sharedscripts give commands to notifiy the application (otherwise it would continue writing to the renamed logfile, in the case of compress would continue writing to the deleted file).
The alternative is copytruncate that copies the file then truncates it (sets the write pointer to 0 that should lead to a rewrite). Writes after the copying but before the truncation are lost. Nevertheless this works quite well with normally opened files. However, specially opened files are only temporarily truncated; the next write will restore (and continue on) the original file - the truncation does not work.