OK, I think I got it now. I'm going to list every source that I used to research this. For more info you may want to read them yourself.
Originally, Unix used a structure called a buffer cache for all file i/o (Bach, 1986 38-58) (Leffler, et al, 1989, 208-213). HP-UX continues to use a buffer cache (Sauers and Weygant, 2000 189-190) but it can dynamically change in size. This is the sort of Unix system that I'm used to. On this kind of Unix system, a page-out is almost always due to paging activity causing by a shortage of free memory. The only exception that I know of is that memory mapped files do all i/o via the paging subsystem (Vahalia, 1996 438-440) and a program can invoke msync() to force the data to disk (Stevens, 1992, 411).
I have always used page-outs to see if a system is paging. No problem there, that's practically the definition of "paging". It may be that I really should be looking at "scan rate". There are two reasons for this: first, when the scanner first starts to run, if it is running a the slowest possible speed, it may take longer that one minute for the first page to be stolen is identified (Mauro and McDougall, 2001 180-182). Also, if the page has not been modified since it was paged-in, it will not be paged-out. Instead it is simply placed on the free list (Bach 1986, 296) (Leffler, et al, 1989, 154). So for a traditional unix system, scan-rate may be the more reliable metric. The only time that you are scanning pages is if you are low on memory.
At some point a new way of doing things emerged. The earliest reference to it that I can find is System V Release 4 (Goodheart and Cox, 1994, 283-287). Solaris has migrated to it (Mauro and McDougall, 167-190, 589-599). Linux also uses this model (Mosberger and Eranian, 2002, 23-25).
With the new way, the buffer cache has shrunk in size. It is only used for metadata. The data from files is stored in pages. All of memory becomes a "buffer cache". Sync operations cause page outs. After memory is mostly full, we need the page stealer to free up pages so new data can be read.
With the new way it is normal to be out of memory. Mauro and McDougall say (page 171) "The free list is generally very small, since most pages that are no longer used by a process or the kernel still keep their vnode/offset information intact." And on page 594: "Have you ever noticed that when you boot your machine there is a lot of free memory, and as the machine is used, memory continues to fall to zero, then just hangs there? Here is why: the file system is using all available memory to cache the reads and writes to each file--and it's completely normal."
This is not say that everything is cool. Sun is having problems with the new approach. On page 596: "If your system seems slow while file I/O is going on, it's because your applications are being paged in and out as a direct result of the file system activity." They go to say that Sun invented a kludge to help this situation called "priority paging" in Solaris 7. You can run this in 2.6 if you get patch 105181-3. And for 2.5.1, you need patch 103640-25. Another book (Musumeci and Loukides, 2002) says on page 116 " The effect is generally excellent. Desktop systems and on-line transaction processing environments tend to feel much more responsive...as much as a 300% performance increase."
I'm just scratching the surface here, but I'm not going to keep typing. Buy some of the books I listed below. And if you're running Solaris, look up "priority paging" on Sun's website.
(Bach 1986) The Design of the Unix Operating System by Maurice J. Bach
(Leffler, et. al., 1989) The Design and Implementation of the 4.3BSD Unix Operating System by Samuel J. Leffler, Marshall Kirk McKusick, Michael J. Karels, and John S. Quarterman
(Sauers and Weygant, 2000) HP-UX Tuning and Performance: Concepts, Tools, and Methods by Robert F. Sauers and Peter S. Weygant
(Vahalia, 1996) Unix Internals, The New Frontiers by Uresh Vahalia
(Stevens, 1992) Advanced Programming in the Unix Environment by W. Richard Stevens
(Goodheart and Cox, 1994) The Magic Garden Explained: The Internals of Unix System V Release 4, An Open Systems Design by Berny Goodheart and James Cox
(Mauro and McDougall, 2001) Solaris Internals Core Kernel Architecture by Jim Mauro and Richard McDougall
(Mossberg and Eranian, 2002) IA-64 Linux Kernel Design and Implementation by David Mosberger and Stephane Eranian
(Musumeci and Loukides, 2002) System Performance Tuning, 2nd Edition by Gian-Paolo D. Musumeci and Mike Loukides
