Huge PI in vmstat

UNIX Solaris
1

This is something nowbody around me can explain:
vmstat (-S 5) shows a huge number of PI but when I try to monitor it in parallel with iostat - there is no IO activity to be seen that would correspond to this.

I have 16G RAM and 32G swap file.
I'll really appreciate if somebody can explain it.

kthr memory page disk faults cpu
r b w swap free si so pi po fr de sr s0 s1 s2 s3 in sy cs us sy id
0 0 68 32064376 282736 0 0 86 0 0 0 0 8 5 0 0 302 176 335 1 4 96
0 0 68 32064376 280200 0 0 6 0 0 0 0 0 0 0 0 286 131 305 0 2 97
0 0 68 32064376 280168 0 0 0 0 0 0 0 0 0 0 0 277 120 296 0 1 99
0 0 68 32064376 280168 0 0 6 0 0 0 0 1 0 0 0 283 132 298 0 2 97
0 0 68 32067088 285552 0 0 8 0 0 0 0 1 0 0 0 285 147 307 0 2 98
0 0 68 32066752 289824 0 0 157896899627 1958 5505 49408 89202 22 19 0 0 348 1282 379 3 6 91
0 0 68 32064072 292312 0 0 0 0 0 29192 0 13 13 0 0 306 125 305 0 3 97
0 0 68 32064072 291120 0 0 8 0 0 17256 0 0 1 0 0 282 136 312 0 2 97
0 0 68 32064072 290784 0 0 0 0 0 10208 0 0 0 0 0 279 123 306 0 2 98
0 0 68 32064072 290208 0 0 13702786999 0 0 6048 0 0 1 0 0 283 146 300 1 1 98
0 0 68 32064072 288408 0 0 14 0 0 3584 0 1 0 0 0 284 190 319 1 2 97
0 0 68 32064072 283880 0 0 6 0 0 2136 0 0 1 0 0 288 140 311 1 2 97
0 0 68 32064072 283792 0 0 0 0 0 1248 0 2 2 0 0 293 116 305 0 4 96
0 0 68 32064072 283608 0 0 5 0 0 448 0 0 0 0 0 291 134 315 0 1 99
0 0 68 32064072 281640 0 0 8 0 0 0 0 0 1 0 0 293 159 322 1 2 97
0 0 68 32064072 280408 0 0 6 0 0 0 0 0 0 0 0 290 130 303 0 2 97
0 0 68 32064072 280272 0 0 16 0 0 0 0 1 1 0 0 289 135 317 0 2 98

2

vmstat output is almost unreadable when you do that. Please just cut and paste the output and surround the output with set of code tags. If you don't understand code tages, just do the cut and paste. If you leave the spaces one of us can add the code tags and make it readable.

My first inclination would be to check that my kernel is patched correctly. I/O to or from a memory mapped file can generate lots of page-in's with no I/O provided the pages are zero-fill-on-demand. But I have never seen numbers that high. Modern solaris version have a -p flag on vmstat and if you use that, zfod pi's will be listed in the mf column.

3

Thank you very much, Perderabo. I do not understand coding, I just changed the size of the font to get it in one line.
Originaly it looks like that ( with -p ) - and it is not api but fpi
We just rebooted the machine and numbers in swap and free was changes - but PI still jumps to skyhigh level - I googled a lot last 3 days but I can not see anybody ever had the same problem :frowning:

   swap  free  re  mf  fr  de  sr  epi  epo  epf  api  apo  apf  fpi  fpo  fpf
 46587928 15396896 23 61 0  0   0    0    0    0    0    0    0    0    0    0
 46583400 15393392 12 93 0  0   0    0    0    0    0    0    0    0    0    0
 46583400 15393376 0 0  0   0   0    0    0    0    0    0    0 13716462422 0 0
 46583400 15393368 0 0  0   0   0    0    0    0    0    0    0    0    0    0
 46583400 15393368 0 0  0   0   0    0    0    0    0    0    0    0    0    0
 46583400 15393368 45 253 2 0   0    0    0    0    0    0    0    0    2    2
 46583400 15393368 0 0  0   0   0    0    0    0    0    0    0    0    0    0
 46583400 15393368 0 0  0   0   0    0    0    0    0    0    0    0    0    0
4

in my views, from the above snapshot, the stats are not consistent, systems that experience performance problems will show symptoms of consistent stats. We not talking about consistent for only 10 or 20 minutes but for hours then only is it consider performance problems.

5

thank you. I see now, we do not have it "for hours" - just 2-10 min, and then it is back to 0. The above example was from machine doing nothing just after restart. It's just something I thought might be a reason why our machine can not function without regular reboot every 2-3 months :slight_smile:

Anyway - thanks, I'll start looking into another direction.

6

Are you running vxfs on any of your disks? If so what version?

There is a version of vxfs that overwrites a memory address that is used by kstat for reporting the vmstat PI values.

7

Oh just found it again, the following version has the problem:
VERSION: 4.1,REV=4.1B18_sol_GA_s10b74L2a

8

Oh!
That's what I have, does it explain the vmstat values?
So it is just a problem of representation then?

# pkginfo -l VRTSvxfs
PKGINST: VRTSvxfs
NAME: VERITAS File System
CATEGORY: system,utilities
ARCH: sparc
VERSION: 4.1,REV=4.1B18_sol_GA_s10b74L2a
BASEDIR: /
VENDOR: VERITAS Software
DESC: Commercial File System
PSTAMP: VERITAS-4.1FS-2005-02-24
INSTDATE: Mar 01 2006 09:49
HOTLINE: (800) 342-0652
EMAIL: support@veritas.com
STATUS: completely installed
FILES: 226 installed pathnames
33 shared pathnames
4 linked files
47 directories
74 executables
5 setuid/setgid executables
49155 blocks used (approx)

9

Yep, those values are incorrect. You'll need to install VRTSvxfs 4.1MP2 or later.

10

Thank you, Solarius. We'll do it.