I'm looking for good web log analysis software for 95% Apache / 5% IIS logs. There's a lot out there, but our needs are unusual. We're a web hosting company with a large number of VPS servers. For a variety of reasons, we like to develop models of virtual server activity. Among other things, this means obtaining Apache logs from a large number of virtual servers and analyzing them for all sorts of characteristics. Each virtual server maintains its own set of Apache logs. These logs are pulled to a central analysis server and analyzed.
We're not really interested in analyzing in a domain-specific way. We're much more interested in creating composite reports on each of many virtual servers. We can then create a "typical virtual server" model from such reports.
The following would be very useful:
1) The ability to calculate mean and standard deviation of page size. This is far from an exact science, of course, but statistically-reasonable estimates would be nice.
2) Same as 1), but for number of elements per page.
3) Transaction rates, preferably per minute or second.
4) All the other "stuff" that's common to most log analyzers.
Are there any packages that can do what I've described here?