My company run a CMS (typo3) on LAMP on RH linux.
Basically all users have same typo3 page as home page and thousands of users starts their browser every morning causing server very slow or even to hang (swap full because of too many apache processes)..
99% of the traffic is this home page which take 2s in a non busy time to be delivered.
I improve this time to 1.5s using php cache. Others mysql/apache tunning have so far not given me any significant improvement (I have been trhu web pages on apache/mysql tunning and test differents things).
As far I have seen mysql is ok and httpd processes are taking all cpu/mem.
My question is :
1 - As this home page (php page, 30K) in 99% of the traffic how can i improve the process not getting 1000 times the same request to my apache server.
I have no way to modify this home page policy, I can just work on the infrastructure side. I was thinking on some proxy/cache server but have no experience on that. Any efficient and open source solution ?
2 - Apart from php cache any hint to help me improve apache performance ?
I did not find anything else number of thread and related stuff in apache.
It is bad that you can't modify optimize the home page, 30k php page sounds like a lot, combined with your users' concurrency (how much is it ? average, peaks ? ). Apache can be tuned in many ways, depending on the usage. Ironically, this is what I'm working on currently, so I will have to ask about the version of apache you're using. Depending on the answer I should be able to provide more info. Also, let us know if you are using SSL somewhere to serve content, and, if it's not secret, what are the hardware parameters of the server. For better PHP handling, I can suggest FastCGI - I have seen very good results with it in the past.
1a. APC cache was the quickest way to get the most improvement in speed on my server. Memcached is also an option, but only has benefits above APC when you can distribute the caching load to backend servers or if you have multiple front-end servers.
1b. Squid. See Squid (software) - Wikipedia, the free encyclopedia. You put your LAMP server on another port, and proxy it using SQUID. Squid will cache as much as can be cached and fetch from your LAMP server. If most of your content is dynamic, this may not help much.
2a. You absolutely must minimize swap in such an environment. If you start going to swap, your system loses.
In sysctl.conf, change:
vm.swappiness = 1
and "tune" your swap to use a swap file that's about 80M. If you need more swap then that, your system will crash. Better to let Apache think it cannot get more memory and return 503 (server busy) errors than to wait forever.
2b. You can play around with the following:
MinSpareServers # keep this at least 5 or 10.
MaxSpareServers # no problem to set this 1/4th max.
ServerLimit # set to what your server can handle, memory-wise
MaxClients # Set to what your server can handle, memory-wise
KeepAlive # keep on
KeepAliveTimeout # keep to 30 seconds or less
HostnameLookups # set to OFF
2c. More tuning at the TCP level. See sysctl parameters such as:
net.core.somaxxconn
net.core.netdev_max_backlog
net.ipv4.tcp_keepalive_intvl
net.ipv4.tcp_tw_reuse
net.core.optmem_max
net.core.rmem_max
net.core.wmem_max
net.ipv4.tcp_rmem
net.ipv4.tcp_wmem
net.ipv4.tcp_congestion_control
Also, LOWER the amount of shared memory, to ensure you have more memory for HTTP. (Note: This may not work if you are running MySql or postgres on the same server!)
kernel.shmmax
kernel.shmall
2d. If you have the memory, put all your HTTP/PHP files onto a ramdisk. Otherwise, install everything on a FAST internal (or SAN) disk. When I say FAST, I mean hardware-RAID1 15k RPM SCSI Fiberchannel. Store all log files on a SEPARATE disk, which can be any junk drive.
My apache version is : Apache/2.0.52 with 4x2.8 mhz xeon with 3gb RAM and 2gb swap.
On dev I got VM machine with 2x1.86Mhz xeon and 2.5gb RAM and 2Gb swap (I do all tests there).
99% of the traffic is when users starts their browser around 9h.
A lot of connections (may be 1000 users).
I do my tests on dev with curl and ab but did not experience any swap pb (i did not start more than 100 thread each time).
I use eaccelerator for cache because of my boss choice, I tried mmcache and got slightly better results.
1a : I will try APC.
1b : as typo3 is php script and home page is index.php I wonder how squid will help. I thought that squid is not easy to setup, is it worth to try ??
2a : I agree with you better having 503 than swapping.
I understand vm.swappiness, but it's not clear on what you say about tunning swap, is it trying different values on swappiness ? It looks tricky
2b : Some parameters are not clear for me especially ServerLimit/maxclient. I am a little bit reluctant to change that as I am not sure of what I am doing. I'l take deeper look in documentation.
keepalive is on and I will try to play with the value.
But most of my connections are 1 time connection, user start browser ask home page and that's all.
For 1 connection like that I was gessing that I do not need a high keepalive as I got only 1 http request, I am right ?
Is there any relation between tcp keepalive and apache keepalive ?
Finally as I tried to emphasize, I got 99% request on the same page which is dynamic page BUT do not change 99% of the time.
Can I cache that kind of things in any proxy or cache to avoid going thru all the layers (browser httpd mysql httpd ...).
And as otheus said I rather like to have 503 but quick response than having server crashing.
Maybe the swap tuning is my best help in this case ???
I'm not really sure. If the ENTIRE home page is from PHP, then it won't. However, if there's a lot of media, static JSS, static CSS on the page, it might help. Then again, if it's just one page, and there's a fixed number of media files, it won't -- HTTP and the OS should cache most of that.
After you move /var/log to its own disk, find the disk least used in your system. Say it's already mounted as /vol3.
That's it. Now go into /etc/fstab and change the /swap mountpoint. Finally, make sure rc.sysinit does the right thing. Do:
grep -n -E "swapon|mount .*-a"
If mount comes before swapon, you're fine.
Nothing to fear here.
This is straight from Apache documentation. In your case, you want to set the maximum servers to something like 1024 (!) and the MinSpareServers to something like 128 or maybe even 512.
You got it! You might not need at all, but I am guessing your home page also consists of graphics, JS source files, CSS files, etc., all coming from your server. So its better to keep it on, but to timeout after only a few seconds.
Only if you have lots of connections all over the internet. If everyone is coming from a LAN or something, you don't need to touch it.
If you know the parameters that change it, then sure! Just pre-generate the page and serve that page. If the condition is determined on IP or COOKIE or some header info, you can use Mod_rewrite to give redirect the user to the dynamic one.
What's MySQL doing in the picture? Is the DB read-only? If so, can you fit it on a RAM-disk?
Run this command before and after each set of changes you make, assuming you make one change per day. Post the early and improved results here.
These numbers are too high and will cause performance problems.
FYI, our site is quite busy (do you have more traffic than this site?) and our numbers are much lower!
Timeout 20
ServerLimit 250
#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On
#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 1000
#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 2
...
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
StartServers 30
MinSpareServers 25
MaxSpareServers 100
MaxClients 250
MaxRequestsPerChild 4000
</IfModule>
I was trying to do some "manual" benchmarking with Apache's flood, but even though I have two instances with 100 threads each, it causes only 4 slave servers to go into the "run" state. Do you know of another tool for load testing?
The only way we can accurately tune this site is on the live traffic over many days. I don't think it is really possible to tune performance very well with tools that flood a server with traffic. I did run ab a few times, but it is really a waste of time. Here is the link:
The trouble is that tuning requires setting many parameters over time, like key buffers, query cache, timeouts, max connections, etc. So many parameters and each parameter is different.
I noticed the other day that someone ran a flooding tool against this site, because we reached maximum connnections, which only happens when a flooding tool is used Please do not do this. Why? Because it completely ruins the tuning analysis. Flooding is not real traffic and it is not even close to helpful.
In other words, what happens with flooding tools is so different than with real world LAMP.
There is some good reading on the net, for example:
But even these recommendations are only useful as background reference, because each site is different, the hardware is different, the traffic is different, the database queries are different, the JavaScript is different, etc etc.
I used to read all of these, and they are useful; but at the end of the day, LAMP tuning is based on knowing the system details, hardware, software, database structure, traffic pattern, query pattern, load etc. Tools cannot come close to simulating the real traffic (load, queries, timeouts, dropped connections, etc), especially on a site like this.
If the typo3 page is the same for most users and does not depend on session-specific or IP-specific data, I think squid will help. Like I said, you can use mod-rewrite to go to squid if there's session/cookie data, and to typo3 if there isn't.
