How do you run a site that gets roughly 3 million daily hits, while not killing server performance?  Well, the first step is to get ride of Apache – far too resource intensive to handle the load.  Lighttpd is a light weight webserver designed to do just this!  I recently spent some time performance tuning the site, and wanted to share the config tweaks here in the event that it may help someone else.

The server we are running is  Debian Lenny 64bit box, hosted by SoftLayer.  Before I even got to tuning Lighty, I was getting synflood errors.  I dug into it and discovered the hosting provider had put some checks in place to help prevent DoS attacks.  In “/etc/sysctl.conf” I change the following:

net.ipv4.tcp_syncookies=0
net.ipv4.tcp_synack_retries = 5

Now that I solved that problem – the next thing I went to do was make sure there would not be a restriction on the number of file handlers that the web process could use.  To set this up, I added the following in “/etc/security/limits.conf“:

www-data        soft    nofile          4096
www-data        hard    nofile          10240

This will allow the lighttpd user (www-data) to have a soft limit of 4096 handlers, with a hard limit of 10240.

In order for this file to work, you have to enabled it in PAM.  In “/etc/pam.d/su“, find and uncomment the following line:

session    required   pam_limits.so

There are two main config files that we need to work with – the first is the main config “/etc/lighttpd/lighttpd.conf” which does not have alot of changes.  The following were added to tune performance:

server.max-fds = 40000
server.max-keep-alive-requests = 100
server.max-keep-alive-idle = 2
server.max-connections = 10000

I also enabled the “compress” and “expire” modules.  The compress module is using the default settings:

#### compress module
compress.cache-dir          = "/var/cache/lighttpd/compress/"
compress.filetype           = ("text/plain", "text/html", "application/x-javascript", "text/css")

Now, in order to take advantage of this, you need to enable compression in php.  Since this is using the cgi build of php, the file to change is “/etc/php5/cgi/php.ini“.  There is only one option to change here:

zlib.output_compression = on

In PHP, I also enabled APC for opcode caching.  I currently have this running with the following settings:

apc.enabled=1
apc.shm_size=128M

The expires module is currently configure as follows:

expire.url                  = ( "/" => "access 6 hours")

The second lighttpd config file that needs to be changed is the fastcgi config “/etc/lighttpd/conf-enabled/10-fastcgi.conf“:

fastcgi.server    = ( ".php" =>
((
"bin-path" => "/usr/bin/php-cgi",
"socket" => "/tmp/php.socket",
"max-procs" => 4,
"idle-timeout" => 20,
"bin-environment" => (
"PHP_FCGI_CHILDREN" => "16",
"PHP_FCGI_MAX_REQUESTS" => "10000"
),
"bin-copy-environment" => (
"PATH", "SHELL", "USER"
),
"broken-scriptfilename" => "enable"
))
)

Well, that about does it.  Hopefully the above notes and insight help others out!  I could not have done this without the help of the Lighttpd community – they are great and always willing to help!