Recent Posts
Sponsors
 |
Optimizing NGINX and PHP-fpm for high traffic sitesAdrian Singer, 04-20-2014 |
After 7 years of using NGINX with PHP, we learned a couple of things about how to best optimize NGINX and PHP-fpm for high traffic sites.
1. TCP Sockets vs UNIX domain sockets
UNIX domain sockets offer slightly better performance than TCP sockets over loopback interface (less copying of data, fewer context switches).
upstream backend
{
# UNIX domain sockets
server unix:/var/run/fastcgi.sock;
# TCP sockets
# server 127.0.0.1:8080;
}
If you need to support more than 1,000 connections per server, use TCP sockets - they scale much better.
2. Adjust Worker Processes
Modern hardware is multiprocessor and NGINX can leverage multiple physical or virtual processors.
In most cases your web server machine will not be configured to handle multiple workloads (like providing services as a Web Server and a Print Server at the same time) so you will want to configure NGINX to use all the available processors since NGINX worker processes are not multi-threaded.
You can determine how many processors your machine has by running:
On Linux -
cat /proc/cpuinfo | grep processor
On FreeBSD -
sysctl dev.cpu | grep location
Set the worker_processes in your nginx.conf file to the number of cores your machine has.
While you're at it, increase the number of worker_connections (how many connections each core should handle) and set "multi_accept" to ON, as well as "epoll" if you're on Linux:
# We have 16 cores
worker_processes 16;
# connections per worker
events
{
worker_connections 4096;
multi_accept on;
}
3. Setup upstream load balancing
In our experience, multiple upstream backends on the same machine, produce higher throughout than a single one.
For example, if you're looking to support 1,000 max children, divide that number across two backends, letting each handle 500 children:
upstream backend {
server unix:/var/run/php5-fpm.sock1 weight=100 max_fails=5 fail_timeout=5;
server unix:/var/run/php5-fpm.sock2 weight=100 max_fails=5 fail_timeout=5;
}
Here are the two pools from php-fpm.conf:
<section name="pool">
<value name="name">www1</value>
<value name="listen_address">/var/run/php5-fpm.sock1</value>
<value name="listen_options">
<value name="backlog">-1</value>
<value name="owner"></value>
<value name="group"></value>
<value name="mode">0666</value>
</value>
<value name="user">www</value>
<value name="group">www</value>
<value name="pm">
<value name="style">static</value>
<value name="max_children">500</value>
</value>
<value name="rlimit_files">50000</value>
<value name="rlimit_core">0</value>
<value name="request_slowlog_timeout">20s</value>
<value name="slowlog">/var/log/php-slow.log</value>
<value name="chroot"></value>
<value name="chdir"></value>
<value name="catch_workers_output">no</value>
<value name="max_requests">5000</value>
<value name="allowed_clients">127.0.0.1</value>
<value name="environment">
<value name="HOSTNAME">$HOSTNAME</value>
<value name="PATH">/usr/local/bin:/usr/bin:/bin</value>
<value name="TMP">/usr/tmp</value>
<value name="TMPDIR">/usr/tmp</value>
<value name="TEMP">/usr/tmp</value>
<value name="OSTYPE">$OSTYPE</value>
<value name="MACHTYPE">$MACHTYPE</value>
<value name="MALLOC_CHECK_">2</value>
</value>
</section>
<section name="pool">
<value name="name">www2</value>
<value name="listen_address">/var/run/php5-fpm.sock2</value>
<value name="listen_options">
<value name="backlog">-1</value>
<value name="owner"></value>
<value name="group"></value>
<value name="mode">0666</value>
</value>
<value name="user">www</value>
<value name="group">www</value>
<value name="pm">
<value name="style">static</value>
<value name="max_children">500</value>
</value>
<value name="rlimit_files">50000</value>
<value name="rlimit_core">0</value>
<value name="request_slowlog_timeout">20s</value>
<value name="slowlog">/var/log/php-slow.log</value>
<value name="chroot"></value>
<value name="chdir"></value>
<value name="catch_workers_output">no</value>
<value name="max_requests">5000</value>
<value name="allowed_clients">127.0.0.1</value>
<value name="environment">
<value name="HOSTNAME">$HOSTNAME</value>
<value name="PATH">/usr/local/bin:/usr/bin:/bin</value>
<value name="TMP">/usr/tmp</value>
<value name="TMPDIR">/usr/tmp</value>
<value name="TEMP">/usr/tmp</value>
<value name="OSTYPE">$OSTYPE</value>
<value name="MACHTYPE">$MACHTYPE</value>
<value name="MALLOC_CHECK_">2</value>
</value>
</section>
4. Disable access log files
This can make a big impact, because log files on high traffic sites involve a lot of I/O that has to be synchronized across all threads.
access_log off;
log_not_found off;
error_log /var/log/nginx-error.log warn;
If you can't afford to turn off access log files, at least buffer them:
access_log /var/log/nginx/access.log main buffer=16k;
5. Enable GZip
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
6. Cache information about frequently accessed files
open_file_cache max=2000 inactive=20s;
open_file_cache_valid 60s;
open_file_cache_min_uses 5;
open_file_cache_errors off;
7. Adjust client timeouts
client_max_body_size 50M;
client_body_buffer_size 1m;
client_body_timeout 15;
client_header_timeout 15;
keepalive_timeout 2 2;
send_timeout 15;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
8. Adjust output buffers
fastcgi_buffers 256 16k;
fastcgi_buffer_size 128k;
fastcgi_connect_timeout 3s;
fastcgi_send_timeout 120s;
fastcgi_read_timeout 120s;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
reset_timedout_connection on;
server_names_hash_bucket_size 100;
9. /etc/sysctl.conf tuning
# Recycle Zombie connections
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.maxtcptw=200000
# Increase number of files
kern.maxfiles=65535
kern.maxfilesperproc=16384
# Increase page share factor per process
vm.pmap.pv_entry_max=54272521
vm.pmap.shpgperproc=20000
# Increase number of connections
vfs.vmiodirenable=1
kern.ipc.somaxconn=3240000
net.inet.tcp.rfc1323=1
net.inet.tcp.delayed_ack=0
net.inet.tcp.restrict_rst=1
kern.ipc.maxsockbuf=2097152
kern.ipc.shmmax=268435456
# Host cache
net.inet.tcp.hostcache.hashsize=4096
net.inet.tcp.hostcache.cachelimit=131072
net.inet.tcp.hostcache.bucketlimit=120
# Increase number of ports
net.inet.ip.portrange.first=2000
net.inet.ip.portrange.last=100000
net.inet.ip.portrange.hifirst=2000
net.inet.ip.portrange.hilast=100000
kern.ipc.semvmx=131068
# Disable Ping-flood attacks
net.inet.tcp.msl=2000
net.inet.icmp.bmcastecho=1
net.inet.icmp.icmplim=1
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
10. Monitor
Continually monitor the number of open connections, free memory and number of waiting threads.
Set alerts to notify you when thresholds exceed. You can build these alerts yourself, or use something like ServerDensity.
Be sure to install the NGINX stub_status module
You'll need to recompile NGINX -
./configure --with-http_ssl_module --with-http_stub_status_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module
make install BATCH=yes
1. TCP Sockets vs UNIX domain sockets
UNIX domain sockets offer slightly better performance than TCP sockets over loopback interface (less copying of data, fewer context switches).
upstream backend
{
# UNIX domain sockets
server unix:/var/run/fastcgi.sock;
# TCP sockets
# server 127.0.0.1:8080;
}
If you need to support more than 1,000 connections per server, use TCP sockets - they scale much better.
2. Adjust Worker Processes
Modern hardware is multiprocessor and NGINX can leverage multiple physical or virtual processors.
In most cases your web server machine will not be configured to handle multiple workloads (like providing services as a Web Server and a Print Server at the same time) so you will want to configure NGINX to use all the available processors since NGINX worker processes are not multi-threaded.
You can determine how many processors your machine has by running:
On Linux -
cat /proc/cpuinfo | grep processor
On FreeBSD -
sysctl dev.cpu | grep location
Set the worker_processes in your nginx.conf file to the number of cores your machine has.
While you're at it, increase the number of worker_connections (how many connections each core should handle) and set "multi_accept" to ON, as well as "epoll" if you're on Linux:
# We have 16 cores
worker_processes 16;
# connections per worker
events
{
worker_connections 4096;
multi_accept on;
}
3. Setup upstream load balancing
In our experience, multiple upstream backends on the same machine, produce higher throughout than a single one.
For example, if you're looking to support 1,000 max children, divide that number across two backends, letting each handle 500 children:
upstream backend {
server unix:/var/run/php5-fpm.sock1 weight=100 max_fails=5 fail_timeout=5;
server unix:/var/run/php5-fpm.sock2 weight=100 max_fails=5 fail_timeout=5;
}
Here are the two pools from php-fpm.conf:
<section name="pool">
<value name="name">www1</value>
<value name="listen_address">/var/run/php5-fpm.sock1</value>
<value name="listen_options">
<value name="backlog">-1</value>
<value name="owner"></value>
<value name="group"></value>
<value name="mode">0666</value>
</value>
<value name="user">www</value>
<value name="group">www</value>
<value name="pm">
<value name="style">static</value>
<value name="max_children">500</value>
</value>
<value name="rlimit_files">50000</value>
<value name="rlimit_core">0</value>
<value name="request_slowlog_timeout">20s</value>
<value name="slowlog">/var/log/php-slow.log</value>
<value name="chroot"></value>
<value name="chdir"></value>
<value name="catch_workers_output">no</value>
<value name="max_requests">5000</value>
<value name="allowed_clients">127.0.0.1</value>
<value name="environment">
<value name="HOSTNAME">$HOSTNAME</value>
<value name="PATH">/usr/local/bin:/usr/bin:/bin</value>
<value name="TMP">/usr/tmp</value>
<value name="TMPDIR">/usr/tmp</value>
<value name="TEMP">/usr/tmp</value>
<value name="OSTYPE">$OSTYPE</value>
<value name="MACHTYPE">$MACHTYPE</value>
<value name="MALLOC_CHECK_">2</value>
</value>
</section>
<section name="pool">
<value name="name">www2</value>
<value name="listen_address">/var/run/php5-fpm.sock2</value>
<value name="listen_options">
<value name="backlog">-1</value>
<value name="owner"></value>
<value name="group"></value>
<value name="mode">0666</value>
</value>
<value name="user">www</value>
<value name="group">www</value>
<value name="pm">
<value name="style">static</value>
<value name="max_children">500</value>
</value>
<value name="rlimit_files">50000</value>
<value name="rlimit_core">0</value>
<value name="request_slowlog_timeout">20s</value>
<value name="slowlog">/var/log/php-slow.log</value>
<value name="chroot"></value>
<value name="chdir"></value>
<value name="catch_workers_output">no</value>
<value name="max_requests">5000</value>
<value name="allowed_clients">127.0.0.1</value>
<value name="environment">
<value name="HOSTNAME">$HOSTNAME</value>
<value name="PATH">/usr/local/bin:/usr/bin:/bin</value>
<value name="TMP">/usr/tmp</value>
<value name="TMPDIR">/usr/tmp</value>
<value name="TEMP">/usr/tmp</value>
<value name="OSTYPE">$OSTYPE</value>
<value name="MACHTYPE">$MACHTYPE</value>
<value name="MALLOC_CHECK_">2</value>
</value>
</section>
4. Disable access log files
This can make a big impact, because log files on high traffic sites involve a lot of I/O that has to be synchronized across all threads.
access_log off;
log_not_found off;
error_log /var/log/nginx-error.log warn;
If you can't afford to turn off access log files, at least buffer them:
access_log /var/log/nginx/access.log main buffer=16k;
5. Enable GZip
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
6. Cache information about frequently accessed files
open_file_cache max=2000 inactive=20s;
open_file_cache_valid 60s;
open_file_cache_min_uses 5;
open_file_cache_errors off;
7. Adjust client timeouts
client_max_body_size 50M;
client_body_buffer_size 1m;
client_body_timeout 15;
client_header_timeout 15;
keepalive_timeout 2 2;
send_timeout 15;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
8. Adjust output buffers
fastcgi_buffers 256 16k;
fastcgi_buffer_size 128k;
fastcgi_connect_timeout 3s;
fastcgi_send_timeout 120s;
fastcgi_read_timeout 120s;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
reset_timedout_connection on;
server_names_hash_bucket_size 100;
9. /etc/sysctl.conf tuning
# Recycle Zombie connections
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.maxtcptw=200000
# Increase number of files
kern.maxfiles=65535
kern.maxfilesperproc=16384
# Increase page share factor per process
vm.pmap.pv_entry_max=54272521
vm.pmap.shpgperproc=20000
# Increase number of connections
vfs.vmiodirenable=1
kern.ipc.somaxconn=3240000
net.inet.tcp.rfc1323=1
net.inet.tcp.delayed_ack=0
net.inet.tcp.restrict_rst=1
kern.ipc.maxsockbuf=2097152
kern.ipc.shmmax=268435456
# Host cache
net.inet.tcp.hostcache.hashsize=4096
net.inet.tcp.hostcache.cachelimit=131072
net.inet.tcp.hostcache.bucketlimit=120
# Increase number of ports
net.inet.ip.portrange.first=2000
net.inet.ip.portrange.last=100000
net.inet.ip.portrange.hifirst=2000
net.inet.ip.portrange.hilast=100000
kern.ipc.semvmx=131068
# Disable Ping-flood attacks
net.inet.tcp.msl=2000
net.inet.icmp.bmcastecho=1
net.inet.icmp.icmplim=1
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
10. Monitor
Continually monitor the number of open connections, free memory and number of waiting threads.
Set alerts to notify you when thresholds exceed. You can build these alerts yourself, or use something like ServerDensity.
Be sure to install the NGINX stub_status module
You'll need to recompile NGINX -
./configure --with-http_ssl_module --with-http_stub_status_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module
make install BATCH=yes
 |
Peter Mescalchin, 04-30-2014 |
Note that the MIME type for JavaScript in mime.types in Nginx 1.6.0+ is now "application/javascript" so your "gzip_types" parameter should be updated.
Also, why the need for the massively large "client_max_body_size" at 500MB?
Also, why the need for the massively large "client_max_body_size" at 500MB?
 |
Stefan, 05-02-2014 |
At least on Linux systems, I'm not sure about #1. The kernel includes a lot of optimizations for network connections to localhost. afair data is copied directly to the inbound buffer of the receiving process, thus the performance of a TCP configuration should be practically the same with UNIX domain sockets.
 |
amanda john, 05-07-2014 |
That's really a good idea. Best way to optimize high traffic websites! NGINX with PHP really makes more sense.
 |
SayB, 06-20-2014 |
Excellent article! ... What catches my eye is how do you propose on using two different sock files for one single server ? I see the settings, but I'm curious, what would I put in fastcgi_pass which sock file to use when I'm trying to use both ?
|
|
SayB, 06-20-2014 |
Figured it out when I read the man page here:
http://wiki.nginx.org/HttpFastcgiModule#fastcgi_pass
Didn't know you could specify the name of the upstream as well.
Thanks for a very insightful article though ... cheers!
http://wiki.nginx.org/HttpFastcgiModule#fastcgi_pass
Didn't know you could specify the name of the upstream as well.
Thanks for a very insightful article though ... cheers!
 |
Robson Sobral, 09-05-2014 |
Thanks for the info.
Testing it now.
Testing it now.
 |
Truong Anh Tuan, 05-15-2015 |
Could you tell me which license this article is released?
Is it any kind of creative commons?
Is it any kind of creative commons?
 |
Nauseous, 09-03-2015 |
After gzip on try:
gzip_disable "MSIE [1-6].(?!.*SV1)";
It's for all version not supporting compress for IE
gzip_disable "MSIE [1-6].(?!.*SV1)";
It's for all version not supporting compress for IE
 |
Ferri Sutanto, 03-13-2016 |
hi ..
how to adjust worker_connections correctly?
i saw on other place thats related with ulimit -n
thanks
how to adjust worker_connections correctly?
i saw on other place thats related with ulimit -n
thanks
 |
Dolo, 05-21-2016 |
Hello nice guide.
just wondering if it s possible to use 2 number as value for keepalive_timeout as u did up there !
keepalive_timeout 2 2;
just wondering if it s possible to use 2 number as value for keepalive_timeout as u did up there !
keepalive_timeout 2 2;
 |
Nicolas Hug, 12-17-2016 |
Ay,
Thanks for this post ! It helps me a lot with my project of hosting 800+ Wordpress instances ! :)
Thanks for this post ! It helps me a lot with my project of hosting 800+ Wordpress instances ! :)
 |
Naura, 10-08-2017 |
Hi Adrian,
Hope you still monitor this old post.
I want to ask, how to calculate the number of max_children?
And why you decide to use pm static not dynamic?
Thanks
Hope you still monitor this old post.
I want to ask, how to calculate the number of max_children?
And why you decide to use pm static not dynamic?
Thanks
 |
Robert Haba, 11-15-2021 |
i saw a big improvement on my website following these :O
 |
Jhordy Barrera, 07-15-2022 |
Thanks for the article, I'm curious about an update to date, I'm starting to migrate from Apache to NGINX and I would like to know updates or improvements to date, Greetings.
|
|
Subscribe Now to receive new posts via Email as soon as they come out.
Comments
Post your comments
