Call us Toll-Free:
1-800-218-1525
Live ChatEmail us

 Sponsors

Optimizing NGINX and PHP-fpm for high traffic sites

Adrian Singer, 04-20-2014
After 7 years of using NGINX with PHP, we learned a couple of things about how to best optimize NGINX and PHP-fpm for high traffic sites.

1. TCP Sockets vs UNIX domain sockets

UNIX domain sockets offer slightly better performance than TCP sockets over loopback interface (less copying of data, fewer context switches).


upstream backend
{
 
# UNIX domain sockets
 
server unix:/var/run/fastcgi.sock;

 
# TCP sockets
  # server 127.0.0.1:8080;
}

If you need to support more than 1,000 connections per server, use TCP sockets - they scale much better.

2. Adjust Worker Processes

Modern hardware is multiprocessor and NGINX can leverage multiple physical or virtual processors.

In most cases your web server machine will not be configured to handle multiple workloads (like providing services as a Web Server and a Print Server at the same time) so you will want to configure NGINX to use all the available processors since NGINX worker processes are not multi-threaded.

You can determine how many processors your machine has by running:

On Linux -


cat
/proc/cpuinfo | grep processor

On FreeBSD -


sysctl dev
.cpu | grep location

Set the worker_processes in your nginx.conf file to the number of cores your machine has.

While you're at it, increase the number of worker_connections (how many connections each core should handle) and set "multi_accept" to ON, as well as "epoll" if you're on Linux:


# We have 16 cores
worker_processes 16;

# connections per worker
events
{
 
worker_connections 4096;
 
multi_accept on;
}

3. Setup upstream load balancing

In our experience, multiple upstream backends on the same machine, produce higher throughout than a single one.

For example, if you're looking to support 1,000 max children, divide that number across two backends, letting each handle 500 children:


upstream backend
{
 
server unix:/var/run/php5-fpm.sock1 weight=100 max_fails=5 fail_timeout=5;
 
server unix:/var/run/php5-fpm.sock2 weight=100 max_fails=5 fail_timeout=5;
}

Here are the two pools from php-fpm.conf:


 
<section name="pool">

      <
value name="name">www1</value>
      <
value name="listen_address">/var/run/php5-fpm.sock1</value>

      <
value name="listen_options">
        <
value name="backlog">-1</value>
        <
value name="owner"></value>
        <
value name="group"></value>
        <
value name="mode">0666</value>
      </
value>

      <
value name="user">www</value>
      <
value name="group">www</value>

      <
value name="pm">
        <
value name="style">static</value>
        <
value name="max_children">500</value>
      </
value>

      <
value name="rlimit_files">50000</value>
      <
value name="rlimit_core">0</value>
      <
value name="request_slowlog_timeout">20s</value>
      <
value name="slowlog">/var/log/php-slow.log</value>
      <
value name="chroot"></value>
      <
value name="chdir"></value>
      <
value name="catch_workers_output">no</value>
      <
value name="max_requests">5000</value>
      <
value name="allowed_clients">127.0.0.1</value>

      <
value name="environment">
        <
value name="HOSTNAME">$HOSTNAME</value>
        <
value name="PATH">/usr/local/bin:/usr/bin:/bin</value>
        <
value name="TMP">/usr/tmp</value>
        <
value name="TMPDIR">/usr/tmp</value>
        <
value name="TEMP">/usr/tmp</value>
        <
value name="OSTYPE">$OSTYPE</value>
        <
value name="MACHTYPE">$MACHTYPE</value>
        <
value name="MALLOC_CHECK_">2</value>
      </
value>

    </
section>

  <
section name="pool">

      <
value name="name">www2</value>
      <
value name="listen_address">/var/run/php5-fpm.sock2</value>
     
      <
value name="listen_options">
        <
value name="backlog">-1</value>
        <
value name="owner"></value>
        <
value name="group"></value>
        <
value name="mode">0666</value>
      </
value>

      <
value name="user">www</value>
      <
value name="group">www</value>

      <
value name="pm">
        <
value name="style">static</value>
        <
value name="max_children">500</value>
      </
value>

      <
value name="rlimit_files">50000</value>
      <
value name="rlimit_core">0</value>
      <
value name="request_slowlog_timeout">20s</value>
      <
value name="slowlog">/var/log/php-slow.log</value>
      <
value name="chroot"></value>
      <
value name="chdir"></value>
      <
value name="catch_workers_output">no</value>
      <
value name="max_requests">5000</value>
      <
value name="allowed_clients">127.0.0.1</value>
     
      <
value name="environment">
        <
value name="HOSTNAME">$HOSTNAME</value>
        <
value name="PATH">/usr/local/bin:/usr/bin:/bin</value>
        <
value name="TMP">/usr/tmp</value>
        <
value name="TMPDIR">/usr/tmp</value>
        <
value name="TEMP">/usr/tmp</value>
        <
value name="OSTYPE">$OSTYPE</value>
        <
value name="MACHTYPE">$MACHTYPE</value>
        <
value name="MALLOC_CHECK_">2</value>
      </
value>

    </
section>       

4. Disable access log files

This can make a big impact, because log files on high traffic sites involve a lot of I/O that has to be synchronized across all threads.


access_log off
;
log_not_found off;
error_log /var/log/nginx-error.log warn;

If you can't afford to turn off access log files, at least buffer them:


access_log
/var/log/nginx/access.log main buffer=16k;

5. Enable GZip


gzip on
;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

6. Cache information about frequently accessed files


open_file_cache max
=2000 inactive=20s;
open_file_cache_valid 60s;
open_file_cache_min_uses 5;
open_file_cache_errors off;

7. Adjust client timeouts


client_max_body_size 50M
;
client_body_buffer_size 1m;
client_body_timeout 15;
client_header_timeout 15;
keepalive_timeout 2 2;
send_timeout 15;
sendfile on;
tcp_nopush on;
tcp_nodelay on;

8. Adjust output buffers


fastcgi_buffers 256 16k
;
fastcgi_buffer_size 128k;
fastcgi_connect_timeout 3s;
fastcgi_send_timeout 120s;
fastcgi_read_timeout 120s;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
reset_timedout_connection on;
server_names_hash_bucket_size 100;

9. /etc/sysctl.conf tuning


# Recycle Zombie connections
net.inet.tcp.fast_finwait2_recycle=1
net
.inet.tcp.maxtcptw=200000

# Increase number of files
kern.maxfiles=65535
kern
.maxfilesperproc=16384

# Increase page share factor per process
vm.pmap.pv_entry_max=54272521
vm
.pmap.shpgperproc=20000

# Increase number of connections
vfs.vmiodirenable=1
kern
.ipc.somaxconn=3240000
net
.inet.tcp.rfc1323=1
net
.inet.tcp.delayed_ack=0
net
.inet.tcp.restrict_rst=1
kern
.ipc.maxsockbuf=2097152
kern
.ipc.shmmax=268435456

# Host cache
net.inet.tcp.hostcache.hashsize=4096
net
.inet.tcp.hostcache.cachelimit=131072
net
.inet.tcp.hostcache.bucketlimit=120

# Increase number of ports
net.inet.ip.portrange.first=2000
net
.inet.ip.portrange.last=100000
net
.inet.ip.portrange.hifirst=2000
net
.inet.ip.portrange.hilast=100000
kern
.ipc.semvmx=131068

# Disable Ping-flood attacks
net.inet.tcp.msl=2000
net
.inet.icmp.bmcastecho=1
net
.inet.icmp.icmplim=1
net
.inet.tcp.blackhole=2
net
.inet.udp.blackhole=1

10. Monitor

Continually monitor the number of open connections, free memory and number of waiting threads.

Set alerts to notify you when thresholds exceed. You can build these alerts yourself, or use something like ServerDensity.

Be sure to install the NGINX stub_status module

You'll need to recompile NGINX -


./configure --with-http_ssl_module --with-http_stub_status_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module
make install BATCH
=yes 

Peter Mescalchin, 04-30-2014
Note that the MIME type for JavaScript in mime.types in Nginx 1.6.0+ is now "application/javascript" so your "gzip_types" parameter should be updated.

Also, why the need for the massively large "client_max_body_size" at 500MB?

Stefan, 05-02-2014
At least on Linux systems, I'm not sure about #1. The kernel includes a lot of optimizations for network connections to localhost. afair data is copied directly to the inbound buffer of the receiving process, thus the performance of a TCP configuration should be practically the same with UNIX domain sockets.

amanda john, 05-07-2014
That's really a good idea. Best way to optimize high traffic websites! NGINX with PHP really makes more sense.

SayB, 06-20-2014
Excellent article! ... What catches my eye is how do you propose on using two different sock files for one single server ? I see the settings, but I'm curious, what would I put in fastcgi_pass which sock file to use when I'm trying to use both ?

SayB, 06-20-2014
Figured it out when I read the man page here:

http://wiki.nginx.org/HttpFastcgiModule#fastcgi_pass

Didn't know you could specify the name of the upstream as well.

Thanks for a very insightful article though ... cheers!

Robson Sobral, 09-05-2014
Thanks for the info.

Testing it now.

Truong Anh Tuan, 05-15-2015
Could you tell me which license this article is released?
Is it any kind of creative commons?

Nauseous, 09-03-2015
After gzip on try:
gzip_disable "MSIE [1-6].(?!.*SV1)";

It's for all version not supporting compress for IE

Ferri Sutanto, 03-13-2016
hi ..

how to adjust worker_connections correctly?

i saw on other place thats related with ulimit -n

thanks

Dolo, 05-21-2016
Hello nice guide.

just wondering if it s possible to use 2 number as value for keepalive_timeout as u did up there !

keepalive_timeout 2 2;

Nicolas Hug, 12-17-2016
Ay,

Thanks for this post ! It helps me a lot with my project of hosting 800+ Wordpress instances ! :)

Naura, 10-08-2017
Hi Adrian,

Hope you still monitor this old post.

I want to ask, how to calculate the number of max_children?

And why you decide to use pm static not dynamic?

Thanks
Enjoyed this post?

Subscribe Now to receive new posts via Email as soon as they come out.

 Comments
Post your comments












Note: No link spamming! If your message contains link/s, it will NOT be published on the site before manually approved by one of our moderators.



About Us  |  Contact us  |  Privacy Policy  |  Terms & Conditions