Call us Toll-Free:
1-800-218-1525
Live ChatEmail us
After 7 years of using NGINX with PHP, we learned a couple of things about how to best optimize NGINX and PHP-fpm for high traffic sites.

Below is a collection of tips and recommendations:

1. Switch from TCP to UNIX domain sockets

UNIX domain sockets offer better performance than TCP sockets over loopback interface (less copying of data, fewer context switches).

One thing to keep in mind is that UNIX sockets are only reachable from programs that are running on the same server (there's no network support, obviously).


upstream backend
{
 
# UNIX domain sockets
 
server unix:/var/run/fastcgi.sock;

 
# TCP sockets
  # server 127.0.0.1:8080;
}

2. Adjust Worker Processes

Modern hardware is multiprocessor and NGINX can leverage multiple physical or virtual processors.

In most cases your web server machine will not be configured to handle multiple workloads (like providing services as a Web Server and a Print Server at the same time) so you will want to configure NGINX to use all the available processors since NGINX worker processes are not multi-threaded.

You can determine how many processors your machine has by running:

On Linux -


cat
/proc/cpuinfo | grep processor

On FreeBSD -


sysctl dev
.cpu | grep location

Set the worker_processes in your nginx.conf file to the number of cores your machine has.

While you're at it, increase the number of worker_connections (how many connections each core should handle) and set "multi_accept" to ON, as well as "epoll" if you're on Linux:


# We have 16 cores
worker_processes 16;

# connections per worker
events
{
 
worker_connections 4096;
 
multi_accept on;
}

3. Setup upstream load balancing

In our experience, multiple upstream backends on the same machine, produce higher throughout than a single one.

For example, if you're looking to support 1,000 max children, divide that number across two backends, letting each handle 500 children:


upstream backend
{
 
server unix:/var/run/php5-fpm.sock1 weight=100 max_fails=5 fail_timeout=5;
 
server unix:/var/run/php5-fpm.sock2 weight=100 max_fails=5 fail_timeout=5;
}

Here are the two pools from php-fpm.conf:


 
<section name="pool">

      <
value name="name">www1</value>
      <
value name="listen_address">/var/run/php5-fpm.sock1</value>

      <
value name="listen_options">
        <
value name="backlog">-1</value>
        <
value name="owner"></value>
        <
value name="group"></value>
        <
value name="mode">0666</value>
      </
value>

      <
value name="user">www</value>
      <
value name="group">www</value>

      <
value name="pm">
        <
value name="style">static</value>
        <
value name="max_children">500</value>
      </
value>

      <
value name="rlimit_files">50000</value>
      <
value name="rlimit_core">0</value>
      <
value name="request_slowlog_timeout">20s</value>
      <
value name="slowlog">/var/log/php-slow.log</value>
      <
value name="chroot"></value>
      <
value name="chdir"></value>
      <
value name="catch_workers_output">no</value>
      <
value name="max_requests">5000</value>
      <
value name="allowed_clients">127.0.0.1</value>

      <
value name="environment">
        <
value name="HOSTNAME">$HOSTNAME</value>
        <
value name="PATH">/usr/local/bin:/usr/bin:/bin</value>
        <
value name="TMP">/usr/tmp</value>
        <
value name="TMPDIR">/usr/tmp</value>
        <
value name="TEMP">/usr/tmp</value>
        <
value name="OSTYPE">$OSTYPE</value>
        <
value name="MACHTYPE">$MACHTYPE</value>
        <
value name="MALLOC_CHECK_">2</value>
      </
value>

    </
section>

  <
section name="pool">

      <
value name="name">www2</value>
      <
value name="listen_address">/var/run/php5-fpm.sock2</value>
     
      <
value name="listen_options">
        <
value name="backlog">-1</value>
        <
value name="owner"></value>
        <
value name="group"></value>
        <
value name="mode">0666</value>
      </
value>

      <
value name="user">www</value>
      <
value name="group">www</value>

      <
value name="pm">
        <
value name="style">static</value>
        <
value name="max_children">500</value>
      </
value>

      <
value name="rlimit_files">50000</value>
      <
value name="rlimit_core">0</value>
      <
value name="request_slowlog_timeout">20s</value>
      <
value name="slowlog">/var/log/php-slow.log</value>
      <
value name="chroot"></value>
      <
value name="chdir"></value>
      <
value name="catch_workers_output">no</value>
      <
value name="max_requests">5000</value>
      <
value name="allowed_clients">127.0.0.1</value>
     
      <
value name="environment">
        <
value name="HOSTNAME">$HOSTNAME</value>
        <
value name="PATH">/usr/local/bin:/usr/bin:/bin</value>
        <
value name="TMP">/usr/tmp</value>
        <
value name="TMPDIR">/usr/tmp</value>
        <
value name="TEMP">/usr/tmp</value>
        <
value name="OSTYPE">$OSTYPE</value>
        <
value name="MACHTYPE">$MACHTYPE</value>
        <
value name="MALLOC_CHECK_">2</value>
      </
value>

    </
section>       

4. Disable access log files

This can make a big impact, because log files on high traffic sites involve a lot of I/O that has to be synchronized across all threads.


access_log off
;
log_not_found off;
error_log /var/log/nginx-error.log warn;

If you can't afford to turn off access log files, at least buffer them:


access_log
/var/log/nginx/access.log main buffer=16k;

5. Enable GZip


gzip on
;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

6. Cache information about frequently accessed files


open_file_cache max
=200000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;

7. Adjust client timeouts


client_max_body_size 500M
;
client_body_buffer_size 1m;
client_body_timeout 15;
client_header_timeout 15;
keepalive_timeout 2 2;
send_timeout 15;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
client_max_body_size 500M;

8. Adjust output buffers


fastcgi_buffers 256 16k
;
fastcgi_buffer_size 128k;
fastcgi_connect_timeout 3s;
fastcgi_send_timeout 120s;
fastcgi_read_timeout 120s;
reset_timedout_connection on;
server_names_hash_bucket_size 100;

9. /etc/sysctl.conf tuning


# Recycle Zombie connections
net.inet.tcp.fast_finwait2_recycle=1
net
.inet.tcp.maxtcptw=200000

# Increase number of files
kern.maxfiles=65535
kern
.maxfilesperproc=16384

# Increase page share factor per process
vm.pmap.pv_entry_max=54272521
vm
.pmap.shpgperproc=20000

# Increase number of connections
vfs.vmiodirenable=1
kern
.ipc.somaxconn=3240000
net
.inet.tcp.rfc1323=1
net
.inet.tcp.delayed_ack=0
net
.inet.tcp.restrict_rst=1
kern
.ipc.maxsockbuf=2097152
kern
.ipc.shmmax=268435456

# Host cache
net.inet.tcp.hostcache.hashsize=4096
net
.inet.tcp.hostcache.cachelimit=131072
net
.inet.tcp.hostcache.bucketlimit=120

# Increase number of ports
net.inet.ip.portrange.first=2000
net
.inet.ip.portrange.last=100000
net
.inet.ip.portrange.hifirst=2000
net
.inet.ip.portrange.hilast=100000
kern
.ipc.semvmx=131068

# Disable Ping-flood attacks
net.inet.tcp.msl=2000
net
.inet.icmp.bmcastecho=1
net
.inet.icmp.icmplim=1
net
.inet.tcp.blackhole=2
net
.inet.udp.blackhole=1

10. Monitor

Continually monitor the number of open connections, free memory and number of waiting threads.

Set alerts to notify you when thresholds exceed. You can build these alerts yourself, or use something like ServerDensity.

Be sure to install the NGINX stub_status module

You'll need to recompile NGINX -


./configure --with-http_ssl_module --with-http_stub_status_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module
make install BATCH
=yes 


View 6 Comment(s)

How to install htop

Adrian Singer, April 20
htop is an interactive process viewer for Linux, replacing the traditional top.

Why htop?

htop provides a more interactive process-viewing experience. You can surf through running processes, scrolling horizontally and vertically to reveal information that would otherwise have been clipped, information such as full command lines.

You can see what files a process has open (click "l"); you can even trace processes using strace. There's also a handy tree view for understanding process ancestry.

Installing on Linux

CentOS:


yum install htop

Debian:


sudo apt
-get install htop

Installing on FreeBSD

Step 1:

Add the following line to /etc/fstab:


linproc
/compat/linux/proc linprocfs rw 0 0

Step 2:

Create a symbolic link for /usr/compat


mkdir
-p /usr/compat/linux/proc ; ln -s /usr/compat /compat ; mount linproc

Step 3:

Compile and install


cd
/usr/ports/sysutils/htop
make install
Ever try to delete a lot of files in a folder, only to have the operation fail with "Argument list too long"?

Here's how to get it done:

FreeBSD

find
. -name "*.pdf" -print0 | xargs -0 rm

Note that this is a recursive search and will find (and delete) files in subdirectories as well.

Linux

find
. -name "*.pdf" -maxdepth 1 -print0 | xargs -0 rm


View 1 Comment(s)
Gluster is an open-source software-only distributed file system designed to run on commodity hardware, scaling to support petabytes of storage.

Gluster supports file system mirroring & replication, striping, load balancing, volume failover, storage quotas and disk caching.

Hesitant with the lack of glowing reviews about Gluster, we were attracted by its feature set and simple architecture.

Over the last six months, we battle-tested Gluster in production, relying on the system to deliver high-availability and geo replication, to power large scale Internet Marketing product launches.

Architecture

The Gluster architecture aggregates compute, storage, and I/O resources into a global namespace. Each server plus attached commodity storage is considered to be a node. Capacity is scaled by adding additional nodes or adding additional storage to each node. Performance is increased by deploying storage among more nodes. High availability is achieved by replicating data n-way between nodes.

Unlike other distributed file systems, Gluster runs on top of your existing file-system, with client-code doing all the work. The clients are stateless and introduce no centralized single point of failure.

Gluster integrates with the local file system using FUSE, delivering wide compatibility across any system that supports extended file attributes - the "local database" where Gluster keeps track of all changes to a file.

The system supports several storage volume configurations:

* None: Files are transparently distributed across servers, with each node adding to the total storage capacity.
* Replica: Files are replicated between two LAN drives (synchronous replication)
* Geo replica: Files are replicated between two remote drives (asynchronous replication, using rsync in the background)
* Stripe: Each file is spread across 4 servers to distribute load.

As of October 2011, development of Gluster is funded by RedHat

Installing Gluster

This is one of the areas where Gluster really shines. You can be up and running in minutes.

Step 1

Installing the FUSE client, which serves as the "glue" between Gluster and your local file system.


wget
"http://sourceforge.net/projects/fuse/files/fuse-2.X/2.9.1/fuse-2.9.1.tar.gz/download"
tar xvfz fuse-2.9.1.tar.gz
cd fuse
-2.9.1
./configure
make all
make install

Step 2

Building Gluster from source


wget
"http://download.gluster.org/pub/gluster/glusterfs/LATEST/glusterfs-3.3.0.tar.gz"
tar xvfz glusterfs-3.3.0.tar.gz
cd glusterfs
-3.3.0
./configure
make all
make install

Starting Gluster and setting it to auto-start on next reboot


/etc/init.d/glusterd start
/usr/sbin/update-rc.d glusterd defaults

Step 3

Configuring your first two nodes as a Replica setup (mirroring)

On node 1 (backup1east):


mkdir
/gfs
mkdir
/gfs/bricks
mkdir
/gfs/backup

gluster peer probe backup2west

gluster volume create backup replica 2 transport tcp backup1east
:/gfs/bricks/vol0 backup2west:/gfs/bricks/vol0

mount
-t glusterfs backup1east:/backup /gfs/backup

On node 2 (backup2west):


mkdir
/gfs
mkdir
/gfs/bricks
mkdir
/gfs/backup

gluster peer probe backup1east

mount
-t glusterfs backup2west:/backup /gfs/backup

Important: Make sure the name of your Gluster volume ('backup' in the example above) is different than the name of the share ('gfs' in the example above) or things will not work properly.

Our Experience

Going into this experiment, we had very high hopes for Gluster. Once proven, the goal was to replace our entire private cloud storage cluster with Gluster.

Unfortunately, we have been very disappointed with Gluster...

In spite of getting a lot of help from the Gluster community, testing different platforms and configurations, results have been consistent.

Like other users reported, we struggled with poor performance, bugs, race conditions when dealing with lots of small files, difficulties in monitoring node health and worst of all - two instances of unexplained data loss.

We ended up completely abandoning Gluster and switching back to our home-grown rsync-based solution.

As always, run your own tests to determine if this is a good fit for your needs.

Proceed with caution.

More Resources

* SlideShare Introduction to GlusterFS
* Gluster Documentation
* Gluster IRC Channel
* Gluster Blog

View 1 Comment(s)
Enjoy our step-by-step guide to configuring PHP 5 with FPM, NGinx Web server, Memcached and MySQL 5.1, on FreeBSD 8:

1. Install FreeBSD 7 compatibility and standard packages


cd
/usr/ports/misc/compat7x
make all
make install

pkg_add
-r libevent
pkg_add
-r libtool
pkg_add
-r m4
pkg_add
-r pcre
pkg_add
-r pdftk
pkg_add
-r rsync
pkg_add
-r vim
pkg_add
-r wget

2. Install ProFTPD


cd
/usr/ports/ftp/proftpd
make all
make install

3. Install NGinx

Make sure you click to enable 'HTTP_GZIP_STATIC_MODULE', 'HTTP_SSL_MODULE' and 'HTTP_ZIP_MODULE'


cd
/usr/ports/www/nginx
make all
make install

echo "nginx_enable=YES" >> /etc/rc.conf
echo "<?php phpinfo(); ?>" >> /usr/local/www/nginx/phpinfo.php

You can always run make config to redo the configuration options

4. Install CURL+LibXML


cd
/usr/ports/ftp/curl
make all
make install

cd
/usr/ports/textproc/libxml
make all
make install

5. Install MySQL client and server


cd
/usr/ports/databases/mysql51-server
make all
make install

cd
/usr/ports/databases/mysql51-client
make all
make install

cd
/usr/tmp
fetch
"http://api6.softwareprojects.com/files/auto/my.cnf"
mv my.cnf /etc/my.cnf

mkdir
/usr/local/mysql
mkdir
/usr/local/mysql/data
chmod 777
/usr/local/mysql
chown
-R mysql:mysql /usr/local/mysql

/usr/local/bin/mysql_install_db
chmod
-R 777 /usr/local/mysql
chown
-R mysql:mysql /usr/local/mysql

6. Install GD


cd
/usr/ports/graphics/ruby-libpng
make all
make install

7. Install PHP 5


cd
/usr/ports/security/libmcrypt
make all
make install

cd
/usr/ports/devel/php5-pcntl
make all
make install

cd
/usr/tmp
fetch
"http://api6.softwareprojects.com/files/auto/php-5.2.8.tar.gz"
tar xvfz php-5.2.8.tar.gz

fetch
"http://api6.softwareprojects.com/files/auto/php-5.2.8-fpm-0.5.10.diff.gz"
gzip -cd php-5.2.8-fpm-0.5.10.diff.gz | patch -d php-5.2.8 -p1

fetch
"http://api6.softwareprojects.com/files/auto/suhosin-patch-5.2.8-0.9.6.3.patch.gz"
gzip -cd suhosin-patch-5.2.8-0.9.6.3.patch.gz | patch -d php-5.2.8 -p1

cd php
-5.2.8
./configure --with-config-file-path=/usr/local/lib/ --enable-pcntl --enable-fastcgi --enable-fpm --enable-calendar --enable-ftp --enable-mbstring --with-mysql --with-curl --with-mcrypt --with-gd --with-iconv --with-jpeg-dir=/usr/lib --with-mysql=/usr/local/mysql --enable-memcache --with-openssl --enable-soap --enable-sockets --with-zlib --enable-zip --enable-bcmath --with-ttf --enable-gd-native-ttf --with-freetype-dir=/usr/local/lib/ --enable-pdo --with-pdo_mysql --enable-suhosin
make all install

fetch
"http://api6.softwareprojects.com/files/auto/php-fpm.conf"
mv php-fpm.conf /usr/local/etc/php-fpm.conf

fetch
"http://api6.softwareprojects.com/files/auto/nginx.conf"
mv nginx.conf /usr/local/etc/nginx/nginx.conf

8. Install Memcached


cd
/usr/ports/databases/memcached
make all
make install

9. Install HAProxy


cd
/usr/ports/net/haproxy
make all
make install

10. Start MySQL and NGinx


/usr/local/bin/mysqld_safe &
/
usr/local/sbin/php-fpm start
/usr/local/etc/rc.d/nginx start

--

Verify MySQL is working properly:

Attempt connecting to MySQL:

/usr/local/bin/mysql -uroot

Verify NGinx is working properly:

Point your browser to http://1.2.3.4/ (replacing 1.2.3.4 with the PUBLIC ip address of the server)

Verify PHP is working properly:

Point your browser to http://1.2.3.4/phpinfo.php (replacing 1.2.3.4 with the PUBLIC ip address of the server).

If you see the PHP info screen, all is well

NTP for Accurate Global Time Synchronization

Mike Peters, February 11, 2011
Running a multi-server architecture?

Keeping your server clocks in-sync is very important, especially when using NoSQL databases like Cassandra.

Cassandra attaches a timestamp to every insert operation. If your server clocks fall out of sync, some updates will be dropped, due to one server taking precedence over others.

Even if your servers are all showing the same time right now, it's important to understand that without continually applying corrections, the different clocks will eventually fall out of sync.

How does Global Time Synchronization work?

Public time servers, update their clocks using hardware based on atom's electrons frequency (aka Atomic Clocks).

Your local machines ping the time server repeatedly, applying corrections so that all clocks are in sync.

NTP

NTP (Network Time Protocol) is an Internet protocol used to synchronize the clocks of computers to a global time reference.

FreeBSD and Linux servers come with an NTPD service that automatically adjusts the local clock based on the selected global time server.

To start NTPD on Linux:
ntpdate pool.ntp.org
service ntpd restart

To start NTPD on FreeBSD:
ntpdate pool.ntp.org
/etc/rc.d/ntpd start

Controlling which time server to use is done by updating /etc/ntp.conf. Example:
server pool.ntp.org prefer
driftfile /var/db/ntpd.drift
logfile /var/log/ntpd.log

To configure NTPD to start on boot automatically on Linux:
chkconfig --level 2345 ntpd on

To configure NTPD to start on boot automatically on FreeBSD:
Add these lines to your /etc/rc.conf file:
ntpd_enable="YES"
ntpdate_enable="YES"
ntpdate_flags="pool.ntp.org"


View 1 Comment(s)

Cassandra for PHP Sessions

Hojda Vasile Dan, January 26, 2011
Building on Dawn's Memcached for PHP sessions post, we've now converted our php-sessions handling from Memcached to Cassandra.

Cassandra supports built-in caching, sharding & replication and scales to infinity, overcoming the shortcomings of the memcached-for-sessions approach.

Click here to download the new dbsession.php and here to download common_cassandra.php
« Previous Posts



About Us  |  Contact us  |  Privacy Policy  |  Terms & Conditions