Call us Toll-Free:
Email us

NTP for Accurate Global Time Synchronization

Mike Peters, February 11, 2011
Running a multi-server architecture?

Keeping your server clocks in-sync is very important, especially when using NoSQL databases like Cassandra.

Cassandra attaches a timestamp to every insert operation. If your server clocks fall out of sync, some updates will be dropped, due to one server taking precedence over others.

Even if your servers are all showing the same time right now, it's important to understand that without continually applying corrections, the different clocks will eventually fall out of sync.

How does Global Time Synchronization work?

Public time servers, update their clocks using hardware based on atom's electrons frequency (aka Atomic Clocks).

Your local machines ping the time server repeatedly, applying corrections so that all clocks are in sync.


NTP (Network Time Protocol) is an Internet protocol used to synchronize the clocks of computers to a global time reference.

FreeBSD and Linux servers come with an NTPD service that automatically adjusts the local clock based on the selected global time server.

To start NTPD on Linux:

To start NTPD on FreeBSD:

Controlling which time server to use is done by updating /etc/ntp.conf. Example:

To configure NTPD to start on boot automatically on Linux:

To configure NTPD to start on boot automatically on FreeBSD:
Add these lines to your /etc/rc.conf file:

View 1 Comment(s)

Cassandra for PHP Sessions

Hojda Vasile Dan, January 26, 2011
Building on Dawn's Memcached for PHP sessions post, we've now converted our php-sessions handling from Memcached to Cassandra.

Cassandra supports built-in caching, sharding & replication and scales to infinity, overcoming the shortcomings of the memcached-for-sessions approach.

Click here to download the new dbsession.php and here to download common_cassandra.php

redis: a persistent key-value store

Mike Peters, November 5, 2010
redis is a key-value store, similar to memcached but with data persistence.

redis supports multiple value types, counters with atomic increment/decrement and built-in key expiration.

To achieve persistence without scarifying speed, like Cassandra, redis performs updates in memory as well as adding them to an append-only file, which is synced to disk from time to time.

redis is fast (110,000 writes per second, 81,000 reads per second), supports sharding and master-slave replication (no master-master yet)

Why redis?

Those of you keeping track, know we've always been big fans of MySQL but at the same time, we keep writing about migrating different parts of our application to Memcached, Cassandra, Lucene and ElasticSearch.

Why do we keep jumping from one storage engine to another? Can't we make up our minds already and settle with the "best" storage engine that meets our needs?

In short, No.

A common misconception is the belief that all storage engines are created equal, all designed to simply "store stuff" and provide fast access to your data. Unless your application performs one clearly defined simple task, it is a dire mistake to expect a single storage engine will effectively fulfill all of your data warehousing and processing needs.

* MySQL is great when you need ad-hoc queries and you're dealing with a relatively small data set.

* Memcached comes into play when you have a read-heavy environment and need a quick volatile cache to avoid querying MySQL a dozen times per page.

* Lucene and ElasticSearch are your friends when you need fulltext search, or when your MySQL data set grows to a point where running the filters and joins in MySQL becomes slow like a snail.

* Cassandra is amazing when you have a write-heavy environment, need to be capable of scaling writes linearly and supporting a huge data set.

* redis works particularly well as a state machine, when you need counters with atomic increment/decrement. Typical uses: "how many users are on my website?" ala ChartBeat, "how many jobs are waiting to be processed" etc.


redis is written in Ansi C and runs as a single light-weight daemon on your machine. All updates are done in memory first and saved to disk later, asynchronously.

Supported languages: C, C#, Erlang, Java, JavaScript, Lua, Perl, PHP, Python, Ruby, Scala, Go, and Tcl.

As of 15 March 2010, development of redis is funded by VMware.

Installing redis

Step 1

Download the redis tarball and extract it

Step 2

Compile redis and install it

Step 3

Run redis

Once running you can use redis-benchmark to run some benchmarks.

redis doesn't come with a config file, it will use all default settings by default. But you're going to want to study the config options and set them up.

Sample redis config file here

Configuration options

A few important redis.conf options you're going to want to set:

* First, if you will only be connecting to a local Redis instance, uncomment the bind configuration in the sample file:

That tells Redis not to listen for external connections.

* redis supports multiple databases, but for most use cases, you're only going to need one. Change the default 16 to 1:

* You can set the maximum number of bytes Redis can allocate, after which it will start purging volatile keys. If it cannot reclaim any more memory it will start refusing write commands. Here's a sample setting for a 100MB limit:

* The server will periodically fork and asynchronously dump the current contents of the database to disk. The dump is actually made to a temporary file and then moved to replace any older dump, so the operation is atomic and won't leave you with a partially dumped database. If Redis is eventually shutdown and reloaded, it will restore from this dump file.

How often it dumps the keys is configureable by the amount of time that passes and the number of changes that have been made to the data. For example, the following settings tell Redis to dump the database after 60 seconds if 100 changes have been made or after five minutes if there has been at least 1 change:

* By default redis starts in foreground mode. To fix that, change demonize option in redis.conf file to "Yes":

To Redis or not to Redis?

If you have a large data set that cannot comfortably fit into RAM, Redis is not the key value store for you to use, but if you have smaller sets, and if you can live with the asynchronous write behavior, then, for me, the answer is definitely "to Redis."

As an alternative, Tokyo Cabinet is very fast for a synchronous key value store, and it does support some features that Redis does not, such as tables. Redis permits a master/slave setup, which can alleviate fears of data loss from failure, but it's not as certain as something like Tokyo Cabinet, which will write the data as soon as it gets it. On the other hand, Redis is blazingly fast, incredibly easy to use, and will support just about anything you can think of doing with your data.

More resources:

* Try redis in your browser
* Download redis
* Retwis - a PHP twitter clone using redis

View 5 Comment(s)

How to hide .php extension in your urls with Nginx

Adrian Singer, November 4, 2010
Looking for clean urls (/hello instead of /hello.php)?

Here's how to set it up:

Step 1

Create a notfound.php script and place it in your root web server folder

Step 2

Update your Nginx nginx.conf file, rewriting all urls where the file is not found, to notfound.php

Note: This is different than doing an error_document 404 redirect. With a 404 redirect, HTTP_POST data is not preserved.

View 5 Comment(s)

Three different ways of handling a problem

Mike Peters, October 25, 2010
Presented with a new problem, I noticed engineers can be divided to three groups:

1. I don't know, don't understand, not familiar with this part of the code

Unless you're on your first few weeks with the company, this statement is totally inexcusable. It's lazy and pathetic.

Be a Problem solver.

Waving your hands in the air announcing to the world you're not familiar with some code, does nothing more than exhibit your incompetence in picking up something new and running with it.

How comfortable will your manager, peers and clients feel, once you've made such a statement?

Be a Problem solver. Study the code, follow the execution process flow, find engineers who are more familiar than you and ask them specific questions.

Don't stop probing and asking until you've mastered the code.

Unless you fully understand the code, problem at hand and the big picture implications, you're better off not touching it at all.

2. Quick and dirty: Let me patch it up real nice. In & out as quickly as possible.

Time is of the essence, right?

How about doing the absolute minimum, so the problem can be patched up and you can move on with more important pressing items on your todo list?


I'm a big proponent of code clarity and elegance.

Taking shortcuts, especially when working in an agile development environment, will come back to bite you in the butt.

Why I hate patches:

* Patches clutter the code, which means the next guy will have to struggle twice as hard to understand what's going on

* Patches are often specific to dealing with one edge case, one facet of the problem. Slightly different variation of the same problem and you're back to square one.

* Patching up code is often a lazy act done by someone who didn't want to take the time to understand the big picture. Which means, chances are the patch can break other perfectly valid scenarios.

Take the time to do things right, the first time around.

The difference between patching up a problem and doing it right? That brings me to the third type of engineers, those who still have a job.

3. The right way: Complete, Elegant and Short

How do you know when you've fully mastered a problem and came up with the best possible solution?

Think Occam's Razor

* When your solution is clean, simple and easy to understand;
* When you cannot make it any better had you had unlimited time on your hands;
* When your solution doesn't handle a single edge case of the problem, but rather completely eliminate it;
* When you feel your code should go up on the code hall of fame;

...that's when you're a real code ninja.

Be careful though. It's very easy to take this the way of over-engineering things.

If you're adding complexity instead of taking things away and simplyfing, you're doing it all wrong.

This doesn't mean you should re-engineer an entire architecture from scratch, only to make things cleaner.

It's okay to take baby steps, just make sure your contribution to the code, is the single most Elegant, Complete and Short solution.


A simple example

MySQL database server used to store timestamps in this format:

For example, 7am October 25th, 2010, would be represented as: 20101025070000

Starting with MySQL 5.1, the way timestamps are represented in the database changed to: YYYY-MM-DD HH:II:SS

The same date, is now represented as: 2010-10-25 07:00:00

When no date is set, MySQL 5.1 used 0000-00-00 00:00:00 whereas previous versions of the MySQL database used 00000000000000.

A few sections of our code had to test whether or not a timestamp was set.

The original php code looked something like this:

With the transition to MySQL 5.1, a new problem surfaced, where new records used "0000-00-00 00:00:00" to indicate no date was set, while old denormalized data still used "00000000000000".

Three different engineers approached this simple problem of supporting both the old and new timestamp formats.

I've included an excerpt of what each had to say below.

Engineer 1:

Engineer 2:

Engineer 3:

* First guy is out.

* Second guy cluttered the code, didn't properly handle other edge cases and didn't think about the future.

* Third one passed with flying colors.

Closing Thoughts

When I originally wrote this piece, I was thinking software development.

I realize now, after reading this again, that the same traits described here, apply to any area in business.

Be a true Problem solver.

It's the single most important skill you'll ever develop.

View 2 Comment(s)

How to mount /proc on FreeBSD

Michel Nadeau, September 27, 2010
There are a few commands under FreeBSD that depend on procfs (process file system).

FreeBSD doesn't mount it by default.

This tutorial describes how to mount /proc on FreeBSD and how to get FreeBSD to do it automatically when rebooting.

1. Mounting /proc

To mount /proc, run the following command:

Applications and commands depending on procfs will now work correctly.

2. Mount /proc automatically when rebooting

To get /proc to be mounted automatically when rebooting, simply add this line in /etc/fstab:

There you go, /proc will now be mounted automatically at boot time.
« Previous Posts » Next Posts

About Us  |  Contact us  |  Privacy Policy  |  Terms & Conditions