Recent Posts
Sponsors
 |
Lucene vs MySQL ShowdownMike Peters, 04-01-2010 |
MySQL is great as a storage-engine. It is one of the cornerstones that fueled the growth of Web 2.0, enabling tens of thousands of startups to get a robust database running overnight, without breaking the bank.
MySQL just has a lot of quirks making it a poor choice when it comes to production sub-second queries on big tables.
Index optimization, database denormalization, replication and sharding are great techniques to squeeze more juice out of MySQL...
But eventually, as your tables and queries grow, you're going to hit a brick-wall.
Storing huge amounts of data with MySQL is easy. But when it comes time to Retrieve those records, using filters, sorts and joins, you'll be lucky if you can ever scale beyond 1 million records while still maintaining high front-end speeds.
Thankfully, these days there are plenty of tools you can use to gradually replace components of your RDBMS world, with ones that are simply put, better tools for the job.
We are now in the process of converting the entire SPI platform, switching all queries, filters and sorting to Lucene, letting MySQL do nothing but primary-key lookups.
The impact on performance has been phenomenal!
In the video below, I demonstrate the effect of switching filters and sorting from MySQL to Lucene:
Next on the agenda, in our never-ending quest for performance and scalability:
* Lucene to Solr
* ActiveMQ replacing our home-grown queue
* Hadoop for processing
* Cassandra for logs and stats
MySQL just has a lot of quirks making it a poor choice when it comes to production sub-second queries on big tables.
Index optimization, database denormalization, replication and sharding are great techniques to squeeze more juice out of MySQL...
But eventually, as your tables and queries grow, you're going to hit a brick-wall.
Storing huge amounts of data with MySQL is easy. But when it comes time to Retrieve those records, using filters, sorts and joins, you'll be lucky if you can ever scale beyond 1 million records while still maintaining high front-end speeds.
Thankfully, these days there are plenty of tools you can use to gradually replace components of your RDBMS world, with ones that are simply put, better tools for the job.
We are now in the process of converting the entire SPI platform, switching all queries, filters and sorting to Lucene, letting MySQL do nothing but primary-key lookups.
The impact on performance has been phenomenal!
In the video below, I demonstrate the effect of switching filters and sorting from MySQL to Lucene:
Next on the agenda, in our never-ending quest for performance and scalability:
* Lucene to Solr
* ActiveMQ replacing our home-grown queue
* Hadoop for processing
* Cassandra for logs and stats
|
|
Subscribe Now to receive new posts via Email as soon as they come out.
Comments
Post your comments
