Full-service Internet Marketing & Web Development
Recent Posts
Recommended Reads
|
How to create a PHP DaemonMike Peters, April 14 -- Posted under Programming |
A daemon is a Linux program that runs in the background.
Most daemons are written in C. While C is faster and more robust than PHP, looking at development time and cost, PHP generally scores a lot better than C.
Converting your C daemons to PHP where applicable, makes the code more accessible to a wider percentage of your development force and allows better re-use and connecting of existing code.
To Facebook, keeping all code accessible to all developers was so important, that they've decided to come up with a way to compile PHP to C in real time. So you get PHP speed of development with C speed. The HipHop toolkit is available on an open source license.
The simple PHP4 functions included here, will let you easily create PHP daemons.
Features:
* Daemon will not start if another instance is already running
* Daemon PID saved to /var/run/
* Daemon log messages saved to /var/log/
* Run in foreground mode with the "-f" flag
Sample daemon:
require_once("common_daemon.php");
// Get parameters from command line
foreach ($argv as $parameter)
{
$run_in_foreground = Strcasecmp($parameter,"-f")==0;
}
// Start our daemon
if (!PHPDaemonStart("mydaemon", $run_in_foreground))
{
// Already running
return;
}
// Do something useful
while (1)
{
PHPDaemonLog("Hello, the time is ".date("h:i:s", time()));
sleep(1);
}
// Never here
The common_daemon.php functions:
// Give us eternity to execute the script.
ini_set("max_execution_time", "0");
ini_set("max_input_time", "0");
set_time_limit(0);
function PHPDaemonStart($name, $run_in_foreground=0)
{
// This must be global so that the handle is kept alive
// after this function exits
global $handle_lockfile;
// Remember the daemon name
global $phpdaemon_name;
$phpdaemon_name = $name;
// Remember if we're running in foreground mode
global $phpdaemon_run_in_foreground;
$phpdaemon_run_in_foreground = $run_in_foreground;
// Open PID file
$tmpfilename = "/var/run/$name.pid";
if (!($handle_lockfile = @fopen($tmpfilename,"a+")))
{
// Script already running - abort
return 0;
}
// Obtain an exlcusive lock on file
// (If script is running this will fail)
if (!@flock( $handle_lockfile, LOCK_EX | LOCK_NB, &$wouldblock) || $wouldblock)
{
// Script already running - abort
@fclose($handle_lockfile);
return 0;
}
// Write our PID
@ftruncate($handle_lockfile,0);
@fseek($handle_lockfile, 0, 0);
@fwrite($handle_lockfile, getmypid());
@fflush($handle_lockfile);
// If we need to run in foreground, we're done
if ($run_in_foreground)
{
return 1;
}
// If we're up to here we need to fork our process
$pid = pcntl_fork();
if ($pid==-1)
{
// Can't fork
return 0;
}
// If we are the parent
if ($pid)
{
// Sleep for 2 seconds, letting the forked process wake up
// and attempt to grab our lock
sleep(2);
// Release the lock
@flock($handle_lockfile, LOCK_UN);
// Close file
fclose($handle_lockfile);
// Kill parent process
// By now the child has the lock
die;
}
// (Else - we are the child)
else
{
// Open file
while (!($handle_lockfile = @fopen($tmpfilename, "a+")))
{
// Try again
usleep(100);
}
// Re-establish lock on file
while (!@flock($handle_lockfile, LOCK_EX, &$wouldblock))
{
// Try again
usleep(100);
}
// Write pid into file
@ftruncate($handle_lockfile,0);
@fseek($handle_lockfile, 0, 0);
@fwrite($handle_lockfile, getmypid());
@fflush($handle_lockfile);
}
// We have a lock - all good
return 1;
}
function PHPDaemonLog($str)
{
// This is the daemon name
global $phpdaemon_name;
// This indicates whether or not we are running in foreground mode
global $phpdaemon_run_in_foreground ;
// Open log file
$handle = @fopen("/var/log/$phpdaemon_name.log", "a");
// Obtain an exclusive lock (so that this is thread safe)
flock($handle, LOCK_EX, &$wouldblock);
// Write
$output = date("Y-m-d h:i:s", time())." $str\r\n";
@fwrite($handle, $output);
// If we are running in foreground mode, output to screen as well
if ($phpdaemon_run_in_foreground) echo $output;
// Release lock
flock($handle, LOCK_UN);
// Close file
fclose($handle);
}
function IsPHPDaemonRunning($name)
{
// Open PID file
$tmpfilename = "/var/run/$name.pid";
if (!($tmpfile = @fopen($tmpfilename,"r")))
{
// Script not running
return 0;
}
// Obtain an exlcusive lock on file
// (If script is running this will fail)
$not_running = flock( $tmpfile, LOCK_EX | LOCK_NB, &$wouldblock);
// Release lock if successful
if ($not_running) flock($tmpfile, LOCK_UN);
// Close file
@fclose($tmpfile);
// Return result
return !$not_running || $wouldblock;
}
View 1 Comment(s)
Most daemons are written in C. While C is faster and more robust than PHP, looking at development time and cost, PHP generally scores a lot better than C.
Converting your C daemons to PHP where applicable, makes the code more accessible to a wider percentage of your development force and allows better re-use and connecting of existing code.
To Facebook, keeping all code accessible to all developers was so important, that they've decided to come up with a way to compile PHP to C in real time. So you get PHP speed of development with C speed. The HipHop toolkit is available on an open source license.
The simple PHP4 functions included here, will let you easily create PHP daemons.
Features:
* Daemon will not start if another instance is already running
* Daemon PID saved to /var/run/
* Daemon log messages saved to /var/log/
* Run in foreground mode with the "-f" flag
Sample daemon:
require_once("common_daemon.php");
// Get parameters from command line
foreach ($argv as $parameter)
{
$run_in_foreground = Strcasecmp($parameter,"-f")==0;
}
// Start our daemon
if (!PHPDaemonStart("mydaemon", $run_in_foreground))
{
// Already running
return;
}
// Do something useful
while (1)
{
PHPDaemonLog("Hello, the time is ".date("h:i:s", time()));
sleep(1);
}
// Never here
The common_daemon.php functions:
// Give us eternity to execute the script.
ini_set("max_execution_time", "0");
ini_set("max_input_time", "0");
set_time_limit(0);
function PHPDaemonStart($name, $run_in_foreground=0)
{
// This must be global so that the handle is kept alive
// after this function exits
global $handle_lockfile;
// Remember the daemon name
global $phpdaemon_name;
$phpdaemon_name = $name;
// Remember if we're running in foreground mode
global $phpdaemon_run_in_foreground;
$phpdaemon_run_in_foreground = $run_in_foreground;
// Open PID file
$tmpfilename = "/var/run/$name.pid";
if (!($handle_lockfile = @fopen($tmpfilename,"a+")))
{
// Script already running - abort
return 0;
}
// Obtain an exlcusive lock on file
// (If script is running this will fail)
if (!@flock( $handle_lockfile, LOCK_EX | LOCK_NB, &$wouldblock) || $wouldblock)
{
// Script already running - abort
@fclose($handle_lockfile);
return 0;
}
// Write our PID
@ftruncate($handle_lockfile,0);
@fseek($handle_lockfile, 0, 0);
@fwrite($handle_lockfile, getmypid());
@fflush($handle_lockfile);
// If we need to run in foreground, we're done
if ($run_in_foreground)
{
return 1;
}
// If we're up to here we need to fork our process
$pid = pcntl_fork();
if ($pid==-1)
{
// Can't fork
return 0;
}
// If we are the parent
if ($pid)
{
// Sleep for 2 seconds, letting the forked process wake up
// and attempt to grab our lock
sleep(2);
// Release the lock
@flock($handle_lockfile, LOCK_UN);
// Close file
fclose($handle_lockfile);
// Kill parent process
// By now the child has the lock
die;
}
// (Else - we are the child)
else
{
// Open file
while (!($handle_lockfile = @fopen($tmpfilename, "a+")))
{
// Try again
usleep(100);
}
// Re-establish lock on file
while (!@flock($handle_lockfile, LOCK_EX, &$wouldblock))
{
// Try again
usleep(100);
}
// Write pid into file
@ftruncate($handle_lockfile,0);
@fseek($handle_lockfile, 0, 0);
@fwrite($handle_lockfile, getmypid());
@fflush($handle_lockfile);
}
// We have a lock - all good
return 1;
}
function PHPDaemonLog($str)
{
// This is the daemon name
global $phpdaemon_name;
// This indicates whether or not we are running in foreground mode
global $phpdaemon_run_in_foreground ;
// Open log file
$handle = @fopen("/var/log/$phpdaemon_name.log", "a");
// Obtain an exclusive lock (so that this is thread safe)
flock($handle, LOCK_EX, &$wouldblock);
// Write
$output = date("Y-m-d h:i:s", time())." $str\r\n";
@fwrite($handle, $output);
// If we are running in foreground mode, output to screen as well
if ($phpdaemon_run_in_foreground) echo $output;
// Release lock
flock($handle, LOCK_UN);
// Close file
fclose($handle);
}
function IsPHPDaemonRunning($name)
{
// Open PID file
$tmpfilename = "/var/run/$name.pid";
if (!($tmpfile = @fopen($tmpfilename,"r")))
{
// Script not running
return 0;
}
// Obtain an exlcusive lock on file
// (If script is running this will fail)
$not_running = flock( $tmpfile, LOCK_EX | LOCK_NB, &$wouldblock);
// Release lock if successful
if ($not_running) flock($tmpfile, LOCK_UN);
// Close file
@fclose($tmpfile);
// Return result
return !$not_running || $wouldblock;
}
View 1 Comment(s)
|
|
Cassandra PHP WrapperMike Peters, April 7 -- Posted under Programming |
Update: Check out the Cassandra PHP Wrapper 0.7
In a previous post, I explained how the Cassandra decentralized database can allow you to scale well beyond what's possible with MySQL.
The Cassandra Data Model and API are very different than traditional RDBMS way-of-thinking. That's why a lot of developers switching from MySQL to Cassandra are finding it difficult to grasp at first.
To simplify the migration from MySQL to Cassandra, we created a high-level PHP wrapper for Cassandra, using function prototypes and variable names that are close in meaning to RDBMS.
While there are several other high level PHP wrappers for Cassandra, none of the existing ones answered our requirements -
* Simple
* Don't throw exceptions, return error codes
* As close as possible to the low level Thrift
* Make it easy for RDBMS developers to adopt
This is how the CassandraDB class was born.
Adding records to the database is as simple as:
// Initialize Cassandra
$cassandra = new CassandraDB("SPI");
// Debug on
$cassandra->SetDisplayErrors(true);
// Insert record ("Columns" in Cassandra)
$record = array();
$record["name"] = "Mike Peters";
$record["email"] = "mike at softwareprojects.com";
if ($cassandra->InsertRecord('mytable', "Mike Peters", $record))
{
echo "Record (Columns) inserted successfully.\r\n";
}
// Print record
$record = $cassandra->GetRecordByKey('mytable', "Mike Peters");
print_r($record);
Output is
Adding record arrays:
// Initialize Cassandra
$cassandra = new CassandraDB("SPI");
// Debug on
$cassandra->SetDisplayErrors(true);
// Insert record array ("SuperColumns" in Cassandra)
$record = array();
$record["Mike Peters"] = array("name" => "Mike Peters", "email" => "Mike at Peters");
$record["Jonathan Ellis"] = array("name" => "Jonathan Ellis", "email" => "Jonathan at Ellis");
if ($cassandra->InsertRecordArray('my_super_table', "People", $record))
{
echo "RecordArray (SuperColumns) inserted successfully.\r\n";
}
// Print record array
$record = $cassandra->GetRecordByKey('my_super_table', 'People');
print_r($record);
Output is
--
Here's the complete Cassandra PHP Wrapper class. Enjoy!
// CassandraDB version 0.1
// Software Projects Inc
// http://www.softwareprojects.com
//
// Includes
$GLOBALS['THRIFT_ROOT'] = '/services/thrift/';
require_once $GLOBALS['THRIFT_ROOT'].'/packages/cassandra/Cassandra.php';
require_once $GLOBALS['THRIFT_ROOT'].'/packages/cassandra/cassandra_types.php';
require_once $GLOBALS['THRIFT_ROOT'].'/transport/TSocket.php';
require_once $GLOBALS['THRIFT_ROOT'].'/protocol/TBinaryProtocol.php';
require_once $GLOBALS['THRIFT_ROOT'].'/transport/TFramedTransport.php';
require_once $GLOBALS['THRIFT_ROOT'].'/transport/TBufferedTransport.php';
class CassandraDB
{
// Internal variables
protected $socket;
protected $client;
protected $keyspace;
protected $transport;
protected $protocol;
protected $err_str = "";
protected $display_errors = 0;
protected $consistency = 1;
protected $parse_columns = 1;
// Functions
// Constructor - Connect to Cassandra via Thrift
function CassandraDB ($keyspace, $host = "127.0.0.1", $port = 9160)
{
// Initialize
$this->err_str = '';
try
{
// Store passed 'keyspace' in object
$this->keyspace = $keyspace;
// Make a connection to the Thrift interface to Cassandra
$this->socket = new TSocket($host, $port);
$this->transport = new TFramedTransport($this->socket, 1024, 1024);
$this->protocol = new TBinaryProtocolAccelerated($this->transport);
$this->client = new CassandraClient($this->protocol);
$this->transport->open();
}
catch (TException $tx)
{
// Error occured
$this->err_str = $tx->why;
$this->Debug($tx->why." ".$tx->getMessage());
}
}
// Insert Column into ColumnFamily
// (Equivalent to RDBMS Insert record to a table)
function InsertRecord ($table /* ColumnFamily */, $key /* ColumnFamily Key */, $record /* Columns */)
{
// Initialize
$this->err_str = '';
try
{
// Timestamp for update
$timestamp = time();
// Build batch mutation
$cfmap = array();
$cfmap[$table] = $this->array_to_supercolumns_or_columns($record, $timestamp);
// Insert
$this->client->batch_insert($this->keyspace, $key, $cfmap, $this->consistency);
// If we're up to here, all is well
$result = 1;
}
catch (TException $tx)
{
// Error occured
$result = 0;
$this->err_str = $tx->why;
$this->Debug($tx->why." ".$tx->getMessage());
}
// Return result
return $result;
}
// Insert SuperColumn into SuperColumnFamily
// (Equivalent to RDMBS Insert record to a "nested table")
function InsertRecordArray ($table /* SuperColumnFamily */, $key_parent /* Super CF */,
$record /* Columns */)
{
// Initialize
$err_str = '';
try
{
// Timestamp for update
$timestamp = time();
// Build batch mutation
$cfmap = array();
$cfmap[$table] = $this->array_to_supercolumns_or_columns($record, $timestamp);
// Insert
$this->client->batch_insert($this->keyspace, $key_parent, $cfmap, $this->consistency);
// If we're up to here, all is well
$result = 1;
}
catch (TException $tx)
{
// Error occured
$result = 0;
$this->err_str = $tx->why;
$this->Debug($tx->why." ".$tx->getMessage());
}
// Return result
return $result;
}
// Get record by key
function GetRecordByKey ($table /* ColumnFamily or SuperColumnFamily */, $key, $start_from="", $end_at="")
{
// Initialize
$err_str = '';
try
{
return $this->get($table, $key, NULL, $start_from, $end_at);
}
catch (TException $tx)
{
// Error occured
$this->err_str = $tx->why;
$this->Debug($tx->why." ".$tx->getMessage());
return array();
}
}
// Print debug message
function Debug ($str)
{
// If verbose is off, we're done
if (!$this->display_errors) return;
// Print
echo date("Y-m-d h:i:s")." CassandraDB ERROR: $str\r\n";
}
// Turn verbose debug on/off (Default is off)
function SetDisplayErrors($flag)
{
$this->display_errors = $flag;
}
// Set Consistency level (Default is 1)
function SetConsistency ($consistency)
{
$this->consistency = $consistency;
}
// Build cf array
function array_to_supercolumns_or_columns($array, $timestamp=null)
{
if(empty($timestamp)) $timestamp = time();
$ret = null;
foreach($array as $name => $value) {
$c_or_sc = new cassandra_ColumnOrSuperColumn();
if(is_array($value)) {
$c_or_sc->super_column = new cassandra_SuperColumn();
$c_or_sc->super_column->name = $this->unparse_column_name($name, true);
$c_or_sc->super_column->columns = $this->array_to_columns($value, $timestamp);
$c_or_sc->super_column->timestamp = $timestamp;
}
else
{
$c_or_sc = new cassandra_ColumnOrSuperColumn();
$c_or_sc->column = new cassandra_Column();
$c_or_sc->column->name = $this->unparse_column_name($name, true);
$c_or_sc->column->value = $value;
$c_or_sc->column->timestamp = $timestamp;
}
$ret[] = $c_or_sc;
}
return $ret;
}
// Parse column names for Cassandra
function parse_column_name($column_name, $is_column=true)
{
if(!$column_name) return NULL;
return $column_name;
}
// Unparse column names for Cassandra
function unparse_column_name($column_name, $is_column=true)
{
if(!$column_name) return NULL;
return $column_name;
}
// Convert supercolumns or columns into an array
function supercolumns_or_columns_to_array($array)
{
$ret = null;
for ($i=0; $i<count($array); $i++)
foreach ($array[$i] as $object)
{
if ($object)
{
// If supercolumn
if (isset($object->columns))
{
$record = array();
for ($j=0; $j<count($object->columns); $j++)
{
$column = $object->columns[$j];
$record[$column->name] = $column->value;
}
$ret[$object->name] = $record;
}
// (Otherwise - not supercolumn)
else
{
$ret[$object->name] = $object->value;
}
}
}
return $ret;
}
// Get record from Cassandra
function get($table, $key, $super_column=NULL, $slice_start="", $slice_finish="")
{
try
{
$column_parent = new cassandra_ColumnParent();
$column_parent->column_family = $table;
$column_parent->super_column = $this->unparse_column_name($super_column, false);
$slice_range = new cassandra_SliceRange();
$slice_range->start = $slice_start;
$slice_range->finish = $slice_finish;
$predicate = new cassandra_SlicePredicate();
$predicate->slice_range = $slice_range;
$resp = $this->client->get_slice($this->keyspace, $key, $column_parent, $predicate, $this->consistency);
return $this->supercolumns_or_columns_to_array($resp);
}
catch (TException $tx)
{
$this->Debug($tx->why." ".$tx->getMessage());
return array();
}
}
// Convert array to columns
function array_to_columns($array, $timestamp=null) {
if(empty($timestamp)) $timestamp = time();
$ret = null;
foreach($array as $name => $value) {
$column = new cassandra_Column();
$column->name = $this->unparse_column_name($name, false);
$column->value = $value;
$column->timestamp = $timestamp;
$ret[] = $column;
}
return $ret;
}
// Get error string
function ErrorStr()
{
return $this->err_str;
}
}
CassandraDB Todo:
* Ability to add multiple nodes and try to connect to next-in-line automatically if connection fails
* Implement the remaining get_ functions
* Implement delete
* Implement update vs insert-fail-on-duplicate-key?
* Implement Order By desc
-
Recommended Further Reading:
* Cassandra in Production at Digg
* WTF is a SuperColumn?
* Cassandra Reads - How do they work
* Cassandra Writes - How do they work
* Cassandra Write Properties (Video)
* Engineering notes by Facebook
* RandomPartitioner vs OrderPreservingPartitioner
View 10 Comment(s)
In a previous post, I explained how the Cassandra decentralized database can allow you to scale well beyond what's possible with MySQL.
The Cassandra Data Model and API are very different than traditional RDBMS way-of-thinking. That's why a lot of developers switching from MySQL to Cassandra are finding it difficult to grasp at first.
To simplify the migration from MySQL to Cassandra, we created a high-level PHP wrapper for Cassandra, using function prototypes and variable names that are close in meaning to RDBMS.
While there are several other high level PHP wrappers for Cassandra, none of the existing ones answered our requirements -
* Simple
* Don't throw exceptions, return error codes
* As close as possible to the low level Thrift
* Make it easy for RDBMS developers to adopt
This is how the CassandraDB class was born.
Adding records to the database is as simple as:
// Initialize Cassandra
$cassandra = new CassandraDB("SPI");
// Debug on
$cassandra->SetDisplayErrors(true);
// Insert record ("Columns" in Cassandra)
$record = array();
$record["name"] = "Mike Peters";
$record["email"] = "mike at softwareprojects.com";
if ($cassandra->InsertRecord('mytable', "Mike Peters", $record))
{
echo "Record (Columns) inserted successfully.\r\n";
}
// Print record
$record = $cassandra->GetRecordByKey('mytable', "Mike Peters");
print_r($record);
Output is
Array
(
[email] => mike at softwareprojects.com
[name] => Mike Peters
)
(
[email] => mike at softwareprojects.com
[name] => Mike Peters
)
Adding record arrays:
// Initialize Cassandra
$cassandra = new CassandraDB("SPI");
// Debug on
$cassandra->SetDisplayErrors(true);
// Insert record array ("SuperColumns" in Cassandra)
$record = array();
$record["Mike Peters"] = array("name" => "Mike Peters", "email" => "Mike at Peters");
$record["Jonathan Ellis"] = array("name" => "Jonathan Ellis", "email" => "Jonathan at Ellis");
if ($cassandra->InsertRecordArray('my_super_table', "People", $record))
{
echo "RecordArray (SuperColumns) inserted successfully.\r\n";
}
// Print record array
$record = $cassandra->GetRecordByKey('my_super_table', 'People');
print_r($record);
Output is
Array
(
[Jonathan Ellis] => Array
(
[email] => Jonathan at Ellis
[name] => Jonathan Ellis
)
[Mike Peters] => Array
(
[email] => Mike at Peters
[name] => Mike Peters
)
)
(
[Jonathan Ellis] => Array
(
[email] => Jonathan at Ellis
[name] => Jonathan Ellis
)
[Mike Peters] => Array
(
[email] => Mike at Peters
[name] => Mike Peters
)
)
--
Here's the complete Cassandra PHP Wrapper class. Enjoy!
// CassandraDB version 0.1
// Software Projects Inc
// http://www.softwareprojects.com
//
// Includes
$GLOBALS['THRIFT_ROOT'] = '/services/thrift/';
require_once $GLOBALS['THRIFT_ROOT'].'/packages/cassandra/Cassandra.php';
require_once $GLOBALS['THRIFT_ROOT'].'/packages/cassandra/cassandra_types.php';
require_once $GLOBALS['THRIFT_ROOT'].'/transport/TSocket.php';
require_once $GLOBALS['THRIFT_ROOT'].'/protocol/TBinaryProtocol.php';
require_once $GLOBALS['THRIFT_ROOT'].'/transport/TFramedTransport.php';
require_once $GLOBALS['THRIFT_ROOT'].'/transport/TBufferedTransport.php';
class CassandraDB
{
// Internal variables
protected $socket;
protected $client;
protected $keyspace;
protected $transport;
protected $protocol;
protected $err_str = "";
protected $display_errors = 0;
protected $consistency = 1;
protected $parse_columns = 1;
// Functions
// Constructor - Connect to Cassandra via Thrift
function CassandraDB ($keyspace, $host = "127.0.0.1", $port = 9160)
{
// Initialize
$this->err_str = '';
try
{
// Store passed 'keyspace' in object
$this->keyspace = $keyspace;
// Make a connection to the Thrift interface to Cassandra
$this->socket = new TSocket($host, $port);
$this->transport = new TFramedTransport($this->socket, 1024, 1024);
$this->protocol = new TBinaryProtocolAccelerated($this->transport);
$this->client = new CassandraClient($this->protocol);
$this->transport->open();
}
catch (TException $tx)
{
// Error occured
$this->err_str = $tx->why;
$this->Debug($tx->why." ".$tx->getMessage());
}
}
// Insert Column into ColumnFamily
// (Equivalent to RDBMS Insert record to a table)
function InsertRecord ($table /* ColumnFamily */, $key /* ColumnFamily Key */, $record /* Columns */)
{
// Initialize
$this->err_str = '';
try
{
// Timestamp for update
$timestamp = time();
// Build batch mutation
$cfmap = array();
$cfmap[$table] = $this->array_to_supercolumns_or_columns($record, $timestamp);
// Insert
$this->client->batch_insert($this->keyspace, $key, $cfmap, $this->consistency);
// If we're up to here, all is well
$result = 1;
}
catch (TException $tx)
{
// Error occured
$result = 0;
$this->err_str = $tx->why;
$this->Debug($tx->why." ".$tx->getMessage());
}
// Return result
return $result;
}
// Insert SuperColumn into SuperColumnFamily
// (Equivalent to RDMBS Insert record to a "nested table")
function InsertRecordArray ($table /* SuperColumnFamily */, $key_parent /* Super CF */,
$record /* Columns */)
{
// Initialize
$err_str = '';
try
{
// Timestamp for update
$timestamp = time();
// Build batch mutation
$cfmap = array();
$cfmap[$table] = $this->array_to_supercolumns_or_columns($record, $timestamp);
// Insert
$this->client->batch_insert($this->keyspace, $key_parent, $cfmap, $this->consistency);
// If we're up to here, all is well
$result = 1;
}
catch (TException $tx)
{
// Error occured
$result = 0;
$this->err_str = $tx->why;
$this->Debug($tx->why." ".$tx->getMessage());
}
// Return result
return $result;
}
// Get record by key
function GetRecordByKey ($table /* ColumnFamily or SuperColumnFamily */, $key, $start_from="", $end_at="")
{
// Initialize
$err_str = '';
try
{
return $this->get($table, $key, NULL, $start_from, $end_at);
}
catch (TException $tx)
{
// Error occured
$this->err_str = $tx->why;
$this->Debug($tx->why." ".$tx->getMessage());
return array();
}
}
// Print debug message
function Debug ($str)
{
// If verbose is off, we're done
if (!$this->display_errors) return;
echo date("Y-m-d h:i:s")." CassandraDB ERROR: $str\r\n";
}
// Turn verbose debug on/off (Default is off)
function SetDisplayErrors($flag)
{
$this->display_errors = $flag;
}
// Set Consistency level (Default is 1)
function SetConsistency ($consistency)
{
$this->consistency = $consistency;
}
// Build cf array
function array_to_supercolumns_or_columns($array, $timestamp=null)
{
if(empty($timestamp)) $timestamp = time();
$ret = null;
foreach($array as $name => $value) {
$c_or_sc = new cassandra_ColumnOrSuperColumn();
if(is_array($value)) {
$c_or_sc->super_column = new cassandra_SuperColumn();
$c_or_sc->super_column->name = $this->unparse_column_name($name, true);
$c_or_sc->super_column->columns = $this->array_to_columns($value, $timestamp);
$c_or_sc->super_column->timestamp = $timestamp;
}
else
{
$c_or_sc = new cassandra_ColumnOrSuperColumn();
$c_or_sc->column = new cassandra_Column();
$c_or_sc->column->name = $this->unparse_column_name($name, true);
$c_or_sc->column->value = $value;
$c_or_sc->column->timestamp = $timestamp;
}
$ret[] = $c_or_sc;
}
return $ret;
}
// Parse column names for Cassandra
function parse_column_name($column_name, $is_column=true)
{
if(!$column_name) return NULL;
return $column_name;
}
// Unparse column names for Cassandra
function unparse_column_name($column_name, $is_column=true)
{
if(!$column_name) return NULL;
return $column_name;
}
// Convert supercolumns or columns into an array
function supercolumns_or_columns_to_array($array)
{
$ret = null;
for ($i=0; $i<count($array); $i++)
foreach ($array[$i] as $object)
{
if ($object)
{
// If supercolumn
if (isset($object->columns))
{
$record = array();
for ($j=0; $j<count($object->columns); $j++)
{
$column = $object->columns[$j];
$record[$column->name] = $column->value;
}
$ret[$object->name] = $record;
}
// (Otherwise - not supercolumn)
else
{
$ret[$object->name] = $object->value;
}
}
}
return $ret;
}
// Get record from Cassandra
function get($table, $key, $super_column=NULL, $slice_start="", $slice_finish="")
{
try
{
$column_parent = new cassandra_ColumnParent();
$column_parent->column_family = $table;
$column_parent->super_column = $this->unparse_column_name($super_column, false);
$slice_range = new cassandra_SliceRange();
$slice_range->start = $slice_start;
$slice_range->finish = $slice_finish;
$predicate = new cassandra_SlicePredicate();
$predicate->slice_range = $slice_range;
$resp = $this->client->get_slice($this->keyspace, $key, $column_parent, $predicate, $this->consistency);
return $this->supercolumns_or_columns_to_array($resp);
}
catch (TException $tx)
{
$this->Debug($tx->why." ".$tx->getMessage());
return array();
}
}
// Convert array to columns
function array_to_columns($array, $timestamp=null) {
if(empty($timestamp)) $timestamp = time();
$ret = null;
foreach($array as $name => $value) {
$column = new cassandra_Column();
$column->name = $this->unparse_column_name($name, false);
$column->value = $value;
$column->timestamp = $timestamp;
$ret[] = $column;
}
return $ret;
}
// Get error string
function ErrorStr()
{
return $this->err_str;
}
}
CassandraDB Todo:
* Ability to add multiple nodes and try to connect to next-in-line automatically if connection fails
* Implement the remaining get_ functions
* Implement delete
* Implement update vs insert-fail-on-duplicate-key?
* Implement Order By desc
-
Recommended Further Reading:
* Cassandra in Production at Digg
* WTF is a SuperColumn?
* Cassandra Reads - How do they work
* Cassandra Writes - How do they work
* Cassandra Write Properties (Video)
* Engineering notes by Facebook
* RandomPartitioner vs OrderPreservingPartitioner
View 10 Comment(s)
|
|
How to install Cassandra + Thrift (and why you should care)Mike Peters, April 5 -- Posted under Programming |
Cassandra is a decentralized (fast reads), highly available (fast writes), fault tolerant database that can allow you to scale out well beyonds what's available with traditional RDBMS like MySQL.
Index optimization, database denormalization, replication and sharding are great techniques to squeeze more juice out of MySQL...
But eventually, as your tables and queries grow, you're going to hit a brick-wall.
Storing huge amounts of data with MySQL is easy. But when it comes time to Retrieve those records, using filters, sorts and joins, you'll be lucky if you can ever scale beyond 1 million records (without aggressive sharding and memcached) while still maintaining high front-end speeds.
With companies like Digg, Facebook, Twitter, switching over from MySQL and betting all cards on Cassandra, this is one technology you should become intimately familiar with.
Key features of Cassandra:
* Fault Tolerant: Data is automatically replicated to multiple nodes for fault-tolerance. Replication across multiple data centers is supported. Failed nodes can be replaced with no downtime.
* Decentralized: Every node in the cluster is identical. There are no network bottlenecks. There are no single points of failure.
* Flexible: Read and write throughput both increase linearly as new machines are added, with no downtime or interruption to applications.
* Highly Available: Writes and reads offer a tunable ConsistencyLevel, all the way from "writes never fail" to "block for all replicas to be readable," with the quorum level in the middle.
Understanding Cassandra Data Model
Keyspace = Database name. (Usually one per application)
Column = One Table cell (field name, field value, timestamp)
Important: Columns are always sorted by their name. Sorting supports BytesType, UTF8Type, LexicalUUIDType, TimeUUIDType, AsciiType and LongType. Each of these options treats the Columns' name as a different data type.
SuperColumn = One row in a table.
Important: Supercolumns are always sorted by their name. Internally the columns inside each super column are also sorted by their name.
ColumnFamily = Table holding Columns. (Structure that contains an infinite number of Rows)
SuperColumnFamily = Table holding SuperColumns. Similar to ColumnFamily, but in this case every "row" holds SuperColumns.
--
1. Getting started
Download the latest version of Cassandra from this page.
Use the Bin version. No need to recompile.
2. Download JRE
Download and install the version of JRE matching your system (32bit or 64bit) from the Java SE Download page.
3. Configure Cassandra
Update the ClusterName to something meaningful.
Update the folders where the database is to be stored: CommitLogDirectories, DataFileDirectories, CalloutLocation and StagingFileDirectory.
4. Set JAVA_HOME
Before you can start Cassandra, it's important to set the JAVA_HOME environment variable, pointing it to the location of the Java JRE bin files on your machine.
For example:
5. Start Cassandra
If you get any errors about ClassNotFound, it means there's something wrong with your JAVA_HOME environment variable. Make sure it is pointing to the proper JRE/JDK directory on your machine.
6. Install Thrift
Now that we have one Cassandra Node running, let's move on to interface with our new database.
Thrift is a software framework for scalable cross-language services development. Cassandra supports Thrift, thereby allowing integration across multiple programming languages and platforms.
As part of this guide we'll use Thrift over PHP.
First, we need to install Boost.
We need automake and autoconf:
Now, assuming you meet Thrift requirements, we can proceed.
You can download the latest version of Thrift from this page.
If you're on FreeBSD, you can simply install Thrift from the FreeBSD ports:
7. Interfacing with Cassandra
Creating a record in Cassandra using Thrift:
Check out our Cassandra PHP Wrapper for an easier way to interface with Cassandra from PHP, or refer to the Thrift Examples.
View 11 Comment(s)
Index optimization, database denormalization, replication and sharding are great techniques to squeeze more juice out of MySQL...
But eventually, as your tables and queries grow, you're going to hit a brick-wall.
Storing huge amounts of data with MySQL is easy. But when it comes time to Retrieve those records, using filters, sorts and joins, you'll be lucky if you can ever scale beyond 1 million records (without aggressive sharding and memcached) while still maintaining high front-end speeds.
With companies like Digg, Facebook, Twitter, switching over from MySQL and betting all cards on Cassandra, this is one technology you should become intimately familiar with.
Key features of Cassandra:
* Fault Tolerant: Data is automatically replicated to multiple nodes for fault-tolerance. Replication across multiple data centers is supported. Failed nodes can be replaced with no downtime.
* Decentralized: Every node in the cluster is identical. There are no network bottlenecks. There are no single points of failure.
* Flexible: Read and write throughput both increase linearly as new machines are added, with no downtime or interruption to applications.
* Highly Available: Writes and reads offer a tunable ConsistencyLevel, all the way from "writes never fail" to "block for all replicas to be readable," with the quorum level in the middle.
Understanding Cassandra Data Model
Keyspace = Database name. (Usually one per application)
Column = One Table cell (field name, field value, timestamp)
{ // this is a column
name: "emailAddress",
value: "arin@example.com",
timestamp: 123456789
}
name: "emailAddress",
value: "arin@example.com",
timestamp: 123456789
}
Important: Columns are always sorted by their name. Sorting supports BytesType, UTF8Type, LexicalUUIDType, TimeUUIDType, AsciiType and LongType. Each of these options treats the Columns' name as a different data type.
SuperColumn = One row in a table.
{ // this is a super column
name: "homeAddress",
// with an infinite list of columns
value: {
street: {name: "street", value: "1234 x street", timestamp: 123456789},
city: {name: "city", value: "san francisco", timestamp: 123456789},
zip: {name: "zip", value: "94107", timestamp: 123456789},
}
}
name: "homeAddress",
// with an infinite list of columns
value: {
street: {name: "street", value: "1234 x street", timestamp: 123456789},
city: {name: "city", value: "san francisco", timestamp: 123456789},
zip: {name: "zip", value: "94107", timestamp: 123456789},
}
}
Important: Supercolumns are always sorted by their name. Internally the columns inside each super column are also sorted by their name.
ColumnFamily = Table holding Columns. (Structure that contains an infinite number of Rows)
UserProfile = { // this is a ColumnFamily
phatduckk: { // this is the key to this Row inside the CF
// now we have an infinite # of columns in this row
username: "phatduckk",
email: "phatduckk@example.com",
phone: "(900) 976-6666"
}, // end row
ieure: { // this is the key to another row in the CF
// now we have another infinite # of columns in this row
username: "ieure",
email: "ieure@example.com",
phone: "(888) 555-1212"
age: "66",
gender: "undecided"
},
}
phatduckk: { // this is the key to this Row inside the CF
// now we have an infinite # of columns in this row
username: "phatduckk",
email: "phatduckk@example.com",
phone: "(900) 976-6666"
}, // end row
ieure: { // this is the key to another row in the CF
// now we have another infinite # of columns in this row
username: "ieure",
email: "ieure@example.com",
phone: "(888) 555-1212"
age: "66",
gender: "undecided"
},
}
SuperColumnFamily = Table holding SuperColumns. Similar to ColumnFamily, but in this case every "row" holds SuperColumns.
AddressBook = { // this is a ColumnFamily of type Super
phatduckk: { // this is the key to this row inside the Super CF
// the key here is the name of the owner of the address book
// now we have an infinite # of super columns in this row
// the keys inside the row are the names for the SuperColumns
// each of these SuperColumns is an address book entry
friend1: {street: "8th street", zip: "90210", city: "Beverley Hills", state: "CA"},
// this is the address book entry for John in phatduckk's address book
John: {street: "Howard street", zip: "94404", city: "FC", state: "CA"},
Kim: {street: "X street", zip: "87876", city: "Balls", state: "VA"},
Tod: {street: "Jerry street", zip: "54556", city: "Cartoon", state: "CO"},
Bob: {street: "Q Blvd", zip: "24252", city: "Nowhere", state: "MN"},
...
// we can have an infinite # of ScuperColumns (aka address book entries)
}, // end row
ieure: { // this is the key to another row in the Super CF
// all the address book entries for ieure
joey: {street: "A ave", zip: "55485", city: "Hell", state: "NV"},
William: {street: "Armpit Dr", zip: "93301", city: "Bakersfield", state: "CA"},
},
}
phatduckk: { // this is the key to this row inside the Super CF
// the key here is the name of the owner of the address book
// now we have an infinite # of super columns in this row
// the keys inside the row are the names for the SuperColumns
// each of these SuperColumns is an address book entry
friend1: {street: "8th street", zip: "90210", city: "Beverley Hills", state: "CA"},
// this is the address book entry for John in phatduckk's address book
John: {street: "Howard street", zip: "94404", city: "FC", state: "CA"},
Kim: {street: "X street", zip: "87876", city: "Balls", state: "VA"},
Tod: {street: "Jerry street", zip: "54556", city: "Cartoon", state: "CO"},
Bob: {street: "Q Blvd", zip: "24252", city: "Nowhere", state: "MN"},
...
// we can have an infinite # of ScuperColumns (aka address book entries)
}, // end row
ieure: { // this is the key to another row in the Super CF
// all the address book entries for ieure
joey: {street: "A ave", zip: "55485", city: "Hell", state: "NV"},
William: {street: "Armpit Dr", zip: "93301", city: "Bakersfield", state: "CA"},
},
}
--
1. Getting started
Download the latest version of Cassandra from this page.
Use the Bin version. No need to recompile.
mkdir /usr/tmp
fetch "http://apache.raffsoftware.com/cassandra/0.5.1/apache-cassandra-0.5.1-bin.tar.gz"
tar xvfz apache-cassandra-0.5.1-bin.tar.gz
fetch "http://apache.raffsoftware.com/cassandra/0.5.1/apache-cassandra-0.5.1-bin.tar.gz"
tar xvfz apache-cassandra-0.5.1-bin.tar.gz
2. Download JRE
Download and install the version of JRE matching your system (32bit or 64bit) from the Java SE Download page.
3. Configure Cassandra
cd /usr/tmp/apache-cassandra-0.5.1/conf
vi storage-conf.xml
vi storage-conf.xml
Update the ClusterName to something meaningful.
Update the folders where the database is to be stored: CommitLogDirectories, DataFileDirectories, CalloutLocation and StagingFileDirectory.
4. Set JAVA_HOME
Before you can start Cassandra, it's important to set the JAVA_HOME environment variable, pointing it to the location of the Java JRE bin files on your machine.
For example:
setenv JAVA_HOME /usr/home/admin/htdocs/services/java/jdk7_64
set JAVA_HOME=/usr/home/admin/htdocs/services/java/jdk7_64
set JAVA_HOME=/usr/home/admin/htdocs/services/java/jdk7_64
5. Start Cassandra
cd /usr/tmp/apache-cassandra-0.5.1/
bin/cassandra -f
bin/cassandra -f
If you get any errors about ClassNotFound, it means there's something wrong with your JAVA_HOME environment variable. Make sure it is pointing to the proper JRE/JDK directory on your machine.
6. Install Thrift
Now that we have one Cassandra Node running, let's move on to interface with our new database.
Thrift is a software framework for scalable cross-language services development. Cassandra supports Thrift, thereby allowing integration across multiple programming languages and platforms.
As part of this guide we'll use Thrift over PHP.
First, we need to install Boost.
cd /usr/ports/devel/boost
make all
make install
make all
make install
We need automake and autoconf:
cd /usr/ports/devel/automake110
make all
make install
/usr/ports/devel/autoconf262
make all
make install
make all
make install
/usr/ports/devel/autoconf262
make all
make install
Now, assuming you meet Thrift requirements, we can proceed.
You can download the latest version of Thrift from this page.
cd /usr/tmp/
fetch "http://apache.raffsoftware.com/incubator/thrift/0.2.0-incubating/thrift-0.2.0-incubating.tar.gz"
tar xvfz thrift-0.2.0-incubating.tar.gz
cd thrift-0.2.0
./bootstrap.sh
./configure --with-boost=/usr/local
make
make install
fetch "http://apache.raffsoftware.com/incubator/thrift/0.2.0-incubating/thrift-0.2.0-incubating.tar.gz"
tar xvfz thrift-0.2.0-incubating.tar.gz
cd thrift-0.2.0
./bootstrap.sh
./configure --with-boost=/usr/local
make
make install
If you're on FreeBSD, you can simply install Thrift from the FreeBSD ports:
cd /usr/ports/devel/thrift
./bootstrap.sh
./configure --with-boost=/usr/local
make all
make install
./bootstrap.sh
./configure --with-boost=/usr/local
make all
make install
7. Interfacing with Cassandra
Creating a record in Cassandra using Thrift:
/* Insert some data into the Standard1 column family from the default config */
// Keyspace specified in storage=conf.xml
$keyspace = 'Keyspace1';
// reference to specific User id
$keyUserId = "1";
// Constructing the column path that we are adding information into.
$columnPath = new cassandra_ColumnPath();
$columnPath->column_family = 'Standard1';
$columnPath->super_column = null;
$columnPath->column = 'email';
// Timestamp for update
$timestamp = time();
// Add the value to be written to the table, User Key, and path.
$value = "foobar@example.com";
$client->insert($keyspace, $keyUserId, $columnPath, $value, $timestamp, $consistency_level);
// Keyspace specified in storage=conf.xml
$keyspace = 'Keyspace1';
// reference to specific User id
$keyUserId = "1";
// Constructing the column path that we are adding information into.
$columnPath = new cassandra_ColumnPath();
$columnPath->column_family = 'Standard1';
$columnPath->super_column = null;
$columnPath->column = 'email';
// Timestamp for update
$timestamp = time();
// Add the value to be written to the table, User Key, and path.
$value = "foobar@example.com";
$client->insert($keyspace, $keyUserId, $columnPath, $value, $timestamp, $consistency_level);
Check out our Cassandra PHP Wrapper for an easier way to interface with Cassandra from PHP, or refer to the Thrift Examples.
View 11 Comment(s)
|
|
Lucene vs MySQL ShowdownMike Peters, April 1 -- Posted under Programming |
MySQL is great as a storage-engine. It is one of the cornerstones that fueled the growth of Web 2.0, enabling tens of thousands of startups to get a robust database running overnight, without breaking the bank.
MySQL just has a lot of quirks making it a poor choice when it comes to production sub-second queries on big tables.
Index optimization, database denormalization, replication and sharding are great techniques to squeeze more juice out of MySQL...
But eventually, as your tables and queries grow, you're going to hit a brick-wall.
Storing huge amounts of data with MySQL is easy. But when it comes time to Retrieve those records, using filters, sorts and joins, you'll be lucky if you can ever scale beyond 1 million records while still maintaining high front-end speeds.
Thankfully, these days there are plenty of tools you can use to gradually replace components of your RDBMS world, with ones that are simply put, better tools for the job.
We are now in the process of converting the entire SPI platform, switching all queries, filters and sorting to Lucene, letting MySQL do nothing but primary-key lookups.
The impact on performance has been phenomenal!
In the video below, I demonstrate the effect of switching filters and sorting from MySQL to Lucene:
Next on the agenda, in our never-ending quest for performance and scalability:
* Lucene to Solr
* ActiveMQ replacing our home-grown queue
* Hadoop for processing
* Cassandra for logs and stats
MySQL just has a lot of quirks making it a poor choice when it comes to production sub-second queries on big tables.
Index optimization, database denormalization, replication and sharding are great techniques to squeeze more juice out of MySQL...
But eventually, as your tables and queries grow, you're going to hit a brick-wall.
Storing huge amounts of data with MySQL is easy. But when it comes time to Retrieve those records, using filters, sorts and joins, you'll be lucky if you can ever scale beyond 1 million records while still maintaining high front-end speeds.
Thankfully, these days there are plenty of tools you can use to gradually replace components of your RDBMS world, with ones that are simply put, better tools for the job.
We are now in the process of converting the entire SPI platform, switching all queries, filters and sorting to Lucene, letting MySQL do nothing but primary-key lookups.
The impact on performance has been phenomenal!
In the video below, I demonstrate the effect of switching filters and sorting from MySQL to Lucene:
Next on the agenda, in our never-ending quest for performance and scalability:
* Lucene to Solr
* ActiveMQ replacing our home-grown queue
* Hadoop for processing
* Cassandra for logs and stats
|
Magnetic Sponsoring giving away an Audi R8Kate Richards, March 30 -- Posted under Traffic |
When it comes to Product Launches, we've seen our share of crazy affiliate giveaways over the years...
From vacation packages, gadgets, show tickets, waterboats, to 27" iMacs with the recent ShoeMoney System product launch SPI orchestrated two months ago.
But this one trumps them all.
Our good friends at Magnetic Sponsoring, launched "What's Working Now" yesterday, giving away an Audi R8 as the giveaway prize for their #1 affiliate!
Check out this video:
We're honored to be working with Mike Dillard, Matt Crystal, TJ and the rest of the gang at Magnetic. Pay close attention to what they're doing.
For more information about becoming a "What's Working Now" affiliate go to http://www.getmagneticsponsoring.com
You can learn all about the "What's Working Now" product at http://www.getwwn.com
From vacation packages, gadgets, show tickets, waterboats, to 27" iMacs with the recent ShoeMoney System product launch SPI orchestrated two months ago.
But this one trumps them all.
Our good friends at Magnetic Sponsoring, launched "What's Working Now" yesterday, giving away an Audi R8 as the giveaway prize for their #1 affiliate!
Check out this video:
We're honored to be working with Mike Dillard, Matt Crystal, TJ and the rest of the gang at Magnetic. Pay close attention to what they're doing.
For more information about becoming a "What's Working Now" affiliate go to http://www.getmagneticsponsoring.com
You can learn all about the "What's Working Now" product at http://www.getwwn.com
|
|
How to fix BugsMike Peters, March 29 -- Posted under Programming |
A few weeks ago I gave a talk at Fort Lauderdale to a group of Computer Science graduates about fixing bugs.
Fixing bugs is an art many engineers still struggle with on a day-to-day basis.
In my presentation I explain the 4 prerequisites you need before you can fix any bug and share our "binary sorting the code" approach to bug fixing.
Fixing bugs is an art many engineers still struggle with on a day-to-day basis.
In my presentation I explain the 4 prerequisites you need before you can fix any bug and share our "binary sorting the code" approach to bug fixing.
| « Previous Posts | » Next Posts |