SoftwareProjects - Internet Marketing and Web Development

MSN Messenger Integration - Java Robot script

Brett Batie — Thu, 02 Sep 2010 21:44:22 GMT

I recently worked on a project that involved integrating with MSN Messenger, creating my very own "IM Robot".

What can an IM Robot do?

Keep track of contacts online/offline status and when they were last seen.

Broadcast a message to all contacts.

Automatically answer common questions.

Notify contacts about new events. (See http://notify.me)

Keep track of code snippets

Which IM Library to Use

Setting up a IM robot can be a bit of work especially if starting from scratch. There are a lot of libraries out there that can be used to help simplify the process. The trouble is a lot of libraries are not kept up to date and fail to work as IM protocols change.

We did some digging and found a library that would provide a good foundation to build a IM Robot that can do just about anything. We say implementations in PHP, C, Java, Perl and Python. After some testing we concluded the Java MSN Library would be a very good fit.

How To Use It

Using this library with java is pretty straight forward. First the library must be added to the classpath. The step to take to complete this will depend on how your developing your java code. The most basic method to add a library to your classpath is to do this at run time with a command such as:

Code:

java -classpath MyLibrary.jar MyPackage.MyClass

A better approach would be to setup the classpath in a manifest file. The manifest file is then placed inside the jar file and tells the executable jar where to look for the libraries. This manifest file should look something like this (note the class-path on line 5):

Code:

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.7.1
Created-By: 14.3-b01 (Sun Microsystems Inc.)
Main-Class: imstatus.Main
Class-Path: lib/jml-1.0b4-full.jar lib/httpcore-4.0.1.jar lib/mysql-co
 nnector-java-5.1.6-bin.jar
X-COMMENT: Main-Class will be added automatically by build

This is setup so that the 3 required libraries are in the lib folder. These 3 libraries are needed for setting up an IM robot and can be downloaded from the following locations:

jml-1.0b4-full.jar

httpcore-4.0.1.jar

mysql-connector-java-5.1.6-bin.jar

Now that the libraries are setup we can begin to use them.

Developing the IM Robot Code

There are a few examples of using the Java MSN Library on the main page. However, they are a tad confusing as it creates a new BasicMessenger class. This is confusing as the library already has a BasicMessenger class which is abstract. The library also has a SimpleMessenger class which is a subclass of BasicMessenger. This class appears to be the correct implementation that we would want to use to create a new IM Robot.

However, the original authors made the constructor protected so that we cannot instantiate the class outside of the original package. Since we want a simple way to create an IM Robot I have modified the original source code to have a public constructor for the SimpleMessenger class.

With this new package we can very easily create a new IM Robot with the following two lines of code (make sure to replace yourLogin and yourPassword):

Code:

SimpleMessenger messenger = new SimpleMessenger(Email.parseStr("yourLogin@msn.com"), "yourPassword");
messenger.login();

With that code in our main funtion we can run it and test that the Robot automatically logs into Windows Live Messenger.

Of course that code just logs the Robot into Windows Live Messenger. The next step is to setup the robot to do something interesting. This is one feature that is very nice about the Java Msn Library as it as listeners for many different events. For example we can detect when the robot has finished logging in with the following:

Code:

messenger.addListener(new MsnAdapter() {
        // Setup the login completed event

        @Override
        public void loginCompleted(MsnMessenger messenger) {
                MsnOwner owner = messenger.getOwner();
                owner.setInitStatus(MsnUserStatus.ONLINE);
                owner.setStatus(MsnUserStatus.ONLINE);

                // Setup the contact list event
                messenger.addContactListListener(new ContactListAdapter());
        }
});

Then we can take this a step further and detect when a status changes for one of the robots contacts with something like the following:

Code:

messenger.addListener(new MsnAdapter() {
        // Setup the login completed event

        @Override
        public void loginCompleted(MsnMessenger messenger) {
                MsnOwner owner = messenger.getOwner();
                owner.setInitStatus(MsnUserStatus.ONLINE);
                owner.setStatus(MsnUserStatus.ONLINE);

                // Setup the contact list event
                messenger.addContactListListener(new ContactListAdapter());
        }
});

The above code will detect when the robot has finished logging in and then setup a new listener to detect when a contacts status has changed. The new listener invokes the ContactListAdapter class when a status has changed. This contactListAdapter class is setup as followes:

Code:

class ContactListAdapter extends MsnContactListAdapter {
        @Override
        public void contactStatusChanged(MsnMessenger messenger, MsnContact contact) {
                System.out.println(contact.getEmail()+" is currently "+contact.getStatus());
                // Can add code here to store the status in a database
        }
}

We can still take this a step further and setup the robot to handle automatically adding contacts when a contact requests it. This logic can be added to the ContactListAdapter class with something like the following:

Code:

class ContactListAdapter extends MsnContactListAdapter {
        @Override
        public void contactListSyncCompleted(MsnMessenger messenger) {
                        MsnContact[] contacts = messenger.getContactList().getContactsInList(MsnList.AL);
                        for (int i = 0; i < contacts.length; i++) {
                                contactStatusChanged(messenger,contacts[i]);
                        }
        }

        @Override
        public void contactAddedMe(MsnMessenger messenger, MsnContact contact) {
                messenger.addFriend(contact.getEmail(), contact.getDisplayName());
        }

        @Override
        public void contactAddedMe(MsnMessenger messenger, MsnContactPending[] pending){
                for(int i=0; i                        messenger.addFriend(pending[i].getEmail(), pending[i].getDisplayName());
                }
        }

        @Override
        public void contactStatusChanged(MsnMessenger messenger, MsnContact contact) {
                System.out.println(contact.getEmail()+" is currently "+contact.getStatus());
                // Can add code here to store the status in a database
        }
}

There you have it! Put all of the above code together and you will have a robot that knows how to automatically add contacts and keep track of when a contact's status changes.

Kajabi vs SoftwareProjects

Adrian Singer — Wed, 01 Sep 2010 14:16:00 GMT

There's a new kid on the block.

A new system designed to "simplify the deployment of Marketing Campaigns, Sales Funnels, and Product Launches from their current technical complexity down to Drag-and-Drop Ease" (Source: Kajabi.com)

Kajabi was created by Andy Jenkins (VideoBoss, formerly with Stompernet), Kenny Rueter and Travis Rosser. With this roster, you can bet your Inbox is about to be flooded with emails about how Kajabi is the best thing since sliced bread.

With this post, I wanted to address a question by one of our clients - how does Kajabi compare with SoftwareProjects' own Shopping Cart solution.

Disclaimer: Kajabi is still under development. The information I gathered for this post was collected by analyzing two recent product launches powered by Kajabi, along with a series of emails from the Kajabi team.

First things first: Kajabi is NOT a Shopping Cart

Kajabi is not designed to be an end-to-end shopping cart system.

They will integrate with all popular shopping cart systems, but it's just not designed to be a CRM / Affiliate & Lead Manager / Fulfillment / Backoffice, like SoftwareProjects or Infusionsoft.

You can expect limited built-in support for selling your content, but you're not going to get an executive dashboard that lets you manage the full 360 of your Internet Business.

Membership System

Kajabi offers a built-in membership system, where paying members login to access the content (videos / tutorials) they paid for.

You can customize the membership system skin to some degree and manage members from the admin panel. Think aMember plus a CMS (Content Management System).

At this point my understanding is there's going to be a single backoffice (unlike SPI's Customer's Backoffice and Affiliate's Backoffice), which will support a single product per Kajabai deployment.

Content Delivery

This is one area that keeps being emphasized in all communications I hear about Kajabi.

Still not clear how this is different from hosting all videos and MultiMedia on Amazon S3 or EdgeCast, like everyone are doing.

I'm guessing they use the same underlying technology and just make it easier to use.

Autoresponder

Kajabi will not have a built-in Autoresponder.

It is said to integrate with popular Autoresponder services such as AWeber, SoftwareProjects, iContact etc.

Help desk

Kajabi will not have a built-in help desk system.

Hosting

Kajabi is said to offer "Cloud Computing Technology, (so that) you can publish anything you create with Kajabi instantly to the web - with theoretically infinitely scalable virtual hardware resources".

They seem to be using Rackspace Cloud hosting, to create virtual instances of your site as needed, to accommodate traffic demands.

This sounds very similar to SoftwareProjects' Multi-Homed hosting, where we do what no other company has ever done before and guarantee 100% uptime, no matter what.

Battle Tested

Kajabi successfully powered the product launches of Andy Jenkins' Video Boss and Frank Kern's List Control.

While still under development, the technology has proven its ability to handle launch-day traffic without too many glitches.

Here at SoftwareProjects, we have been powering the majority of the big product launches you hear about, including Magnetic Sponsoring WWN, the ShoeMoney System, ListBuilding, PPC Classroom, Affiliate Classroom, OnlineMLMSecrets and others.

Then again... we've been doing this since 1998.

In Summary

We really look forward to working with the Kajabi team on the integration side.

Where Kajabi shines is simplifying the process of getting an info-marketing product up in no time.

I would describe it as an InfoMarketing CMS with a membership backoffice. It may not have all the bells and whistles, but as long as the integration is straightforward, you get the best of both worlds.

-

More on this topic:
* Kajabi's home page
* Kajabi Beta Testing portal
* Follow Kajabi on Twitter
* Andy Jenkins blog

Troubleshooting Cassandra

Mike Peters — Tue, 31 Aug 2010 17:38:50 GMT

Keynotes from a great presentation titled Cassandra Troubleshooting: out of the shadows, presented by Benjamin Black at the Cassandra Summit in San Francisco two weeks ago.

The slides are here

-

Is your Ring unbalanced?

That's because when you add one node at a time using RandomPartitioner, the new nodes takes over half of the most balanced node:

Code:

32
16 16
8 8 16
8 8 8 8
4 4 8 8 8
4 4 4 4 8 8

Note that as long as you're doubling-up the size of your cluster, everything will be balanced. But when you're growing one node at
a time, the cluster will be unbalanced.

To fix: Manually assign tokens.

How do you know which tokens to assign? Use this Python script:

Code:

def tokens(nodes)
 0.upto(nodes - 1) do {n}
   p (n * (2**127 - 1) / nodes)
 end
end

Writes are slow

Make sure your commitlog is on a separate drive.

Writes are fast. Reads keep getting slower

Step 1:

Look at iostat -x to see if you're maxing out utilization

If you are, get more nodes

Step 2:

Look at nodetool tpstats

Focus on the middle column (pending) and specifically:
* Row-Read-STage
* Message-Deserializer-pool

If these two are high (4096 is the max), it means your client is sending too many reads to this node.

Update your client or get more nodes to distribute reads.

Step 3:

Adjust memtable settings

When does a memtable get flushed to disk?

Size: When it gets to a certain size
Time: If it hasn't been flushed in x seconds
Operations: When certain operations occur

If you're flushing memtables too often, you're triggering follow-up effects (compactions, sstable merges) that is consuming a lot of bandwidth.

You want less frequent memtable flush, which leads to less frequent compaction and less disk bandwidth demand.

If memtable is not compatible with your data needs, you begin consuming huge amounts of your bandwidth on compactions.

once a minute = bad

Step 4:

Use SSDs for the disk drives. Makes no difference on the commit log drive.

I inserted a bunch of data, now my nodes are flapping

Flapping = nodes are marked down/up

Step 1:

Monitor swap (vmstat on linux, swapinfo on freebsd)

mmap takes 2gb per segment.

Swapping can delay gossip long enough to cause a node
to be marked down.

Swapping is bad.

To fix: Change DiskAccessMode in the Cassandra config file, to mmap_index_only

We avoid risking driving ourselves into swap by the JVM
allocating large chunks of mmap blocks.

Step 2:

Tell the O/S you want to avoid swapping if possible.
On FreeBSD: add this line to /etc/sysctl.conf

Code:

vm.swap_enabled=0

On Linux, echo 0 into /proc/sys/vm/swappiness

Scaling in the real world

Mike Peters — Tue, 31 Aug 2010 12:55:58 GMT

In our ongoing journey to deliver faster, more reliable shopping carts, email marketing and hosting solutions, we're always on the lookout for new technologies and techniques to deal with a lot of data.

I'd like to share a few presentations we recently came across, that talk about scaling in the real world:

* MySpace: Massive scaling a .Net website with the Microsoft Robotic Studio
* Twitter: Big Data in Real-Time at Twitter
* Scale at Facebook
* Yahoo: Selectively Materializing User's Event Feeds
* Four Kitchens: Scalable queuing with flexible service levels
* Cassandra at Mahalo
* Cassandra at CloudKick

PHP Thrift library for Cassandra

Mike Peters — Fri, 06 Aug 2010 15:50:20 GMT

If you're using Cassandra over PHP, you are probably aware of several bugs in the PHP Thrift interface that get in the way.

Courtesy of Jonathan Ellis, here's a list of known patches you need to apply to your PHP Thrift library:

* https://issues.apache.org/jira/browse/THRIFT-347 (TSocket Timeout)
* https://issues.apache.org/jira/browse/THRIFT-638 (Block until recv timeout)
* https://issues.apache.org/jira/browse/THRIFT-780 (Prevent aborts)
* https://issues.apache.org/jira/browse/THRIFT-788 (Multiget more than 17 keys)

To make your life easier, you can download a patched version of Thrift 0.2 for PHP, with all patches applied, here.

-

A great presentation by Cliff Moon: Dealing with the Thrift API

Cassandra PHP Wrapper 0.7

Mike Peters — Thu, 29 Jul 2010 12:32:19 GMT

A new version of the SoftwareProjects Cassandra PHP Wrapper (db_cassandra.php) is now available, for Cassandra 0.7.

This version of the PHP Wrapper adds support for:
* Adding Keyspaces and Column families on the fly
* Better error logging
* Disconnect on destructor to prevent java fd leaks
* Reconnect on timeout, up to 10 times
* Deletes
* Updated all API calls to 0.7

You can download the Cassandra 0.7 PHP wrapper, modify and distribute it freely.

Please let us know if you find any issues.

Usage Examples

Adding a new Keyspace with one column family

// Initialize Cassandra
$cassandra = new CassandraDB("mytest_keyspace");

// Debug on
$cassandra->SetDisplayErrors(true);

// Prepare Column Family
// (You can add as many as you need to the cfdef array)
$cfdef = null;
$record = array();
$record['keyspace'] = "mytest_keyspace";
$record['name'] = "mytable";
$record['column_type'] = "Standard";
$record['comparator_type'] = "BytesType";
$record['row_cache_size'] = 0;
$record['key_cache_size'] = 0.01;
$cfdef[] = new cassandra_CfDef($record);

// Add keyspace with one columnfamily
$result = $cassandra->AddKeyspace($cfdef);

Adding records and reading back values

// Initialize Cassandra
$cassandra = new CassandraDB("mytest_keyspace");

// Debug on
$cassandra->SetDisplayErrors(true);

// Set our key
$key = "Mike at softwareprojects.com";

// Insert record ("Columns" in Cassandra)
$record = array();
$record["name"] = "Mike Peters";
$record["email"] = "mike at softwareprojects.com";
if ($cassandra->InsertRecord('mytable', $key, $record))
{
echo "Record (Columns) inserted successfully.\r\n";
}

// Print record
$record = $cassandra->GetRecordByKey('mytable', $key);
print_r($record);

Setup tips

The PHP Wrapper was tested with the 0.7 trunk version, June 30, 2010 build.

Make sure you patch your Thrift TSocket as described here (comments section).

-

Further Reading

If you're just getting started with Cassandra, you'll enjoy the selection of hand picked Cassandra tutorials listed below. These are the best of the best:

= Tutorials
* Start here: WTF is a SuperColumn
* Cassandra Basics - Indexing
* Introduction to Cassandra
* History of Cassandra (Video)
* The shift to NoSQL

= Case Studies
* Scaling Twitter with Cassandra
* Cassandra at Digg
* Cassandra at RackSpace

= Internals
* How Reads work in Cassandra
* How Writes work in Cassandra
* Cassandra Replication and Consistency
* Random Partitioner vs Order Preserving Partitioner (OPP)
* Cassandra 101
* Do you really need SQL to do it all in Cassandra
* Let's play with Cassandra

Mercurial Hook: PHP Syntax Check

Mike Peters — Wed, 16 Jun 2010 04:40:18 GMT

In a previous post, we wrote about PHP real time syntax checking. A method to send email alerts when any critical scripts on your live servers fail to pass PHP syntax validation.

The problem with this approach is that it alerts you of a problem after-the-fact.

Someone uploads a bad PHP script which breaks a critical module, you are alerted, but by that time users are already affected.

Today we'll take it this concept one step further by writing a hook for the Mercurial version control system, preventing developers from checking-in scripts that don't pass syntax-checking.

Installing PHP Syntax Check Mercurial hook

Step 1: The shell script

Save this shell script under your Mercurial folder, calling it: php_syntax.sh

Code:

#!/usr/local/bin/bash
echo "STARTING PHP SYNTAX CHECK..."
# create a random temp file
temp_file=`/usr/bin/mktemp -t php_syntax_files`

# get all modified files and remove duplicate's
#note: use file_mods,file_adds instead
hg log -r $HG_NODE:tip --template "{files}\n" | sort | uniq > $temp_file

# Walk through each line
#for line in "$temp_file"; do
for line in $(< $temp_file); do
    # Make sure it is a php file
    if [ `echo $line |  grep -E "\.(php)|(php4)|(php5)$"` ]
    then
        # create a random temp file
        php_file=`/usr/bin/mktemp -t php_syntax_check`

        # save the contents of this file (latest commit) to the temp file
        hg cat -r tip $line > $php_file

        # check the syntax
        php_syntax_output=`/usr/local/bin/php-cgi -l -d display_errors=1 -d error_reporting=4 -d html_errors=0 < $php_file`;

        # remove the temp file
        rm -f $php_file;

        test_syntax=`echo $php_syntax_output | grep "Parse error"`
        if [ "$test_syntax" ];then
            exit 1;
        fi
    fi
done

rm -f "$temp_file"

Step 2: Add hook to Mercurial config file

Update your Mercurial hgweb.config, adding the new hook under the 'hooks' section, like this:

[hooks]
pretxnchangegroup.syntax_check = /usr/home/mercurial/php_syntax.sh

--

Replace '/usr/home/mercurial' with the path where you saved php_syntax.sh

Attempting to push changes to the repository that don't pass php-syntax check, are now blocked.

Users will be presented with an error message and the transaction rolled back, until the user fixes the problem and pushes changes again.

How to install APC on Linux / RedHat

Mike Peters — Mon, 14 Jun 2010 18:32:53 GMT

APC (Alternative PHP Cache) is an opcode cache for PHP that pre-compiles commonly used PHP scripts to machine-language, speeding up execution.

We previously used eAccelarator which is another opcode cache for PHP. We decided to switch from eAccelarator to APC due to stability issues and the fact that APC was chosen as the best opcode cache engine to be included with PHP6.

In this post, I'll cover the steps to install APC:

1. Type: 'pecl config-set preferred-state stable'
2. Type: 'pecl install apc'
3. When prompted type 'all' to change configuration
4. Type 'no' to change the option
5. Hit the enter key to run the installation script.
6. When complete, open /usr/local/lib/php.ini for editing
7. Comment out all eaccelerator lines if they exist
8. Add the following block:

Code:

 
extension=apc.so
apc.enabled="1"
apc.shm_segments="1"
apc.shm_size="100"
apc.ttl="5"
apc.user_ttl="5"
apc.gc_ttl="5"
apc.file_update_protection="2"
apc.enable_cli="1"
apc.max_file_size="1M"
apc.write-lock="1"
apc.report_autofilter="0"
apc.include_once_override="0"

Additional settings and explanations for each can be found at: http://www.php.net/manual/en/apc.configuration.php

9. Save the file and exit the editor

10. Change the system setting for shared memory as follows:
a. edit /etc/sysctl.conf
b. add this line:

Code:

kern.ipc.shmmax=134217728

c. the run the following command:

Code:

sysctl -w kern.ipc.shmmax=134217728

11. Restart FastCGI manager with:

Code:

/usr/local/bin/php-fpm restart

Temporary File Names in C

Mike Peters — Thu, 27 May 2010 10:24:03 GMT

There's a lot of confusion among developers about the best way to generate a unique temporary file name in C.

If you're using C, the most suitable function that comes to mind is tmpnam():

Code:

char *tmpnam(char *str);

// Usage
printf ("My temporary file name is: %s\r\n", tmpnam("/usr/tmp"));

While it seems like a great fit, you should never ever use tmpnam.

I'll repeat it again - avoid using tmpnam() at all cost.

The reason is - tmpnam() suffers from a race condition:

Since the temporary file never gets created, if you have multiple threads/processes calling tmpnam() at the same time, it is very possible for two instances to end up with the same exact temporary file name... And the consequences can be fatal.

The tmpnam() function should be deprecated. That's probably why it was never ported over to PHP.

In PHP you should use tempnam() or tmpfile(), both of which create the temporary file before returning the name, so you are guaranteed no two instances will ever end up with the same temporary file name.

Here's the correct way to get a temporary file name in C:

Code:

    char sTempfile[] = "/usr/tmp/mytmpfileXXXXXX"; // The X's are important
    int tmp_handle;

    if ( (tmp_handle=mkstemp(sTempfile)) < 1)
    {
        return 0;
    }            
    close(tmp_handle);

    // We now have the temporary filename in sTempfile
    printf ("My temporary file name is: %s\r\n", sTempfile);

MySQL InnoDB Deadlocks and Duplicate key errors (1213, 1205, 1062)

Mike Peters — Thu, 06 May 2010 03:41:28 GMT

InnoDB is one of the best MySQL storage engines when you're looking for concurrent writes, transactions support and ACID reliability.

Switching from MyISAM to InnoDB is very straightforward (alter table MYTABLE engine=innodb), but there are a few pitfalls you should be prepared for.

Review the full list of InnoDB restrictions, understand them and make sure nothing collides with your existing design... Watch out for SELECT COUNT(*) on InnoDB - they require full table scans.

As part of this post I'd like to touch on three key issues with InnoDB that many developers fail to properly deal with:

#1062 Duplicate entry for key 'PRIMARY' error

This is bug 26316 that has been around for quite some time and is still alive and kicking as of version 5.1.43 of MySQL that we're using.

Not sure if it is really linked to triggers or not. Out of nowhere, MySQL will throw a 1062 duplicate key error on auto_increment keys. Something which by design is impossible.

To fix, you have to retry the transaction and it will go through on the second attempt.

#1213 Deadlock found when trying to get lock error

InnoDB locks rows and starts transactions internally as needed.

From time to time, particularly when concurrent threads are hitting the same rows, you're going to experience a deadlock.

Deadlocks happen when two transactions wait on each other to acquire a lock. For example:

Tx 1: lock A, then B
Tx 2: lock B, then A

Because InnoDB starts transactions on the internally, you -are- going to experience deadlocks.

No way of escaping it.

Fortunately when deadlocks do happen with this error 1213, all you have to do is retry the query until it goes through.

#1205 Lock wait timeout exceeded error

Similar to error 1213, this one will occur whether or not you are manually starting a transaction.

This error is more complicated to deal with -

If the transaction was triggered by InnoDB (atomic to the query you're running), you can simply retry the query and it will eventually go through.

If however this error shows up inside a transaction you started, it means any uncommitted inserts/updates are lost and will have to be resent in a new transaction block.

--

Below are two simple PHP MySQL wrapper functions you can use to gracefully handle the three issues I described, while logging all errors to a file.

Thickbox - ColorBox - FancyBox - ShadowBox Comparison

Brett Batie — Wed, 28 Apr 2010 19:21:22 GMT

There are a large number of pop-up/dialog box libraries that are now available. Many of the pop-up dialog options out there support the same features. However, in our testing we found that some failed terribly.

As part of this post, I'd like to cover a quick comparison of what I found in hopes that it will help others in their search for the ideal pop-up library.

The popup controls I tested:

jQueryUI Dialog

DOM Window

Thickbox

fancybox

Colorbox

Shadowbox.js

At a first glance all of the above looked like they would work well. However we did find some major flaws with the above libraries. Here is what we found:

jQueryUI Dialog - This is by far our favorite pop-up dialog. It supports features such as modal, dragging it to other locations on the screen and allows the user to resize it. The features that jQuery offers makes it a top choice. However, the major flaw is it makes IE6 hang for about 10 seconds when the page is loading. That one flaw forced us to look at other options.

Dom window - For us this package is too basic. It does support the basic needs of a pop-up but when compared to some of the other libraries it was not nearly as good.

Thickbox - This library had all the features needed. We have used it in the past and have found it very easy to work with. The major pitfall with this library is it is no longer supported. The library works correctly today but there is no guarantee it will continue to work into the future and therefore is not a great choice.

FancyBox - This library has a good set of features unfortunately it loads very slowly in IE6 and can cause the browser to crash.

ColorBox - This library works very well, supports a good amount of features and does load correctly in all browsers including IE6. It would be nice if this library had all of the features of the jQueryUI Dialog and some may consider it a flaw that jQuery is required in order to use this library.

ShadowBox - This library works well in all of the major browsers and supports most features that are necessary for a nice pop-up. It does not have all of the features of jQueryUI Dialog. This library includes everything needed in one javascript and css file.

Based on my research I found the two best libraries to be colorbox and shadowbox.

These libraries offer very similar features and they both work well with all of the major browsers. For us ColorBox was the right choice because it has is under an MIT license, the size of the library is smaller than ShadowBox and it supports auto-resize-to-fit-content which ShadowBox didn't have.

RabbitMQ Message Queue

Mike Peters — Fri, 16 Apr 2010 22:13:02 GMT

RabbitMQ is an open source messaging server, allowing you to scale by queuing jobs to be processed offline.

If you're looking to scale your service, integrating message-queues into your application is a MUST. Without offline processing, you'll quickly hit a glass ceiling on maximum throughput.

In the world of message-queues, there are lots of different solutions. From the over-simplistic PHP Dropr, through RabbitMQ, ActiveMQ, Fuse, ZeroMQ and commercial enterprise queue systems.

There's a great post by the makers of SecondLife, sharing their evaluation of 9 popular message-queue systems here. Take the time to review available message-queues and test which one best suits your needs.

Here at SPI, our original implementation of distributed message-queues was MySQL based. Two master-master databases, sharing a queue table. While it did work for our needs, databases suck for message queues and we needed a better solution.

We had a short run with ActiveMQ -- another popular message-queue solution, but were not happy with the test results. RabbitMQ is a lot faster (x40 times), supports AMQP and is better suited for distributed clusters.

Let's cover the steps to install RabbitMQ on your server.

1. Getting started

Download the latest version of RabbitMQ from this page.

Use the Bin version. No need to recompile.

Code:

unsetenv JAVA_HOME
mkdir /usr/tmp
fetch "http://www.rabbitmq.com/releases/rabbitmq-server/v1.7.2/rabbitmq-server-generic-unix-1.7.2.tar.gz"
tar xvfz rabbitmq-server-generic-unix-1.7.2.tar.gz

2. Download JRE

Download and install the version of JRE matching your system (32bit or 64bit) from the Java SE Download page.

3. Install Erlang

If you're on FreeBSD, simply do:

Code:

cd /usr/ports/lang/erlang
make all
make install

Otherwise, Download Erlang.

4. Start RabbitMQ

Changedir to the path where you installed RabbitMQ and start it with:

Code:

cd bin
./rabbitmq-server  -detached

Simple right? You now have the RabbitMQ up and running (in the background), ready to relay messages.

It's important to understand the terminology and building blocks (Queues, Exchanges, Bindings) of RabbitMQ.

Take the time to review the Getting Started Slides and FAQ.

5. Communicating with RabbitMQ

Now that we have RabbitMQ up and running we can start the fun part of writing and reading messages.

For the purpose of this guide, we'll use PHP to communicate with RabbitMQ.

There are several PHP-wrappers for RabbitMQ. Our favorite one is the clean and simple PHP amqp lib.

The code below uses php-amqplib to connect and send messages to a RabbitMQ server.

The publisher (writes messages to queue):

require_once('amqp.inc');

$HOST = 'localhost';
$PORT = 5672;
$USER = 'guest';
$PASS = 'guest';
$VHOST = '/';
$EXCHANGE = 'router';
$QUEUE = 'msgs';

$conn = new AMQPConnection($HOST, $PORT, $USER, $PASS);
$ch = $conn->channel();
$ch->access_request($VHOST, false, false, true, true);

$msg_body = implode(' ', array_slice($argv, 1));
$msg = new AMQPMessage($msg_body, array('content_type' => 'text/plain'));
$ch->basic_publish($msg, $EXCHANGE);

$ch->close();
$conn->close();

The consumer (reads messages from queue):

require_once('amqp.inc');

$HOST = 'localhost';
$PORT = 5672;
$USER = 'guest';
$PASS = 'guest';
$VHOST = '/';
$EXCHANGE = 'router';
$QUEUE = 'msgs';
$CONSUMER_TAG = 'consumer';

$conn = new AMQPConnection($HOST, $PORT, $USER, $PASS);
$ch = $conn->channel();
$ch->access_request($VHOST, false, false, true, true);

$ch->queue_declare($QUEUE);
$ch->exchange_declare($EXCHANGE, 'direct', false, false, false);
$ch->queue_bind($QUEUE, $EXCHANGE);

function process_message($msg) {
global $ch, $CONSUMER_TAG;

echo "\n--------\n";
echo $msg->body;
echo "\n--------\n";

$ch->basic_ack($msg->delivery_info['delivery_tag']);

// Cancel callback
if ($msg->body === 'quit') {
$ch->basic_cancel($CONSUMER_TAG);
}
}

$ch->basic_consume($QUEUE, $CONSUMER_TAG, false, false, false, false, 'process_message');

// Loop as long as the channel has callbacks registered
while(count($ch->callbacks)) {
$ch->wait();
}

$ch->close();
$conn->close();

--

Further reading:
* Why Databases suck for messaging
* Introduction to Message Queues and how RabbitMQ works
* SecondLife (the game) evaluation of 9 message-queue solutions
* RabbitMQ Administration
* RabbitMQ Clustering

Realtime PHP Syntax Checking

Mike Peters — Wed, 14 Apr 2010 17:08:35 GMT

If you love Agile Development and PHP as much as we do, you're probably going to have developers editing critical PHP scripts on live machines.

While many PHP editors include built-in syntax checking, it doesn't hurt to have a fail-safe in place. We wanted to come up with a way to apply php -l SCRIPT syntax checking on any PHP script as soon as it is updated.

At first the idea was to write a plugin for Mercurial - our version control system, so that whenever a changeset is pushed, the syntax will be checked.

Unfortunately this doesn't cover changes done via SSH or files uploaded via FTP by clients.

We came up with a shell script, executed by cron every minute, that iterates through all files changed within the last 3 days, applying "php -l" on those files and sending an email to the appropriate developer with the information about the error.

Regardless of how files are updated, we now have an automated way to syntax-check every single change to a PHP script.

The emails generated by the script follow this format:

Code:

Oops! Found a FATAL syntax error in a production PHP script...

DIRECTORY: /magnetic/

---------------------------------------------
FILE: ./affiliate/offline_credit_commission.php
---------------------------------------------
Parse error: syntax error, unexpected '<' in /offline_credit_commission.php on line 8
changeset:   2673:fe54a1e4dbe0
user:        Doug 
date:        Wed Apr 14 00:29:26 2010 +0300
summary:     Change offline credit affiliate commission

CheckSyntax cronjob:

Code:

#!/bin/sh

# Change to source directory directory
cd $(dirname $0)

# Generate report
./CheckSyntax.sh >log.$$
a=$?

# Email report
# TODO: Update these with your settings
if [ $a = 1 ]
then
    ./sendEmail -f CheckSyntax@mycompany.com \
                -t developers@mycompany.com  \
                -u "Check Syntax" \
                -s mail.mymailserver.com \
                -m < log.$$
fi

# Remove temporary file
rm -rf log.$$

# (eof)

CheckSyntax.sh:

Code:

#!/usr/local/bin/bash

##################################
# Config
##################################
#

# TODO: Update these with your settings
php="/usr/local/bin/php"
hg="/usr/local/bin/hg"
def="/magnetic" 

#
##################################

# Directory passed in parameters?
if [ "$1" != "" ]
then
    # Get the directory from parameters
    dir=$1
else
    # No parameter, use default
    dir=$def
fi

# Change to the directory we need to check
cd $dir

# Prepare tmp file
tmp=/tmp/CheckSyntax.$$

# Header
echo "Oops! Found a FATAL syntax error in a production PHP script..."
echo ""
echo "DIRECTORY: $dir"

# Find PHP files
#   Note: '-newerct' only workds under FreeBSD
find . -newerct '3 days ago' ! -path "*/temp/*" | grep "\.php$" >$tmp

# Loop files
cnt=0
err=0
tot=0
while read i
do
    # If file isn't a file, skip
    [ ! -f "$i" ] && continue;


    ((tot++))

    # Syntax check
    a=$($php -l $i | grep "^No syntax errors detected in")

    # Is it OK?
    if [ "$a" = "" ]
    then
        # Not OK, display details
        echo ""
        echo "----------------------------------------------------------------------------------------------"
        echo "FILE: $i"
        echo "----------------------------------------------------------------------------------------------"
        a=$($php -d display_errors=1 -l $i | grep "Parse error")
        echo $a | sed "s/<[a-zA-Z\/][^>]*>//g"
        $hg log $i | head -n 5 | grep -v "^$"
        ((cnt++))
        err=1
    fi
done < $tmp

echo ""
echo "--"
echo ""
echo "$cnt BAD FILES"
echo "$tot TOTAL FILES"
echo ""

# Remove tmp file
rm -rf $tmp

# 0 - no error | 1 - error
exit $err

# (eof)

-

To install, save the two files (CheckSyntax and CheckSyntax.sh), chmod a+rx, update the email settings in CheckSyntax and add this line to your cronjob:

Code:

# CheckSyntax
# TODO: Update path with the location of CheckSyntax
*/1 * * * * /home/CheckSyntax

How to create a PHP Daemon

Mike Peters — Wed, 14 Apr 2010 16:15:19 GMT

A daemon is a Linux program that runs in the background.

Most daemons are written in C. While C is faster and more robust than PHP, looking at development time and cost, PHP generally scores a lot better than C.

Converting your C daemons to PHP where applicable, makes the code more accessible to a wider percentage of your development force and allows better re-use and connecting of existing code.

To Facebook, keeping all code accessible to all developers was so important, that they've decided to come up with a way to compile PHP to C in real time. So you get PHP speed of development with C speed. The HipHop toolkit is available on an open source license.

The simple PHP4 functions included here, will let you easily create PHP daemons.

Features:
* Daemon will not start if another instance is already running
* Daemon PID saved to /var/run/
* Daemon log messages saved to /var/log/
* Run in foreground mode with the "-f" flag

Sample daemon:

require_once("common_daemon.php");

// Get parameters from command line
foreach ($argv as $parameter)
{
$run_in_foreground = Strcasecmp($parameter,"-f")==0;
}

// Start our daemon
if (!PHPDaemonStart("mydaemon", $run_in_foreground))
{
// Already running
return;
}

// Do something useful
while (1)
{
PHPDaemonLog("Hello, the time is ".date("h:i:s", time()));
sleep(1);
}

// Never here

The common_daemon.php functions:

// Give us eternity to execute the script.
ini_set("max_execution_time", "0");
ini_set("max_input_time", "0");
set_time_limit(0);

function PHPDaemonStart($name, $run_in_foreground=0)
{
// This must be global so that the handle is kept alive
// after this function exits
global $handle_lockfile;

// Remember the daemon name
global $phpdaemon_name;
$phpdaemon_name = $name;

// Remember if we're running in foreground mode
global $phpdaemon_run_in_foreground;
$phpdaemon_run_in_foreground = $run_in_foreground;

// Open PID file
$tmpfilename = "/var/run/$name.pid";
if (!($handle_lockfile = @fopen($tmpfilename,"a+")))
{
// Script already running - abort
return 0;
}

// Obtain an exlcusive lock on file
// (If script is running this will fail)
if (!@flock( $handle_lockfile, LOCK_EX | LOCK_NB, &$wouldblock) || $wouldblock)
{
// Script already running - abort
@fclose($handle_lockfile);
return 0;
}

// Write our PID
@ftruncate($handle_lockfile,0);
@fseek($handle_lockfile, 0, 0);
@fwrite($handle_lockfile, getmypid());
@fflush($handle_lockfile);

// If we need to run in foreground, we're done
if ($run_in_foreground)
{
return 1;
}

// If we're up to here we need to fork our process
$pid = pcntl_fork();
if ($pid==-1)
{
// Can't fork
return 0;
}

// If we are the parent
if ($pid)
{
// Sleep for 2 seconds, letting the forked process wake up
// and attempt to grab our lock
sleep(2);

// Release the lock
@flock($handle_lockfile, LOCK_UN);

// Close file
fclose($handle_lockfile);

// Kill parent process
// By now the child has the lock
die;
}
// (Else - we are the child)
else
{
// Open file
while (!($handle_lockfile = @fopen($tmpfilename, "a+")))
{
// Try again
usleep(100);
}

// Re-establish lock on file
while (!@flock($handle_lockfile, LOCK_EX, &$wouldblock))
{
// Try again
usleep(100);
}

// Write pid into file
@ftruncate($handle_lockfile,0);
@fseek($handle_lockfile, 0, 0);
@fwrite($handle_lockfile, getmypid());
@fflush($handle_lockfile);
}

// We have a lock - all good
return 1;
}

function PHPDaemonLog($str)
{
// This is the daemon name
global $phpdaemon_name;

// This indicates whether or not we are running in foreground mode
global $phpdaemon_run_in_foreground ;

// Open log file
$handle = @fopen("/var/log/$phpdaemon_name.log", "a");

// Obtain an exclusive lock (so that this is thread safe)
flock($handle, LOCK_EX, &$wouldblock);

// Write
$output = date("Y-m-d h:i:s", time())." $str\r\n";
@fwrite($handle, $output);

// If we are running in foreground mode, output to screen as well
if ($phpdaemon_run_in_foreground) echo $output;

// Release lock
flock($handle, LOCK_UN);

// Close file
fclose($handle);
}

function IsPHPDaemonRunning($name)
{
// Open PID file
$tmpfilename = "/var/run/$name.pid";
if (!($tmpfile = @fopen($tmpfilename,"r")))
{
// Script not running
return 0;
}

// Obtain an exlcusive lock on file
// (If script is running this will fail)
$not_running = flock( $tmpfile, LOCK_EX | LOCK_NB, &$wouldblock);

// Release lock if successful
if ($not_running) flock($tmpfile, LOCK_UN);

// Close file
@fclose($tmpfile);

// Return result
return !$not_running || $wouldblock;
}

Cassandra PHP Wrapper

Mike Peters — Wed, 07 Apr 2010 14:40:29 GMT

Update: Check out the Cassandra PHP Wrapper 0.7

In a previous post, I explained how the Cassandra decentralized database can allow you to scale well beyond what's possible with MySQL.

The Cassandra Data Model and API are very different than traditional RDBMS way-of-thinking. That's why a lot of developers switching from MySQL to Cassandra are finding it difficult to grasp at first.

To simplify the migration from MySQL to Cassandra, we created a high-level PHP wrapper for Cassandra, using function prototypes and variable names that are close in meaning to RDBMS.

While there are several other high level PHP wrappers for Cassandra, none of the existing ones answered our requirements -

* Simple
* Don't throw exceptions, return error codes
* As close as possible to the low level Thrift
* Make it easy for RDBMS developers to adopt

This is how the CassandraDB class was born.

Adding records to the database is as simple as:

// Initialize Cassandra
$cassandra = new CassandraDB("SPI");

// Debug on
$cassandra->SetDisplayErrors(true);

// Insert record ("Columns" in Cassandra)
$record = array();
$record["name"] = "Mike Peters";
$record["email"] = "mike at softwareprojects.com";
if ($cassandra->InsertRecord('mytable', "Mike Peters", $record))
{
echo "Record (Columns) inserted successfully.\r\n";
}

// Print record
$record = $cassandra->GetRecordByKey('mytable', "Mike Peters");
print_r($record);

Output is

Code:

Array
(
    [email] => mike at softwareprojects.com
    [name] => Mike Peters
)

Adding record arrays:

// Initialize Cassandra
$cassandra = new CassandraDB("SPI");

// Debug on
$cassandra->SetDisplayErrors(true);

// Insert record array ("SuperColumns" in Cassandra)
$record = array();
$record["Mike Peters"] = array("name" => "Mike Peters", "email" => "Mike at Peters");
$record["Jonathan Ellis"] = array("name" => "Jonathan Ellis", "email" => "Jonathan at Ellis");
if ($cassandra->InsertRecordArray('my_super_table', "People", $record))
{
echo "RecordArray (SuperColumns) inserted successfully.\r\n";
}

// Print record array
$record = $cassandra->GetRecordByKey('my_super_table', 'People');
print_r($record);

Output is

Code:

Array
(
    [Jonathan Ellis] => Array
        (
            [email] => Jonathan at Ellis
            [name] => Jonathan Ellis
        )

    [Mike Peters] => Array
        (
            [email] => Mike at Peters
            [name] => Mike Peters
        )

)

--

Here's the complete Cassandra PHP Wrapper class. Enjoy!

// CassandraDB version 0.1
// Software Projects Inc
// http://www.softwareprojects.com
//

// Includes
$GLOBALS['THRIFT_ROOT'] = '/services/thrift/';
require_once $GLOBALS['THRIFT_ROOT'].'/packages/cassandra/Cassandra.php';
require_once $GLOBALS['THRIFT_ROOT'].'/packages/cassandra/cassandra_types.php';
require_once $GLOBALS['THRIFT_ROOT'].'/transport/TSocket.php';
require_once $GLOBALS['THRIFT_ROOT'].'/protocol/TBinaryProtocol.php';
require_once $GLOBALS['THRIFT_ROOT'].'/transport/TFramedTransport.php';
require_once $GLOBALS['THRIFT_ROOT'].'/transport/TBufferedTransport.php';

class CassandraDB
{
// Internal variables
protected $socket;
protected $client;
protected $keyspace;
protected $transport;
protected $protocol;
protected $err_str = "";
protected $display_errors = 0;
protected $consistency = 1;
protected $parse_columns = 1;

// Functions

// Constructor - Connect to Cassandra via Thrift
function CassandraDB ($keyspace, $host = "127.0.0.1", $port = 9160)
{
// Initialize
$this->err_str = '';

try
{
// Store passed 'keyspace' in object
$this->keyspace = $keyspace;

// Make a connection to the Thrift interface to Cassandra
$this->socket = new TSocket($host, $port);
$this->transport = new TFramedTransport($this->socket, 1024, 1024);
$this->protocol = new TBinaryProtocolAccelerated($this->transport);
$this->client = new CassandraClient($this->protocol);
$this->transport->open();
}
catch (TException $tx)
{
// Error occured
$this->err_str = $tx->why;
$this->Debug($tx->why." ".$tx->getMessage());
}
}

// Insert Column into ColumnFamily
// (Equivalent to RDBMS Insert record to a table)
function InsertRecord ($table /* ColumnFamily */, $key /* ColumnFamily Key */, $record /* Columns */)
{
// Initialize
$this->err_str = '';

try
{
// Timestamp for update
$timestamp = time();

// Build batch mutation
$cfmap = array();
$cfmap[$table] = $this->array_to_supercolumns_or_columns($record, $timestamp);

// Insert
$this->client->batch_insert($this->keyspace, $key, $cfmap, $this->consistency);

// If we're up to here, all is well
$result = 1;
}
catch (TException $tx)
{
// Error occured
$result = 0;
$this->err_str = $tx->why;
$this->Debug($tx->why." ".$tx->getMessage());
}

// Return result
return $result;
}

// Insert SuperColumn into SuperColumnFamily
// (Equivalent to RDMBS Insert record to a "nested table")
function InsertRecordArray ($table /* SuperColumnFamily */, $key_parent /* Super CF */,
$record /* Columns */)
{
// Initialize
$err_str = '';

try
{
// Timestamp for update
$timestamp = time();

// Build batch mutation
$cfmap = array();
$cfmap[$table] = $this->array_to_supercolumns_or_columns($record, $timestamp);

// Insert
$this->client->batch_insert($this->keyspace, $key_parent, $cfmap, $this->consistency);

// If we're up to here, all is well
$result = 1;
}
catch (TException $tx)
{
// Error occured
$result = 0;
$this->err_str = $tx->why;
$this->Debug($tx->why." ".$tx->getMessage());
}

// Return result
return $result;
}

// Get record by key
function GetRecordByKey ($table /* ColumnFamily or SuperColumnFamily */, $key, $start_from="", $end_at="")
{
// Initialize
$err_str = '';

try
{
return $this->get($table, $key, NULL, $start_from, $end_at);
}
catch (TException $tx)
{
// Error occured
$this->err_str = $tx->why;
$this->Debug($tx->why." ".$tx->getMessage());
return array();
}
}

// Print debug message
function Debug ($str)
{
// If verbose is off, we're done
if (!$this->display_errors) return;

// Print
echo date("Y-m-d h:i:s")." CassandraDB ERROR: $str\r\n";
}

// Turn verbose debug on/off (Default is off)
function SetDisplayErrors($flag)
{
$this->display_errors = $flag;
}

// Set Consistency level (Default is 1)
function SetConsistency ($consistency)
{
$this->consistency = $consistency;
}

// Build cf array
function array_to_supercolumns_or_columns($array, $timestamp=null)
{
if(empty($timestamp)) $timestamp = time();

$ret = null;
foreach($array as $name => $value) {
$c_or_sc = new cassandra_ColumnOrSuperColumn();
if(is_array($value)) {
$c_or_sc->super_column = new cassandra_SuperColumn();
$c_or_sc->super_column->name = $this->unparse_column_name($name, true);
$c_or_sc->super_column->columns = $this->array_to_columns($value, $timestamp);
$c_or_sc->super_column->timestamp = $timestamp;
}
else
{
$c_or_sc = new cassandra_ColumnOrSuperColumn();
$c_or_sc->column = new cassandra_Column();
$c_or_sc->column->name = $this->unparse_column_name($name, true);
$c_or_sc->column->value = $value;
$c_or_sc->column->timestamp = $timestamp;
}
$ret[] = $c_or_sc;
}

return $ret;
}

// Parse column names for Cassandra
function parse_column_name($column_name, $is_column=true)
{
if(!$column_name) return NULL;

return $column_name;
}

// Unparse column names for Cassandra
function unparse_column_name($column_name, $is_column=true)
{
if(!$column_name) return NULL;

return $column_name;
}

// Convert supercolumns or columns into an array
function supercolumns_or_columns_to_array($array)
{
$ret = null;
for ($i=0; $i<count($array); $i++)
foreach ($array[$i] as $object)
{
if ($object)
{
// If supercolumn
if (isset($object->columns))
{
$record = array();
for ($j=0; $j<count($object->columns); $j++)
{
$column = $object->columns[$j];
$record[$column->name] = $column->value;
}
$ret[$object->name] = $record;
}
// (Otherwise - not supercolumn)
else
{
$ret[$object->name] = $object->value;
}
}
}

return $ret;
}

// Get record from Cassandra
function get($table, $key, $super_column=NULL, $slice_start="", $slice_finish="")
{
try
{
$column_parent = new cassandra_ColumnParent();
$column_parent->column_family = $table;
$column_parent->super_column = $this->unparse_column_name($super_column, false);

$slice_range = new cassandra_SliceRange();
$slice_range->start = $slice_start;
$slice_range->finish = $slice_finish;
$predicate = new cassandra_SlicePredicate();
$predicate->slice_range = $slice_range;

$resp = $this->client->get_slice($this->keyspace, $key, $column_parent, $predicate, $this->consistency);

return $this->supercolumns_or_columns_to_array($resp);
}
catch (TException $tx)
{
$this->Debug($tx->why." ".$tx->getMessage());
return array();
}
}

// Convert array to columns
function array_to_columns($array, $timestamp=null) {
if(empty($timestamp)) $timestamp = time();

$ret = null;
foreach($array as $name => $value) {
$column = new cassandra_Column();
$column->name = $this->unparse_column_name($name, false);
$column->value = $value;
$column->timestamp = $timestamp;

$ret[] = $column;
}
return $ret;
}

// Get error string
function ErrorStr()
{
return $this->err_str;
}
}

CassandraDB Todo:

* Ability to add multiple nodes and try to connect to next-in-line automatically if connection fails
* Implement the remaining get_ functions
* Implement delete
* Implement update vs insert-fail-on-duplicate-key?
* Implement Order By desc

-

Recommended Further Reading:

* Cassandra in Production at Digg
* WTF is a SuperColumn?
* Cassandra Reads - How do they work
* Cassandra Writes - How do they work
* Cassandra Write Properties (Video)
* Engineering notes by Facebook
* RandomPartitioner vs OrderPreservingPartitioner