SoftwareProjects - Internet Marketing and Web Development

How to count unique visitors in an nginx access.log

Mike Peters — Thu, 05 Nov 2009 01:25:06 GMT

This nifty one-liner will count the number of unique visitors in an NGinx web-server access.log file, for a given day:

Code:

grep "\[04/Nov/2009" access.log | cut -d" " -f1 | sort | uniq | wc -l

Replace 04/Nov/2009 with the date you'd like to count the number of unique visitors for.

FreeBSD No Ports Fix

Mike Peters — Wed, 04 Nov 2009 23:52:34 GMT

When installing a FreeBSD system, you're asked if you would like to install the Ports Collection. If you chose no, you can follow these instructions to obtain the ports collection:

Option 1 - CVSup

Code:

csup -L 2 -h cvsup.FreeBSD.org /usr/share/examples/cvsup/ports-supfile
csup

Option 2 - Portsnap

Code:

portsnap fetch
portsnap extract

How to: Add a Lead Opt-in Web Form

Adrian Singer — Wed, 28 Oct 2009 09:21:38 GMT

A lead opt-in web form is a short form, asking the user to provide name, emailaddress, phone and/or other optional fields, for the purpose of signing up for a program, or receiving more information about your offering.

Unlike other shopping carts, the SPI cart doesn't limit you to a single form style or any specific fields.

You can create whatever html form you like, using any design and combination of fields (text, radio button, checkboxes, textareas) as needed.

The only requirement is to have at least the emailaddress field, whenever saving a new lead to the database.

Step 1 - Design your opt-in form

Use your favorite HTML editor to design your lead opt-in form.

For simplicity, use 'name' as the fieldname for the user's name (or 'first_name' + 'last_name'), use 'emailaddress' as the fieldname for the user's email address and use 'phone' as the fieldname for the user's phone number.

Here's a typical lead opt-in form with the basic name,emailaddress,phone fields as well as a few additional custom ones:

<form action="#" method="post">
<p>First Name: <input type="text" name="first_name" />p>
<p>Last Name: <input type="text" name="last_name" />p>
<p>Best Email: <input type="text" name="email" />p>
<p>Best Phone Number: <input type="text" name="phone" />p>
<p>Have you traded individual stocks?
Yes <input name="traded_indi_stock" type="radio" value="Yes" /> No <input name="traded_indi_stock" type="radio" value="No" />
p>
<p>Have you traded options
Yes <input name="traded_option" type="radio" value="Yes" /> No <input name="traded_option" type="radio" value="No" />
p>
<input type="submit" name="submit" value="Submit" />
form>

Step 2 - Add custom fields to Leads table

Since we're using a few custom fields in this example (traded_indi_stock, traded_option), we need to add those fields to the 'leads' table.

Connect to your SPI database using PHPMyAdmin, locate the leads table and add the two new fields. You only have to do this once every time you're adding a new custom field to a table.

Make sure you're adding the custom fields to the 'leads' table and pay close attention to the fieldnames. The fieldnames you choose are going to have to match the fieldnames you pass to the do_addlead() function.

Step 3 - Save the lead to the database

Now that we have the custom leads in the 'leads' table, saving the lead to the database is the easy part.

The code below renders the opt-in form, saves the lead to the database and redirects the user to a thankyou.html page.

php
// Include SPICart shopping cart API
require_once("spicart.php");

// If this is a lead submit
if (!empty($_POST['email']))
{
// Save lead to the database
$input = array();
$input['first_name'] = $_POST['first_name'];
$input['last_name'] = $_POST['last_name'];
$input['emailaddress'] = $_POST['emailaddres'];
$input['phone'] = $_POST['phone'];
$input['traded_indi_stock'] = $_POST['traded_indi_stock'];
$input['traded_option'] = $_POST['traded_option'];
do_addlead($input, &$output, &$result_str);
Header("Location: thankyou.html");
die;
}
?>

Note about Affiliate Tracking

Often times, affiliates will be sending their traffic to your lead opt-in forms. You're going to want to know which affiliate generated which leads.

Luckily the system takes care of this automatically.

One of the fields in the 'leads' table is aff_id. This is the ID of the referring affiliate and serves as the glue, associating leads with the affiliates who referred them.

When a user clicks on an affiliate link to visit the opt-in form on your website, they typically follow a link that looks like this:

www.YourDomain.com/?aff_id=1234

Where 1234 is the affiliate ID.

As soon as the user lands on the lead opt-in form, the spicart.php include at the top of your file, identifies the aff_id url-parameter and saves it in a cookie on the end-user's machine.

Later when you call do_addlead(), this method picks up the aff_id from the end-user's cookie and passes it along, so that when the lead is added, the referring aff_id is saved as well.

Do It Yourself Web Server and Database monitoring

Mike Peters — Wed, 21 Oct 2009 06:44:12 GMT

When you have production web servers and databases, it is absolutely critical to have a monitoring system in-place that will alert you whenever service goes down.

Getting Started with website monitoring

One of our favorite monitoring tools is Pingdom. It's a $9.95/month easy-to-use service that will ping 5 of your servers, once every 60 seconds, alerting you via email & SMS whenever any server goes down.

In addition to downtime notifications, the service also provides response-time trends, raw data and uptime-over-time reports:

If you don't have any monitoring in-place right one, you should definitely try out Pingdom or a similar service.

In about 5 minutes of setting it up, you'll have the peace of mind, knowing exactly when your merchant website, affiliate landing page or payment gateway goes down.

Beyond simple HTTP monitoring

While Pingdom and other similar tools are great in letting you know once your website is -already- down, a lot of times the outage can be prevented in the first place, by detecting a decrease available memory, cpu, disk space etc.

At the pinnacle of server monitoring are tools such as Nagios and Munin.

Munin will monitor every single resource on your server, including: memory, cpu utilization, available handles, open connections, average load, running processes and more.

Nagios sends alerts to groups of users based on predetermined rules.

Do It Yourself Server Monitoring

In this post, I'd like to focus on how to create a simple monitoring script, that will survey remote servers for a few critical metrics, displaying the results on the screen.

Unlike Nagios+Munin, such a setup doesn't require installing a monitoring component on each server. Your monitoring script can run remotely, surveying all servers once every 60 seconds on a cronjob, sending an email/sms whenever detecting abnormal behavior.

For the purpose of this example, our script will monitor the following metrics:

= MySQL Database server
* Number of running threads
* Number of threads connected
* Queries per second
* Open tables
* Sample query response
* Number of tables in database

= Any FreeBSD/Linux server
* CPU load over the last 5 minutes
* Available memory
* Open sockets
* Number of running processes

// Set these for easier access
$USERNAME = "root";
$SERVER = "www.mydomain.com";
$DBHOST = "www.mydatabase.com";
$DBUSER = "root";
$DBPASS = "dbpassword";
$DBNAME = "dbdatabasename";

// Get load average, available memory and number of running processes
$load_average = "N/A";
$available_memory = "N/A";
$running_processes = "N/A";
$CRLF = "\\r\"\n";
exec("ssh -l$USERNAME $SERVER \"top -b -n 1\"",&$buf);
for ($i=0,$str=""; $i<count($buf); $i++) $str .= $buf[$i].$CRLF;
if (($pos=strpos($str,"load averages:"))!==false)
{
$str = trim(substr($str, $pos+strlen("load averages:")));
if (($pos=strpos($str,$CRLF))!==false)
{
$load_average = substr($str,0,$pos);
if (($pos=strpos($str,"up"))!==false) $load_average = substr($load_average,0,$pos);
}
}
else
if (($pos=strpos($str,"load average:"))!==false)
{
$str = trim(substr($str, $pos+strlen("load average:")));
if (($pos=strpos($str,$CRLF))!==false)
{
$load_average = substr($str,0,$pos);
}
}
if (($pos=strpos($str,"processes:"))!==false)
{
$str = trim(substr($str, $pos+strlen("processes:")));
if (($pos=strpos($str,","))!==false)
{
$running_processes = substr($str,0,$pos);
}
}
else
if (($pos=strpos($str,"Tasks:"))!==false)
{
$str = trim(substr($str, $pos+strlen("Tasks:")));
if (($pos=strpos($str,","))!==false)
$str = trim(substr($str, $pos+1));
if (($pos=strpos($str,"run"))!==false)
{
$running_processes = substr($str,0,$pos);
}
}
if (($pos=strpos($str,"Mem:"))!==false)
{
$str = trim(substr($str, $pos+strlen("Mem:")));
if (($pos=strpos($str,$CRLF))!==false)
{
$available_memory = substr($str, 0, $pos);
}
}

// Get open sockets
$open_sockets = "N/A";
$buf = array();
exec("ssh -l$USERNAME $SERVER "netstat -n | wc -l"",&$buf);
$open_sockets = trim($buf[0]);

// Display server metrics
echo "== Server\r\n";
echo " Load average: $load_average\r\n";
echo " Open Sockets: $open_sockets\r\n";
echo " Running Processes: $running_processes\r\n";
echo " Available memory: $available_memory\r\n";
echo "\\r\"\n";

// Get running and connected threads
$db_threads_connected = "N/A";
$db_running_threads = "N/A";
mysql_connect($DBHOST,$DBUSER,$DBPASS);
$Result = @mysql_query("show status like '%thread%' ");
$cnt = @mysql_num_rows($Result);
while ($cnt)
{
$cnt--;
if (!($Row = @mysql_fetch_array($Result))) continue;

if (empty($Row['Value'])) continue;

if (Strcasecmp($Row['Variable_name'],'Threads_connected')==0)
$db_threads_connected = $Row['Value'];
else
if (Strcasecmp($Row['Variable_name'],'Threads_running')==0)
$db_running_threads = $Row['Value'];
}

// Get queries per second
$Result = @mysql_query("show status like 'questions' ");
$Row = @mysql_fetch_array($Result);
$questions = $Row['Value']+0;
$Result = @mysql_query("show status like 'uptime' ");
$Row = @mysql_fetch_array($Result);
$uptime = $Row['Value']+0;
$db_queries_per_second = number_format($questions/$uptime,0);

// Get open tables
$db_open_tables = "N/A";
$Result = @mysql_query("show status like '%tables%' ");
$cnt = @mysql_num_rows($Result);
while ($cnt)
{
$cnt--;
if (!($Row = @mysql_fetch_array($Result))) continue;

if (empty($Row['Value'])) continue;
if (Strcasecmp($Row['Variable_name'],'Open_tables')==0)
$db_open_tables = $Row['Value'];
}

// Get query response
$Result = @mysql_query("SELECT 1");
$Row = @mysql_fetch_row($Result);
$db_query_response = $Row[0]=='1' ? "Good" : "BAD";

// Get total number of tables
@mysql_query("use $DBNAME");
$Result = @mysql_query("show tables");
$db_tables_in_database = @mysql_num_rows($Result);

// Display database metrics
echo "== Database\r\n";
echo " Running Threads: $db_running_threads\r\n";
echo " Threads Connected: $db_threads_connected\r\n";
echo " Queries Per Second: $db_queries_per_second\r\n";
echo " Open Tables: $db_open_tables\r\n";
echo " Query Response: $db_query_response\r\n";
echo " Number of tables: $db_tables_in_database\r\n";

The output of the script will look something like this:

Code:

== Server
        Load average: 0.09,  0.23,  0.23  
        Open Sockets: 226
   Running Processes: 1 running
    Available memory: 2203M Active, 112M Buf, 4668K Free

== Database
     Running Threads: 3
   Threads Connected: 16
  Queries Per Second: 320
         Open Tables: 200
      Query Response: Good
    Number of tables: 1648

Tip: If you're looking to monitor dozens of servers, consider applying our ssh no password approach, to save the hassle of having to exchange private keys with each server.

Product Launch Marketing: Quickest path to a 6 figure monthly cashflow

Kate Richards — Wed, 21 Oct 2009 04:49:28 GMT

A Product Launch is the process of building anticipation and emphasizing scarcity in a hungry market, towards the release of a new product / service, positioned to have the potential to make a significant impact on the target customer's life.

In the Internet Marketing world, Product Launches are powerful marketing vehicles used by Information-Marketers to drive a massive amount of traffic on a specific day ("Launch day"), with the goal of converting the wave of anonymous traffic into qualified leads and then paying customers.

For the past 5 years, SPI has been instrumental in orchestrating some of the largest most-successful product launches online, working with leading information marketers on applying time-tested techniques to generate hundreds of millions of dollars in product revenues.

Some of the recent product launches we handled include ListBuilding, GoogleCashDetective, 7FigurePPCSecrets, LaunchTree, AffiliateClassroom, PPCClassroom, DirectResponseMarketing.

Why does it work

Product Launch Marketing works so well because they build on two fundamental human traits:

#1. Herd Mentality - Follow the leaders
In the weeks building to the launch, opinion leaders and peers within your industry, will all glorify how great this product is going to be and why you absolutely must have it.

#2. Scarcity - Desire to want what we can't have
Using a limited number of units, fixed close date, early-bird discounts, one-time-offers and other tactics, the company will build the impression you must act fast or lose the opportunity to own the product forever.

Examples of great well-planned product launches: The iPhone, "The Secret" DVD, StomperNet 3.0, Gary Vaynerchuk's CrushIt book.

In Numbers

A typical Internet Marketing Product Launch these days, yields $1m (One Million Dollars) on launch day, followed by $200k in monthly residual payments from a continuity program.

It is far easier to convince 2,000 prospects to pay $99 a month, than it is to direct-sell a few large accounts in order to generate the equivalent of $200k in monthly revenues.

1,000 users paying $99 a month = $100k a month
2,000 users paying $50 a month = $100k a month
4,000 users paying $25 a month = $100k a month

50% of the revenues are often paid back to affiliates.
20% is the average three-months drop-out rate.

The SPI Product Launch Program

With the "Product Launch Program", we are now helping dozens of companies in various industries, capitalize on this phenomenal marketing strategy.

The SPI Product Launch Program, is not a how-to tutorial, or an online course.

This program is a joint-venture between you and SPI, whereby our company will work with you, hands-on, on every aspect of your product launch in a joint effort to make it a big success.

What you can expect from SPI, during this process:

* Hands-on consulting
* Introduction to leading information-marketing affiliates
* Access to case studies, analytics and proven tactics, used by leading information marketers
* Access to live front-end and back-office systems, so that you can learn what works best
* Professional services: Landing page design, Hosting, Affiliate system, Payment solutions
* PLR: Videos and mp3 files to complement your member's-only content
* Membership system integration (Wordpress+plugin / aMember / SPI)
* Load testing, to ensure your setup can handle launch-day traffic
* Affiliate emails autoresponder kit
* Corporate address, Call center, Help desk
* Ongoing Software development / website customizations / graphic design

Application Process

Our success over the years has made this service very popular. Committed to maintaining our strong reputation, SPI dedicates a lot of resources to every individual product launch we are involved with.

Unfortunately we cannot blindly accept new clients into this program.

Please contact your account manager (or email us here) for details on how to apply.

SSH no password without any private keys. It's magic!

Michel Nadeau — Thu, 08 Oct 2009 14:18:17 GMT

We all know that one can configure SSH to login automatically by adding the client's public key to the server's ~/.ssh/authorized_keys file. But what if you need to run commands on 200 machines and don't want to login to every single machine to add the key? Search no more, this tutorial has the answer!

1. Installing sshpass

sshpass is a tool for non-interactivly performing password authentication with SSH's so called "interactive keyboard password authentication".

Here's a standard SSH connect command:

Code:

debian_I:~# ssh -l root localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is b4:e9:e7:56:a2:b4:89:9b:d8:fd:7e:8e:f1:e4:1d:9f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
root@localhost's password:
Last login: Thu Oct  8 09:35:41 2009 from localhost
Linux debian_I 2.6.18-6-686 #1 SMP i686
debian_I:~#

First you have to answer "yes" to accept the host key and then to enter the password.

With sshpass, you are able to specify the password on the command line and skip this step. Here's how to install it:

Code:

mkdir -p /usr/local/src/
cd /usr/local/src/
wget http://downloads.sourceforge.net/project/sshpass/sshpass/1.04/sshpass-1.04.tar.gz?use_mirror=iweb
tar xvfz sshpass-1.04.tar.gz
cd sshpass-1.04
./configure
make
make install

sshpass is now ready to be used!

2. Using sshpass

The sshpass' syntax is:

Code:

sshpass -p [password] [ssh command]

So instead of doing:

Code:

ssh -l root localhost

You can simply do:

Code:

sshpass -p myrootpass ssh -l root localhost

You will be automatically logged in, without any password prompt:

Code:

debian_I:~# sshpass -p myrootpass ssh -l root localhost
Last login: Thu Oct  8 09:52:04 2009 from localhost
Linux debian_I 2.6.18-6-686 #1 SMP i686
debian_I:~#

3. Automatically accepting host keys

The last problem is this prompt:

Code:

The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is b4:e9:e7:56:a2:b4:89:9b:d8:fd:7e:8e:f1:e4:1d:9f.
Are you sure you want to continue connecting (yes/no)? yes

When you're using sshpass to connect on a single machine that you use often, it's not a big deal because you will get the prompt once and never again after. But if you want to connect to 200 machines, you definitely don't want to type "yes" 200 times.

To fix this issue, simply add this line in /etc/ssh/ssh_config on the CLIENT machine:

Code:

StrictHostKeyChecking=no

With this setting enabled, SSH will automatically accept the host keys and will not prompt you about it.

4. Automating the process

This section will show you how to easily automate the process of running commands on any number of machines you want.

First you will need a file containing the hostname, username and password for each of the servers you want to run commands on, in a CSV format. For example:

ssh_magic.csv:

Code:

someserver.com,root,123456
someotherserver.com,sshadmin,abcdef
onelastserver.com,root,123456

Then you will create this bash script:

ssh_magic.sh:

#!/bin/sh

# Loop ssh_magic.csv
for LINE in $(cat ssh_magic.csv)
do
# Split line
host=$(echo $LINE | cut -d "," -f1)
user=$(echo $LINE | cut -d "," -f2)
pass=$(echo $LINE | cut -d "," -f3)

# Display server info
echo ""
echo "HOSTNAME: $host"
echo ""

# Run commands
sshpass -p $pass ssh -l $user $host "uname -a"
sshpass -p $pass ssh -l $user $host "df -h"

echo ""
echo "===================================="
done

echo ""
echo "DONE"
echo ""

Simply place ssh_magic.csv and ssh_magic.sh together and run the script.

Conclusion

You now know how to run commands on any number of SSH machines, without any prompt! Make sure you check the other ways to pass the password to sshpass - they offer more secure ways than directly on the command line with the -p option.

We're Hiring! Looking for Technical Account Managers, Team Leaders

Mike Peters — Thu, 08 Oct 2009 02:30:13 GMT

Software Projects, a New York based full-service Internet Marketing firm, is seeking a bright, articulate, detail-oriented, and technical applicant interested in joining our group of software professionals.

As a Technical Account Manager, you will work directly with clients, translating client vision into technical requirements, submit weekly plans, manage teams of engineers and oversee all software development work.

You will brain-storm with engineers, think outside the box, provide direction and leadership, while directly interacting with some of the best minds in our industry.

== Requirements:
* Excellent problem solving skills
* Leader, Motivator, Confidence, Energetic
* Superb communication skills with the know-how to verbalize ideas into technical requirements
* Minimum 2 years programming experience (PHP / C / Perl)
* Experience with FreeBSD/Linux a great plus

== Responsibilities:
* Translate client wants into requirements
* Plan projects, oversee software development
* Solve Problems
* Update clients on progress
* Maintain high client satisfaction

== Location:
You will be working from your home office for the first six months. After that time, you will have the option of relocating to one of our offices or continuing to work from home.

== Next steps:
Send an email to hr@softwareprojects.com with your resume and let us know when is a good time to chat.

Didn't update your resume in the last 2 years? That's fine. It's all about your experience and skills! Send us a quick cover-letter and we'll follow-up.

How to install basic Squid on FreeBSD

Michel Nadeau — Fri, 02 Oct 2009 19:51:35 GMT

Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing frequently-requested web pages. Squid has extensive access controls and makes a great server accelerator. It runs on most available operating systems, including Windows and is licensed under the GNU GPL.

This tutorial describes how to install a very simple implementation of Squid on your network.

1. Getting started

The first thing you want to do is to download Squid. In this tutorial, we're going to work in the /usr/local/src directory.

Code:

$ mkdir -p /usr/local/src
$ cd /usr/local/src
$ wget http://www.squid-cache.org/Versions/v3/3.0/squid-3.0.STABLE19.tar.gz

Then you want to extract Squid and change to its directory:

Code:

$ tar xvfz squid-3.0.STABLE19.tar.gz
$ cd squid-3.0.STABLE19

We're now ready to compile and install Squid!

2. Compiling/installing Squid

Compiling and installing Squid is very easy:

Code:

$ ./configure --prefix=/usr/local/squid
$ make all
$ make install

If the configure command fails because you don't have Perl, you can simply install it like this:

Code:

$ pkg_add -r perl

Then re-run the configure command.

3. Configuring Squid

First of all, you need to add the "visible_hostname" setting in your Squid configuration file. The main configuration file is:

Code:

/usr/local/squid/etc/squid.conf

Open it with your favorite editor and find this block:

Code:

# TAG: visible_hostname

At the end of the block (before the next "TAG" block), insert a new line and put something like this:

Code:

visible_hostname freebsd

Replace "freebsd" with any hostname you want Squid to use - it can be a local hostname or a fully qualified domain name. Save the file.

Now you need to adjust some permissions before Squid can be initialized...

Code:

$ mkdir -p /usr/local/squid/var/logs/
$ chmod 777 /usr/local/squid/var/logs/

$ mkdir -p /usr/local/squid/var/cache/
$ chmod 777 /usr/local/squid/var/cache/

Squid can now be initialized. Use this command:

Code:

/usr/local/squid/sbin/squid -z

Squid is now ready to run!

4. Starting/stopping Squid

Starting Squid:

Code:

/usr/local/squid/sbin/squid

Stopping Squid:

Code:

kill -9 `cat /usr/local/squid/var/logs/squid.pid`

You can start Squid automatically at boot time by creating the /usr/local/etc/rc.d/squid.sh file with this content:

Code:

#!/bin/sh
/usr/local/squid/sbin/squid

You also need to allow execution of this file:

Code:

$ chmod 755 /usr/local/etc/rc.d/squid.sh

5. Using Squid with your browser

To use Squid with your browser, you simply need to set it as a proxy in your browser's settings. Squid is listening on TCP port 3128. So if your FreeBSD machine's IP is 1.2.3.4, you will configure your proxy to be 1.2.3.4, port 3128.

Once configured, all the traffic over the chosen protocols (usually your browser lets you choose for which protocols you want to use a proxy) will go through your Squid server.

6. "Overriding" Web sites addresses

The first place where Squid is looking when it comes to resolving URL's to IP addresses is in the /etc/hosts file. If you want to "override" Web sites addresses, simply add them in the /etc/hosts file along with the IP address where you want to redirect traffic.

For example, let's say that you want your Squid's users to be sent to 1.2.3.4 when they request www.google.com instead of to the real google.com. To do so, you will add this line in your /etc/hosts file:

Code:

1.2.3.4 google.com www.google.com

Replace "1.2.3.4" with the IP of the machine where you want to redirect traffic. This isn't really a "redirect" as the users will never see 1.2.3.4 in their browser. For them, it will look like just as if they were really on www.google.com, not on 1.2.3.4.

NOTE: you NEED to restart Squid when you make changes in /etc/hosts.

Conclusion

Squid is very simple to install and use. Though, it's also VERY powerful and flexible: it has literally thousands of options! This tutorial only covered the very basic use - refer to the Squid users guide if you want to customize squid.conf.

Resources

* Squid
* Configuration guide
* Configuration examples
* Users guide

How to convert mbox mailboxes to the maildir format

Adrian Singer — Sun, 20 Sep 2009 07:35:13 GMT

This weekend we converted all SPI mutlihomed hosting email mailboxes from the mbox format to maildir.

What is the difference between mbox and maildir?

Mbox is the traditional way of storing mail messages in the Unix world. In this format, a regular text file which serves as the mail user's mailbox file is created.

Pros: Format is universally supported, Appending a new mail into the mailbox file is fast, Searching text inside a single mailbox file is fast.

Cons: Has file locking problems, Has problems when used with network file systems, Format is prone to corruption.

Maildir is a new way of storing mail messages. In this format, a directory usually named Maildir is created for each mail user. Under this directory are three more directories named new, cur and tmp.

Pros: Locating, retrieving and deleting a specific mail is fast, Minimal to no file locking needed, Immune to mailbox corruption (assuming the hardware will not fail).

Cons: Some filesystems may not efficiently handle a large number of small files, Searching text, which requires all mail files to be opened is slow.

Why we chose Maildir

We've decided to upgrade to Maildir because its more reliable (entire mailbox never fully corrupts) and thanks to no locking it is noticeably much faster to access your mail over Maildir.

Ongoing mbox file locking issues were driving us nuts.

SPI is using the Dovecot mail server on all machines. Dovecot is one of the highest performing IMAP servers. Luckily it supports both mbox and maildir, so all we had to do was: Convert old email messages from mbox to maildir.

Mb2md

Mb2md is a Perl script that takes one or more Mbox format mailbox files in a directory and convert them to Maildir format mailboxes.

Replace USERNAME below with the unix username whose mailbox you are converting

Step 1: Set environment variables for Mb2md

Code:

setenv MAIL /var/mail/USERNAME

Step 2: Convert the inbox folder and create Maildir folder

Code:

sudo -u USERNAME mb2md -m

Step 3: Convert other folders the user created

Code:

sudo -u USERNAME mb2md -s /home/USERNAME/mail/ -R

Step 4: Copy IMAP subscriptions

Code:

cp /home/USERNAME/mail/.subscriptions /home/USERNAME/Maildir/subscriptions

Step 5: Tell sendmail to deliver new mail to the user's /Maildir

Code:

cd /home/USERNAME
echo '"|IFS='"' '"' && exec /usr/local/bin/procmail -f || exit 75 #YOUR EMAIL NAME"' >> .forward
chmod 644 .forward
echo "DEFAULT="'$'"HOME/Maildir/" >> .procmailrc
chmod 644 .procmailrc
chown USERNAME:USERNAME .forward
chown USERNAME:USERNAME .procmailrc

Step 6: Delete old mbox files

Code:

rm /var/mail/USERNAME
rm -fdr /home/USERNAME/mail

Testing & Troubleshooting

To test, use sendmail to send a test message to your user:

Code:

sendmail USEREMAIL
subject: Test, please ignore

This is a test
.

If all works well, your email message will be delivered to the user's Maildir.

If you see the mail appended to /var/mail/USERNAME (do a tail -f /var/mail/USERNAME), this means sendmail is failing to invoke .procmailrc and deliver the email message to Maildir.

This is usually due to the /home or /home/USERNAME folders not having the correct file permissions.

Run this command to troubleshoot:

Code:

sendmail -v -Am -d11 USERNAME < /dev/null

How to run FreeBSD on Windows using VMware Server

Michel Nadeau — Tue, 15 Sep 2009 15:31:25 GMT

In a recent post, we explained how to run FreeBSD on Windows using VirtualBox. Today we will show you how to achieve the same thing but using VMware Server. VirtualBox is a very nice application: it's fast, simple and lightweight; but VMware Server has some very interesting features to offer:

* It can run any number of virtual machines in the background. You don't need to keep any window opened and you can simply access your virtual machines using SSH if you wish.

* It supports any guest operating system, including 64-bit versions.

* All the administration is done via a great Web interface.

As VirtualBox, VMware Server is also 100% free. Another product that is competing VMware Server, and that is also free, is Microsoft VirtualServer. It has the same features but is harder to install, harder to use and is a lot more fragile. After months of testing, VMware Server proved to be the best on performance, stability and usability.

1. Download VMware Server

The first step is obviously to download VMware Server. Go there: http://www.vmware.com/products/server/ and click on the "Download" button on the left. You will need to register to download the software. Once you're registered and logged in, copy the "VMware Server 2 for Windows" serial number displayed in the "Licensing" section of the page and paste it somewhere - you will need it later. Then grab the "VMware Server 2" EXE image under the "Binaries" section. The file is around 500MB.

2. Install VMware Server

Installing VMware Server is very straightforward: basically, you simply run the setup file and hit next, next, next, next... without changing anything. There's no real options except which shortcuts you want to create.

At the end of the setup, you will be asked to enter your serial number. Because of the virtual network adapters VMware Server installs, you will also be required to reboot: you need to do it immediately if you want to go further in this tutorial.

3. Download FreeBSD

In this tutorial, we'll be installing FreeBSD 7.2 minimal. If you have a 64-bit CPU, download this one (amd64 release):

ftp://ftp.freebsd.org/pub/FreeBSD/re...md64-disc1.iso

If you have a 32-bit CPU, download this one (i386 release):

ftp://ftp.freebsd.org/pub/FreeBSD/re...i386-disc1.iso

The ISO file is around 600MB.

4. Create the FreeBSD Virtual Machine

Start the VMware Server Web Access using the Start Menu shortcut (called "Web Access") or use your favorite browser and go to this URL: http://127.0.0.1:8308/ui/

Log into the VMware Server Web Access using your Windows account.

The first thing to do is to add a Datastore to your VMware Server configuration. To do so, click on your machine name on the left panel and click on the "Add Datastore" link on the right panel. The Datastore is a folder where everything about virtual machines will get created and stored. Simply give it a name and specify the directory (example: name: VM, directory: d:\VM).

Click on the "Create Virtual Machine" on the right panel:

* Name the virtual machine "FreeBSD", select the Datastore you've just created and hit next.

* Choose "Other operating systems", choose "FreeBSD" (32 or 64-bit, depending on your CPU) in the combo box and hit next.

* Assign the amount of memory you want your virtual machine to have (usually the recommended amount is quite good, unless you need a high-performance virtual machine). If you have a multi-core CPU (or many CPU's), you can choose the number of CPU's you want your virtual machine to use. Hit next when you're satisfied with your settings.

* Select "Create a New Virtual Disk" and hit next.

* Choose the size you want your virtual disk to have and hit next.

* Select "Add a Network Adapter" and hit next.

* Leave the settings to "Bridged" and hit next.

* In the CD/DVD section, select "Use an ISO Image" and hit next.

* Move the FreeBSD 7.2 ISO image you've just downloaded somewhere under the directory you're using as your Datastore. Then select it and hit OK. Then hit next.

* Select "Don't Add a Floppy Drive"

* Select "Don't Add a USB Controller"

* Check "Power on your new virtual machine now" and Hit "Finish"

5. Installing FreeBSD

Now that your FreeBSD virtual machine is created, it's time to install FreeBSD! First of all, in the VMware Server Web Access, click on the FreeBSD virtual machine on the left panel. Then click on the "Console" tab on the right, and click anywhere in the black screen. A small console window will popup: that's your FreeBSD machine, just like if you were sitting in front of a computer running FreeBSD.

NOTE: the console window "steals the focus"; at any time, press CTRL+ALT to release focus from this window. And simply click into the console window to use it.

It should boot from the ISO image you've specified in the setup: it's now time to install FreeBSD:

* Choose "United States" and hit enter.

* Choose "Standard" and hit enter.

* In the Disk Geometry screen, hit "a", then "q".

* Choose "BootMgr" and hit enter.

* In the Partition screen, hit "a", then "q".

* In the "Choose Distributions" screen, select "Minimal" and hit space to check it. Then hit tab, then enter on the OK button.

* Choose CD/DVD and hit enter. The base system will install.

* Network:

Hit "yes" to configure network.
Choose the "em0" network interface.
Hit "yes" to configure using DHCP.
Enter an hostname and hit enter many times.
Choose "no" to use this machine as a gateway.
Hit "no" to enable inetd.
Hit "yes" to enable SSH.
Hit "no" to anonymous FTP.
Hit "no" to NFS server.
Hit "no" to NFS client.

* Last settings:

Hit "no" to customize console.
Hit "yes" to timezone and choose your timezone.
Hit "no" to PS/2.
Hit "no" to package collection.
Hit "no" to create a user.
Set the "root" password.
Hit "no" to visit the options menu.

At the main menu, hit tab to go to the "Exit Install" option, hit enter and then hit "yes". Close the console. In the VMware Server Web Access, select the FreeBSD Virtual Machine on the left panel and hit the red "stop" sign at the top. Then hit the green "play" sign and return in the console. Your FreeBSD system should boot normally.

6. Mounting a shared folder using Samba

It's very convenient to be able to easily and quickly share files between your Windows and FreeBSD machine.

First of all, share a folder on your Windows machine (ex: share). It can be any folder, just share it normally. Then choose a mount point for that folder on your FreeBSD (ex: /mnt/share). We're now going to configure your FreeBSD machine so the shared folder will be automatically mounted at boot time.

Here's the sample informations (replace with yours):

Windows machine: win
Windows IP addr: 192.168.0.100
Windows share: share
Windows user: administrator
Windows password: mypass
FreeBSD mount point: /mnt/share (do not forget to create the directory!)

In /etc/nsmb.conf, add:
NOTE: the caps are important

Code:

[WIN]
addr=192.168.0.100

[WIN:ADMINISTRATOR]
password=mypass

Then in /etc/fstab, add:

Code:

//administrator@win/share /mnt/share smbfs rw 0 0

NOTE: before rebooting FreeBSD to test, I recommend testing the mounting manually with a command like this:

Code:

mkdir -p /mnt/tmp
mount_smbfs //administrator@win/share /mnt/tmp

Finally, reboot your FreeBSD machine and then go check in your mount point (like /mnt/share) if it works! Try creating folders from FreeBSD and from Windows and verify if it's working correctly. If it's not, the problem is probably on the authentication side.

7. Starting the virtual machine automatically

One other great VMware Server feature is the ability to start virtual machines automatically when you boot Windows.

To configure this, in the VMware Server Web Access, click on your Windows machine name on the left panel, and then click on "Edit Virtual Machine Startup/Shutdown Settings". Select your FreeBSD virtual machine and click "Move Up" to put it in the "Any Order" category.

Conclusion

You now have a FreeBSD virtual machine, acting just like if it was a real server sitting somewhere in your house! Virtual machines are great for development and testing. Please let me know if you have any problems getting this tutorial working!

How to run FreeBSD on Windows using VirtualBox

Mike Peters — Sat, 05 Sep 2009 01:15:58 GMT

Step 1 - Download and install VirtualBox

VirtualBox is a general-purpose full virtualizer for x86 hardware. It allows running any flavor Unix system on a Windows box.

You can download the latest version of VirtualBox for Windows Hosts here

Step 2 - Download FreeBSD 6

Download the single-file dvd iso here FreeBSD 6.4 download repository

Download 7-zip and extract the 6.4-RELEASE-i386-dvd1.iso.gz file to a folder on your machine

Step 3 - Create a new FreeBSD VirtualBox machine

Create a new VirtualBox machine, selecting BSD as the operating system and FreeBSD as the version.

Click Next to accept all defaults, but when you get to the 'Virtual Disk Location and Size' screen, change the disk size to 50GB.

Double-click the new machine to start it.

Step 4 - Install FreeBSD on new Virtual Machine

You will be greeted with a 'Welcome to the first run' wizard.

A 'Select Installation Media' screen will then popup. Pick the 'Image File' option, click the icon and locate the folder on your machine where you previously saved the FreeBSD iso file.

Click next a few times, keeping all default options.

FreeBSD install will boot in the VirtualBox

Select OK for United States
Select Standard Installation
Click A for 'Use Entire Disk', followed by Q
Click Enter for BootMgr
Click A for 'Auto Defaults', followed by Q
Select option 4 (Developer) and Yes to install ports
Select 1 for Install from CD

You will see a message saying All file information saved successfully. FreeBSD install will then begin running.

Step 5 - Configuration

Select YES to configure Ethernet network devices and pick your Ethernet card on the next screen

Select YES for DHCP

Select NO for network gateway and NO for iNetD

Select YES to enable SSH login

Select NO for FTP and NFS server/client

Select NO for all remaining questions

Pick a password for your root user

From the VirtualBox menu, select Devices - then Unmount CDRom

Enter X to exit install and reboot the FreeBSD virtual machine

SoftLayer down for 2 hours due to a power outage

Kate Richards — Fri, 04 Sep 2009 13:04:31 GMT

Earlier tonight, at 1:30am EST, SoftLayer - one of the tier 1 data centers we use, experienced a power outage in their primary facility, rendering thousands of sites dead.

SPI engineers were alerted to the outage within seconds (we take monitoring very seriously). After establishing that the core problem was a total data-center failure, we verified load balancers are properly redirecting traffic to other locations.

The Power Outage lasted a total of 2 hours and 3 minutes according to our friends at Pingdom:

All SPI client sites are multi-homed, which means we host everything in two independent data centers: Rackspace, Softlayer, NTT Verio and iWeb.

SPI utilizes redundant global load balancing, to immediately redirect end users from west coast to east coast, when such global outages happen.

How it works

New end users hitting your mydomain.com are diverted to active servers.

Old users / scripts (which keep a cached version of your website ip address) continue getting routed to the old server until their local DNS cache is cleared.

What's Next

While the DNS ttl (time to live) is setup as 60 seconds, some browsers and ISPs don't honor that and can take up to 2 hours until the DNS entry is refreshed.

As a workaround for these type of catastrophes where active users have cached a malfunctioning server's ip address, we'll be configuring the load balancers to reply back with all available server ip addresses, letting an end-user's machine rotate ips internally.

We understand the importance of keeping your sites up at all times and although our current setup ensures 100% uptime for new users, we are determined to come up with a solution for users with a cached version of a malfunctioning server ip address.

Your SPI account manager will contact you with any specific steps you have to take, in order to enable this setup for your domains.

How to recover from MySQL replication Event too small corruption

Dawn Rossi — Fri, 04 Sep 2009 10:37:39 GMT

If you're using MySQL InnoDB tables with these two lines in your /etc/my.cnf, for the most part, your database should gracefully handle unexpected shutdowns without data corruption:

Code:

innodb_flush_log_at_trx_commit=1
sync_binlog=1

I was working on such a database today, that experienced a power outage.

While all InnoDB data remained intact (I verified the checksum to ensure database is in sync), replication relay logs got corrupted.

Trying to SLAVE START failed with this nasty error message:

Code:

Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.

Here's how to recover from such an error -

Step 1: Identify the corrupted file

Issue a SHOW SLAVE STATUS followed by a SHOW MASTER STATUS and record the relay log filenames and positions.

In my case those were: mysql-bin.000313 position 1037717439 and mysql-bin.000249 position 326501604

Running mysqlbinlog to read the first block in each of these files, starting at the last position, immediately reveals the problematic file:

Code:

/usr/local/mysql/bin/mysqlbinlog -Hvv mysql-bin.000313 --start-position=1037717439 | more

ERROR: Error in Log_event::read_log_event(): 'Event too small', data_len: 0, event_type: 0
ERROR: Could not read entry at offset 326501604: Error in log format or read error.
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;

Notice the ERROR on the first line. This tells us we've found the corrupted replication file

Step 2 - Skip the bad block

This involves some trial and error, but what we want to do is use mysqlbinlog to manually parse MySQL replication log file to the end, bypassing the 'Event too small' bad block.

Once done, we have to update relay-log.info, so that MySQL doesn't try to process this file again.

Replaced this:

Code:

cat relay-log.info

./mysql-bin.000313 
1037717439 
mysql-bin.000249
113612

With:

Code:

./mysql-bin.000314 
0
mysql-bin.000249
113612

(Telling MySQL to advance to the next file mysql-bin.000314)

Step 3 - Restart database

While this may be a bit of an overkill, I prefer restarting the entire database, before issuing a SLAVE START

Code:

mysqladmin shutdown
mysqld_safe &

How to serve Mercurial HG repository over NGinx

Adrian Singer — Thu, 03 Sep 2009 14:31:08 GMT

Mercurial server can be setup in two modes:

* hg serve - a built in lightweight web server that is provided as part of the Mercurial package. hg serve can be used to quickly serve a single repository, but it's not a good choice for production systems.

hg serve doesn't support user authentication, multiple repositories and is limited to handling a single connection at a time.

* hgweb.cgi - a Python script that is faster, supports user authentication, multiple repositories, styles and web console.

As part of this post I'll walk you through setting up hgweb.cgi on an NGinx web server.

Step 1 - Install Python

Code:

cd /usr/ports/lang/python25
make all install

Step 2 - Install hgweb.cgi

Save this file under your web server main direcory.

hgweb.cgi:

Code:

#!/usr/bin/env python

import cgitb
from mercurial.hgweb.hgwebdir_mod import hgwebdir
from mercurial.hgweb.request import wsgiapplication
from flup.server.fcgi import WSGIServer

cgitb.enable()

def make_web_app():
 return hgwebdir("/home/admin/htdocs/hgweb.config")

WSGIServer(wsgiapplication(make_web_app)).run()

Replace /home/admin/htdocs/ with your web server folder

Step 3 - Configure hgweb

Save this config file under your web server main directory.

hgweb.config:

Code:

[ui]
username = root
editor = vim

[hooks]
changegroup = hg update >&2

[web]
baseurl =/.
style= gitweb
push_ssl = false
allow_push = *

[paths]
Clients = /home/
SPI = /home/admin/htdocs/

Replace mydomain with your web server url

Create this global settings file under /etc/mercurial/hgrc:

Code:

[web]
style= gitweb

Step 4 - Install Spawn-fcgi

Before we can tell NGinx to handle the hgweb.cgi Python script, we have to install spawn-fcgi

Follow these steps:

Code:

wget "http://www.lighttpd.net/download/spawn-fcgi-1.6.2.tar.gz"
tar -xvf spawn-fcgi-1.6.2.tar.gz
cd spawn-fcgi-1.6.2
./configure
make all
make install

Start spawn-fcgi:

Code:

/usr/local/bin/spawn-fcgi -f /home/code/hgweb.cgi -a 127.0.0.1 -p 8081

Try running hgweb.cgi by issuing: python hgweb.cgi
If you get a message about missing flup, you'll have to download flup and install it:

Code:

wget "http://www.saddi.com/software/flup/dist/flup-1.0.1.tar.gz"
tar -xvf flup-1.0.1.tar.gz
cd flup-1.0.1
python setup.py install

--

Now create a new file under /usr/local/etc/rc.d/start-spawnfcgi.sh to start spawn-fcgi automatically after a reboot.

Save the same line above under /usr/local/etc/rc.d/start-spawnfcgi.sh

Step 5 - Configure NGinx

Add this to your nginx.conf, replacing mydomain.com with your web server domain name and 1.2.3.4 with the dedicated ip address:

Code:

    server {             
        listen       1.2.3.4:80;
        server_name  mydomain.com;
    
    server_name_in_redirect off;

  auth_basic            "Restricted";
  auth_basic_user_file  /home/hg/.htpasswd;

        gzip on;               

    location / {

            root   /home/hg/; 
            index  index.php index.html index.htm;

            fastcgi_pass   127.0.0.1:8081;     
            fastcgi_param  SCRIPT_FILENAME  /home/hg/$fastcgi_script_name;
            include        /etc/nginx/fastcgi_top.conf;
            fastcgi_param  DOCUMENT_ROOT    /home/hg/;
            include        /etc/nginx/fastcgi_bottom.conf;

          fastcgi_param  QUERY_STRING     $query_string;
          fastcgi_param  REQUEST_METHOD   $request_method;
          fastcgi_param  CONTENT_TYPE     $content_type;
          fastcgi_param  CONTENT_LENGTH   $content_length;
        }


    }

Note that we are password protecting the repository by using an .htpasswd file. You can generate one using publicly available htpasswd generators like this one.

Tell NGinx to reload its configuration file:

Code:

kill -HUP `cat /usr/local/nginx/logs/nginx.pid`

If all goes well, you should be able to point your browser to mydomain.com and see a list of Mercurial repositories.

Mercurial Version Control

Adrian Singer — Wed, 02 Sep 2009 19:13:37 GMT

Mercurial (also called HG) is a cross-platform, super-fast, distributed revision control system for software developers.

Unlike other version control tools such as CVS, SVN, Preforce etc., Mercurial is a distributed system, where every node can act as both a client and server.

We've tested a lot of competing revision control systems, before Mercurial won us over.

The advantages Mercurial brings to the table include:

* Allows users to work on a local copy of the files, even when not connected to a network

* Most operation are faster since no network is involved

* Allows private work, so you can use your revision control system even for early drafts you don't want to publish

* Avoids relying on a single physical machine. A server disk crash is a non-event with distributed revision control

* Allows participation in projects without requiring permissions from project authorities, instead of requiring "committer" status

Graphical User Interface

Mercurial comes with great desktop integration for Windows and Mac, using a tool called Tortoise. Once installed, it adds a right-click context-menu, allowing one-click commit, file diff, sync, directly from the file explorer.

Tortoise includes a green icon next to all files that are up-to-date and an orange icon next to files that have been updated and need to be uploaded.

How to install Mercurial

An all-in-one Mercurial+Tortoise installer is available here

Once installed, Mercurial will add a right-click context-menu, as shown in the image above.

Before you move forward, it is important that you setup a Username, so that all Commits you upload, include your name.

Right click and select 'Global Options' from HG menu. Then enter your full name under Username

Next - click on the 'Sync' tab and select Update as the 'after pull operation'

Connecting to a live repository

Create a new folder on your machine where you'd like to save all source files, right click and select 'Create Repository Here' from Mercurial's menu

This will create two files under the current directory:
.hg subfolder
.hgignore text file

To avoid any conflicts, go ahead and delete the .hgignore text file.

Now that the repository is created, we can download all files from the server. Right click again and select 'Synchronize' from the menu.

This popup window will open:

Enter the live repository url and click the 'Pull' button to download all source files from the live repository.

Note: You can repeat this process of synchronizing with the live server, whenever you'd like to download updates from the server.

The first time you synchornize, it's going to take a while. But subsequent pulls will be very quick.

Updating Files

Step 1: Open the local copy of the file in your favorite editor, make changes and hit Save.

Mercurial detects the change and automatically updates the file icon to orange.

Step 2: Once you're done making changes, right-click and select HG Commit.

Enter a comment describing the changes you've just implemented. For example:
"Project 384283 - Added opt-in form on instant pay increase"

Good comments are the basis of a solid version control system. Your comments help other developers understand the nature of your changes, as they download them to their local repositories.

Step 3: Unlike a centralized cvs, the 'Commit' operation is a local one. Once changes are committed to your local repository, Right click and select 'Synchronize'. Enter the live repository url and click the 'Push' button. This takes care of uploading your changes to the server.

Adding new files to the project

In order to add a new file, you have to take one additional step -

Highlight the new files, right click and select 'Add Files'

Then, follow the steps listed under "Updating Files", committing changes and synchronizing with the server.