Call us Toll-Free:
1-800-218-1525
Email us

How to count unique visitors in an nginx access.log

Mike Peters, November 5    --    Filed under Programming
This nifty one-liner will count the number of unique visitors in an NGinx web-server access.log file, for a given day:

grep "\[04/Nov/2009" access.log | cut -d" " -f1 | sort | uniq | wc -l

Replace 04/Nov/2009 with the date you'd like to count the number of unique visitors for.

FreeBSD No Ports Fix

Mike Peters, November 4    --    Filed under Programming
When installing a FreeBSD system, you're asked if you would like to install the Ports Collection. If you chose no, you can follow these instructions to obtain the ports collection:

Option 1 - CVSup

csup -L 2 -h cvsup.FreeBSD.org /usr/share/examples/cvsup/ports-supfile
csup

Option 2 - Portsnap

portsnap fetch
portsnap extract

How to: Add a Lead Opt-in Web Form

Adrian Singer, October 28    --    Filed under SoftwareProjects Products
A lead opt-in web form is a short form, asking the user to provide name, emailaddress, phone and/or other optional fields, for the purpose of signing up for a program, or receiving more information about your offering.



Unlike other shopping carts, the SPI cart doesn't limit you to a single form style or any specific fields.

You can create whatever html form you like, using any design and combination of fields (text, radio button, checkboxes, textareas) as needed.

The only requirement is to have at least the emailaddress field, whenever saving a new lead to the database.

Step 1 - Design your opt-in form

Use your favorite HTML editor to design your lead opt-in form.

For simplicity, use 'name' as the fieldname for the user's name (or 'first_name' + 'last_name'), use 'emailaddress' as the fieldname for the user's email address and use 'phone' as the fieldname for the user's phone number.

Here's a typical lead opt-in form with the basic name,emailaddress,phone fields as well as a few additional custom ones:


<form action="#" method="post">
<
p>First Name: <input type="text" name="first_name" /></p>
<
p>Last Name: <input type="text" name="last_name" /></p>
<
p>Best Email: <input type="text" name="email" /></p>
<
p>Best Phone Number: <input type="text" name="phone" /></p>
<
p>Have you traded individual stocks?
Yes <input name="traded_indi_stock" type="radio" value="Yes" /> No <input name="traded_indi_stock" type="radio" value="No" />
</
p>
<
p>Have you traded options
Yes
<input name="traded_option" type="radio" value="Yes" /> No <input name="traded_option" type="radio" value="No" />
</
p>
<
input type="submit" name="submit" value="Submit" />
</
form>

Step 2 - Add custom fields to Leads table

Since we're using a few custom fields in this example (traded_indi_stock, traded_option), we need to add those fields to the 'leads' table.

Connect to your SPI database using PHPMyAdmin, locate the leads table and add the two new fields. You only have to do this once every time you're adding a new custom field to a table.



Make sure you're adding the custom fields to the 'leads' table and pay close attention to the fieldnames. The fieldnames you choose are going to have to match the fieldnames you pass to the do_addlead() function.

Step 3 - Save the lead to the database

Now that we have the custom leads in the 'leads' table, saving the lead to the database is the easy part.

The code below renders the opt-in form, saves the lead to the database and redirects the user to a thankyou.html page.


<?php
// Include SPICart shopping cart API
require_once("spicart.php");

// If this is a lead submit
if (!empty($_POST['email']))
{
// Save lead to the database
$input = array();
$input['first_name'] = $_POST['first_name'];
$input['last_name'] = $_POST['last_name'];
$input['emailaddress'] = $_POST['emailaddres'];
$input['phone'] = $_POST['phone'];
$input['traded_indi_stock'] = $_POST['traded_indi_stock'];
$input['traded_option'] = $_POST['traded_option'];
do_addlead($input, &$output, &$result_str);
Header("Location: thankyou.html");
die;
}
?>
<form action="#" method="post">
<p>First Name: <input type="text" name="first_name" /></p>
<p>Last Name: <input type="text" name="last_name" /></p>
<p>Best Email: <input type="text" name="emailaddress" /></p>
<p>Best Phone Number: <input type="text" name="phone" /></p>
<p>Have you traded individual stocks?
Yes <input name="traded_indi_stock" type="radio" value="Yes" /> No <input name="traded_indi_stock" type="radio" value="No" />
</p>
<p>Have you traded options
Yes <input name="traded_option" type="radio" value="Yes" /> No <input name="traded_option" type="radio" value="No" />
</p>
<input type="submit" name="submit" value="Submit" />
</form>

Note about Affiliate Tracking

Often times, affiliates will be sending their traffic to your lead opt-in forms. You're going to want to know which affiliate generated which leads.

Luckily the system takes care of this automatically.

One of the fields in the 'leads' table is aff_id. This is the ID of the referring affiliate and serves as the glue, associating leads with the affiliates who referred them.

When a user clicks on an affiliate link to visit the opt-in form on your website, they typically follow a link that looks like this:

www.YourDomain.com/?aff_id=1234

Where 1234 is the affiliate ID.

As soon as the user lands on the lead opt-in form, the spicart.php include at the top of your file, identifies the aff_id url-parameter and saves it in a cookie on the end-user's machine.

Later when you call do_addlead(), this method picks up the aff_id from the end-user's cookie and passes it along, so that when the lead is added, the referring aff_id is saved as well.

Do It Yourself Web Server and Database monitoring

Mike Peters, October 21    --    Filed under Programming
When you have production web servers and databases, it is absolutely critical to have a monitoring system in-place that will alert you whenever service goes down.

Getting Started with website monitoring

One of our favorite monitoring tools is Pingdom. It's a $9.95/month easy-to-use service that will ping 5 of your servers, once every 60 seconds, alerting you via email & SMS whenever any server goes down.

In addition to downtime notifications, the service also provides response-time trends, raw data and uptime-over-time reports:



If you don't have any monitoring in-place right one, you should definitely try out Pingdom or a similar service.

In about 5 minutes of setting it up, you'll have the peace of mind, knowing exactly when your merchant website, affiliate landing page or payment gateway goes down.

Beyond simple HTTP monitoring

While Pingdom and other similar tools are great in letting you know once your website is -already- down, a lot of times the outage can be prevented in the first place, by detecting a decrease available memory, cpu, disk space etc.

At the pinnacle of server monitoring are tools such as Nagios and Munin.

Munin will monitor every single resource on your server, including: memory, cpu utilization, available handles, open connections, average load, running processes and more.

Nagios sends alerts to groups of users based on predetermined rules.



Do It Yourself Server Monitoring

In this post, I'd like to focus on how to create a simple monitoring script, that will survey remote servers for a few critical metrics, displaying the results on the screen.

Unlike Nagios+Munin, such a setup doesn't require installing a monitoring component on each server. Your monitoring script can run remotely, surveying all servers once every 60 seconds on a cronjob, sending an email/sms whenever detecting abnormal behavior.

For the purpose of this example, our script will monitor the following metrics:

= MySQL Database server
* Number of running threads
* Number of threads connected
* Queries per second
* Open tables
* Sample query response
* Number of tables in database

= Any FreeBSD/Linux server
* CPU load over the last 5 minutes
* Available memory
* Open sockets
* Number of running processes


// Set these for easier access
$USERNAME  = "root";
$SERVER  = "www.mydomain.com";
$DBHOST    = "www.mydatabase.com";
$DBUSER    = "root";
$DBPASS    = "dbpassword";
$DBNAME    = "dbdatabasename";

// Get load average, available memory and number of running processes
$load_average = "N/A";
$available_memory = "N/A";
$running_processes = "N/A";
$CRLF = "\\r\"\n";
exec("ssh -l$USERNAME $SERVER \"top -b -n 1\"",&$buf);
for (
$i=0,$str=""; $i<count($buf); $i++) $str .= $buf[$i].$CRLF;
if ((
$pos=strpos($str,"load averages:"))!==false)
{
   
$str = trim(substr($str, $pos+strlen("load averages:")));
    if ((
$pos=strpos($str,$CRLF))!==false)
    {
       
$load_average = substr($str,0,$pos);
        if ((
$pos=strpos($str,"up"))!==false) $load_average = substr($load_average,0,$pos);
    }
}
else
if ((
$pos=strpos($str,"load average:"))!==false)
{
   
$str = trim(substr($str, $pos+strlen("load average:")));
    if ((
$pos=strpos($str,$CRLF))!==false)
    {
       
$load_average = substr($str,0,$pos);
    }
}
if ((
$pos=strpos($str,"processes:"))!==false)
{
 
$str = trim(substr($str, $pos+strlen("processes:")));
  if ((
$pos=strpos($str,","))!==false)
  {
   
$running_processes = substr($str,0,$pos);
  }
}
else
if ((
$pos=strpos($str,"Tasks:"))!==false)
{
   
$str = trim(substr($str, $pos+strlen("Tasks:")));
    if ((
$pos=strpos($str,","))!==false)
   
$str = trim(substr($str, $pos+1));
    if ((
$pos=strpos($str,"run"))!==false)
    {
       
$running_processes = substr($str,0,$pos);
    }
}
if ((
$pos=strpos($str,"Mem:"))!==false)
{
 
$str = trim(substr($str, $pos+strlen("Mem:")));
  if ((
$pos=strpos($str,$CRLF))!==false)
  {
   
$available_memory = substr($str, 0, $pos);
  }
}

// Get open sockets
$open_sockets = "N/A";
$buf = array();
exec("ssh -l$USERNAME $SERVER "netstat -n | wc -l"",&$buf); 
$open_sockets = trim($buf[0]);     

// Display server metrics
echo "== Server\r\n";
echo
"    Load average: $load_average\r\n";
echo
"    Open Sockets: $open_sockets\r\n";
echo
" Running Processes: $running_processes\r\n";
echo
"  Available memory: $available_memory\r\n";
echo
"\\r\"\n";

// Get running and connected threads
$db_threads_connected = "N/A";
$db_running_threads = "N/A";
mysql_connect($DBHOST,$DBUSER,$DBPASS);
$Result = @mysql_query("show status like '%thread%' ");
$cnt = @mysql_num_rows($Result);
while (
$cnt)

   
$cnt--;
    if (!(
$Row = @mysql_fetch_array($Result))) continue;

    if (empty(
$Row['Value'])) continue;

    if (
Strcasecmp($Row['Variable_name'],'Threads_connected')==0)
   
$db_threads_connected = $Row['Value'];
    else
    if (
Strcasecmp($Row['Variable_name'],'Threads_running')==0)
   
$db_running_threads = $Row['Value'];
}

// Get queries per second
$Result = @mysql_query("show status like 'questions' ");
$Row = @mysql_fetch_array($Result);
$questions = $Row['Value']+0;
$Result = @mysql_query("show status like 'uptime' ");
$Row = @mysql_fetch_array($Result);
$uptime = $Row['Value']+0;
$db_queries_per_second = number_format($questions/$uptime,0);

// Get open tables
$db_open_tables = "N/A";
$Result = @mysql_query("show status like '%tables%' ");
$cnt = @mysql_num_rows($Result);
while (
$cnt)
{
   
$cnt--;
    if (!(
$Row = @mysql_fetch_array($Result))) continue;

    if (empty(
$Row['Value'])) continue;
    if (
Strcasecmp($Row['Variable_name'],'Open_tables')==0)
   
$db_open_tables = $Row['Value'];
}

// Get query response
$Result = @mysql_query("SELECT 1");
$Row = @mysql_fetch_row($Result);
$db_query_response = $Row[0]=='1' ? "Good" : "BAD";

// Get total number of tables
@mysql_query("use $DBNAME");
$Result = @mysql_query("show tables");
$db_tables_in_database = @mysql_num_rows($Result);

// Display database metrics
echo "== Database\r\n";
echo
"  Running Threads: $db_running_threads\r\n";
echo
" Threads Connected: $db_threads_connected\r\n";
echo
" Queries Per Second: $db_queries_per_second\r\n";
echo
"    Open Tables: $db_open_tables\r\n";
echo
"  Query Response: $db_query_response\r\n";
echo
"  Number of tables: $db_tables_in_database\r\n";

The output of the script will look something like this:

== Server
Load average: 0.09, 0.23, 0.23
Open Sockets: 226
Running Processes: 1 running
Available memory: 2203M Active, 112M Buf, 4668K Free

== Database
Running Threads: 3
Threads Connected: 16
Queries Per Second: 320
Open Tables: 200
Query Response: Good
Number of tables: 1648

Tip: If you're looking to monitor dozens of servers, consider applying our ssh no password approach, to save the hassle of having to exchange private keys with each server.

Product Launch Marketing: Quickest path to a 6 figure monthly cashflow

Kate Richards, October 21    --    Filed under Traffic
A Product Launch is the process of building anticipation and emphasizing scarcity in a hungry market, towards the release of a new product / service, positioned to have the potential to make a significant impact on the target customer's life.

In the Internet Marketing world, Product Launches are powerful marketing vehicles used by Information-Marketers to drive a massive amount of traffic on a specific day ("Launch day"), with the goal of converting the wave of anonymous traffic into qualified leads and then paying customers.

For the past 5 years, SPI has been instrumental in orchestrating some of the largest most-successful product launches online, working with leading information marketers on applying time-tested techniques to generate hundreds of millions of dollars in product revenues.

Some of the recent product launches we handled include ListBuilding, GoogleCashDetective, 7FigurePPCSecrets, LaunchTree, AffiliateClassroom, PPCClassroom, DirectResponseMarketing.

Why does it work

Product Launch Marketing works so well because they build on two fundamental human traits:

#1. Herd Mentality - Follow the leaders
In the weeks building to the launch, opinion leaders and peers within your industry, will all glorify how great this product is going to be and why you absolutely must have it.

#2. Scarcity - Desire to want what we can't have
Using a limited number of units, fixed close date, early-bird discounts, one-time-offers and other tactics, the company will build the impression you must act fast or lose the opportunity to own the product forever.

Examples of great well-planned product launches: The iPhone, "The Secret" DVD, StomperNet 3.0, Gary Vaynerchuk's CrushIt book.

In Numbers

A typical Internet Marketing Product Launch these days, yields $1m (One Million Dollars) on launch day, followed by $200k in monthly residual payments from a continuity program.

It is far easier to convince 2,000 prospects to pay $99 a month, than it is to direct-sell a few large accounts in order to generate the equivalent of $200k in monthly revenues.

1,000 users paying $99 a month = $100k a month
2,000 users paying $50 a month = $100k a month
4,000 users paying $25 a month = $100k a month

50% of the revenues are often paid back to affiliates.
20% is the average three-months drop-out rate.

The SPI Product Launch Program

With the "Product Launch Program", we are now helping dozens of companies in various industries, capitalize on this phenomenal marketing strategy.

The SPI Product Launch Program, is not a how-to tutorial, or an online course.

This program is a joint-venture between you and SPI, whereby our company will work with you, hands-on, on every aspect of your product launch in a joint effort to make it a big success.

What you can expect from SPI, during this process:

* Hands-on consulting
* Introduction to leading information-marketing affiliates
* Access to case studies, analytics and proven tactics, used by leading information marketers
* Access to live front-end and back-office systems, so that you can learn what works best
* Professional services: Landing page design, Hosting, Affiliate system, Payment solutions
* PLR: Videos and mp3 files to complement your member's-only content
* Membership system integration (Wordpress+plugin / aMember / SPI)
* Load testing, to ensure your setup can handle launch-day traffic
* Affiliate emails autoresponder kit
* Corporate address, Call center, Help desk
* Ongoing Software development / website customizations / graphic design

Application Process

Our success over the years has made this service very popular. Committed to maintaining our strong reputation, SPI dedicates a lot of resources to every individual product launch we are involved with.

Unfortunately we cannot blindly accept new clients into this program.

Please contact your account manager (or email us here) for details on how to apply.

SSH no password without any private keys. It's magic!

Michel Nadeau, October 8    --    Filed under Programming
We all know that one can configure SSH to login automatically by adding the client's public key to the server's ~/.ssh/authorized_keys file. But what if you need to run commands on 200 machines and don't want to login to every single machine to add the key? Search no more, this tutorial has the answer!

1. Installing sshpass

sshpass is a tool for non-interactivly performing password authentication with SSH's so called "interactive keyboard password authentication".

Here's a standard SSH connect command:

debian_I:~# ssh -l root localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is b4:e9:e7:56:a2:b4:89:9b:d8:fd:7e:8e:f1:e4:1d:9f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
root@localhost's password:
Last login: Thu Oct 8 09:35:41 2009 from localhost
Linux debian_I 2.6.18-6-686 #1 SMP i686
debian_I:~#

First you have to answer "yes" to accept the host key and then to enter the password.

With sshpass, you are able to specify the password on the command line and skip this step. Here's how to install it:

mkdir -p /usr/local/src/
cd /usr/local/src/
wget http://downloads.sourceforge.net/project/sshpass/sshpass/1.04/sshpass-1.04.tar.gz?use_mirror=iweb
tar xvfz sshpass-1.04.tar.gz
cd sshpass-1.04
./configure
make
make install

sshpass is now ready to be used!

2. Using sshpass

The sshpass' syntax is:

sshpass -p [password] [ssh command]

So instead of doing:

ssh -l root localhost

You can simply do:

sshpass -p myrootpass ssh -l root localhost

You will be automatically logged in, without any password prompt:

debian_I:~# sshpass -p myrootpass ssh -l root localhost
Last login: Thu Oct 8 09:52:04 2009 from localhost
Linux debian_I 2.6.18-6-686 #1 SMP i686
debian_I:~#

3. Automatically accepting host keys

The last problem is this prompt:

The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is b4:e9:e7:56:a2:b4:89:9b:d8:fd:7e:8e:f1:e4:1d:9f.
Are you sure you want to continue connecting (yes/no)? yes

When you're using sshpass to connect on a single machine that you use often, it's not a big deal because you will get the prompt once and never again after. But if you want to connect to 200 machines, you definitely don't want to type "yes" 200 times.

To fix this issue, simply add this line in /etc/ssh/ssh_config on the CLIENT machine:

StrictHostKeyChecking=no

With this setting enabled, SSH will automatically accept the host keys and will not prompt you about it.

4. Automating the process

This section will show you how to easily automate the process of running commands on any number of machines you want.

First you will need a file containing the hostname, username and password for each of the servers you want to run commands on, in a CSV format. For example:

ssh_magic.csv:
someserver.com,root,123456
someotherserver.com,sshadmin,abcdef
onelastserver.com,root,123456

Then you will create this bash script:

ssh_magic.sh:
#!/bin/sh

# Loop ssh_magic.csv
for LINE in $(cat ssh_magic.csv)
do
   
# Split line
   
host=$(echo $LINE | cut -d "," -f1)
   
user=$(echo $LINE | cut -d "," -f2)
   
pass=$(echo $LINE | cut -d "," -f3)
   
   
# Display server info
   
echo ""
   
echo "HOSTNAME: $host"
   
echo ""
   
   
# Run commands
   
sshpass -p $pass ssh -l $user $host "uname -a"
   
sshpass -p $pass ssh -l $user $host "df -h"
   
   
echo ""
   
echo "===================================="
done

echo ""
echo "DONE"
echo ""

Simply place ssh_magic.csv and ssh_magic.sh together and run the script.

Conclusion

You now know how to run commands on any number of SSH machines, without any prompt! Make sure you check the other ways to pass the password to sshpass - they offer more secure ways than directly on the command line with the -p option.

We're Hiring! Looking for Technical Account Managers, Team Leaders

Mike Peters, October 7    --    Filed under Basics
Software Projects, a New York based full-service Internet Marketing firm, is seeking a bright, articulate, detail-oriented, and technical applicant interested in joining our group of software professionals.

As a Technical Account Manager, you will work directly with clients, translating client vision into technical requirements, submit weekly plans, manage teams of engineers and oversee all software development work.

You will brain-storm with engineers, think outside the box, provide direction and leadership, while directly interacting with some of the best minds in our industry.

== Requirements:
* Excellent problem solving skills
* Leader, Motivator, Confidence, Energetic
* Superb communication skills with the know-how to verbalize ideas into technical requirements
* Minimum 2 years programming experience (PHP / C / Perl)
* Experience with FreeBSD/Linux a great plus

== Responsibilities:
* Translate client wants into requirements
* Plan projects, oversee software development
* Solve Problems
* Update clients on progress
* Maintain high client satisfaction

== Location:
You will be working from your home office for the first six months. After that time, you will have the option of relocating to one of our offices or continuing to work from home.

== Next steps:
Send an email to hr@softwareprojects.com with your resume and let us know when is a good time to chat.

Didn't update your resume in the last 2 years? That's fine. It's all about your experience and skills! Send us a quick cover-letter and we'll follow-up.
« Previous Posts



About Us  |  Contact us  |  Privacy Policy  |  Terms & Conditions