Full-service Internet Marketing & Web Development
Recent Posts
Sponsors
 |
Do It Yourself Web Server and Database monitoringMike Peters, 10-21-2009 |
When you have production web servers and databases, it is absolutely critical to have a monitoring system in-place that will alert you whenever service goes down.
Getting Started with website monitoring
One of our favorite monitoring tools is Pingdom. It's a $9.95/month easy-to-use service that will ping 5 of your servers, once every 60 seconds, alerting you via email & SMS whenever any server goes down.
In addition to downtime notifications, the service also provides response-time trends, raw data and uptime-over-time reports:
If you don't have any monitoring in-place right one, you should definitely try out Pingdom or a similar service.
In about 5 minutes of setting it up, you'll have the peace of mind, knowing exactly when your merchant website, affiliate landing page or payment gateway goes down.
Beyond simple HTTP monitoring
While Pingdom and other similar tools are great in letting you know once your website is -already- down, a lot of times the outage can be prevented in the first place, by detecting a decrease available memory, cpu, disk space etc.
At the pinnacle of server monitoring are tools such as Nagios and Munin.
Munin will monitor every single resource on your server, including: memory, cpu utilization, available handles, open connections, average load, running processes and more.
Nagios sends alerts to groups of users based on predetermined rules.
Do It Yourself Server Monitoring
In this post, I'd like to focus on how to create a simple monitoring script, that will survey remote servers for a few critical metrics, displaying the results on the screen.
Unlike Nagios+Munin, such a setup doesn't require installing a monitoring component on each server. Your monitoring script can run remotely, surveying all servers once every 60 seconds on a cronjob, sending an email/sms whenever detecting abnormal behavior.
For the purpose of this example, our script will monitor the following metrics:
= MySQL Database server
* Number of running threads
* Number of threads connected
* Queries per second
* Open tables
* Sample query response
* Number of tables in database
= Any FreeBSD/Linux server
* CPU load over the last 5 minutes
* Available memory
* Open sockets
* Number of running processes
// Set these for easier access
$USERNAME = "root";
$SERVER = "www.mydomain.com";
$DBHOST = "www.mydatabase.com";
$DBUSER = "root";
$DBPASS = "dbpassword";
$DBNAME = "dbdatabasename";
// Get load average, available memory and number of running processes
$load_average = "N/A";
$available_memory = "N/A";
$running_processes = "N/A";
$CRLF = "\r\n";
exec("ssh -l$USERNAME $SERVER \"top -b -n 1\"",&$buf);
for ($i=0,$str=""; $i<count($buf); $i++) $str .= $buf[$i].$CRLF;
if (($pos=strpos($str,"load averages:"))!==false)
{
$str = trim(substr($str, $pos+strlen("load averages:")));
if (($pos=strpos($str,$CRLF))!==false)
{
$load_average = substr($str,0,$pos);
if (($pos=strpos($str,"up"))!==false) $load_average = substr($load_average,0,$pos);
}
}
else
if (($pos=strpos($str,"load average:"))!==false)
{
$str = trim(substr($str, $pos+strlen("load average:")));
if (($pos=strpos($str,$CRLF))!==false)
{
$load_average = substr($str,0,$pos);
}
}
if (($pos=strpos($str,"processes:"))!==false)
{
$str = trim(substr($str, $pos+strlen("processes:")));
if (($pos=strpos($str,","))!==false)
{
$running_processes = substr($str,0,$pos);
}
}
else
if (($pos=strpos($str,"Tasks:"))!==false)
{
$str = trim(substr($str, $pos+strlen("Tasks:")));
if (($pos=strpos($str,","))!==false)
$str = trim(substr($str, $pos+1));
if (($pos=strpos($str,"run"))!==false)
{
$running_processes = substr($str,0,$pos);
}
}
if (($pos=strpos($str,"Mem:"))!==false)
{
$str = trim(substr($str, $pos+strlen("Mem:")));
if (($pos=strpos($str,$CRLF))!==false)
{
$available_memory = substr($str, 0, $pos);
}
}
// Get open sockets
$open_sockets = "N/A";
$buf = array();
exec("ssh -l$USERNAME $SERVER "netstat -n | wc -l"",&$buf);
$open_sockets = trim($buf[0]);
// Display server metrics
echo "== Server\r\n";
echo " Load average: $load_average\r\n";
echo " Open Sockets: $open_sockets\r\n";
echo " Running Processes: $running_processes\r\n";
echo " Available memory: $available_memory\r\n";
echo "\r\n";
// Get running and connected threads
$db_threads_connected = "N/A";
$db_running_threads = "N/A";
mysql_connect($DBHOST,$DBUSER,$DBPASS);
$Result = @mysql_query("show status like '%thread%' ");
$cnt = @mysql_num_rows($Result);
while ($cnt)
{
$cnt--;
if (!($Row = @mysql_fetch_array($Result))) continue;
if (empty($Row['Value'])) continue;
if (Strcasecmp($Row['Variable_name'],'Threads_connected')==0)
$db_threads_connected = $Row['Value'];
else
if (Strcasecmp($Row['Variable_name'],'Threads_running')==0)
$db_running_threads = $Row['Value'];
}
// Get queries per second
$Result = @mysql_query("show status like 'questions' ");
$Row = @mysql_fetch_array($Result);
$questions = $Row['Value']+0;
$Result = @mysql_query("show status like 'uptime' ");
$Row = @mysql_fetch_array($Result);
$uptime = $Row['Value']+0;
$db_queries_per_second = number_format($questions/$uptime,0);
// Get open tables
$db_open_tables = "N/A";
$Result = @mysql_query("show status like '%tables%' ");
$cnt = @mysql_num_rows($Result);
while ($cnt)
{
$cnt--;
if (!($Row = @mysql_fetch_array($Result))) continue;
if (empty($Row['Value'])) continue;
if (Strcasecmp($Row['Variable_name'],'Open_tables')==0)
$db_open_tables = $Row['Value'];
}
// Get query response
$Result = @mysql_query("SELECT 1");
$Row = @mysql_fetch_row($Result);
$db_query_response = $Row[0]=='1' ? "Good" : "BAD";
// Get total number of tables
@mysql_query("use $DBNAME");
$Result = @mysql_query("show tables");
$db_tables_in_database = @mysql_num_rows($Result);
// Display database metrics
echo "== Database\r\n";
echo " Running Threads: $db_running_threads\r\n";
echo " Threads Connected: $db_threads_connected\r\n";
echo " Queries Per Second: $db_queries_per_second\r\n";
echo " Open Tables: $db_open_tables\r\n";
echo " Query Response: $db_query_response\r\n";
echo " Number of tables: $db_tables_in_database\r\n";
The output of the script will look something like this:
Tip: If you're looking to monitor dozens of servers, consider applying our ssh no password approach, to save the hassle of having to exchange private keys with each server.
Getting Started with website monitoring
One of our favorite monitoring tools is Pingdom. It's a $9.95/month easy-to-use service that will ping 5 of your servers, once every 60 seconds, alerting you via email & SMS whenever any server goes down.
In addition to downtime notifications, the service also provides response-time trends, raw data and uptime-over-time reports:
If you don't have any monitoring in-place right one, you should definitely try out Pingdom or a similar service.
In about 5 minutes of setting it up, you'll have the peace of mind, knowing exactly when your merchant website, affiliate landing page or payment gateway goes down.
Beyond simple HTTP monitoring
While Pingdom and other similar tools are great in letting you know once your website is -already- down, a lot of times the outage can be prevented in the first place, by detecting a decrease available memory, cpu, disk space etc.
At the pinnacle of server monitoring are tools such as Nagios and Munin.
Munin will monitor every single resource on your server, including: memory, cpu utilization, available handles, open connections, average load, running processes and more.
Nagios sends alerts to groups of users based on predetermined rules.
Do It Yourself Server Monitoring
In this post, I'd like to focus on how to create a simple monitoring script, that will survey remote servers for a few critical metrics, displaying the results on the screen.
Unlike Nagios+Munin, such a setup doesn't require installing a monitoring component on each server. Your monitoring script can run remotely, surveying all servers once every 60 seconds on a cronjob, sending an email/sms whenever detecting abnormal behavior.
For the purpose of this example, our script will monitor the following metrics:
= MySQL Database server
* Number of running threads
* Number of threads connected
* Queries per second
* Open tables
* Sample query response
* Number of tables in database
= Any FreeBSD/Linux server
* CPU load over the last 5 minutes
* Available memory
* Open sockets
* Number of running processes
// Set these for easier access
$USERNAME = "root";
$SERVER = "www.mydomain.com";
$DBHOST = "www.mydatabase.com";
$DBUSER = "root";
$DBPASS = "dbpassword";
$DBNAME = "dbdatabasename";
// Get load average, available memory and number of running processes
$load_average = "N/A";
$available_memory = "N/A";
$running_processes = "N/A";
$CRLF = "\r\n";
exec("ssh -l$USERNAME $SERVER \"top -b -n 1\"",&$buf);
for ($i=0,$str=""; $i<count($buf); $i++) $str .= $buf[$i].$CRLF;
if (($pos=strpos($str,"load averages:"))!==false)
{
$str = trim(substr($str, $pos+strlen("load averages:")));
if (($pos=strpos($str,$CRLF))!==false)
{
$load_average = substr($str,0,$pos);
if (($pos=strpos($str,"up"))!==false) $load_average = substr($load_average,0,$pos);
}
}
else
if (($pos=strpos($str,"load average:"))!==false)
{
$str = trim(substr($str, $pos+strlen("load average:")));
if (($pos=strpos($str,$CRLF))!==false)
{
$load_average = substr($str,0,$pos);
}
}
if (($pos=strpos($str,"processes:"))!==false)
{
$str = trim(substr($str, $pos+strlen("processes:")));
if (($pos=strpos($str,","))!==false)
{
$running_processes = substr($str,0,$pos);
}
}
else
if (($pos=strpos($str,"Tasks:"))!==false)
{
$str = trim(substr($str, $pos+strlen("Tasks:")));
if (($pos=strpos($str,","))!==false)
$str = trim(substr($str, $pos+1));
if (($pos=strpos($str,"run"))!==false)
{
$running_processes = substr($str,0,$pos);
}
}
if (($pos=strpos($str,"Mem:"))!==false)
{
$str = trim(substr($str, $pos+strlen("Mem:")));
if (($pos=strpos($str,$CRLF))!==false)
{
$available_memory = substr($str, 0, $pos);
}
}
// Get open sockets
$open_sockets = "N/A";
$buf = array();
exec("ssh -l$USERNAME $SERVER "netstat -n | wc -l"",&$buf);
$open_sockets = trim($buf[0]);
// Display server metrics
echo "== Server\r\n";
echo " Load average: $load_average\r\n";
echo " Open Sockets: $open_sockets\r\n";
echo " Running Processes: $running_processes\r\n";
echo " Available memory: $available_memory\r\n";
echo "\r\n";
// Get running and connected threads
$db_threads_connected = "N/A";
$db_running_threads = "N/A";
mysql_connect($DBHOST,$DBUSER,$DBPASS);
$Result = @mysql_query("show status like '%thread%' ");
$cnt = @mysql_num_rows($Result);
while ($cnt)
{
$cnt--;
if (!($Row = @mysql_fetch_array($Result))) continue;
if (empty($Row['Value'])) continue;
if (Strcasecmp($Row['Variable_name'],'Threads_connected')==0)
$db_threads_connected = $Row['Value'];
else
if (Strcasecmp($Row['Variable_name'],'Threads_running')==0)
$db_running_threads = $Row['Value'];
}
// Get queries per second
$Result = @mysql_query("show status like 'questions' ");
$Row = @mysql_fetch_array($Result);
$questions = $Row['Value']+0;
$Result = @mysql_query("show status like 'uptime' ");
$Row = @mysql_fetch_array($Result);
$uptime = $Row['Value']+0;
$db_queries_per_second = number_format($questions/$uptime,0);
// Get open tables
$db_open_tables = "N/A";
$Result = @mysql_query("show status like '%tables%' ");
$cnt = @mysql_num_rows($Result);
while ($cnt)
{
$cnt--;
if (!($Row = @mysql_fetch_array($Result))) continue;
if (empty($Row['Value'])) continue;
if (Strcasecmp($Row['Variable_name'],'Open_tables')==0)
$db_open_tables = $Row['Value'];
}
// Get query response
$Result = @mysql_query("SELECT 1");
$Row = @mysql_fetch_row($Result);
$db_query_response = $Row[0]=='1' ? "Good" : "BAD";
// Get total number of tables
@mysql_query("use $DBNAME");
$Result = @mysql_query("show tables");
$db_tables_in_database = @mysql_num_rows($Result);
// Display database metrics
echo "== Database\r\n";
echo " Running Threads: $db_running_threads\r\n";
echo " Threads Connected: $db_threads_connected\r\n";
echo " Queries Per Second: $db_queries_per_second\r\n";
echo " Open Tables: $db_open_tables\r\n";
echo " Query Response: $db_query_response\r\n";
echo " Number of tables: $db_tables_in_database\r\n";
The output of the script will look something like this:
== Server
Load average: 0.09, 0.23, 0.23
Open Sockets: 226
Running Processes: 1 running
Available memory: 2203M Active, 112M Buf, 4668K Free
== Database
Running Threads: 3
Threads Connected: 16
Queries Per Second: 320
Open Tables: 200
Query Response: Good
Number of tables: 1648
Load average: 0.09, 0.23, 0.23
Open Sockets: 226
Running Processes: 1 running
Available memory: 2203M Active, 112M Buf, 4668K Free
== Database
Running Threads: 3
Threads Connected: 16
Queries Per Second: 320
Open Tables: 200
Query Response: Good
Number of tables: 1648
Tip: If you're looking to monitor dozens of servers, consider applying our ssh no password approach, to save the hassle of having to exchange private keys with each server.
|
|
Subscribe Now to receive new posts via Email as soon as they come out.
Comments
Post your comments