Earlier tonight, at 1:30am EST, SoftLayer - one of the tier 1 data centers we use, experienced a power outage in their primary facility, rendering thousands of sites dead.



SPI engineers were alerted to the outage within seconds (we take monitoring very seriously). After establishing that the core problem was a total data-center failure, we verified load balancers are properly redirecting traffic to other locations.

The Power Outage lasted a total of 2 hours and 3 minutes according to our friends at Pingdom:



All SPI client sites are multi-homed, which means we host everything in two independent data centers: Rackspace, Softlayer, NTT Verio and iWeb.

SPI utilizes redundant global load balancing, to immediately redirect end users from west coast to east coast, when such global outages happen.

How it works

New end users hitting your mydomain.com are diverted to active servers.

Old users / scripts (which keep a cached version of your website ip address) continue getting routed to the old server until their local DNS cache is cleared.

What's Next

While the DNS ttl (time to live) is setup as 60 seconds, some browsers and ISPs don't honor that and can take up to 2 hours until the DNS entry is refreshed.

As a workaround for these type of catastrophes where active users have cached a malfunctioning server's ip address, we'll be configuring the load balancers to reply back with all available server ip addresses, letting an end-user's machine rotate ips internally.

We understand the importance of keeping your sites up at all times and although our current setup ensures 100% uptime for new users, we are determined to come up with a solution for users with a cached version of a malfunctioning server ip address.

Your SPI account manager will contact you with any specific steps you have to take, in order to enable this setup for your domains.

If you're using MySQL InnoDB tables with these two lines in your /etc/my.cnf, for the most part, your database should gracefully handle unexpected shutdowns without data corruption:


I was working on such a database today, that experienced a power outage.

While all InnoDB data remained intact (I verified the checksum to ensure database is in sync), replication relay logs got corrupted.

Trying to SLAVE START failed with this nasty error message:


Here's how to recover from such an error -

Step 1: Identify the corrupted file

Issue a SHOW SLAVE STATUS followed by a SHOW MASTER STATUS and record the relay log filenames and positions.

In my case those were: mysql-bin.000313 position 1037717439 and mysql-bin.000249 position 326501604

Running mysqlbinlog to read the first block in each of these files, starting at the last position, immediately reveals the problematic file:


Notice the ERROR on the first line. This tells us we've found the corrupted replication file

Step 2 - Skip the bad block

This involves some trial and error, but what we want to do is use mysqlbinlog to manually parse MySQL replication log file to the end, bypassing the 'Event too small' bad block.

Once done, we have to update relay-log.info, so that MySQL doesn't try to process this file again.

Replaced this:


With:


(Telling MySQL to advance to the next file mysql-bin.000314)

Step 3 - Restart database

While this may be a bit of an overkill, I prefer restarting the entire database, before issuing a SLAVE START

Mercurial server can be setup in two modes:

* hg serve - a built in lightweight web server that is provided as part of the Mercurial package. hg serve can be used to quickly serve a single repository, but it's not a good choice for production systems.

hg serve doesn't support user authentication, multiple repositories and is limited to handling a single connection at a time.

* hgweb.cgi - a Python script that is faster, supports user authentication, multiple repositories, styles and web console.

As part of this post I'll walk you through setting up hgweb.cgi on an NGinx web server.

Step 1 - Install Python


Step 2 - Install hgweb.cgi

Save this file under your web server main direcory.

hgweb.cgi:

Replace /home/admin/htdocs/ with your web server folder

Step 3 - Configure hgweb

Save this config file under your web server main directory.

hgweb.config:

Replace mydomain with your web server url

Create this global settings file under /etc/mercurial/hgrc:


Step 4 - Install Spawn-fcgi

Before we can tell NGinx to handle the hgweb.cgi Python script, we have to install spawn-fcgi

Follow these steps:

Start spawn-fcgi:

Try running hgweb.cgi by issuing: python hgweb.cgi
If you get a message about missing flup, you'll have to download flup and install it:


--

Now create a new file under /usr/local/etc/rc.d/start-spawnfcgi.sh to start spawn-fcgi automatically after a reboot.

Save the same line above under /usr/local/etc/rc.d/start-spawnfcgi.sh

Step 5 - Configure NGinx

Add this to your nginx.conf, replacing mydomain.com with your web server domain name and 1.2.3.4 with the dedicated ip address:


Note that we are password protecting the repository by using an .htpasswd file. You can generate one using publicly available htpasswd generators like this one.

Tell NGinx to reload its configuration file:

If all goes well, you should be able to point your browser to mydomain.com and see a list of Mercurial repositories.

View 2 Comment(s)

Mercurial (also called HG) is a cross-platform, super-fast, distributed revision control system for software developers.

Unlike other version control tools such as CVS, SVN, Preforce etc., Mercurial is a distributed system, where every node can act as both a client and server.

We've tested a lot of competing revision control systems, before Mercurial won us over.

The advantages Mercurial brings to the table include:

* Allows users to work on a local copy of the files, even when not connected to a network

* Most operation are faster since no network is involved

* Allows private work, so you can use your revision control system even for early drafts you don't want to publish

* Avoids relying on a single physical machine. A server disk crash is a non-event with distributed revision control

* Allows participation in projects without requiring permissions from project authorities, instead of requiring "committer" status

Graphical User Interface

Mercurial comes with great desktop integration for Windows and Mac, using a tool called Tortoise. Once installed, it adds a right-click context-menu, allowing one-click commit, file diff, sync, directly from the file explorer.

Tortoise includes a green icon next to all files that are up-to-date and an orange icon next to files that have been updated and need to be uploaded.



How to install Mercurial

An all-in-one Mercurial+Tortoise installer is available here

Once installed, Mercurial will add a right-click context-menu, as shown in the image above.

Before you move forward, it is important that you setup a Username, so that all Commits you upload, include your name.

Right click and select 'Global Options' from HG menu. Then enter your full name under Username



Next - click on the 'Sync' tab and select Update as the 'after pull operation'



Connecting to a live repository

Create a new folder on your machine where you'd like to save all source files, right click and select 'Create Repository Here' from Mercurial's menu



This will create two files under the current directory:
.hg subfolder
.hgignore text file

To avoid any conflicts, go ahead and delete the .hgignore text file.

Now that the repository is created, we can download all files from the server. Right click again and select 'Synchronize' from the menu.

This popup window will open:



Enter the live repository url and click the 'Pull' button to download all source files from the live repository.

Note: You can repeat this process of synchronizing with the live server, whenever you'd like to download updates from the server.

The first time you synchornize, it's going to take a while. But subsequent pulls will be very quick.

Updating Files

Step 1: Open the local copy of the file in your favorite editor, make changes and hit Save.

Mercurial detects the change and automatically updates the file icon to orange.

Step 2: Once you're done making changes, right-click and select HG Commit.

Enter a comment describing the changes you've just implemented. For example:
"Project 384283 - Added opt-in form on instant pay increase"

Good comments are the basis of a solid version control system. Your comments help other developers understand the nature of your changes, as they download them to their local repositories.

Step 3: Unlike a centralized cvs, the 'Commit' operation is a local one. Once changes are committed to your local repository, Right click and select 'Synchronize'. Enter the live repository url and click the 'Push' button. This takes care of uploading your changes to the server.

Adding new files to the project

In order to add a new file, you have to take one additional step -

Highlight the new files, right click and select 'Add Files'



Then, follow the steps listed under "Updating Files", committing changes and synchronizing with the server.

View 3 Comment(s)

vsftpd is a very light-weight, full-featured FTP server that can be installed and ready to use in only a couple of minutes. It's used by many huge FTP sites like RedHat, Debian, GNU, ISC and many more!

NOTE: you can use this tutorial to install vsftpd on almost any UNIX/Linux distribution, except for the step 4.

1. Installing vsftpd

Installing vsftpd is very simple. First, you need to download and extract it:


Then, you're ready to compile it:


And then to install it:


2. Configuration

vsftpd uses the /etc/vsftpd.conf configuration file. A good idea is to start with the default one:


Here's a complete reference guide to customize each options of vsftpd.conf:

http://vsftpd.beasts.org/vsftpd_conf.html

3. Start/stop vsftpd

To start vsftpd, simply run:


To stop it, you have to kill it:


4. Start vsftpd automatically on boot

To start vsftpd automatically on boot, simply create the /usr/local/etc/rc.d/vsftpd.sh file, change its mode to 755, and put this content in it:


Conclusion

vsftpd is faster than proftpd, easier to install and to configure. It's very secure, stable and flexible: try it and give me your opinion about it!

As part of this post I will walk you through the process of setting up shared hosting, using an NGinx web server, ProFTPD ftp and Bind DNS.

Before you get started, you're going to need three things:

* Domain name to setup vhost under
* A dedicated ip address to use for the vhost
* Select username and password for ftp

Step 1 - Point to DNS servers

Login to your domain registrar (GoDaddy, eNom etc.), click on the designated domain and then update the DNS servers managed by this domain to DNS servers that you control.

Here at SPI, we use AlwaysUpDNS, so the two DNS servers to setup with the new domain, are going to be:

dns1.alwaysupdns.com
dns2.alwaysupdns.com

Once you update the DNS servers, it may take anywhere from 1 to 48 hours for the changes to propagate.

To check whether or not DNS propagated, you can run this command


Step 2 - Setup DNS hosting

Add new DNS hosting entry for the designated domain.

Here at SPI we use a custom version of Bind, so the way this is done is:

(a) Create a copy of a template 'blank' DNS hosting record


Replace mydomainname with the domain name

The end file should look like this:


(b) Edit /etc/alwaysupdns/mydomainname.dns in your favorite editor and update the A record to point to the new dedicated ip address (replacing 1.2.3.4).

Also while you're at it, replace all occurrences of 'mydomain' with your domain name.

You can leave everything else as is.

(c) Edit /var/named/etc/namedb/named.conf and add this block to the end of the file


Replace 'mydomain' everywhere with the domain name.

The zone-trafser details, make it possible to use DynDNS as a secondary level dns server.

(d) Restart Bind DNS by issuing:


(e) Test Bind is returning the correct response:


Step 3 - Add vhost record to Nginx

Open the NGinx config file (/usr/local/nginx/conf/nginx.conf) in your favorite editor and add this section to the bottom of the file, right before the closing }


Replace 1.2.3.4 with the new dedicated ip address

Replace mydomain with the new domain name and create a new folder under /home/mydomain.com to host the new site

Once done, tell NGinx to reload its config file, by issuing


Step 4 - Create FTP user

First we have to create a system user, by issuing an adduser and providing a username and password you've preselected for this website.

Be sure to specify the folder as /home/mydomain.com (replacing mydomain with the chosen domain) and enter ftponly as the new user shell.

For example:


Step 5 - Jail FTP user

To prevent the FTP user from gaining access to any other folders on this server, we need to jail the user to the home directory for the new domain.

Open ProFTPd config file (/usr/local/etc/proftpd.conf) and locate the block that begins with DefaultRoot

Add this line below, replacing mydomain with the new domain name


Restart ProFTPD by issuing


Then connect to the new site via FTP and verify it is working properly, not letting you access any parent folders above the current directory.

Step 6 - Update permissions

The last step is to update the new folder permissions, so that our FTP user will be able to create / delete files.


Replace mydomain with the chosen domain name