Call us Toll-Free:
Email us


SoftLayer down for 2 hours due to a power outage

Kate Richards, 09-04-2009
Earlier tonight, at 1:30am EST, SoftLayer - one of the tier 1 data centers we use, experienced a power outage in their primary facility, rendering thousands of sites dead.


SPI engineers were alerted to the outage within seconds (we take monitoring very seriously). After establishing that the core problem was a total data-center failure, we verified load balancers are properly redirecting traffic to other locations.

The Power Outage lasted a total of 2 hours and 3 minutes according to our friends at Pingdom:


All SPI client sites are multi-homed, which means we host everything in two independent data centers: Rackspace, Softlayer, NTT Verio and iWeb.

SPI utilizes redundant global load balancing, to immediately redirect end users from west coast to east coast, when such global outages happen.

How it works

New end users hitting your are diverted to active servers.

Old users / scripts (which keep a cached version of your website ip address) continue getting routed to the old server until their local DNS cache is cleared.

What's Next

While the DNS ttl (time to live) is setup as 60 seconds, some browsers and ISPs don't honor that and can take up to 2 hours until the DNS entry is refreshed.

As a workaround for these type of catastrophes where active users have cached a malfunctioning server's ip address, we'll be configuring the load balancers to reply back with all available server ip addresses, letting an end-user's machine rotate ips internally.

We understand the importance of keeping your sites up at all times and although our current setup ensures 100% uptime for new users, we are determined to come up with a solution for users with a cached version of a malfunctioning server ip address.

Your SPI account manager will contact you with any specific steps you have to take, in order to enable this setup for your domains.
Enjoyed this post?

Subscribe Now to receive new posts via Email as soon as they come out.

Post your comments

Note: No link spamming! If your message contains link/s, it will NOT be published on the site before manually approved by one of our moderators.

About Us  |  Contact us  |  Privacy Policy  |  Terms & Conditions