Recent Posts
Sponsors
 |
SoftLayer down for 2 hours due to a power outageKate Richards, 09-04-2009 |
Earlier tonight, at 1:30am EST, SoftLayer - one of the tier 1 data centers we use, experienced a power outage in their primary facility, rendering thousands of sites dead.
SPI engineers were alerted to the outage within seconds (we take monitoring very seriously). After establishing that the core problem was a total data-center failure, we verified load balancers are properly redirecting traffic to other locations.
The Power Outage lasted a total of 2 hours and 3 minutes according to our friends at Pingdom:
All SPI client sites are multi-homed, which means we host everything in two independent data centers: Rackspace, Softlayer, NTT Verio and iWeb.
SPI utilizes redundant global load balancing, to immediately redirect end users from west coast to east coast, when such global outages happen.
How it works
New end users hitting your mydomain.com are diverted to active servers.
Old users / scripts (which keep a cached version of your website ip address) continue getting routed to the old server until their local DNS cache is cleared.
What's Next
While the DNS ttl (time to live) is setup as 60 seconds, some browsers and ISPs don't honor that and can take up to 2 hours until the DNS entry is refreshed.
As a workaround for these type of catastrophes where active users have cached a malfunctioning server's ip address, we'll be configuring the load balancers to reply back with all available server ip addresses, letting an end-user's machine rotate ips internally.
We understand the importance of keeping your sites up at all times and although our current setup ensures 100% uptime for new users, we are determined to come up with a solution for users with a cached version of a malfunctioning server ip address.
Your SPI account manager will contact you with any specific steps you have to take, in order to enable this setup for your domains.
SPI engineers were alerted to the outage within seconds (we take monitoring very seriously). After establishing that the core problem was a total data-center failure, we verified load balancers are properly redirecting traffic to other locations.
The Power Outage lasted a total of 2 hours and 3 minutes according to our friends at Pingdom:
All SPI client sites are multi-homed, which means we host everything in two independent data centers: Rackspace, Softlayer, NTT Verio and iWeb.
SPI utilizes redundant global load balancing, to immediately redirect end users from west coast to east coast, when such global outages happen.
How it works
New end users hitting your mydomain.com are diverted to active servers.
Old users / scripts (which keep a cached version of your website ip address) continue getting routed to the old server until their local DNS cache is cleared.
What's Next
While the DNS ttl (time to live) is setup as 60 seconds, some browsers and ISPs don't honor that and can take up to 2 hours until the DNS entry is refreshed.
As a workaround for these type of catastrophes where active users have cached a malfunctioning server's ip address, we'll be configuring the load balancers to reply back with all available server ip addresses, letting an end-user's machine rotate ips internally.
We understand the importance of keeping your sites up at all times and although our current setup ensures 100% uptime for new users, we are determined to come up with a solution for users with a cached version of a malfunctioning server ip address.
Your SPI account manager will contact you with any specific steps you have to take, in order to enable this setup for your domains.
|
|
Subscribe Now to receive new posts via Email as soon as they come out.
Comments
Post your comments
