Moved production environment from NJ to Texas... from Seattle

I just finished a major "kitchen remodel" of BigOven.com, moving the entire production environment (that's business-geek-speak for "all the electronics that that run the site") from New Jersey to Texas.  The old host was UplinkEarth, and the new hosting provider for BigOven is Rackspace. 

The time had come to make the switch, because UplinkEarth's reliability and responsiveness on the dedicated hosting side had suffered.  I wanted expanded storage capacity and a whole lot of bandwidth and backup.  While I was happy with UplinkEarth when I first started with them 3 years ago, I became increasingly disappointed in their fault-tolerance (there was a 4-6 hour site outage last year for many west-coast users due to their lack up backup connectivity).  Service responsiveness was suffering as well -- if I logged a ticket with them on Friday evening, it sometimes wouldn't be until Monday morning that I'd get any kind of response.  Simply put, banker's hours don't work for Websites, and I needed an alternative.

After much comparison and consideration I went with RackSpace, which has many locations worldwide but is headquartered in Texas.  They were roughly triple the price, but in the migration, I've also dramatically expanded the hardware and software configuration, and they offer a level of ticket-response, hardware setup, and customer service that is exceptional.  (They call it "Fanatical" customer service, and have implemented a very good ticketing system whereby I have to close out every service ticket with a grade.  So far, I'm about 92% "Fanatical", 8% "Very Good" in my feedback.)

My oh my was the transfer itself a hassle.  (But worth it.)  The basic method was to:

• Get the new boxes up and running with SQL Server 2005, ASP.NET, all the various code and all the third-party tools I've used to create BigOven.com
• Test the new boxes against the (old) live production server environment, to ensure the code worked
• Lots of firewall configuration issues on the new site
• Make a backup copy of the (old) SQL Server 2000 databases that run BigOven.  This includes all the recipe, ingredient, photo, and member data.  It's a modestly large set of data now, since BigOven has 60,000 users and 160,000+ recipes, and thousands of photos, etc. 
• FTP the data to the new server farm.  (This took forever, because it's several gigabytes of data.)
• Get the new SQL Server 2005 servers running with the snapshot of the data. 
• On the actual cutover day, we took the production environment offline, did a differential backup of the data, then moved this data to the new servers. 
• Put a "we've moved" sign on the old environment, with instructions on how to update the HOSTS file (while DNS takes a week or so to catch up across the Internet)
• Transfer all the photos to the new site
• Redirected all DNS servers for www.bigoven.com and the mail servers, etc. to the new site

During the transfer, there were lots of niggly little steps that had to take place.  For one thing, the SQL Server "Copy Database" wizard seemed like a good idea, but was useless because it did INSERTS on the data, and many of the BigOven tables have "identity" fields.  This kind of migration technique means that all the primary/foreign keys in the database get orphaned (e.g., because on the production environment, recipes, ingredients, users, etc. might have been deleted, so the primary key might be 1..2..3..6..8..9, and it would try to insert them into the new database as 1..2..3..4..5..6).  The right way to transfer the data turned out to be the backup method, and the production site needs to be shut down so that no new data is added to the database in the process.   

Another major hassle centered around deploying my SMTP/POP (email) server in the new location.  I was relying upon a shared SMTP server of UplinkEarth.  It's amazing how many settings have to be just right in order for inbound and outbound email to go through.  (Thanks, spammers!  You've created a fun world.)  Now, although I have a masters in computer science and worked for Microsoft and Expedia (for a total of more than a decade), I've never had to deal with things called SPF records, reverse DNS lookup entries, PTR records in a "zone file", etc.  What  a hassle!  Can't you just set up a server, tell it to be "bigoven.com", and you're done?  Nope!   Here's a very, very useful site to make sure your DNS entries are up and running:  http://www.dnsstuff.com

But it's all starting to settle in.  The result of this migration is a much faster and higher capacity site with much better 24x7 service levels.  I'm happy as a clam, and happy to see the site being very snappy.  One new feature that wouldn't have been possible without this migration is BigOven Video, coming soon to the site.  In all, the site was down for about 3 1/2 hours on a Tuesday late afternoon.  Longer than I would have liked, but not disasterous.

The fact that I could accomplish this entire migration without a plane ticket, managing it all from Seattle, when the production environments moved over 1,000 miles from NJ to Texas, is absolutely amazing to me. 

Comments

[S L Friedman]

Hi, Steve -

Interesting that you just switched servers due to reliability issues - I am completely unable to access BigOven.com (and I was all set to join up, too!).

Could you post a status update? I tried to ask RackSpace in a chat session if they could provide a status update on their servers, but they do not.

Thanks,

Lee

[Steve Murch]

Hi Lee,

Thanks for your note -- yes, we just had a momentary (2-5 min) downtime on the main site, but this was entirely our fault in a configuration setting. We've corrected it and the site is back online.

BTW you can follow site status updates at @bigovenstatus on twitter -- http://twitter.com/bigovenstatus -- thanks!

[S L Friedman]

Just realized that now - I can log in fine. What timing!

Thanks,

Lee