Moving Tripleseat off Amazon EC2 – Drawbacks and Lessons Learned

April 7th, 2009 by kevin

Way back in August of 2008 when we launched Tripleseat, our web-based restaurant CRM and event management software, we chose Amazon EC2 as our hosting platform. A few weeks ago, we moved it off of Amazon and onto one of the the top Australian web hosts, and our other favorite hosting service, Ipage. Before I go into why we switched, here are the advantages and disadvantages of EC2 from our perspective.

Advantages of EC2

We chose EC2 primarily because it would give us the ability to have automated and cheap server scaling. Specifically, you can create and shut down server instances through the EC2 API. This allows you to programmatically scale your web application from zero to any number of servers based on necessity (if you got techcrunched, for instance, you’d probably need a lot) and you only pay for what you use. If you’re interested in more details about EC2 scaling, check out Dusty’s presentation.

Other good things about EC2:

  • Ability to build an image (AMI) of an existing instance. Once built, you can create any number of clones of your existing instance in just a few minutes. Since it’s a copy, you don’t have to install OS libraries, packages, patches, or otherwise on the instance. Very groovy.
  • Very active developer community around hosting on EC2. There are a lot of people using EC2 and building stuff around it.
  • It’s very affordable and competitively priced. One small EC2 instance runs about $100/month if you keep it on all the time.

Drawbacks and Disadvantages of EC2

Here are a few of the disadvatanges we’ve seen and experienced with EC2.

Server Configuration Woes

As it turns out, it can be a real pain in the ass to set up and configure a Rails app in the cloud all by yourself. We leveraged a really sweet Rails plugin called EC2OnRails, which automates the setting up an EC2 instance for Rails hosting. EC2OnRails, doesn’t, however, help you do any sort of automatic dynamic scaling. For that, you’d need to go for something like Scalr or RightScale.

Unrecoverable Instances

If you manage to mess up your instance’s boot sequence, networking or routing configurations, or SSH server, you can easily exile yourself from an instance. More specifically, you won’t be able to log into it and fix whatever you broke. Your only options are to get support from Amazon (generally unreliable and slow from our experience) or build a new instance. Yeah, I had this happen to me a few times – one time in particular it was as a result of installing some innocent Ubuntu OS updates. That was a real bummer.

Random Loss of your Instances!

We’ve never had an instance vanish on us; we’ve actually had a few instances that have been running for upwards of a year now. However, there is always the looming possibility of one just going poof and being gone forever. So, if you want to roll in the EC2 cloud, you need to be able to quickly rebuild a lost instance.

What Caused us to Move Tripleseat

When we launched Tripleseat, we really had no idea how many users we’d have on the site all at once and how much they’d tax the server. After selling the app for several months, it became apparently that Tripleseat’s growth was going to be fairly steady and predictable. No techcrunching or YouTube-esque traffic patterns. As a result, the primary reason we chose EC2–the ability for Tripleseat to scale an infinite number of servers–wasn’t justified. 37Signals basecamp app grew in a similar fashion:

“We ordered a dedicated server from our original web hosts at Tilted and turned it on. Basecamp ran on that single dedicated box for a year. One server, one year.” -Jason Fried

Hosting Your App on One EC2 Instance is Dangerous

Back in December, our Tripleseat EC2 instance disappeared from the Internet for around an hour. There was no reason for it–we just couldn’t access tripleseat.com–and neither could our customers. Our recovery strategy was to launch a new instance from a prebuilt image, deploy the code to it, and populate the new database from backups on Amazon S3. As it turns out, this strategy means you are destined to have downtime for at least a while. We lucked out, though, as our original EC2 instance magically reappeared. But it was certainly a scary situation, and our customers were less than happy.

Overall Lessons Learned

  1. If you don’t know how much traffic or bandwidth your app will require, go with a more simple and secure hosting solution. You can always ramp up. (Slicehost, dreamhost, mediatemple are all good.)
  2. If you want to roll on the EC2 Cloud, don’t store data on the instance or, if you do, back it up frequently (like every 5 minutes). Additionally, make an AMI of your instance so they can be easily recreated and/or cloned.
  3. Unless you can dedicate quite a bit of time to configuring, running, and monitoring EC2 instances, I’d recommend a less risky hosting solution that provides immediate support and full, automated server backups.