Archive for May, 2011

On Mace Windu’s Two Catastrophic Errors

I probably spend way too much time thinking about this, but every time I watch either Episodes 2 or 3 I’m left with the feeling that the key tragic sequences in the series both involve Mace Windu.

The first one is the most obvious. In Episode 3 he had a chance to kill Palpatine straight up but hesitated, leading to his bungee jump out of the Chancellor’s office without a bungee cord after Anakin’s betrayal. Cool cat, but died like a punk really.

The less obvious one is his beheading of Jango Fett in Episode 2. Don’t get me wrong, the Geonosis sequence is probably one of the best fight scenes in the prequels. But if you think about it, Jango was the weak link in all of Palpatine’s plans. There were probably only 3 characters in the whole galaxy that knew the details of Palpatine’s gambit to rule the galaxy by playing both the Republic and the Separatists against the Jedi. The first was Palpatine himself. The second was arguably Dooku, who lead the straw-man Separatist movement until Anakin relieved his shoulders of his head. The last had to be Jango, who was the source of the clones used by Palpatine as well as Dooku’s bodyguard on Geonosis. Capturing instead of killing Jango could have unraveled the whole plot.

But even still, it strikes me as unbelievable that at least Obi-Wan, if not Yoda and Mace, wouldn’t have been able to connect the dots on Jango. Obi-wan confronted him on Kamino and then sees him again on Geonosis. It probably doesn’t take too much brain power to wonder why the guy’s popping up on both sides of the fence.


The Future is DotCloud

Something about the Donors Choose Hacking Education contest appealed to me so I started brainstorming ideas. After a eureka moment (more to come on this later), I was determined to dig in and start coding. With all the hubbub about fan favorite Slicehost dissolving into Rackspace in the near future, I thought the time was right to explore other options.

Enter DotCloud. Similar to Heroku, PHP Fog, or even Google App Engine, DotCloud is a platform as a service. Like Heroku and PHP Fog it’s backed by Amazon AWS. Unlike those other platforms though DotCloud is committed to supporting a variety of open source languages, databases, and message buses. Not just Ruby. Not just PHP. Don’t even get me started about some proprietary environment.

Love. It.

You can mix and match these environments to suit your taste. Want a PHP server with a MongoDB backend? No problem. Spin up a PHP instance and a MongoDB instance and then configure your PHP connection settings. Need some memcached? No problem, throw that in as well. All the instance types are specialized but interoperable, usually requiring nothing more than a little config to connect the dots between the components.

I’m thoroughly impressed with DotCloud. For a developer there’s much to like, and maybe a few concerns to express. Here’s where I stand:

The Good

  • Easy setup: It’s not an exaggeration to say you can have a site up in minutes.  Their tutorials are clear.  I control the code locally and then push it up to DotCloud when it’s ready. They deploy it automatically for me. Perfect.
  • Easy management: I can’t tell you how much time I wasted worrying over minutia in my Slicehost setups.  What Linux flavor and version should I choose?  What ports should I open?  How many user accounts should I create?  Where should I put my web root?  Is my nginx config optimized?  Is the box locked down enough?  Although it was a great learning experience, the fact of the matter is that I’m not and never will be a sysadmin.  I’m a developer.  The good people at DotCloud are (hopefully) competent sysadmins.  Let them worry about it.  Me, I’ll just focus on the code.  Mission accomplished.
  • Truly technology agnostic – Here’s my favorite line from the help docs: “Q: Do I need to use Git to use DotCloud? A: No. We want to help all developers – and not all developers use git.” Fuck yea. Makes Heroku seem snobby by comparison (git only). Look, I know what they’re hoping to accomplish isn’t easy. The reason why Heroku is so popular is that it’s made for Ruby geeks by Ruby geeks. It’s not easy to perfect one language or platform, much less multiple ones. And let you mix and match them at will. But that’s where the next point comes in.
  • Great tools: They’ve hit the perfect balance between “we’ll handle that” and “you can do that if you need to”. They’ll set up a MySQL instance for you. But if you need to get in there to hack some data, you can do that. You can deploy without needing to ssh into your web box, but you can if you want to. Flexibility is the key here. I was thrilled to find that I could customize my nginx.conf file to clean up my CodeIgniter URLs (removing index.php). Last night I spent about 20 minutes configuring an SMTP instance so it will automatically use a gmail account for relay when I call mail() from my code. And yes, you can set up cron jobs. They even provide an SSL cert if you need it (as long as you use their domain). It seems they’ve thought of everything.
  • Command line: Their single command line utility “dotcloud” has everything built into it, but everything makes sense. The commands have become 2nd nature to me after only a few weeks of usage. They nailed it.

The Bad and The Unknown

  • Support: I had a problem pushing some files to my instance recently. Being a beta service, I waited about 6 hours before contacting them about it. There was no notice on their site, and my email went unanswered. But eventually the issue did resolve itself. Still, a little scary. I would expect them to be on top of stuff like that.
  • Background processes: I’m still not really sure where the best place to run background processes is. I typically like to hack up some Perl to do batch updates and stuff like that. Although I can ssh into the instance and set up cron to run some random Perl script, I don’t have permissions to pull down new modules from CPAN, so I’m SOL to some degree. For now I’m running the background processes on a Slicehost box.
  • Pricing: Right now it’s Free (as in Beta). There’s no indication of what the eventual pricing model would look like. My main concern here is that for my small app I’m using 4 separate instances (php, mysql, static, and smtp). A per-instance pricing model is going to kill me.
  • Virtual hosts: Related to pricing, I’m not clear how I would set up virtual hosts. On one of my Slicehost slices I’m hosting 3 instances of WordPress and a random other site, all on unique domains all with the same nginx configuration. I’d hate to have to manage 4 separate instances for this.
  • Scaling: Everything seems to work nicely in a single-server environment. What’s not clear to me yet is how they will handle horizontal scaling of front end web servers running the same code. I’m confident they’ll figure this out, but hopefully it won’t complicate their architecture too much.
  • Performance: Since they’re EC2-backed I’m not as concerned about web server performance. My concern is more about internal network performance, i.e. web server to database. I’m not even sure how to monitor this or what can be done to improve it. If they’ve done their jobs right this shouldn’t be an issue, but it’s likely to come up at some point.
  • Monitoring: I haven’t found any built-in tools for system monitoring yet. These would be nice but again, since it’s EC2-backed it’s possible you can use third party tools for this.
  • Source Control: It might be nice if they offered a git or mercurial instance type so you can keep your code in the cloud as well. I would consider it but I’m not sure how many others would. Right now I’m using CodebaseHQ, which is a poor man’s version of FogBugz/Kiln.

The Future
I’m becoming more comfortable with the idea of deploying a “real” site on DotCloud. Although I don’t imagine R/GA’s clients selecting it any time soon, it seems ideal for everything from rapid prototyping through early startup. Migration to another platform should be simple because you’re not locked into any proprietary languages or workflows. DotCloud is the next logical step in the evolution begun by Amazon years ago – commoditize what can be commoditized, customize the rest. I’m sold.


Tron is My Copilot

Your site just went down. Don’t panic. Think. Could be anything, but you need to start somewhere. Check the database? Check DNS? Scratch that, get out a checklist.

Atul Gawande’s “The Checklist Manifesto: How to Get Things Right” has stormed through popular culture leaving a lot of heads spinning in its wake. The premise is all the more enticing for its simplicity – can something as simple as a checklist reduce common errors and let us focus on the important stuff? Mr. Gawande details the use of checklists in the operating room, where they have slashed complications from surgery more than 30% in some hospitals. That impressive track record is bound to draw attention from leaders across many industries. Reading between the lines I found some important lessons to consider before adopting checklists at your workplace:

1. Checklists Must Become Part of Your Routine

If you rely on yourself to remember to use a checklist you probably won’t. You must change your processes so that checklists are fully integrated. Consider adding it into your production deployment scripts so that you have to work through the list before you can push code. Or consider making it part of your daily scrum so that everyone’s on the same page. Whatever the case, however you manage to do it, you can’t rely on human intervention or it’ll fall out of practice.

2. Checklists Really Matter When Mistakes are Very Costly

Before you go checklist crazy make sure a checklist is the right fit for the type of problem you’re trying to solve. It turns out that the value of a checklist is directly tied to how costly the mistakes it prevents are. A pilot who loses lift at thirty thousand feet doesn’t have a lot of time to debug the situation. While not as life-or-death, running through a simple checklist before restoring a backup might go a long way to ensuring that you don’t clobber your user table for good.

3. Checklists Always Involve At Least 2 Parties

A checklist works best when there’s someone reading from the list (e.g. the copilot or the nurse) and someone else performing the checks (e.g. the pilot or the brain surgeon). It’s kind of like Extreme Programming in that way. So you may consider having your QA team be involved in a pre-launch checklist. Or you may have your tech leads involved in a pre-vacation checklist to ensure all work has been handed over cleanly. Whatever the case, splitting up the work between two people helps focus each on his or her role in the process.

So back to your site. Thanks to your finely tuned checklist you’re reminded to check for recent production pushes on a related but separate project. They didn’t mean to do it, but thankfully the checklist helped you recover quickly. Time to go update your unit tests.