Your site just went down. Don’t panic. Think. Could be anything, but you need to start somewhere. Check the database? Check DNS? Scratch that, get out a checklist.

Atul Gawande’s “The Checklist Manifesto: How to Get Things Right” has stormed through popular culture leaving a lot of heads spinning in its wake. The premise is all the more enticing for its simplicity – can something as simple as a checklist reduce common errors and let us focus on the important stuff? Mr. Gawande details the use of checklists in the operating room, where they have slashed complications from surgery more than 30% in some hospitals. That impressive track record is bound to draw attention from leaders across many industries. Reading between the lines I found some important lessons to consider before adopting checklists at your workplace:

1. Checklists Must Become Part of Your Routine

If you rely on yourself to remember to use a checklist you probably won’t. You must change your processes so that checklists are fully integrated. Consider adding it into your production deployment scripts so that you have to work through the list before you can push code. Or consider making it part of your daily scrum so that everyone’s on the same page. Whatever the case, however you manage to do it, you can’t rely on human intervention or it’ll fall out of practice.

2. Checklists Really Matter When Mistakes are Very Costly

Before you go checklist crazy make sure a checklist is the right fit for the type of problem you’re trying to solve. It turns out that the value of a checklist is directly tied to how costly the mistakes it prevents are. A pilot who loses lift at thirty thousand feet doesn’t have a lot of time to debug the situation. While not as life-or-death, running through a simple checklist before restoring a backup might go a long way to ensuring that you don’t clobber your user table for good.

3. Checklists Always Involve At Least 2 Parties

A checklist works best when there’s someone reading from the list (e.g. the copilot or the nurse) and someone else performing the checks (e.g. the pilot or the brain surgeon). It’s kind of like Extreme Programming in that way. So you may consider having your QA team be involved in a pre-launch checklist. Or you may have your tech leads involved in a pre-vacation checklist to ensure all work has been handed over cleanly. Whatever the case, splitting up the work between two people helps focus each on his or her role in the process.

So back to your site. Thanks to your finely tuned checklist you’re reminded to check for recent production pushes on a related but separate project. They didn’t mean to do it, but thankfully the checklist helped you recover quickly. Time to go update your unit tests.

For anyone other than my mother still reading this blog, I’ll be blogging to support R/GA’s new techblog. I’ll point to any posts I author there when I get around to it. Since this blog is all about me, it’s pretty much just a complicated way to bookmark my R/GA posts.

To get it started, here’s a repost of an article R/GA helped me publish through infoTech

This article was originally posted here.

Enjoy!

It seems trite but it’s important to consider how technology enables things in ways that you can’t predict. Back in my TSN days I had an idea for a game that would be played during a sporting event. Since most of fantasy sports takes place before or after the event, it made sense that someone would try to fill that gap.

The problem was that I envisioned it as a web-based game because there was no such thing as an iPhone back then. Turns out a web-based game that you play while watching a sporting event on your television at home isn’t that appealing to people. Especially if it’s a NASCAR game (because at the time there were questions about whether MLB would win the CDM case). But with with the advent of the iPhone that no longer sounds insane. As a matter of fact it kinda sounds interesting again. If you had a very lightweight game mechanic (think predictive questions) and some fun side-effects, you could probably make a decent fantasy game out of just that. Food for thought.

As I mentioned previously I’ve been involved in the Winter 2009 semester of the Founder Institute in NYC. I thought I’d write a little bit about what I’ve learned to help others get the most out of the experience. Some of these are practical, some of these are just things to keep in mind as you progress through the semester. In no particular order….

• You’ll get the most out of it if you have an idea of what you want to do before you start. To its credit, the Institute will accept you if you don’t have an idea, but to some degree I think this is giving people just enough rope to hang themselves given the aggressive nature of the curriculum. The fact of the matter is that those of us who got the most out of the Founder Institute came in already pretty much knowing what we wanted to do and spent a lot of time tuning and focusing it. Yes, the first couple of sessions will be about how to develop an idea, how to research the opportunity, etc. I found the information to be very useful, but more in a “good to know for next time” way. It’s really hard to keep up with the later sessions if you spend the first 3 or 4 weeks vetting 5 different ideas.
• Therefore, getting in is about you as a founder, not your idea. But staying in is about how well you execute on your idea. Don’t forget that you are expected to have an incorporated company with a real business model and a real plan for success at the end of this. It’s not as easy as it looks.
• Actively seek out those who will challenge your assumptions sooner rather than later. It took me a while to realize that those who didn’t want to talk about their idea (including myself) were doing so because they didn’t want to hear the hard truth about what someone else thought. I know it’s not easy to hear. We’ve all convinced ourselves that whatever we’re thinking of doing can’t possibly fail and that anyone who doesn’t see it that way is wasting our time. That’s expected. But keeping it to yourself doesn’t prove anything. You join this program because you want your idea to succeed.
• Sign up for office hours. It’s free advice from people who have done this before and want to help you succeed. If you’re serious about building a business there’s no excuse for passing that opportunity up.
• Keep an open mind at all times. That pain you feel when someone criticizes your idea – that’s called growth. No one’s going to plunk down a bag of cash for a clever idea. At times someone “just won’t get it”. When that happens you can either find a way to convince them or become defiant and disengage. You can probably guess which one will work in your favor long term. Take the feedback (good and bad) and get better.
• That being said, know when to take advice and when to leave it. It’s your company. Don’t ever forget that. It was your company before the Founder Institute. It will be your company after the Founder Institute. If something really matters to you, stand your ground. Realize that you may still be wrong, but at least you learned something valuable.
• Volunteer to be president of your working group. People act like being the president of the group is like getting the plague or something. It’s not that bad. For the most part you’re just organizing meetings and taking notes. Yes, there are some founders who can’t keep up and you have to find a way to motivate them. It can be time consuming in a lot of ways, but it’s good practice and there are a number of upsides that come with the role. I was president for the 2nd and 3rd groups and found it to be valuable.
• Time is short, be ready to work. Kinda goes without saying. There’s not a lot of time in the program and a lot to do. If you’re going to do this right it’ll need to be a priority in your life for 3 or 4 months. Make sure your spouse or significant other understands this as well.
• Network, network, network. Get to know the other founders. Go out for drinks after each session. Find out about their business. Tell them about yours. Organize on LinkedIn. Connect with mentors where appropriate. These people have a stake in making sure you’re successful and you have a stake in making sure they’re successful. A little extra effort will pay off for everyone.
• The working groups can make or break your experience. A good working group can help you in more ways than you expect. A bad working group can really tax your energy. If things aren’t going well in your group, talk to your president. If he or she doesn’t listen talk to your local facilitator. Don’t let it slide. It’s up to you to get as much value as you can out of the experience, and every week counts.
• The mentors are great but aren’t everything. I came in expecting a one-on-one mentorship. It was a bit naive. The mentors are both successful and busy. They are all nice, and all helpful, but don’t expect too much. Honestly I don’t think I connected on a personal level with many of them. Part of it is that none of them really know my space very well, and I can’t really blame them that for that. It might speak more about my company than I want to think about.

Feel free to ask questions in the comments. If appropriate I will contact you directly via the email address you supply. As I’m still participating in the Founder Institute I’ve tried to avoid any conclusive statements. There’s still a lot of time left in this semester and a lot has changed even since my last post.

We recently bought my mother a digital photo frame for her birthday. The product itself was really nice – connect via USB, WiFi, flash cards, etc. Shows pictures, plays movies, plays music. Perfect gift for a tech-savvy mom who has a growing family.

But the real star of the show was actually Picasa. If you don’t know, recent versions of Picasa will actually scan faces in your photos and present them to you so you can provide names. It does the hard work, and you can coach it along by accepting its suggestions or providing a correction. It then indexes your images so that you can find all images with Aunt Sally, or all images with your brother *and* Aunt Sally.

With this it became trivial to preload my mom’s digital photo frame with family photos. Go through the index for reach family member and copy over some memorable shots. Easy and amazing.

I know it’s not new, and I know Google didn’t do it alone, but this seems to me to be a major milestone for the digital era. Wheels were important, and a lot of things can be done with a wheel, but history truly took a turn when someone strapped a motor to 4 of them and made a car. I feel the same way about digital cameras now. Important, but the dawn of facial recognition on the desktop makes this something else. I can only imagine what comes next. I’m sure someone out there has been spending a lot of time on exactly that. I can’t wait to see it.

One of the things I love about being a software developer is the fractal nature of our work. When we design a system we are almost always taking some piece of the universe and attempting to deconstruct it and model it so that it can run inside a computer. Examples of good (or bad) design are all around us, and our work demands that we draw on these examples to create a working piece of software. And software itself is nothing more than a bunch of bits and registers and some electricity that’s pretending to be more than the sum of its parts.

So I found myself reading Coders at Work on the 8:06 train the other day. I don’t usually catch the 8:06. The 8:06 is a double-decker train. And watching the conductor come through to collect our tickets I realized he represented a real-world example of a mutex.

This day there was only 1 conductor for both levels of the double-decker. It dawned on me that it would be very easy for someone to avoid having to pay by hanging out in the upper level and waiting for the conductor to collect the tickets from the lower level, then sneaking down to the lower level while the conductor moves to the upper level.

The lone conductor represented a flawed algorithm. There was no lock on the resource (exit door = I/O stream?). Adding another conductor could solve the leakage problem and lock the resource. But that would limit (or serialize) the free flow of passengers to and from the car.

I could probably go on exaggerating this example for while but I think you probably get the point.

As I’ve mentioned before, if you develop web sites for a living and haven’t read High Performance Web Sites yet you should be ashamed of yourself. The book’s title unfortunately includes the words “Front-End Engineers” in it, which will cause it to be tuned out by many back-end developers. That’s a mistake on their part. The book does contain information on best practices to improve the experience of a visitor to your site, but many of these solutions require the active participation of backend developers. Other solutions are just important for backend developers to be aware of.

Around the same time the book was released the fellows at Yahoo released the Yahoo Y Slow Plugin for Firefox. It requires the Firebug plugin, which all serious web developers should have installed anyway. The plugin will give you a grade on your compliance with the rules – 0 to 100, just like grade school.

My goal is to have each page in my site score at least 90 in the Y Slow rankings (again, just like grade school). This isn’t terribly hard to do if you’re disciplined. I run a Y Slow check on my pages infrequently to verify that I’m maintaining that goal. So I was a little ticked to see the home page of WhizKidSports.com take a hit when I decided to show the DigiCert badge I purchased (see related post here).

The issue was that 2 images included by DigiCert’s JavaScript. Y Slow was complaining that neither had a far futures expires header or ETags configured. That left my score south of 90, so I decided I’d fire off an email to DigiCert customer support asking if there was any way I could convince them to fix it on their side. I wasn’t expecting much, but figured I should give it a shot anyway. That was at 1am Sunday morning.

Around 11am that same morning I got a response from the CTO of DigiCert, Paul Tiemann. Cool fact #1 – the CTO of DigiCert is scanning customer service emails at 11am on Sunday. Seriously.

He profusely thanked me for noticing this and suggesting it to them. Cool fact #2 – the CTO of DigiCert was willing accept suggestions for improving their service from one of their clients. Seriously.

He got it immediately. As he pointed out, following Y Slow rules not only help visitors to my site, it also reduced bandwidth costs for DigiCert. So he had reconfigured the servers to address the issue. Cool fact #3 – the CTO of DigiCert is still close enough to technology to know how to configure ETags and expires headers on the production servers. Seriously.

I told him that I ran the site back through Y Slow and the news was good. I was back above a grade of 90. And, thanks to this tremendous example of a good business run by good people, I’m a proud DigiCert customer for life.

I wanted to do a couple of smaller posts around my efforts to obtain and make effective use of a secure certificate for WhizKidSports.com. The smaller posts will let me expand on some of the finer points where those familiar with the process might be able to give feedback.

The first task was to select an SSL issuer. I narrowed my choices down to 2 – GoDaddy.com and InstantSSL. I was leaning towards InstantSSL until I found a chart that shows the SSL issuers for Y Combinator companies. This had some value to me because I figured these companies are generally at a similar place as my company in terms of size and technical requirements. Strangely, after seeing the adoption rates of Godaddy and Comodo (who runs instantssl.com) being two of the top ones, I decided to go with DigiCert anyway.

In terms of GoDaddy, I generally just don’t think too highly of them. I use them for domain registration, but otherwise I tend not to trust them. They’re a little spammy, and I’ve read articles and blog posts over the years with people who have gotten the shaft because of their policies and practices. Few of these articles tend to be flattering. Also, they have a reputation for bargain basement prices and a ton of questionably valuable products. This is something of the antithesis of what I want people to think when they see a secure certificate on my site.

In terms of Comodo, I found the array of products to be a red flag. I was looking at the InstantSSL product, which seemed to suit my needs. The price was reasonable. But something nagged at me. The only differences that I could detect between this product and the InstantSSL Pro product (which is $25 more per year) is telephone support and a larger warranty. Honestly, I don’t expect to need either, but the point was that I also don’t tend to trust companies who invent arbitrary reasons to justify price differences between very similar products. The other research I turned up was good but not incredible, so I didn’t feel they really closed the deal on my business.

And I know this doesn’t have even close to anything to do with the quality of the product, but both GoDaddy and Comodo suffer from psychotic web page design syndrome (that’s a topic for another post). In short, I’ve learned that a company’s home page is usually the best indicator of the soul of that company. Call it crazy.

Whatever the case, I finally decided on SSL Plus certificate from DigiCert. Maybe a little more expensive, but still reasonable. The reviews I found were glowing. And once I saw their instructions for installing the certs on all major web servers – including nginx, I was sold. After I went through the typical purchase flow a real human contacted me for some documents to verify my ownership of the domain. As soon I got them what they needed they issued the certificate. It all went incredibly smoothly and professionally. They even had a cool little wizard that generated the appropriate OpenSSL command to run on the command line. Not essential but nice.

So far so good with DigiCert. Next up I’ll discuss installation, which hit a few tiny snags but was also pretty painless.

(See Part II of this series here)

So right now I’m looking at a table that has at least 3 different columns that control whether the particular row is displayed on the front end. In some cases that’s unavoidable, but it has to be kept in check.

Maybe you can tell me what the difference is between the intent of these columns: status (e.g. pending, active, canceled) and should_display (0 or 1). In addition to that, there’s one part of the code that will ignore a record if one of the FK columns is null but will consider it if it’s non null.

This is madness. I now have to piece together which columns are significant to which consumers of the data. And then I have to figure out the magical combination of values to make the row appear on the front end. This leads me to some quick rules for database flags:

• Limit the number of display flags to as few as possible. I usually use a is_active or display_order column to determine whether the row should be retrieved. There will be cases where the row should be retrieved by one consumer and not another, but there should never be more than one column that does almost the same thing.
• Use descriptive column names. The ones above are too general. is_active tells me exactly what I need to know.
• You can use a nullable timestamp column to do both boolean checks and date-triggered checks. In other words, if the column is null it means the column is still valid. If it’s not null you have to check it against the current timestamp. This saves a duplicated column and is fairly easy to get across.

If there’s one concept I’ve fought my entire career it’s that there can be, or even should be, a way to make everything work “automagically”, a term the afflicted developers use lovingly. I recently christened this the “Magic Button Syndrome”.

Usually a bunch of fairly smart developers sit in a room and start dreaming of how a system might work. “We have to make sure we can easily modify the configuration,” one might say. “We should have a means to generate the configuration based on some other configuration file,” another might respond. “Let’s use annotations to make sure that the configurations stay in sync across versions,” someone else might suggest. Yet another person might think it’d be wonderfully cool if you could auto-inject annotations somehow.

Their triumphant moment comes when the CTO is standing over their shoulder screaming about something that needs to be fixed ASAP and they nonchallantly say, “Oh, I can fix that, one second.” They turn to their machines dramatically, edit one or two lines somewhere, smack the return key, twiddle their thumbs, reload the page, and then smile. “No big deal,” they’ll say with a smirk on their face. That’s it. That’s what they live for. They want that one Magic Button moment.

It sounds foolish, but there are plenty of developers like that out there. For these people, it makes perfect sense that if you can automate something little, automating something bigger containing tons of moving parts must be even better. Eventually the automation will reach singularity in the Magic Button.

The problem is that automation suffers from the same law of diminishing returns as does traveling at the speed of light. It takes an infinite amount of energy to accelerate a particle with any mass to the speed of light. In the same way, it takes an infinite amount of energy to create that Magic Button. Not that it stops people from trying. Sure, changing one of the thousands of options that are contained in a config file or database is easy. But if you’ve worked on systems like these you know that doing anything outside of the realm of what the system was designed to do is absolutely, unbearably painful. That Magic Button hides layers of abstraction upon abstraction upon abstraction. Just when you begin to understand what a peice of code does you realize you forgot what code is calling it. In the effort to make something of uber-value, no single component makes any sense.

You find it takes people months to really understand the system. Changes take weeks to test, and lead to reprecussions that no one really ever expected. Once all the original developers are gone everyone starts to realize the system needs to be redesigned. It’s become like the pyramids – beautiful, absolutely brilliantly designed, but a total mystery. This time we’ll do it differently. In Ruby maybe. And auto generate all the documentation using XML…