How the President Used the Cloud to Beat Romney (Part 2)

Gathering Clouds: Obviously, the groundswell during the last campaign was a very palpable cultural moment, and a lot of the technologies that you were using this time around weren’t as widely available last time. How did these technologies change? What was ultimately the benefit for you guys?

Dylan Richard: The ’08 campaign is widely talked about as having been incredibly good at using technology, and one of the things that we set out to do in this campaign was to be good at building technology to try to continue the work that had been happening since ’08 at the DNC. We tried to continue that and use the ’12 campaign as an opportunity to build a battle-tested infrastructure.

The Obama Tech Team (From left to right, Harper Reed, Dan Wagner, Dylan Richard, and Andrew Claster. Photo by Bjarne Jonasson, courtesy of Time Magazine)

None of this would have been possible without a mandate from the top. This whole thing happened only because Jim Messina [President Obama’s campaign manager and former White House Deputy Chief of Staff] and the President saw the value in building a data-oriented campaign and building technology for the democratic process. So there was, from the campaign side, a desire to push in that direction, of building sustainable technology.

It is also four years later. Frankly, four years ago, I don’t know if AWS would have been an obvious choice. The cloud has grown up a great deal and the world is actually different from then, from a technology standpoint. Four years ago, Facebook was less than half the size it is now. On Twitter, we had a tenth of the followers that we had this time around.

The technology is remarkably different. The options that were afforded us this time made it so whereas things that seemed like obvious choices just weren’t possible in ’08. Frankly, AWS being one of them.

GC: What’s your perspective then on Do-It-Yourself (DIY) vs. as-a-service? A lot of the ’08 approach was to use what was available, not create from the ground up. But as someone who’s involved in this space more broadly, what is the inherit benefit you see to doing something DIY vs. doing it as-a-service. Many businesses use software-as-a-service (SaaS) or infrastructure-as-a-service (IaaS) or platform-as-a-service (PaaS) to great benefit. But what characterizes your belief in doing it yourself and customizing it to that degree?

DR: It’s really a matter of what you deem as important. For what we were doing in this specific case, we needed to be close to the cloud metal, as it were. We needed the ability to say, “I know that this is normally how it goes, but we actually need to spin up 10X in a way that if we were using something off the shelf or a little more “as-a-service” as it were, it would not be as feasible.

AWS: Providing a blend of DIY and As-a-Service tools that helped the campaign support its infrastructure.

That being said, we didn’t go all in on DIY – we didn’t go and get our own servers. We leveraged all of the things that we did not invest in our team. So, we invested in the infrastructure and used it in such a way that it is redundant and scalable. It is all of these things because we made it so. After all, you can just as easily use Amazon and do it poorly and not have all of that.

For what we had, we needed to push the blend a little bit more towards DIY. Depending on what fits everybody’s needs, there are certainly times where PaaS is absolutely the right thing to do, and so DIY fits more correctly than it did. In our case, there are platforms that can account for that kind of scale change. For example, there are no software systems that say, “Okay, we’re going to go from X users to 1000X tomorrow and then, the same thing the next day.” So because we needed to build towards that and hit that, we needed to own as much of that as we could.

GC: So what happens to the technology now?

DR: Well, most of it is still up and running.

GC: What are your perspectives on how technology for this campaign has changed how the next election cycle is going to look? Obviously, we can’t fully anticipate the next four years, but what’s your perspective on what this means to the future of politics in America?

DR: For both its impact on campaign politics and campaign connectivity, you can see an evolution, and the continuation of that evolution just makes sense. The campaign tech team in ’08 was focused more on using services and adding little bit of glue where they could since they didn’t have the time that we had – I mean we had the luxury of spending 18 months building stuff.

This time around, the next step in that evolution was to build a much larger team and spend a lot more time writing software and building infrastructure to power the campaign. I don’t think that trend is going to change. I don’t think that it’s possible that the next time around, there’s going to be a smaller set of technologists working on this.

On one hand, yes, what we accomplished certainly changed the way that technologists are viewing politics and political technology where hopefully it’s shed a little light onto some of the interesting challenges that exist and more and more people will be involved in building political technology. But also, I think that campaigns have been evolving towards having more technology, having more analytics and more data and more “connectedness” and I think that will continue in the future.

GC: So what does this mean overall? There’s always this focus on best practices, but in an evolving landscape, it seems to me that best practices are dictated by certain brands that own a market space, to a certain extent. But for you, in this field of political technology especially, are there now best practices? Do you guys own that or does that phrase even apply anymore?

DR: I think there are certainly best practices and the best practices are not actually unique to politics. I think that they are a lot of the same best practices that you would have anywhere, where when you are dealing with technology, failure is always something that can happen. So what does that mean? It means you need to be redundant, you need to have failovers, and you need to have backups. And you need to have all those things regardless of whether its politics or another industry, nothing changes those needs. The only thing that it changes is the things that are important. So when you’re dealing with e-commerce, if things are getting difficult for whatever reason, you want to make sure that your payment stack stays up. Then, after that, you want to make sure that your catalog stack stays up. Beyond that, whatever social features or anything on top of that, need to remain functional. You have the same hierarchy of things that you need to keep up in politics, it’s just different things for each of the groups in the organization, but the technology and practices are the same.

GC: The cloud space is very dynamic. There’s a lot changing. As you mentioned, there’s quite a bit of difference between four years ago and today. How do you view the choices that you guys made in the broader landscape of the brands that are available? Is the mentality one of being in a race to pick the best technology combinations available, or do you view what you choose as more cultural fit, or is it purely determined by the needs of your goals?

DR: I think it’s sort of a twofold consideration. On one hand, there’s a very strict or very tight coupling to the need and goals. On the other, as with any technology decisions, at some point, that decision is no longer going to be right. As soon as you write code, it’s technical debt. As soon as you invest in some infrastructure, it has some cost and it will serve you for some time, but it needs to be able to evolve. So I think that the choices that we made for this worked out great, but we need to make sure that we are building everything with an eye towards evolution because the choices that we make now need to evolve into better choices in the future, simply because the landscape changes so much.

The work of the Obama Campaign’s tech team paid off: The President was reelected.

GC: So the result comes in. Obama is declared the winner. What was your reaction at that moment? What did you do the next day?

DR: So at that moment, honestly, it was something very close to confusion. At the point he won, we had been doing “Get Out The Vote” initiatives for four days straight and I think that I had slept about maybe nine hours over those four days combined. Our 40-person team had spent 18 months building up to this moment and when it happened, I was actually on my way to the McCormick Center in Chicago surrounded by a team of amazing technologists and amazing campaign people. So, it was a sort of mixture of exhilaration and exhaustion. We had been so focused on the finish line, that once we had passed it, it was like, “Oh, now what?” And then it was like, “Oh, everything is amazing.”

So that’s the night and the next day. Honestly, we went into work to kind of figure out what was going to happen next, and scale some things down, since we’d scaled up quite a bit. We change things from “Help Get Out the Vote” to “Thanks for Your Support,” and tried to figure out what the next steps should be. And then President Obama walked in.  He talked to us, and that video has circulated around the Internet. It was incredibly moving and the perfect apex to incredible effort. Then, he gave each one of us a hug.

Dylan Richard, formerly the Director of Engineering for Obama for America, has an impressive track record in the IT space. Having worked with a number of startups, most notably skinnyCorp (where he worked under Harper Reed, later CTO of Obama for America), Dylan is currently the founder of his own company, Appropriate, LLC.

This segment is part 2 in the series : How the President Used the Cloud to Beat Romney
1 2