Then there was that time that I helped Randy Pitchford bring down Gearbox, for a little while at least. I play a serious amount of Borderlands 2, you see. Maybe not in the top percentile of hours played, but with three of five characters maxed out at level 50 and 300+ hours invested, I’m not exactly a noob. I mentioned this to Pitchford during a chat in his palatial Hard Rock Hotel & Casino villa at the 2013 D.I.C.E. Summit and he was immediately inspired to tweet out a Golden Key code.
“Doing an interview @geminibros who has played over 300 hours of Borderlands 2. This inspires me to drop a SHiFT code. Thanks @geminibros!”
Minutes later, Pitchford’s phone buzzed with an incoming text. A frantic series of messages from the Gearbox SHiFT team. First, “Don’t tweet a code right now!” Then, “Updating service, almost done.” Finally, resignation. “Oh well, guess you had it all queued up.”
Pitchford proceeded from there to explain the narrative behind those three texts. Gearbox’s Golden Keys are made possible by the stat-collecting framework that launched alongside Borderlands 2 last year, Shift. It’s a still-developing beast, so there’s very much a situation at the studio of finding and addressing a new set of challenges each day. The disruption on the consumer side is minimal, but chaos internally breeds some scary-yet-valuable lessons.
“There were probably thousands of people pinging the server as Martin was updating,” Pitchford explained. “This is a good lesson for us, because there’s a human failure there of the Twitter feed not saying ‘Hey we’re gonna update something. Stay tuned.’ This is a problem we have not experienced yet. It was just created spontaneously because I had an idea.”
“Hopefully people don’t get too upset at me that the Shift codes might not be working. They get my tweet, they drop everything that they’re doing, they go and put the disc in, and then the shit’s down and they’re like, ‘Gahhh! Randy! I dropped everything I’m doing because of this and now it doesn’t even work!’ I probably just made some enemies. That was not my intent, I was trying to do good.”
This, he says, is one of the dangers inherent to the high-velocity culture that exists at the Gearbox offices. It’s also just as much a vitally important philosophy to embrace in an effort to please a fanbase that is constantly craving more. Making mistakes is a natural product of this breakneck pace, but addressing those mistakes in a smart way is just as fundamental to the Gearbox philosophy. Go fast, always, but don’t be afraid to fail again and fail better.
“We are going very fast, and the feeling that we’re getting from our hardcore customers is we’re not going fast enough.”
“We are going very fast, and the feeling that we’re getting from our hardcore customers is we’re not going fast enough. The key is when we make mistakes, we want to be able to correct them and we also want to be able to learn from them so that we do not make the same mistake twice.”
The high-speed approach isn’t without its risks. Look at the Mister Torgue’s Campaign of Carnage DLC for Borderlands 2. It’s not offensively bad, but it falls short in a number key spots, one of which is the abundance of motorcycles in the new environments that are entirely off-limits to players. The same goes for the vanilla game’s flying Buzzard gunships. These aren’t bugs; they’re features that perhaps weren’t as fleshed out as they could have been.
“That’s a mistake,” Pitchford admitted. “In Borderlands 1 you’ll notice there’s nothing that ever gets off the ground. In Borderlands 2 they really wanted to do things like the Buzzards and I kept saying ‘It’s a mistake. You can’t show me something like that and not let me get in it.'”
“Sometimes the motivation to play with this stuff in one context overcomes the realization that it will create expectations that can’t be fulfilled, and that’s unfortunate. Sometimes we can’t help ourselves. One of the neat things about Gearbox that I love is, we don’t look at our designs as a totalitarian regime and we’re all really happy to let designers and creators within the studio explore in our space.”
That sense of experimentation isn’t always a bad thing, however. Badass Ranks were something that Pitchford specifically campaigned for, and the profile-wide upgrades that players can earn have been a huge hit. Then there’s also the DLC, every iteration of which continues to test the waters in terms of fleshing out the Borderlands 2 endgame. Even while work continues on making the level cap boost a reality, players are still being given new systems to learn and toy with in each piece of add-on content.
“I feel like there’s a lot of work to be done there to perfect that,” Pitchford said of the ongoing endgame fiddling. “When we race ahead, when we run fast on ground we’ve never run on before, sometimes we stumble, and we’ve got to just pick ourselves up and look at our feet a little more, run a little better. I love that we did that, and that [Gearbox producer Mike Wardwell] and I really pushed for that. Because there’s something there, it’s getting more momentum within the studio and I think it’s going to lead to some things that are going to be even better and better and better as we apply our lessons.”
Undercooked content is one issue that sometimes crops up in Gearbox’s fast-paced work environment, but even something as simple as a small coding error can trip things up. The team has had to develop certain processes and techniques that allow for a rapid response. In Borderlands 2, that boils down to the implementation of pre-patch hotfixes.
“One of the things we invested in heavily right after Borderlands 1 launched was the development of an online platform that would allow the game to be connected to us. We wanted our online infrastructure to allow us to iterate and interact with the game to be ubiquitous and to be multi-platform,” Pitchford explained, laying out the broad strokes for one component of the company’s Shift infrastructure.
“We have this system now where we can deploy updates and fix flaws in real time through hotfixes,” he continued. “When you start the game and press the Start button, we are streaming new software to your game, changing the way the code works. It’s temporary, it’s a hotfix.” He went on to describe a specific case in which, shortly after launch, players discovered that an Assassin class mod included a bonus for a Gunzerker skill. Clearly a bit of mistaken coding on the dev side.
“The moment we noticed the bug, it was actually very easy for us to fix. We fixed it in seconds or minutes and then recompiled, and it’s fixed. But we can’t deploy it [as a patch], because there’s 7-9 weeks of lag sometimes between when we have something and when it can go through quality assurance and be submitted for certification. It’s excruciating when you know you can fix a flaw in moments and it can take [multiple] weeks for the player to get the benefits of that fix,” Pitchford said.
“With a hotfix we can deploy it instantaneously. The catch is, it only works if you’re online. When you start the game, it gets that software, updates, and fixes the flaw, and it stays in memory while you’re playing. When you quit out, the memory is cleared. It’s gone. It’s not a patch, it’s a live fix because our servers are actually affecting what’s happening on your machine in real time.”
Hotfixes are more common than you might think in the video game industry, especially when you’re talking about a studio that has the resources to invest in such things. Having the immediacy of a fire-and-forget safety net is a huge boon in Gearbox’s fast-paced work environment, however. It encourages the sort of experimentation that Pitchford espouses, and it creates a situation in which bug and stress testing becomes a possibility post-release.
“The evidence is: quality is what people buy. What do you know?! Do something good, and people want that! Do something less good, and people want it a little less! Who knew?!”
What’s neat about Shift is that, as young as it is, certain ideas have developed organically and grown into fan-favorite components of the player community. Take Golden Keys. According to Pitchford, the in-game consumables obtained by typing in Shift codes were conceived as a testing tool, but they’ve since grown into something much more valuable.
“I think we’ve gleaned almost all of the functional value out of using Golden Keys, but in doing that as a functional testing system we’ve created an ecosystem there that we didn’t anticipate,” Pitchford said. “We’re going to continue to serve that.”
Serve that he did during our D.I.C.E. chat. It makes for an amusing anecdote, but it also highlights perfectly both the strengths and the weaknesses of Gearbox’s high-velocity work ethic. A minor disaster ensued, something that I later learned never ended up impacting the player community, but a lesson was gleaned and processes subsequently evolved. Gearbox might not be perfect, but that isn’t the goal. This is a team that is constantly trying to one-up itself in a race to please a fanbase that constantly craves more.
They key, as they’ve learned, is really rather simple: quality sells. “In Borderlands 1, the best-selling DLC was [General Knoxx]. That was the one that everybody thought was the best,” Pitchford said. The sales numbers for both games in their vanilla form alone speaks volumes. “The evidence is: quality is what people buy. What do you know?! Do something good, and people want that! Do something less good, and people want it a little less! Who knew?! There’s no trend lines that work in entertainment. You can break any trend line by offering value that we as consumers of content want.”
