In the course of Defiance’s development, there have been two Presidential elections, three Olympics, the economy of the world nearly collapsed, and the government has passed countless laws. Well, a few laws. Well, they’ve been there. Then in April the big day came. After a half-decade of hard work, Trion Worlds was finally ready to release its ambitious new game to the world. It was the first of its kind on multiple levels: the first MMO developed jointly for both consoles (excluding the Wii U) and PC, the first third-person shooter MMO, and the first game to be made in conjunction with a television show. Defiance is an ambitious game, made with the blood sweat and tears of hundreds of people. It was just a shame that no one could play it at launch.
“When the tidal wave of humanity hit us, we needed a bigger boat…”
“Tuesday the second, [during the] mass hit, people started stressing out, and smoke started coming out of the servers and all this sort of stuff, and it was like a triage scene from a military movie where they set up the hospital,” Nick Beliaeff, Senior Vice President of Development for Defiance told Digital Trends. “There’s clusters of people talking about issues, it’s noisy, there’s high energy, and it’s like ‘We gotta get shit done.’”
MMO launches are a bit like children. They are unpredictable and wild, occasionally destructive and thoughtless, but most of them inevitably grow out of it. Rough MMO launches are so common that most developers launching one don’t even bother to hope for the best, they prepare for the worst and then work to correct the thousands of problems that they couldn’t test for.
“I can give you a phone book of the things we were prepared for, but that was all the stuff that went well,” Beliaeff stated.
A studio can spend years testing every conceivable problem that could plague an online game, but there is no way to test what will happen when tens of thousands, even hundreds of thousands of gamers all jump online at once. Even when a game has an alpha test lasting six months and an open beta running for three months, as Defiance did, that can’t accurately predict the weight of gamers in a full launch.
“We had, or have so many more people playing at once than we did even in the beta – and we had a huge amount of people in the beta – that a lot of the issues that we had were things that only scale can bring,” Beliaeff said.
I didn’t help that much of what Defiance was attempting had never been done before. Being a pioneer is great in theory, but being a trendsetter also means you are the first ones to face the problems confronting you. This type of game simply wasn’t technically feasible even a few years back, so there is no precedent. And that means more problems.
“We’re doing so many things that haven’t been done before,” Beliaeff told us. “There was no roadmap for us to look too – like ‘Hey, this is a clever company and we like their games. How did they deal with that issue?’ We didn’t have the opportunity in a lot of spaces, so there was a decent amount of trial and error.”
That painful process began on April 2. Trion knew there were likely to be complications, but early tests had been promising. The game actually had a soft launch on Monday, April 1, and the assembled engineers, designers, and dozens of others at Trion stared at computer screens, monitoring every piece of raw data generated in the game. What people were playing, where they were going, the stability of the servers – it all looked good. Most left work that night still somewhat nervous, but generally content with what they saw. Then the proverbial fan took a hit.
“When the tidal wave of humanity hit us, we needed a bigger boat,” Beliaeff said.
“It’s so rare that you can say that you did something first, and it’s not a lot of hyperbole or marketing bullshit…”
To its credit, Trion was well aware of this, and also understood that the relationship between an MMO developer and a gamer goes beyond the traditional model. Gamers are investing in more than just a game, they are investing in a long-term hobby. The best MMO can keep people playing for years, so there is a certain amount of trust and respect that has to be reciprocated. Trion understands this, and rather than hiding behind a litany of press releases like a certain other MMO recently released that faced problems (it’s SimCity, if that was too vague), the developers were completely open about the problems and what they were doing to fix them.
“I should mention that I have been feeling sad lately, then immensely happy, then frustrated but in the end I’m excited and ecstatic that I’m here,” Nathan Richardson, the game’s executive producer said in a blog post before going into the technical issues they recognized and what they were doing to fix them. “Because this is what it feels like launching this little game, an emotional rollercoaster.”
At Trion the scene was tense, but employees were determined. The teams of engineers were joined by the Platform teams – the groups tasked with handling account authorizations, logins, and microtransactions – and the Net and System Ops teams – the people in charge of the hardware. All told there were over 200 people working behind-the-scenes on Defiance immediately after the launch.
The weekend presented Trion with a new influx of gamers, and an even heavier load on the servers. Hundreds of problems, big and small, had been addressed since the Tuesday launch, and hundreds more were created by the new mass of gamers. Once again, the servers went down.
The triage mentality continued throughout the weekend, with the game’s engineers facing the worst of it, putting in consecutive 20+ hour days. Some would go home, shower, eat, take a short nap, then return. Others would just sleep in the office. It got to the point that the managers had to force people to go home.
“We wanted to have a pristine first weekend, and we didn’t,” Beliaeff said. “Everyone was so excited for launch and so excited to be done, and they really wanted everyone to enjoy all the work they had been doing. And it was not optimal.”
The problem was that there was no one fix that would correct everything, nor was there just one problem that needed to be solved. When corrective patches came, they were part of a trial and error system. One fix might alleviate some issues but not others, while some could create even more problems. The resolutions came incrementally and slowly.
“A lot of the feeling was that we don’t have a magic answer for this, because we’ve never seen it before, because we’ve never had this many players playing at once,” Beliaeff said. “That’s why we started opening the dialogue with the players. We figured the more open we are about what it is that we’re doing and why we are doing it, the more understanding they will be. Not everyone is going to listen, but those that listen like having dialogue.”
The general tone of the comments was actually very understanding and appreciative. Gamers are typically a smart lot, and most would rather have an honest bit of bad news than a false reassurance. It always backfires to try to lie gamers. Always. The dialogue continued through the weekend, and is still ongoing.
It wasn’t until around the middle of the game’s second week when Trion began to breathe easy. Not everything is flawless though, and the development process continues. There will be several “quality of life” patches to improve non-essential things like clunky menus and item management, but the majority of the fires have been put out.
“From a stability standpoint and things like that, I think we’re in a pretty good space,” Beliaeff said. “ Are we perfect? No, there’s still issues there. But have we taken care of the big ones? Yeah, absolutely.”
Trion isn’t releasing the numbers yet beyond confirming that the Xbox 360 version is the most popular, but it did say that the game’s sales exceeded expectations. With the TV show ratings looking good as well – the show’s debut was SyFy’s most-watched scripted show in seven years – the experiment is looking like a success, and memories of the triage-like first week should fade from the minds of gamers. Trion, however, will remember.
A team of full time employees in the Network Operating Center monitor Trion’s games constantly out of Texas. When they go home, another team from Russia takes over and watches for issues in the game and looks for customer complaints to pass on the developers. More than 40 people have the job of making sure Trion’s games function properly.
Defiance did something no one else did. In fact, it did several things that no one else did, and that will benefit gaming as a whole. The lessons learned will be valuable for any company looking to emulate Trion’s formula, and even as the next gen comes the problems solved in Defiance will be used in all similar games going forward. And despite the rocky start, Trion was there first.
“It’s so rare that you can say that you did something first, and it’s not a lot of hyperbole or marketing bullshit,” Beliaeff stated. “It’s nice to be able to hang your hat on that. But oh my God, it’s hard. Hard things are definitely worth doing.”
(Images © 2006-2013 Trion Worlds, Inc. All Rights Reserved)
