Speeds & Feeds

By Jennifer Warnick and Lukas Velush

VANCOUVER, British Columbia – A small squad of Microsoft IT professionals has done for “Gears of War 4” what Delta Squad did for humanity. No, they didn’t fight off swarms of subterranean reptilian hominids, but in their quiet, geeky way, they totally saved the day.

It is late afternoon on a Wednesday at The Coalition, the Vancouver, British Columbia-based Microsoft gaming studio, and it is eerily quiet, save for the occasional ding of the microwave heating a late lunch and the distant buzz of—what is that, a chainsaw?

Correction: It’s the distant buzz of a Lancer, the signature weapon of the Gears of War franchise and a gloriously effective tool for dispatching the game’s hideous Locust Horde. It’s August, and with the biggest moment in the studio’s history mere weeks away, the Gears team is working with urgent, caffeinated intensity.

Gears of War 4,” the latest installment of the blockbuster series, comes out on October 11. It’s the first original game since Microsoft acquired the franchise in January 2014, and the folks at The Coalition are on a mission to prove to fans that the Gears universe is not only in good hands, but is more kick-ass than ever.

“No pressure,” jokes Joe Vogt, IT services manager for The Coalition. The name of the game studio, The Coalition, is a reference to the Coalition of Ordered Governments (C.O.G.), a political faction from Gears of War lore. Vogt and his team are the dedicated, in-studio IT professionals entrusted with supporting the production of Gears of War and protecting what he calls “the crown jewels.” The inhouse team at The Coalition works fairly independently from the corporate IT organization at Microsoft, something that could potentially cause friction.

“We keep the entire Gears of War franchise safe,” Vogt says. “A few wrong clicks and we could blow the whole thing away. The priorities here are all about keeping the game code safe, keeping people productive, and getting the new game out the door on time.”

Almost two years ago, as work on the upcoming game title ramped up in earnest, Vogt became increasingly focused on making sure the studio’s IT infrastructure would support the staggering demand.

“Necessity is the mother of invention around here. We always answer ‘yes,’ and then find a way to make it happen and deal with the problems as they come up,” Vogt says.

For starters, The Coalition’s headquarters in downtown Vancouver—though stylish—was not designed for the kind of networking it takes to support a major video game studio. To make sure Vogt could scale up to provide the oomph to power 700 Xboxes, 500 PC workstations, and 300 build and render servers for the teams who build the game, he had constructed a clever hodgepodge of network switches and servers (accompanied by a few portable air conditioners to cool the machines) and stashed them in random closets and corners.

700

Xboxes

+

500

PCs

+

300

Workers

+

“I was grabbing everything I could—recycled Microsoft equipment, network switches from decommissioned buildings in Vancouver—and there was still no way I could keep up with demand,” Vogt says.

Another problem: Designers and programmers often deploy builds of the game many times a day—massive deployments of hundreds of gigabytes, each of which takes an hour or more to transfer.

“It’s no good to have hundreds of people watching progress bars all day,” Vogt says. “We want our artists and programmers to spend less time waiting for bits to move and more time creating monsters and writing code.”

Vogt was ready to go to whatever lengths necessary to help The Coalition meet its ambitious timelines for building the new game. He knew where to find the networking expertise he needed: Microsoft IT. But he was reluctant to make the call to headquarters. Nearly as old as the ability to network two computers is the tension between the steadfast keepers of corporate IT standards and the passionate, protective squads doing specialized IT for product groups.

The Coalition, home of Gears of War
Vancouver, British Columbia.

“We have our own way of doing things, and our own trusted IT guys here,” says Vogt, who fully admits to being a tough customer. Vogt has worked in video games IT for 19 years, and his dislikes have come to include: waiting, asking for permission, waiting, being told how to run his highly specialized IT team, and waiting.

A perfect storm was brewing. It was a rock-and-a-hard-place moment for Vogt, a lone wolf who prefers to go his own way but who clearly needed help. He decided to take the leap. If this were “IT: The Video Game,” this is the part where Sean Adams and Trevor Clark from Microsoft IT would step dramatically from the mists of a forbidden-looking forest carrying tattered burlap sacks full of loot, stuff like Cisco Nexus 3172 switches and mile-long spools of 10-gigabit copper coaxial cable.

But production on “Gears of War 4” was already underway. Was it even possible to update the network in the middle of designing a major video game?

Spoilers dead ahead: Yes.

Vogt, Clark, and Adams—with wisdom from an IT brother-in-arms at Redmond-based 343 Industries, keepers of the Halo franchise—would go on to form an unlikely alliance and successfully land a daring digital transformation for The Coalition.

“There’s certainly no way we’d have been able to ship our product on time without this,” Vogt says. “It was basically the IT equivalent of swapping engines while the jumbo jet was mid-flight.”

Part 01

He Carried Our Flag

Sean Adams and Trevor Clark are the types you want there to dismantle the bomb—the guys with steady hands and reassuring words, but also the technical chops to know exactly which wire to cut as the timer ticks toward zero. When Vogt decided to enlist help from Microsoft IT, his first call was to the Canadian Microsoft IT Site Operations Team, where Clark is a senior field manager serving Western Canada.

Trevor Clark

Senior Field Manager, Microsoft IT

Sean Adams

Service Engineer, Microsoft IT

“There’s a reason they have local IT staff. The Coalition has some unique needs, and some pretty high requirements—higher than almost anywhere else in the company. But the hard cases are where the innovation starts to happen,” Clark says. “There can be a tendency for something like this to become one of those ‘us versus them’ things, but that didn’t happen.”

Clark’s team put Vogt in touch with Adams, a Redmond-based service engineer who manages Microsoft IT deployments large and small in Canada, the United States, and Latin America. Adams sensed Vogt’s reluctance, which only made him more determined to find a solution.

“In our early conversations, it felt like Joe was reaching out because he had to, not because he wanted to. There wasn’t a lot of trust that Microsoft IT could make everything work and fit the needs of the group,” Adams says. “But Joe knows his business better than anyone else, and he set the stage for us to understand how important this was, and how we needed to deliver something big, but with as little impact as possible.”

Though Adams has a team to help him plan and execute IT deployments, it was clear The Coalition would require some special attention. Adams started making the trip from Redmond to Vancouver, showing up in the hallways of The Coalition every few weeks.

“It seemed like he was one of us, and it was obvious he wanted the best for us,” Vogt says. “As an advocate, Sean was not only technical, but he went to battle for us. He carried our flag.”

Adams made commitments, and kept them. He asked questions. And he listened—really listened. Even so, the first several solutions weren’t the right ones.

“We’re generally hard and fast in our IT standards, and we stick to them, because of the sheer size of our environment at Microsoft. But in doing that, we also don’t want to miss the opportunity to engage with our product groups at a deeper level and give them the solutions they are looking for,” Adams says.

Luckily, Adams had been working on a similar infrastructure project with Dan Price, IT service engineering manager for 343 Industries. Though 343 is a much larger studio, both organizations had similar needs, including performance and speed.

“It’s like an amusement park. If there are 100 people in the queue waiting to get on a ride, maybe you need more rides,” Price says. IT service engineering manager for 343 Industries. “We always do the math—how much per hour does it cost while someone is waiting? Let’s go spend or build something to make sure everybody can be productive at the same time.”

With help from Price, Microsoft IT designed and built 343 Industries a high performance network featuring Cisco Nexus switching at each level of connectivity, the same design Microsoft traditionally deploys in datacenters where the focus is on raw performance over features.

Borrowing heavily from the design (and success) of the Halo project, The Coalition’s new network took six months to refine, install, and test. Adams even placed cameras in the server room so he could keep an eye on the new equipment from his desk 120 miles to the south.

“It was the best $100 I could have spent,” Adams says. “We could move the cameras with a web browser, and there was enough resolution to see where things were connected, to understand the ports and the sequence and how to move them around. It was a way for us to get real-time information without constantly asking Joe to take a picture.”

Adams’ willingness to show up, listen, and keep his commitments eased any tension or reluctance Vogt may have had, and a trusted, ongoing partnership was born.

Clark says this is something he’s seeing more and more in IT.

“I’m seeing the focus shift toward a different type of dialogue,” Clark says. “I think the door is definitely open, and we’re working with product groups in all new ways—not only the things we think they need, but things they want and can rely on us to help them get.”

Part 02

Culture as Catalyst of Transformation

Microsoft IT manages applications, infrastructure, and security—the same keep-the-trains-running and lights-on work that all IT shops do. But increasingly, the organization’s role as the first and best user of the company’s rapidly evolving products and services has it playing a pivotal role in helping the company transform itself, says Jim DuBois, chief information officer at Microsoft.

DuBois says his organization’s ability to test how Microsoft offerings work—and sometimes don’t work —together give it a unique footing in the company, especially since its product update cycles have gone from years to weeks.

“We’re working much differently now than in the past, and it’s creating some opportunities to be really disruptive,” he says. “It’s the combination of breadth and depth that gives Microsoft IT a unique position from which to help power the company’s digital transformation.”

But before IT can fully grow into this role of transformation influencer, it first has to undergo its own digital transformation, something that is underway now, DuBois says, pointing to the help that IT is providing to the Gears of War team as an example.

“The old IT would have forced a solution on them,” he says. “We would build something we thought they needed, and they would have had to find a way to make it work or work around it.”

That way of thinking is disappearing.

“This is a really important culture change for us,” DuBois says. “By culture, I mean relationships, vision, teamwork, communication—things of that nature. I've seen a dramatic change within the company and within our group, and we have to continue to get everyone on board.”

DuBois and his leadership team are living and breathing the change. At a recent meeting, his team picked songs they felt best represented the organization’s digital transformation. Some were humorously apt ("Video Killed the Radio Star”) but others were reflective.

“Multiple teams picked Michael Jackson's ‘Man in the Mirror’ to represent that the change starts with ourselves,” DuBois says. “And it really does. This transformation is something that every one of us has the opportunity to impact and embrace for ourselves.”

“We’re working much differently now than in the past, and it’s creating some opportunities to be really disruptive,” DuBois says.

Price and Vogt, admitted skeptics, say they give the organization props for taking the opportunity to do things differently and ultimately creating a new, more flexible IT ethos.

“I totally understand the need for an IT standard. When you run an infrastructure at scale across a massive, global company, you need it,” Price says. “But not every group is going to have the same needs. The biggest win from this project was building a new standard for users like the ones in our studios. Partnering with Microsoft IT to build this for our studios was a huge step forward.”

Recognizing and embracing that independent spirit is crucial to the company and IT, DuBois says. He called out how those operating on the fringe are often the ones who come up with big ideas that help mature companies like Microsoft keep reinventing themselves.

“Fostering IT innovation across the company, and increased cooperation amongst IT, engineering and business groups, will pay off for our customers,” he says. “We think of this cooperation as One IT.”

As for Adams, the culture change DuBois talks about became real when he was able to get Vogt and Price to trust him, and through him, the larger IT organization. He says the best part about the project was the relationship he built with the pair.

Apart from getting his speedy new network, Vogt agrees.

“I appreciated the knowledge, experience, and resources,” Vogt says. “There are so many smart people in IT with all of this tribal knowledge—how things work, who to talk to, who’s done this before. That’s where the value comes out.”

And that collaborative spirit extends to how The Coalition team works with 343 Industries and other big Microsoft gaming studios, including Turn 10 Studios, responsible for the Forza Motorsport series.

“There has been much better cooperation between our IT teams since we started upgrading our network, especially in the last year,” Vogt says. “The relationships between gaming studio IT groups is as strong as I’ve seen in the last five years. We engage with Microsoft IT more often and have more positive outcomes than ever before.”

He says they set up a Studio IT group email for day-to-day idea swapping, monthly calls to talk shop between just the three studios, and a quarterly cadence of in-person gatherings for bonding. Collaborating with Price and 343 Industries has been especially fruitful because they have both been going through some of the same changes.

“343 was able to loan us a large number of Azure-based servers to help us get through our heavy production period this summer,” Vogt says. “They also shared many tips and techniques with us—technical as well as operational—on who to call, how to escalate, and hard technology stuff.”

In turn, The Coalition was able to give back, sharing a multicast-based network transport system that it built for deploying large amounts of data to many PCs simultaneously.

Part 03

Speeds and Feeds

“This is holy grail stuff,” Vogt says, running his hand gently over neat loops of white cable in the Coalition’s updated server room in downtown Vancouver. “From an IT point of view, this stuff is beautiful—and rare.”

Each desk at The Coalition (and before it, 343 Industries) now enjoys multiple, dedicated, gigabit access ports that are not oversubscribed. As Price put it, it’s the difference between a bike lane and a wide-open expressway.

“Moving data around during game development, we used to have the equivalent of four to ten people trying to share the same lane during rush hour as opposed to having a dedicated eight-lane freeway for every user,” Price says.

The Coalition is now working at approximately 10 times its former speed. A game build that used to take 90 minutes to transfer now moves in the time it takes for a designer to stretch and get a drink of water.

“Speeds and feeds,” Vogt says, grinning.

34% complete Copying 1 item from gears of war to test folder 34% complete
Before
34% complete Copying 1 item from gears of war to test folder 34% complete
After

Good timing, too. As launch date nears, Vogt and his team have had to support a ‘round-the-clock, ‘round-the-globe development cycle; when the Pacific Northwest is sleeping, teams in Malaysia are hard at work, and when they go to bed, folks in England take over. A team has even peeled off to begin work on a future installment of the game.

After months—years—of being heads down, the new version of the game is starting to show up in commercials, magazine covers, and online teasers. Every time Vogt sees a mention in the wild for “Gears of War 4,” he runs into the office to make sure the backups are good, and to double-check that there are three copies of everything.

“I get nervous. So many people have put so much talent into this, and IT—we’re the ones who are entrusted to hold the thing in our hands and not drop it,” he says. “We just need to make sure nothing gets in the way of them hitting the finish line.”

He’s jittery, but he’s also proud.

“You’re always kind of hoping one day you’re going to work on something the size and scale of ‘Halo’ or ‘Gears of War,’” Vogt says. “If people ask us what we do, we say we make video games. Whether it’s IT, or artists, or programmers, we are all playing a role. There’s no wall. We’re all part of the same team here.”

Photos by Stuart Isett / © Microsoft

Share this article on:

initializing