Finding rogue access points on Microsoft’s network has become an important mission inside the company.
Wireless gadgets have come to all but dominate the IT world. Employees within Microsoft, and many other large corporations, regularly bring in their own wireless devices. Bringing in a home office wireless router or a wireless speaker system might seem harmless, but such “rogue access points,” or rogue APs, pose serious security risks.
An unauthorized user could be sitting in the parking lot and you just knowingly or unknowingly gave them access to the corporate network.
– Pete Fortman, principal engineer, Microsoft
In the case of a wireless router designed for home use, it might have a default password that is literally “password” or the name of the brand. That could give drive-by hackers’ easy access to an enterprise’s network.
“An unauthorized user could be sitting in the parking lot and you just knowingly or unknowingly gave them access to the corporate network,” says Pete Fortman, a principal engineer for Microsoft who focuses on security.
The danger of rogue APs
Once inside, bad actors can wreak havoc. They can steal intellectual property, flood a network with useless data, or set up conversations between people who think they are talking to each other when they are in fact talking to the attacker.
Beyond that, rogue APs can interfere with legitimate wireless traffic—often simply by competing for airtime with the unwanted device. “It’s like a conference room with 18 seats, but 50 people are in the room and they’re all trying to stream something wirelessly,” Fortman says.
Microsoft has fought to keep rogue APs off its network for years. But as devices become more complex, they’ve also become more difficult to detect.
To combat that, Microsoft is now has applying machine learning and other advanced techniques to track them down.
When Microsoft began examining additional telemetry to find rogue access points in 2019, Fortman was surprised by what was uncovered.
“We had rogue devices all over the place,” Fortman says. “We kept the data private for a while to prevent adversaries from knowing what we can and cannot detect. When the data was shared more broadly, there was a collective gasp when people realized what was going on.”
Tracking down rogues
Obviously, rouge AP access point vulnerabilities are not good at a company that relies on “Zero Trust” to ensure security.
An engineering team within Microsoft Digital, the organization that powers, protects, and transforms Microsoft, took on the challenge of identifying and removing rogue devices.
Finding rogue access points posed a substantial engineering challenge. Potentially thousands of devices might be on the loose in the corporate network, from a wide range of manufacturers all using different wireless protocols.
“Gathering all this information into one place was a feat unto itself,” says Vincent Bersagol, a senior software engineer for Microsoft. “And we had to do it twice for two different data sets. Then we had to correlate the data sets together, and then look at suppression technology.”
Microsoft’s data tools, such as Microsoft Power BI, Microsoft Azure Data Lake, and Microsoft Azure Synapse, played a key role in collecting and correlating the data. “That was a great way to visualize all this data for folks to have a look at it,” Bersagol says.
Microsoft’s expertise in machine learning also proved helpful for finding rogue access points and was used to sort through the correlations between wired and wireless devices.
“We used a clustering algorithm that allowed us to tease out all the media access control (MAC) addresses that were statistically related to each other in a way that humans couldn’t see,” Bersagol says.
Many access points have common identifiable designs that can be determined by looking at multiple sets of network telemetry, including the MAC addresses. Finding these identifiable designs began with a manual examination of already discovered rogue APs. Requiring a sample of every type of rogue AP to generate a manual identification to find new patterns was not going to scale.
But collecting all the wired and wireless telemetry to hunt for new rogue AP designs isn’t enough. “That’s too much data for humans to sift through,” Bersagol says.
Instead, it’s possible to run a script that matches the two telemetry sets across all machines encountered. If any wireless and wired data were found to be correlated, the odds were very high that they came from the same device—a rogue AP. Further confidence that a rogue AP has been tracked down is gained when the correlated addresses come from within the same building.
So far, so good.
But some devices may have designs that elude direct correlation using the existing telemetry. Fortman says that by using additional telemetry sources they’ll be able to unearth more currently undetectable devices.
Still, even finding the simpler devices yields an impressive collection.
In October 2019, for instance, a sweep of about 100 buildings on the Microsoft campus unearthed more than 1,000 rogue APs.
Pulling the plug
It’s possible to take a very fine-grained approach to finding rogue access points and booting them off a network, such as assigning traffic through their ports to a virtual local area network (VLAN), or by blocking the devices’ MAC addresses.
In this case, Microsoft opted for a somewhat sterner approach: Shutting down a port connected to a rogue AP. This proved simple and effective, and safer than trying gentler approaches.
There is what Fortman calls “collateral damage,” because when a port is shut down, its user may lose network connectivity for other devices in their office, and Microsoft loses visibility to anything connected to that port.
“Shutting down a port is a basic capability of wired access,” Fortman says. “Ideally, we would like to leverage network access controls to just block the MAC address of the targeted device.”
COVID-19 plays a role (of course)
The COVID-19 pandemic that erupted in early 2020 had several impacts on the team tasked with finding rogue access points. Perhaps ironically, many rogue devices disappeared from the network—their owners were working from home.
The disruption also challenged some of the engineers working on the problem.
Blaze Kotsenburg, a software engineer, began work on the project in June 2020—his first month with Microsoft (he also was an intern for the same team in 2019). But onboarding, meeting new team members, and getting up to speed on the rogue AP project all took place over Microsoft Teams.
Working with two teams in parallel worked even better because of the remote situation.
– Diego Baccino, principal software engineering manager, Microsoft
“I couldn’t go to my mentor Vincent and ask him for a 15-minute whiteboard,” Kotsenburg says. “I’d work on something for a few hours, then ping him and say, ‘Hey, I need some help.’”
The entire team found new ways to collaborate and recreate the in-office dynamic. In fact, says Diego Baccino, a principal software engineering manager, the virtual work environment helped create a single team, rather than one team led by Fortman and one by Baccino.
“Working with two teams in parallel worked even better because of the remote situation,” Baccino says. “If I were to do this over again, I’d put even more emphasis on communication between everyone involved.”
Baccino hopes to retain this strong collaborative stance when employees return to the office.
Security is a constant challenge, as new threats emerge and old ones find new ways to cause problems. For Microsoft, preventing unwanted intruders is a top priority, and now one more avenue that bad actors might use has been turned off with the help of some digital sleuthing.