G&J Pepsi reflects on overcoming a ransomware attack, offers insights from the other side

“The heroics of our entire IT team brought us back from a cyberattack. No one would want to experience what we went through.”

Those are the words of Eric McKinney, Enterprise Infrastructure Director at G&J Pepsi-Cola Bottlers, the largest family-owned and -operated independent Pepsi franchise bottler in the United States. Happily, his dedicated team skillfully restored the company’s recent backups on Microsoft Azure Backup, avoiding what might have been a much worse nightmare when a ransomware attack struck. But McKinney can’t forget the 3 AM call days after the Labor Day weekend in September 2021, when his team member on the other end of the phone told him in what would turn out to be an understatement, “Something doesn’t look right.”

Resisting a Cobalt Strike ransomware attack

When McKinney’s attempts to sign into a few company virtual machines (VMs) from home were blocked, he realized that his user profile had been encrypted. His middle-of-the-night dash to the office sparked what would be a two-week marathon of mitigation by his entire eight-person team to recover from what would later be identified as a Cobalt Strike ransomware attack. Ironically, Cobalt Strike is a well-known and popular penetration testing tool that companies use to identify network vulnerabilities. By 2021, the use of this tool by ransomware gangs to infiltrate corporate systems had boomed in part because of its ability to mimic an actual installed tool. Malicious hackers gain entry to systems by deploying an agent, or beacon, on a specific device.

The company was one of many victims that year in what McKinney describes as a textbook attack. “It wouldn’t create a flag on most devices because it installs as a legitimate copy of monitoring software,” he says. “Our team cleaned it up, but we couldn’t find the beacons. Then the ransomware gang must have issued a kill command to start moving laterally.” While not a huge corporation, G&J Pepsi-Cola Bottlers had a sizeable attack surface to protect: 100 apps accessed by more than 1600 employees on about 2000 devices.

Arriving at headquarters, McKinney witnessed the progression as a growing number of files on the company’s servers became encrypted. Engaging with the hackers—much less paying them—was never an option for the company. An hour and a half after the ransomware was discovered, G&J Pepsi’s security team used Microsoft Defender for Endpoint to identify and shut down all its compromised VMs, isolating every device suspected of being targeted for lateral movement by the hackers. Rebuilding their servers began immediately. 

Thanks to expert operations management with Azure Backup, G&J Pepsi-Cola Bottlers had a path forward because it had weekly backup copies for every server. The company began with a methodical restoration of each server, starting with domain controllers and progressing to systems in order of criticality. That’s when the team found the culprit—a VM infected with the faux penetration tester. Fortunately, the attack was confined to a single server at that point.

As Mckinney puts it, “One of the advantages we had in our fight to overcome the attack was our ability to communicate with our organization. Because our communication and collaboration systems were already migrated to Office 365 and Microsoft Teams, we were able to let the organization know with real-time situation reports and next steps on our progress. Our ability to coordinate with various departments impacted helped speed up our recovery efforts and count on them for assistance to help recover.”

In the end, G&J Pepsi-Cola Bottlers managed to prevail. “We got our environment up and running in seven hours, thanks to the data we saved with Azure Backup,” says McKinney. “And we didn’t pay a cent to the attackers.” In fact, over 90 percent of the company had no idea an attack occurred and were not impacted by the threat. Some of the company’s most critical systems were older applications running Azure. G&J Pepsi-Cola Bottlers lost no data and were able to keep the refreshing beverages flowing. But as soon as normalcy was restored, McKinney and team began an intensified cybersecurity program, refining an already sophisticated security posture and extending the use of Microsoft Defender for Endpoint and Microsoft Intune.

The G&J Pepsi-Cola Bottlers IT team then took time out to reconsider its cybersecurity approach. “We were very reactive,” admits McKinney. “That’s what bit us when the ransomware attack happened, because we were only seeing the highest-level security events.” Now the team takes a proactive slant, emphasizing a single consolidated view of global activity across the estate, with close attention to endpoints. “If I could go back in time to the months leading up to our ransomware attack, I’d tell myself to strengthen our endpoint policies, fully delving into all the capabilities of Defender for Endpoint and Intune,” he continues. “I don’t view our recovery as a victory so much as a call to double down on security.”

McKinney offers the following advice to give other organizations the benefit of his experiences and learnings: 

• Create unified security. A long-time user of Microsoft 365 E5, G&J Pepsi had deployed and benefited from Microsoft 365 Defender, a leading XDR solution. That tool set provides security across all key user workloads in the organization, including endpoints, email, documents, cloud apps, and identities. It was the base G&J Pepsi needed to expand security over after recovering from the ransomware attack. By correlating signals across all of these areas, Microsoft 365 Defender is uniquely able to help detect and respond to ransomware threats like the one G&J Pepsi experienced. "Having a strong security posture focused on protecting physical security and the security of devices, identities, and data is critical to company stability and were key components to a successful defense against cyberattacks,” says McKinney. “We also recommend using all the recommendations found in Microsoft solutions, like the security tools in Azure and Defender for Endpoint.”

• Focus on backup capability. G&J Pepsi’s first step was to increase backup frequency, running Azure Backup on every Azure Virtual Machine device nightly and extending backup retention. McKinney trusts Azure Backup because of its different approach from other backup solutions, which back up device images to a Server Message Block (SMB) file share—a protocol that permits applications on the network to read and write to files, facilitating the lateral movement that’s so dangerous during an attack. “Azure Backup differs from a lot of other backup systems because it’s installed on the platform layer, preventing attackers from moving laterally into the rest of the estate to extract backup files,” says McKinney. “Thankfully, we were able to completely restore our environment to its pre-attack state.” 

• Connect the dots. The company revisited its Microsoft Security solutions, activating more capabilities. McKinney emphasized the visibility that G&J Pepsi can get with Microsoft Defender for Cloud. The cloud security posture management solution zeros in on weak spots in the company’s configuration and protects workloads across multicloud and hybrid environments. It generates 90 percent of the signals that indicate a possible misuse of information—invaluable clues the team can drill into to see events like users downloading large numbers of files or documents containing payment card industry (PCI) data. “The cloud agents used by the solution require little to no setup from the team,” adds McKinney. His team combines Defender for Cloud data with instantly grasped threat information via Microsoft Graph Data Connect for overall visibility. “It’s important to activate all the security monitoring layers within Defender for Endpoint and Defender for Cloud,” he insists. “By feeding those signals into a graph like Microsoft Graph Data Connect, the impact will be visible. That’s a key part of an effective security strategy.”

• Roll out the most effective security tools—no exceptions, no apologies. Again, McKinney emphasizes endpoint security in G&J Pepsi’s revamped security approach, acknowledging that the inevitable clash between productivity and convenience with security must be addressed. “My advice for any cybersecurity team is to focus on endpoints and identity,” he says. “Make the fullest possible use of an endpoint detection and response tool like Intune, strengthen those policies, and don’t be apologetic about it.” He insists that G&J Pepsi employees use the Microsoft productivity apps to create and store content, collaborating more safely with SharePoint and OneDrive.

• Go cloud as much as possible. The usual arguments in favor of the cloud—improved scalability and capability at an attractive price tag—resonate with McKinney. But he believes that emphasis misses a critical benefit. “Long story short, get to the cloud,” he advises. “Migrate as much as you can to platform and software as a service (SaaS) offerings. G&J Pepsi has gotten a wide range of security benefits, such as platform-based backups, cloud-based identities, and multifactor authentication, leveraging native tools that help recommend and identify risk.” Those advantages couldn’t realistically be replicated without the cloud. “If we had to host systems to do this ourselves, they’d run on-premises or on VMs,” he continues. “And malicious actors would compromise them eventually. They target vulnerabilities in applications and operating systems to move laterally and expedite their attacks.”