Technical Story
With nearly 450 convenience stores, SPAR NL offers quick, tasty options for shoppers in the Netherlands. SPAR’s success is based on a cooperative philosophy in which independent operators can run and manage stores that serve the specific needs of their communities. To streamline operations across its sprawling network, SPAR NL adopted Azure Integration Services and a cloud-first approach to enterprise resource planning (ERP). But that meant replacing several legacy systems, including a pivotal automation solution that wasn’t available as a cloud service.
Working closely with Microsoft and technical partner HSO Global Services, SPAR NL built an event-driven, serverless Azure integration platform that connects existing on-premises services and scalable cloud apps. Azure Logic Apps and Azure Functions play a key role in orchestrating a reliable, secure workflow. The new platform gives SPAR the agility it needs to respond much faster to changes, while providing end-to-end monitoring for quick diagnostics and insights.
“This Azure integration solution makes it possible to get an order right, get the scalability right, and get an order directly to the right systems.”
Joran van den Brink, IT Integration Manager, SPAR NL
Retailer replaces black box systems
According to SPAR NL IT Integration Manager Joran van den Brink, the company had a legacy of omnichannel retail systems and custom point-of-sale add-ons that made it challenging to stay on top of industry trends. “We wanted a best-of-breed architecture where you can make changes when something better comes on the market, like a food service solution or online delivery opportunity. In the architecture we had at SPAR, we had no ability to create that.”
Moreover, the IT team at the Netherlands headquarters couldn’t easily see the huge inventory of orders and products traversing the spiderweb of retail, finance, and warehousing systems.
“The difficulty at SPAR was that we didn't have an overview of where the data was heading to or where the errors were popping up,” van den Brink explains. “So we didn’t have insights into our system. It was a black box of different data streams.”
SPAR NL operates five unique types of stores to serve various communities—from larger grocers in cities and smaller shops in villages or near university campuses, to gas station convenience stores. If a store misses a shipment or fails to receive pricing updates or other information, SPAR NL headquarters hears about it. But the old omnichannel retail systems made it difficult to see where a breakdown occurred.
“Triage time is limited,” points out Steef-Jan Wiggers, the HSO Technical Integration Architect who worked closely with van den Brink’s team on a new solution. “We wanted to build a new architecture with complete transparency for SPAR, including an incident management system that would notify the correct people of any problems straightaway.”
An Azure-first approach to systems integration
The new data integration project began with a move to Microsoft Dynamics 365, a suite of intelligent business applications that link people, data, and processes across an organization. Dynamics 365 replaced SPAR’s earlier ERP tools and paved the way to connect more assets in the cloud—an important goal for SPAR NL.
“If you want to be future-proof as a retailer, you want to be able to get the same information about the customer online and offline,” van den Brink observes. “That means having all your data and information in the cloud—going from the omnichannel strategy to a cloud strategy so you can easily adapt the new innovations as they appear.”
Van den Brink notes that SPAR’s experience with Dynamics 365 motivated the company to say, “It's Azure first.” So his team worked with HSO to build an Azure replacement for a legacy middleware broker. In doing so, the teams created an integration solution with all the transparency and monitoring capabilities that SPAR had previously lacked.
“We use Azure services as building blocks to create a perfect solution for our customers in the cloud.”
Steef-Jan Wiggers, Technical Integration Architect, HSO
A composable, event-driven system
SPAR NL and HSO replaced the old message-brokering system with a custom solution built on an Azure serverless computing platform with seamless integration services. The new data integration platform is, at the simplest level, an event-driven system, with producers (like Dynamics 365) that generate a stream of events, and consumers (such as point-of-sale systems) that listen for those events.
SPAR’s global systems include nearly 50 producers and consumers—both cloud and on-premises services—each with a unique interface, sending and receiving event messages on a per-minute, daily, or weekly basis. To streamline this flow of events, SPAR followed the Pipes and Filters pattern—a design pattern that improves performance, scalability, and reusability by breaking down complex processes into discrete tasks that are deployed and scaled independently.
“Our approach was to use something like the Microsoft microservices paradigm,” recalls Wiggers. Instead of microservices, however, his team used Azure Functions to build one function per task, such as validation, mapping, or distribution. “We needed an event-driven system that was durable and reliable but also efficient. That’s why we chose Azure Functions. It also allowed us to turn the knobs a bit under the hood.”
Those “knobs” include flexible Azure Functions pricing plans. For example, some workloads, like product orders, can be huge, but latency shouldn’t suffer. The Premium plan ensures that orders are processed and distributed smoothly by allowing Azure to provide additional computational resources as needed to run the functions. But for most workloads, SPAR can use the cost-efficient Consumption plan and pay only for the time that its function code runs.
“Because everything is done by small functions, we can change interfaces easily and add steps when we need to. We have the composability we were looking for,” Wiggers explains.
Architecture for fast, flexible workflows
The team used Azure Logic Apps to create the workflows and connect dozens of producers and consumers sending messages in the cloud and on-premises. An event triggers Azure Functions to execute any business logic required by a consumer. Other logic apps pick up the messages and send them to the correct destination via Logic Apps connectors.
Wiggers says Logic Apps was the key to keeping systems connected—including a few file-based components. “Some of our management systems are not API-enabled, and we want to connect in a secure manner. We default to using a Logic Apps connector because that’s a good connector. And we can easily standardize.”
For fast access to file-based systems, the platform uses Azure File Sync. In addition, Azure Event Grid helps keep things simple by providing a fully managed service for handling event routing across all the event producers and consumers. “There’s always a service in Azure to do what you need to do,” Wiggers observes.
To organize the many interfaces, HSO developed a tagging and naming convention that enables SPAR to easily monitor all systems. Interfaces are grouped into five common sense domains that indicate where data originates. The order interfaces represent the all-important orders sent by a given store type and distributed to Dynamics 365 Finance and to warehouse management systems. The price and articles interfaces represent information stored in Dynamics 365 that must be sent to Azure Storage and other systems, including point-of-sale registers. The assortment interfaces deal with the configuration of a shelf that holds various articles or the types of articles available in a given store. Generic interfaces control the distribution and archiving of invoices.
“The new data integration platform gave us the opportunity to create complete documentation, making systems much easier to troubleshoot, and the tagging and naming conventions enable us to monitor everything,” van den Brink reports. If an issue arises within a particular domain, the platform notifies the right person, who can resolve it quickly.
As Wiggers says, “It’s all about observability. Azure gave us the means to design a transparent system, and it’s a tremendous value for SPAR.”
The new platform also provides a unified integration layer for all the interfaces, making it easier for SPAR to scale up for peak volumes. Each shop owner—and there are hundreds—can send dozens of orders per day directly to multiple suppliers. Messages from a particular domain, such as assortment or articles, are sent in batches on a schedule or on demand, and then distributed to the appropriate system. Additionally, orders are sent throughout the day as messages—ranging in volume from approximately 6,000 to 10,000 daily.
What happens next depends on the domain. For example, every day the platform receives a batch message consisting of hundreds of articles and stores it in Azure Storage. This event triggers a pipeline of functions that filter the messages, as illustrated in the following diagram. The first function validates the batch message, triggering a second function that maps the message to an intermediate and a third that maps it to the target system format. The final function sends the message to the destination.
This design accommodates a variety of incoming formats, including flat file, XML, EDI, JSON, and other file-based traffic that is still commonly used in the retail industry for pricing data. For example, an on-premises SPAR file share that stores prices as XML data is synced daily with Azure Storage. The XML values enter the pipeline of functions to be processed and then are sent to either another file share in Azure or another system through a logic app and SFTP adapter.
“Azure Functions and Azure Logic Apps gave us the flexibility we needed to address these really huge data volumes with the performance we needed,” Wiggers concludes.
Integration services to future-proof a platform
Unlike the black box systems of the past, SPAR’s new data integration platform clarifies complex workflows and creates opportunities for new business models. Not only is the new platform meeting SPAR’s Azure-first goal, but also it provides better data quality, thanks to the validity checks in the event-driven functions.
“Now we can troubleshoot proactively and see potential failures up front, before our customers—the shop owners—do. We correct them on the spot, mostly on the same day,” explains van den Brink. “This Azure integration solution makes it possible to get an order right, get the scalability right, and get an order directly to the right systems, whether that’s Dynamics 365 for financials or the warehouse management system, so all the orders flow on time. It’s a big win for the company.”
“Our Azure-first strategy enables us to look forward while phasing out the legacy systems. We’re already fostering new business opportunities and piloting new applications.”
Joran van den Brink, IT Integration Manager, SPAR NL
Follow Microsoft