Among stock exchanges, low latency—the speed at which a stock trade can be processed—is supreme. Direct Edge wanted to reduce the already low latency of its system, while supporting vastly larger trading volumes. It accomplished those goals and more by rejecting Linux or UNIX choices, instead building its new exchange on technology from Microsoft and Informatica. Since deploying Windows Server 2008 R2, Microsoft SQL Server 2008, and Informatica Ultra Messaging, Direct Edge has reduced latency by 83 percent, to just 340 microseconds, enabling a 580 percent increase in throughput, and it envisions further cuts in latency. Uptime is 100 percent to date. Since implementation, the company’s market share has increased based in part on its choice of platform. A 50 percent reduction in time to market meant a savings of U.S.$14 million in reduced operational costs. And a 25 percent smaller hardware footprint saved another $1 million.Situation
How fast is fast enough? Some measure speed in seconds; others in the blink of an eye. For Direct Edge, however, anything as slow as a blink would likely cost it its business.
In 2010, Direct Edge became the newest stock exchange in the United States, as well as the third-largest, after the New York Stock Exchange and NASDAQ. That milestone represented a rapid rise for the company, which had been founded just five years earlier as an electronic communications network (ECN), a platform that matches buyers and sellers. By 2007, Direct Edge was handling 100 million trades a day—with each transaction requiring it to send messages among components of its exchange application to take an order, find a match, and settle the trade. That volume gave it about 3 percent of the 7 billion share annual market.
To maintain that volume of business, Direct Edge had to offer its customers the fastest, most reliable service it could. “Among financial exchanges, reliability is a baseline requirement, and competition is based largely on low latency—the speed at which you can execute orders,” says Steve Bonanno, Chief Technology Officer at Direct Edge. “That’s because the amount of profit that your customers make depends on the speed at which a transaction is cleared. Latency also determines how many transactions an exchange can handle in a day. The higher the transaction volume, the greater the profits for the exchange.”
||When we put our stock exchange on Windows, our number one goal was to achieve latency low enough to compete against the biggest exchanges in the country. At 340 microseconds, Windows delivered.
Chief Technology Officer, Direct Edge
For Direct Edge, latency was a matter of milliseconds (thousandths of a second); the company handled 95 percent of its orders in fewer than 2 milliseconds. And the ECN was highly reliable.
By 2008, Direct Edge needed to handle higher transaction volumes with even less latency. It was partly a matter of supporting continued growth, but two other factors were at play, as well.
First, Direct Edge acquired the operations of the ISE stock exchange. That required Direct Edge to handle the additional business volume as effectively—and as cost-effectively—as possible. Second, Direct Edge was preparing to convert from an ECN to a full-fledged, licensed stock exchange. The company stood to benefit in a couple of ways. As a stock exchange, it would reduce operational expense by minimizing clearing and settlement costs. Also, the company would have a higher marketing profile and could increase revenues—if it could handle higher transaction volumes without increasing latency.
“Being a stock exchange rather than an ECN means more regulation, but also more recognition,” says Bonanno. “It’s better branding for us. It enables us to do more deals.”
Direct Edge needed a technology platform that would support more-stringent technical requirements, and that it could use to go head to head with the leading U.S. stock exchanges. In the financial services industry, those requirements frequently meant an exchange based on the UNIX or Linux operating systems. But Direct Edge had built its ECN on the Windows Server 2003 operating system, Microsoft SQL Server 2005 data management software, and daemon and broker-based messaging middleware. Microsoft technology had sufficed when Direct Edge was smaller and younger. Now, facing the potential for vastly larger trading volumes, and the need for flawless reliability and lower latency, Direct Edge couldn’t afford to make the wrong technology choices. Solution
This is the point in the story at which most companies would consider their technology options—but not Direct Edge.
“Both the Direct Edge and ISE platforms had been built on Windows Server technology,” says Bonanno. “It’s where our expertise lies. Based on our experience with those systems over five years, we felt that Microsoft had what it took to be a mission-critical, high-performance platform. It had always been a requirement of our earlier systems to handle massive numbers of transactions, and to do so quickly. For example, we built a real-time processing engine on Microsoft technology that published several hundred thousand transaction messages per second. If Microsoft hadn’t been able to meet our needs, we’d have looked elsewhere—but there was no need to.”
Bonanno and his colleagues anticipated that their new exchange platform would need to handle trading volumes of 1 billion messages a day in short order. If they were successful, they figured, they could see trading volume of 2.5 billion daily messages or more within a few years. They also wanted to better the 2 millisecond order-handling time, bringing it down to microseconds (millionths of a second).
||Being a stock exchange rather than an ECN means more regulation, but also more recognition. It’s better branding for us. It enables us to do more deals.
Chief Technology Officer, Direct Edge
Another requirement of the new exchange platform was fast time to market. Rapid application development was essential so that the new platform would be available before the existing systems reached capacity. It was also essential because every day that Direct Edge continued as an ECN rather than as a formal stock exchange meant higher operational costs and lost revenues. Given the large and complex nature of the anticipated architecture—including two exchange platforms with multiple layers of failover redundancy and a highly complex application—Direct Edge set a one-year deadline for time to market and began work in mid-2009.
“It was an aggressive schedule,” says John Ryan, Chief Architect at Direct Edge.
Turning to a Familiar Choice
To meet the deadline, Ryan and his 12-person development team adopted the Microsoft tools with which they were already familiar: the Microsoft .NET Framework 3.5, Microsoft Visual Studio Team System 2008 Team Suite, and the C# managed development language. The team used the Microsoft tools to build the exchange application subsystems from reusable application blocks. That minimized the need for time-consuming coding during development and, the developers anticipate, will minimize the amount of maintenance required over the life of the exchange application.
Rather than upgrade application components from their existing platform, the developers created new components designed to exploit the scalability and throughput technologies in Windows Server 2008 R2 and SQL Server 2008. Those new components include the matching engine that reconciles buy and sell orders, and the routing engine that manages the message flow among subsystem components.
By choosing Windows Server 2008 R2, Direct Edge was able to use additional cores on its multiprocessor, multicore computer servers for processing network traffic. That, in turn, enabled more application functions to be handled on a single computer rather than being sent over the network to other servers for processing. “We really benefitted from the new Microsoft networking stack,” says Ryan. “It contributed directly to the low latency we were seeking.”
Considering Messaging Middleware
Direct Edge also reviewed its daemon and broker-based messaging middleware technologies to further reduce latency and boost availability. The company chose Ultra Messaging from Informatica to create a messaging bus that enables low-latency, highly available publish- and subscribe-based communication between application components spanning the front-end order gateways, the matching engines, and the back-end compliance systems. The bus serves as the enterprise messaging backbone for all mission-critical traffic transiting the exchange, including market data, order, and trade flow.
“Relying on legacy messaging solutions in today’s capital markets is a path to non-competitiveness,” says Ryan. “Beyond low latency today, we wanted enterprise-class scalability, reliability, and a vendor with a vision. That’s what makes Informatica so valuable to us.”
Ryan and his colleagues used Ultra Messaging to develop a peer-to-peer design for efficient communications between applications within the exchange platform. They also used the technology to eliminate daemon/broker messaging components susceptible to failure.
Direct Edge boosted availability by replicating its order-stream in real time for zero latency failover of trading applications. This complemented the failover clustering and disaster recovery capability at the data center level that Direct Edge gained through Windows Server and SQL Server. For continuity between its two data centers, Direct Edge uses hardware replication.
In mid-2010, one year after it started design and development work, Direct Edge brought its new exchange platform into production. It also completed the regulatory process for conversion from an ECN and became a full-fledged stock exchange.Benefits
By almost any measure—including lower latency, high availability, increased volume, faster time to market, and reduced hardware costs—Direct Edge has achieved stunning results with its new exchange platform.
Decreases Latency by 83 Percent, to 340 Microseconds
Direct Edge has succeeded in reducing its already low latency by 83 percent, bringing the 2 millisecond average end-to-end transaction time down to 340 microseconds—a 580 percent increase in throughput. Bonanno and Ryan envision using Microsoft technology to reduce latency further.
||Beyond low latency today, we wanted enterprise-class scalability, reliability and a vendor with a vision. That’s what makes Informatica so valuable to us.
In addition to the sharp reduction in latency, Direct Edge has also achieved low variation in latency. That means that customers and potential customers who are evaluating the new performance numbers can understand how likely they are to achieve the benchmark speeds on their own stock trades.
“When we put our stock exchange on Windows, our number one goal was to achieve latency low enough to compete against the biggest exchanges in the country,” says Bonanno. “At 340 microseconds, Windows delivered.”
The exchange has already seen an increase in transaction volume from current customers, which it attributes to the reduced latency. “Customers are increasing the amount of business they do with us because of this breakthrough,” says Bonanno.
Delivers 100 Percent Uptime—a “Must-Have” for Business Development
High availability is crucial to Direct Edge. “While everyone wants high reliability, in this business, if you’re down, the entire world knows about it—there’s no hiding,” says Bonanno. He points to news media inquiries that would immediately follow any outages on the predecessor platform.
Direct Edge met this goal with the new exchange platform. Through its use of failover technology at both the messaging and data management levels, Direct Edge has gained 100 percent scheduled uptime to date.
“In our business, 100 percent availability isn’t a ‘nice to have,’ it’s a ‘must have’ in order to increase business,” says Bonanno. “With the Microsoft and Informatica technologies, we have it—a fully failover-redundant system that we didn’t have before. Now, we can devote more of our resources—both people and technology—to areas of competitive advantage.”
Bonanno says that the Informatica technology, for example, contributes “persistence without penalty.” Persistence of the message store is a requirement for recovery after a failover. Typical high-availability systems achieve this persistence by writing to disk, which is effective—but at the cost of a “latency penalty” because it takes time to read from disk to effect a recovery.
Because Ultra Messaging uses dual, real-time messaging streams rather than writing to disk, there is no latency penalty—hence, “persistence without penalty.”
Saves More than $1 Million in Hardware Costs
Reduced operating costs and lower latency share a contributing factor: reduced hardware needs. The current platform reduces the server count by 25 percent compared to the number of servers that would have been required to scale the predecessor platform to support Direct Edge’s current business volume. That’s a savings of between U.S.$1 million and $1.2 million—a savings that will be repeated with each hardware replacement cycle.
Ryan attributes the reduced server needs to several factors. Informatica Ultra Messaging technology, for example, reduces the messaging footprint by 50 percent by eliminating the need for messaging servers and storage area network devices. SQL Server data compression rates of up to eight to one drive down the need for storage throughout the platform.
The reduced hardware footprint doesn’t adversely affect scalability—far from it. Direct Edge projects that the system can support up to 5 billion shares traded daily with its current configuration and at least 10 billion shares traded daily with hardware additions to the present architecture.
“Microsoft and Informatica give us the ability to support vastly greater trading volumes than we could before,” says Bonanno. “And that ability is part of what attracts new and expanded business to us.”
Reduces Time to Market by 50 Percent, Saves $14 Million
Direct Edge met its “aggressive” one-year development schedule by adopting Microsoft development technologies instead of alternatives, according to Ryan. He estimates the time savings at 50 percent—or an additional year of development time and budget.
But the savings in development budget may be the least of the savings that Direct Edge saw from its rapid application development, he says. For example, Direct Edge saved between $3.5 million and $5 million in reduced operating costs by bringing the current platform to market a year sooner. It also saved an estimated $10 million—the one-year differential in costs related to clearing transactions.
“For us, $14 million is a significant savings from a technology choice,” notes Ryan.
Microsoft Server Product PortfolioFor More Information
For more information about the Microsoft server product portfolio, go to:
For more information about Microsoft products and services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Information Centre at (877) 568-2495. Customers in the United States and Canada who are deaf or hard-of-hearing can reach Microsoft text telephone (TTY/TDD) services at (800) 892-5234. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information using the World Wide Web, go to:
For more information about Informatica products and services, visit the website at:
For more information about Direct Edge, visit the website at: