Add to social bookmarks:
4-page Case Study - Posted 4/27/2011
Views: 477
Rate This Evidence:

SGS

Mining Services Company Implements Parallel Processing with Only Six Lines of Code

SGS Geometallurgy, a division of SGS Canada, Inc., helps mining companies optimize plant design, forecast production, and minimize risk. FLEET™, the group’s old solution for simulating the flow of ore through a processing plant, required seven seconds per mine block regardless of how powerful the system running the application was. SGS used the new parallel processing tools in Microsoft Visual Studio 2010 and the .NET Framework 4 to build its Integrated Geometallurgical Simulator, or IGS™, which simulates 20 blocks per second per processor core—more than a million blocks per hour on a modern four-processor server. SGS adopted the new parallel processing tools in just a few days and can now easily parallelize its applications. Ease of parallelization has also enabled the company to simplify its code and deliver compelling new features, giving SGS a greater edge in winning new business.

Situation

SGS SA and its affiliates provide inspection, verification, testing, and certification services through a network of more than 1,250 offices and laboratories around the world. Established in 1878, the company started by offering agricultural inspection services to grain traders in Europe. Today, SGS helps companies around the world improve quality, increase productivity, and reduce risk through expertise and activities in the areas of agricultural services; oil, gas, and chemicals services; minerals services; life sciences services; systems and services certification; industrial services; environmental services; automotive services; and governments and institutions services.

*
* We replaced hundreds of lines of handwritten multithreading code with just six lines of PLINQ code, which is functionally equivalent and delivers better performance. *

Hugo Rumayor
Lead Software Developer
SGS Geometallurgy

*
Offerings from the SGS Minerals Services group range from quality and quantity inspection to services for optimizing the recovery of metals in processing plants. One business unit within the group is SGS Geometallurgy. A fusing of geology and metallurgy, the science of geometallurgy focuses on the geologically informed selection of a number of test samples to determine metallurgical parameters and the distribution of these parameters through an ore body using accepted geostatistical techniques to support metallurgical process modeling.

“Geometallurgy quantifies the variability of an ore deposit in terms of process parameters such as hardness, flotation, leach response, and environmental impact,” explains Doug Hatfield, Technical Manager, Development, at SGS Geometallurgy. “The data is then applied to the mine’s geographic block model or the mine plan, after which mathematical models are used to generate economic parameters such as throughput, grind size, grade, and recovery. These parameters are then applied to the mine’s block model—as required to optimize plant design and performance, forecast production parameters with respect to ore variability, and minimize overall risk.”

FLEET, the company’s previous solution for performing such simulations, was developed about 10 years ago using the Microsoft Visual Basic 6 programming language. Because the application took seven seconds to run a simulation on a single mine block, it was useful only for analyzing a small fraction of the blocks in a mine model. “Typical geographic mine models are made up of hundreds of thousands to several million individual blocks of earth, meaning that we could only analyze performance for a small subset of samples,” says Hatfield. “As a result, projections derived from those simulations were only realistic for highly homogeneous ore bodies, which are becoming less and less common as all the relatively easy-to-mine ore bodies are used up.”

In 2008, with support for Visual Basic 6 coming to an end, SGS decided it was time to replace FLEET with a new solution. One goal was to reduce simulation times to less than 100 milliseconds per block, as needed to efficiently run simulations on all blocks in a mine model instead of just a small subset of samples. Toward the same goal, the company wanted to implement parallel processing—as required to fully take advantage of modern multicore processors and multiprocessor servers. “Solving plant models that have a large number of process units of different types and recycle streams in the process flow presents a significant computational challenge,” says Hatfield. “We knew that even highly optimized, single-threaded code would only get us so far and were counting on a parallel processing engine that we designed on a whiteboard to deliver the rest of the performance we needed.”

Solution

SGS used the Microsoft Visual Studio 2008 development system and the Microsoft .NET Framework 3.5 to develop its Integrated Geometallurgical Simulator (IGS), implementing a basic multithreading engine by hand—an effort that delivered multicore scalability and 100-millisecond block-processing times. Following that, the company upgraded to Visual Studio 2010 and the .NET Framework 4, taking advantage of the new parallel processing tools they provided to vastly simplify its multithreading code and further reduce simulation times.

Figure 1
Figure 1. SGS used parallel processing to optimize the performance of its simulation tool.
With a user interface based on Windows Presentation Foundation (see Figure 1), IGS enables metallurgists, project managers, mine managers, and engineers to build a graphical model of a processing plant. After the model is built, they can use IGS to simulate how changes to any part of the plant’s grinding or flotation circuit (used to separate certain minerals from others) will affect overall performance and output. It does this with full granularity, eliminating guesswork and risk by simulating each and every block in a million-block model in about four hours on a modern quad-core desktop PC, or in about an hour on a modern four-processor server.

“IGS simulates comminution and flotation circuits either separately or as combined processes,” says Hatfield. “Users can manipulate operating conditions, easily perform circuit layout changes, and see how the changes affect the circuit throughput and final product quality over the life of the mine. Through such capabilities, SGS customers can optimize plant design, energy usage, and production forecasting—as required to minimize risk and run their operations at peak efficiency.”

Early Efforts at Parallelization

*
* Support for parallel programming was the main reason to migrate to Visual Studio 2010 and the .NET Framework 4. In the beginning, I thought it would take a lot of time, but it turned out to be quick and easy. *

Hugo Rumayor
Lead Software Developer
SGS Geometallurgy

*
Development began in November 2008. Hugo Rumayor, Lead Software Developer at SGS Geometallurgy, was the lone developer on the project. He spent about six weeks implementing a multithreading engine by hand, of which three weeks were spent coding and another three weeks debugging. “I implemented a custom thread-pool that received multiple work items, processing a number of blocks sequentially while communicating task status and completion,” says Rumayor. “Executing tasks in batches helped offset the startup cost of our algorithm. It was fairly rudimentary, but it met our limited needs.”

By mid-2010, Rumayor had reached code-complete and met his original goal of 100 milliseconds per block, per processor core. However, he was less than satisfied. “We wanted to parallelize other parts of the application, but doing so would have been too complicated,” he says. “The multithreading code was already highly complex, which I knew would make it hard to maintain and even more difficult for other developers to learn. Throughout the team, there was a genuine concern that any effort to further parallelize the application could get out of hand.”

New Parallel Programming Tools

Fortunately, Microsoft had recently released Visual Studio 2010 and the .NET Framework 4, which included new parallelization tools that enabled Rumayor to address the remaining challenges. He modified his code to use the new parallel programming libraries provided in the .NET Framework 4, which are supported by new features in Visual Studio 2010. The parallel programming libraries provided in the .NET Framework 4 include:

Task Parallel Library (TPL), which includes parallel implementations of for and foreach loops (For and For Each in Visual Basic) as well as lower-level types for task-based parallelism. Implemented as a set of public types and APIs in the System.Threading.Tasks namespace, the TPL relies on a task scheduler that is integrated with the .NET ThreadPool and that scales the degree of concurrency dynamically so that all available processors and processing cores are used most efficiently.

Parallel Language-Integrated Query (PLINQ), a parallel implementation of LINQ to Objects that combines the simplicity and readability of LINQ syntax with the power of parallel programming. PLINQ implements the full set of LINQ standard query operators as extension methods in the System.Linq namespace, along with additional operators to control the execution of parallel operations. As with code that targets the Task Parallel Library, PLINQ queries scale in the degree of concurrency according to the capabilities of the host computer.

Data Structures for Parallel Programming, which introduces several new types that are useful in parallel programming—including a set of concurrent collection classes that are scalable and thread-safe, lightweight synchronization primitives, and types for lazy initialization. Developers can use these new types with any multithreaded application code, including that which uses the Task Parallel Library and PLINQ.

*
* Not only was the PLINQ code functionally equivalent to the multithreading engine I had implemented by hand, but it delivered significant performance gains with much simpler code. *

Hugo Rumayor
Lead Software Developer
SGS Geometallurgy

*
Parallel programming libraries in the .NET Framework 4 are complemented by new parallel diagnostic tools in Visual Studio 2010, which includes new Parallel Stacks and Parallel Tasks windows for debugging code. Visual Studio 2010 Premium and Ultimate also have a new Concurrency Visualizer, which is integrated with the profiler. The visualizations provide graphical, tabular, and numerical data about how a multithreaded application interacts with itself and other programs, enabling developers to quickly identify areas of concern and navigate through call stacks and to relevant call sites in the source code.

“Support for parallel programming was the main reason to migrate to Visual Studio 2010 and the .NET Framework 4,” says Rumayor. “In the beginning, I thought it would take a lot of time, but it turned out to be quick and easy. As a first step, I migrated our automatic tests to the .NET Framework 4 and was able to get it running in just a few hours.”

Migrating IGS to the .NET Framework 4 and replacing his hand-coded multithreading engine with the built-in parallel programming libraries was just as easy, taking Rumayor only a few days. In the end, Rumayor was able to replace his entire hand-coded thread pool—consisting of several hundred lines of low-level code—with the code shown in Figure 2. (In the code sample, PLINQ is implemented by using AsParallel(). Also noteworthy is the use of the ThreadLocal<> type, which makes it easy to maintain a thread-local value that is initialized on demand.)

At first, Rumayor attempted to use Parallel.ForEach instead of PLINQ as a means of simplifying his code. However, after a brief email exchange with someone at Microsoft, he realized that the use of PLINQ was a better approach. Stephen Toub, Principal Architect on the Parallel Computing Platform team at Microsoft, explains further: “The change to PLINQ alleviated his need to do explicit locking to store the result set, as PLINQ encompasses those aggregation patterns already. More importantly, it enabled Hugo to cleanly express what he wanted to accomplish rather than focus on the details of how it should be done.”

“Not only was the PLINQ code functionally equivalent to the multithreading engine I had implemented by hand, but it delivered significant performance gains with much simpler code,” says Rumayor. “Improved performance was due to two things. First, we reduced the startup cost to once per thread by avoiding custom task batching. Second, processor time is utilized more efficiently due to better workload distribution: Our jobs have unpredictable execution times that can range from 20 to 400 milliseconds, so it’s better to distribute them individually instead of using custom task batches. These performance gains were easy to achieve with the .NET Framework 4 but would have been difficult to do with our original multithreading implementation.

“Considering that the entire move to the .NET Framework 4 took just a few days, it was well worth the effort,” Rumayor says.

Figure 2
Figure 2. Parallelized geometallurgical simulation using PLINQ.
With parallelism made easy, SGS decided it was feasible to parallelize other parts of the application—and to add new parallel enabled features. “Before we moved to the .NET Framework 4, IGS only supported single-threaded optimization,” says Hatfield. “To find an optimal solution, users had to run a simulation, manually adjust the input parameters, rerun the simulation, and then compare the results,” says Hatfield. “With the support for parallel programming provided in the .NET Framework 4, we were able to implement a parallelized genetic algorithm that automates the trial-and-error process required to find an optimal solution, massively reducing execution times.”

SGS Geometallurgy released IGS for use by customers in December 2010. A demo version with sample input data can be found at www.met.sgs.com/integrated-geometallurgical-simulator-igs.

Benefits

By taking advantage of the parallel programming tools in Visual Studio 2010 and the .NET Framework 4, SGS was able to quickly and easily maximize performance and simplify its code base. Simplified parallel development also enabled the company to deliver compelling additional features, making IGS that much more useful to SGS consultants and customers who rely on it to help optimize plant design and performance, forecast production parameters, and minimize risk.

Ease of Adoption

Rumayor was able to quickly and easily move to the .NET Framework 4 and take advantage of its support for parallel programming, with the entire process taking only a few days. This also simplified the application’s code base, thereby reducing the learning curve for others who will work on the code. “In the past, working on multithreaded code was highly complex—something I preferred to let others handle,” says Software Developer Jean Morin, who recently joined the IGS development team. “With the support for parallel programming provided in Visual Studio 2010 and the .NET Framework 4, I was able to come up to speed in a matter of days. All it took was a bit of reading and examination of a few code samples.”

Increased Developer Productivity

*
* In the past, working on multithreaded code was highly complex—something I preferred to let others handle. With the support for parallel programming provided in Visual Studio 2010 and the .NET Framework 4, I was able to come up to speed in a matter of days. *

Jean Morin
Software Developer
SGS Geometallurgy

*
Adoption of Visual Studio 2010 and the .NET Framework 4 has also increased developer productivity by enabling SGS to parallelize IGS—and future applications—with far less effort than in the past. “The abstraction of multithreading in the .NET Framework 4 lets you focus on the computation at hand, instead of low-level constructs that are difficult to code correctly,” says Rumayor. “You still have to be aware of multithreading issues, but the number of things that you have to worry about is greatly reduced. We replaced hundreds of lines of handwritten multithreading code with just six lines of PLINQ code, which is functionally equivalent and delivers better performance.”

Significant Performance Gains

By optimizing its application for parallel processing, SGS was able to take full advantage of the performance potential of modern multicore processors and multiprocessor servers. “With our old single-threaded solution, it took seven seconds to simulate a single mine block, regardless of the number of processors or cores,” says Hatfield. “Today, with IGS, we can simulate 20 blocks per second per processor core. That’s 288,000 blocks per hour on a new quad-core desktop PC, or more than a million blocks per hour on a modern four-processor server.”

The application’s strong performance has also made Hatfield and others at SGS believers in the power of the Visual C# programming language. “In the beginning, I was skeptical that C# could meet our performance needs due to its automatic garbage collection and level of abstraction from the ‘bare metal,’ ” recalls Hatfield. “I must admit that C# has surprised me, as it’s not often considered appropriate for performance-sensitive applications by the scientific computing community—a misperception that we’ve clearly proven to be wrong.”

Should additional performance be needed, SGS may look at running the solution in the cloud, on the Windows Azure platform. “Most simulation jobs are processor intensive but of relatively short duration,” says Hatfield. “With Windows Azure, we can spin up 100 cores, run a simulation for five minutes, and we’re done. It’s a lot more cost-effective than buying and supporting the same hardware.”

Powerful New Features

The performance gains delivered through parallelization—and the ease of achieving them—have enabled SGS to add additional features to IGS, such as the genetic algorithm that enables users to quickly find an optimal solution. “Through use of a genetic algorithm, we enable users to find an optimal solution in a few hours instead of a few days,” says Hatfield. “This powerful new feature was only made possible due to the performance gains and ease of parallelization provided by Visual Studio 2010 and the .NET Framework 4. We had considered adding it before, but had deemed it too complex to parallelize by hand.”

Positive Business Results

*
* With our old single-threaded solution, it took seven seconds to simulate a single mine block, regardless of the number of processors or cores. Today, with IGS, we can simulate 20 blocks per second per processor core. *

Doug Hatfield
Technical Manager, Development
SGS Geometallurgy

*
IGS has enabled SGS Geometallurgy to offer new consulting services and is helping the group to retain and expand its client base. “IGS is proving extremely useful, speeding some of our consulting jobs by a factor of 10,” says Hatfield. “When you combine that productivity gain with the way that IGS enables us to help customers optimize plant performance, forecast production, and reduce risk, it’s easy to understand how IGS will help us to expand our business.”

  

Microsoft Visual Studio 2010

Microsoft Visual Studio 2010 is an integrated development system that helps simplify the entire development process from design to deployment. Unleash your creativity with powerful prototyping, modeling, and design tools that help you bring your vision to life. Work within a personalized environment that helps accelerate the coding process and supports the use of your existing skills, and target a growing number of platforms, including Microsoft SharePoint Server 2010 and cloud services. Also, work more efficiently thanks to integrated testing and debugging tools that you can use to find and fix bugs quickly and easily to help ensure high-quality solutions.

For more information about Visual Studio 2010, go to:

www.msdn.microsoft.com/vstudio

For More Information

For more information about Microsoft products and services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Information Centre at (877) 568-2495. Customers in the United States and Canada who are deaf or hard-of-hearing can reach Microsoft text telephone (TTY/TDD) services at (800) 892-5234. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information using the World Wide Web, go to:

www.microsoft.com

For more information about SGS, visit the website at:
www.sgs.com

This case study is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.
Solution Overview



Organization Size: 64000 employees

Organization Profile

SGS provides inspection, verification, testing, and certification services across a broad range of industries. The company has more than 64,000 employees and is based in Geneva, Switzerland.


Business Situation

SGS needed to enable its new simulation application for parallel processing but found implementing a multithreaded solution to be relatively difficult and overly complex.


Solution

The company adopted new parallel processing aids in Visual Studio 2010 and the .NET Framework 4, replacing hundreds of lines of custom multithreading code with just six lines of new code.


Benefits

  • Ease of adoption
  • Increased developer productivity
  • Significant performance gains
  • New features made possible
  • Positive business results


Software and Services
  • Microsoft Visual Studio 2010 Ultimate
  • Microsoft .NET Framework 4
  • Microsoft Azure

Vertical Industries
Life Sciences, Chemicals; Oil & Gas, Mining

Country / Region
Canada

IT Issues
Development

Languages
English