Microsoft Rocket for Live Video Analytics

Microsoft Rocket for Live Video Analytics


News & features

News & features

News & features

News & features


This page is about Microsoft’s research work on building efficient live video analytics. To learn about Microsoft’s commercial product addressing this need, see Live Video Analytics from Azure Media Services. We also provide instructions and code samples to set up Microsoft Rocket containers for analyzing videos using Live Video Analytics.

Cameras are now everywhere. Large-scale video processing is a grand challenge representing an important frontier for analytics, what with videos from factory floors, traffic intersections, police vehicles, and retail shops. It’s the golden era for computer vision, AI, and machine learning – it’s a great time now to extract value from videos to impact science, society, and business!

Project Rocket‘s goal is to democratize video analytics: build a system for real-time, low-cost, accurate analysis of live videos. This system will work across a geo-distributed hierarchy of intelligent edges and large clouds, with the ultimate goal of making it easy and affordable for anyone with a camera stream to benefit from video analytics. For information regarding our project, please see our publications.

Rocket: a powerful configurable platform for live video analytics

Microsoft Rocket Video Analytics Platform is now available on GitHub!
icon: award ribbon Download from GitHub✍ Learn about the key features

Rocket is an extensible software stack for democratizing video analytics: making it easy and affordable for anyone with a camera stream to benefit from computer vision and machine learning algorithms. It comes with a default object counting pipeline that includes a cascade of DNNs. With Rocket, you can plug in any TensorFlow, Darknet or ONNX DNN model, including custom-built models. You can also augment the above pipeline with simpler motion filters based on OpenCV background subtraction, as shown in the figure below. Rocket’s pipelined architecture can be easily configured to execute over a distributed infrastructure, potentially spanning specialized edge hardware (e.g., Azure Stack Edge) and the cloud (e.g., Azure Machine Learning and Cognitive Services).

Video Analytics Stack graphic

For developers, Rocket allows for plugging in new analytics modules to video analytics pipelines. Customized modules can be developed for various applications (as shown in the figure below). These modules can be written to consume and process the data from upstream modules and pass their outputs to the downstream modules.
Video Analytics Stack graphic

Video analytics for Vision Zero

One of the verticals this project is focused on is video streams from cameras at traffic intersections. Traffic-related accidents are among the top 10 reasons for fatalities worldwide. This project partners with jurisdictions to identify traffic details—vehicles, pedestrians, bikes—that impact traffic planning and safety.

We conducted a pilot study in Bellevue, Washington for active traffic monitoring of traffic intersections live 24×7. We hosted a traffic dashboard powered by Rocket’s video analytics live at Bellevue’s Traffic Management Center. The dashboard alerts the traffic authorities on abnormal traffic volumes. Read our case study report.

Screenshot: dashboard of traffic analysis in Bellevue, WA


icon: award ribbon“Safer Cities, Safer People” US Department of Transportation Award
icon: award ribbonInstitute of Transportation Engineering 2017 Achievements Award – “Video Analytics for Vision Zero
icon: award ribbonACM MobiSys 2017 Best Demo
icon: award ribbonMicrosoft 2017 Hackathon Grand Prize Winner
icon: award ribbonACM MobiSys 2019 Best Demo (Runner-up)
icon: award ribbonACM Symposium on Edge Computing 2020 Best Paper Award
icon: award ribbonCSAW 2020 Applied Research Competition Award (Runner-up)





  • Portrait of Haoyu Zhang

    Haoyu Zhang

    Intern - 2015

    Princeton University

  • Portrait of Shubham Jain

    Shubham Jain

    Intern - 2015, 2016

    Rutgers University

  • Portrait of Yao Lu

    Yao Lu

    Intern - 2015, 2016

    University of Washington

  • Portrait of Michael Hung

    Michael Hung

    Intern - 2016

    University of Southern California

  • Portrait of Giulio Grassi

    Giulio Grassi

    Intern - 2015, 2016

    Sorbonne Université / LIP6

  • Portrait of Kevin Hsieh

    Kevin Hsieh

    Intern - 2017

    Carnegie Mellon University

  • Portrait of Enrique  Saurez Apuy

    Enrique Saurez Apuy

    Intern - 2017

    Georgia Tech

Public talks

Keynotes, seminars, conferences

Keynote talks

  • IEEE 14th International Conference on Mobile Ad Hoc and Sensor Systems (October 23rd, 2017) Victor Bahl, “Live Video Analytics
  • 3rd IEEE International Conference on Collaboration and Internet Computing (October 15th, 2017) Victor Bahl, “Democratizing Video Analytics
  • Emerging Topics in Computing Symposium, University of Buffalo Computer Systems Engineering Dept. 50th Anniversary (September 29th, 2017) Victor Bahl, “Live Video Analytics the Perfect Edge Computing Application
  • 35th IEEE International Performance Computing and Communications Conference (December 10th, 2016) Victor Bahl, “Distributed Video Analytics

University department seminars

  • ETH Zurich (Aug 2017) Ganesh Ananthanarayanan, “Taming the Video Star! Real-time Video Analytics at Scale”
  • University of California at Berkeley (May 2017) Ganesh Ananthanarayanan, “Taming the Video Star! Real-time Video Analytics at Scale”
  • Washington University of St. Louis (April 28, 2017) Victor Bahl, “Live Video Analytics the Perfect Edge Computing Application”
  • Cornell University (April 2017) Ganesh Ananthanarayanan, “Taming the Video Star! Real-time Video Analytics at Scale”

Miscellaneous Invited Talks

  • Ganesh Ananthanarayanan, “Video Analytics for Vision Zero”, Microsoft Office of the CTO Summit (February 2017)
  • Victor Bahl, “Distributed Video Analytics”, The First IEEE/ACM Symposium on Edge Computing, Washington DC, USA (October 28th 2016)
  • Peter Bodik, “Cameras everywhere! Video Analytics at Scale”, Microsoft Research Faculty Summit, Redmond, WA (July 13th, 2016)


  • Haoyu Zhang, “Live Video Analytics at Scale with Approximation and Delay-Tolerance”, USENIX NSDI, Boston, MA, 2017.
  • Aakanksha Chowdhery, “The Design and Implementation of a Wireless Video Surveillance System”, ACM MobiCom, Paris, France, 2015.