Microsoft Indoor Localization Competition – IPSN 2018

Microsoft Indoor Localization Competition – IPSN 2018

Official Results

The slide deck that was used during the official results announcement session can be found HERE.

The detailed ground truth measurements and error visualization for each team, as well as the 3D point cloud of the evaluation area can be found HERE. Each team is assigned a short acronym comprised of 2-4 letters. There is an excel file for each category (2D and 3D) that maps the full submissions to these acronyms. Important note: these results have been further refined since the official announcement of the results to better synchronize the system under test and the ground truth system. As a result, some teams might see slight differences in the reported accuracies in this data when compared to the graphs below. These differences have no impact on the ranking of the top teams. Ranking only changed for a few teams towards the tail.

 

 

Note 1: Organizers considered Naviguy and Ariel University to have the same accuracy as the difference of 10cm is well within the error of this type of technologies. Given that Naviguy had to be explicitly initialized to a known location at the beginning of the evaluation, Ariel University takes the first place, and Naviguy the second place.

Note 2: Given that we had only 7 submissions in the 2D category, only the top 2 teams are eligible for awards.

Note 3: Rea et al. had 2 of their 10 APs fail during the evaluation. In addition, they used the wrong system configuration during evaluation. After fixing some of the issues, and re-processing the data after the competition, the team reported an improved accuracy of 3.39m.

 

Notes: Organizers considered SND Smart Ltd and Yodel Labs to have the same accuracy.

 

This is the final list of winners and their prizes:

 

Call for Contesters

Accurate indoor localization has the potential to transform the way people navigate indoors in a similar way that GPS transformed the way people navigate outdoors. Over the last 15 years, several indoor localization technologies have been proposed and experimented by both academia and industry, but we have yet to see large scale deployments. This competition aims to bring together real-time or near real-time indoor location technologies and compare their performance in the same space.

Eligibility

Both academia and industry submissions are encouraged. All location techniques, such as ranging, fingerprinting, infrastructure, or device free, are welcome, except those that require end users’ manual measurements. Contesters can deploy their own infrastructure of up to 10 devices (this number is tentative and it will be finalized after the submission deadline). Normal RF interference is expected, but no jammers from other deployments are allowed. The results must be shown on a portable device, such as a phone or a tablet/laptop that a person can easily carry around.

Demo submissions that do not meet one or more of the guidelines above will be included in the poster session and will be evaluated as a regular submission, but they will not be considered for prizes.

The competition will take place if at least 5 teams respond to this preliminary call for competition.

Competition Categories

Depending on the nature and number of submissions multiple categories might be defined based on the accuracy (i.e., point-based vs. area based), the size, the cost, or the type (i.e., software vs. hardware) of the proposed solution.

Given our past experience, this year we expect to have the following categories (note that these categories have not been finalized. The final categories will be announced shortly after the registration deadline and they will depend on the number and type of submissions received):

  1. Commercial off-the-shelf (COTS) Technologies: Submissions in this category should be able to work with unmodified commercial off-the-shelf devices such as laptops, phones, and tablets. In this category, the unmodified COTS device is localized. Teams in this category will not be allowed to interface any custom hardware to the COTS devices (i.e., UWB or ultrasound hardware). Only changes to the software of the devices will be allowed. Submissions in this category could be further classified to infrastructure-based and infrastructure-free depending on their requirement to deploy custom hardware (i.e., BLE beacons) in the evaluation area.
  2. Commercial off-the-shelf (COTS) Technologies with Initialization: Submissions in this category should meet all the criteria of the COTS Technologies category, with the additional requirement to initialize the location of the COTS device being localized to a ground truth location before the evaluation.
  3. Modified Commercial off-the-shelf (COTS) Technologies: Submissions in this category could interface custom hardware to the COTS devices to be localized (i.e., UWB or ultrasound hardware). Submissions in this category could be further classified to infrastructure-based and infrastructure-free depending on their requirement to deploy custom hardware (i.e., BLE beacons) in the evaluation area. Most, if not all, submissions in this category are expected to be infrastructure-based.

For instance:

  • WiFi fingerprinting approaches that also leverage inertial sensors will be classified as infrastructure-free COTS Technologies as they do not require to interface any custom hardware to the COTS device being localized.
  • WiFi fingerprinting approaches that also deploy custom BLE beacons in the evaluation area will be classified as infrastructure-based COTS Technologies as they don’t modify the COTS device being localized, but they require the deployment of custom hardware.
  • Approaches based on inertial sensing that require to be initialized to a ground truth location before being evaluated will be classified as COTS Technologies with Initialization.
  • UWB approaches will be classified as infrastructure-based modified COTS Technologies as they require modifications to the COTS device being localized, and they need to deploy custom hardware in the evaluation area.

In this year’s competition, teams will be required to report 3D locations (X,Y,Z). The evaluation area we have reserved, even though it does not contain multiple floors, it includes locations with different elevation characteristics. On top of that, we plan to place the devices to be localized at different heights during the competition to evaluate the 3D localization accuracy of each team.

Not all teams will have to report 3D locations. We understand that some technologies are not suited for 3D localization (i.e., WiFi/geo-magnetic fingerprinting, dead reckoning, BLE-based etc.), but they are still excellent candidates for commercial indoor location systems. With this in mind, the organizers will determine shortly after the registration deadline which teams will be classified as 3D localization teams. It is not up to the individual teams to decide if they will report 2D or 3D locations. The organizers will ensure that all the teams in a given category will either report 2D or 3D locations. There will be no categories containing both 2D and 3D localization teams.

Acceptable Technologies

LiDAR-based systems will not be accepted to the competition this year. LiDAR is reserved for ground truth measurements. Also, the use of laser range finders is allowed only during the setup of each system and not during its evaluation.

Other than the LiDAR and laser range finder restrictions outlined above, there are no other technology restrictions.

Evaluation and Prizes

Results are judged based on accuracy, and an award will be given to the top 3 teams in each category. When accuracy ties, infrastructure requirements will be used for tie breaking. The winning teams in each category will be invited to present their approach at the conference, and receive a cash award. The exact accuracy metrics that will be used during evaluation will be announced shortly before the competition takes place.

Traditionally, teams were evaluated on their accuracy in estimating locations of individual points. This year, teams will be evaluated on their tracking accuracy as opposed to point accuracy. Each system under test will be taken through a pre-determined path and will output its estimated location at a predetermined rate while walking along the path. The system that will be able to approximate the ground truth path with the highest overall accuracy wins. Organizers will provide more details about the evaluation process and its logistics as we get closer to the competition.

Poster Session

A poster session dedicated to all competition participants will be organized during the conference. Participants will have the opportunity to explain their system to conference attendees.

Submission Guidelines

Contesters must submit an abstract describing their approach and deployment requirements by the contest registration deadline. Submissions are treated as confidential until the competition. Submissions must be at most two (2) single-spaced 8.5″ x 11″ pages, including figures, tables, and references. Submission should follow the exact same format as regular, full IPSN 2018 papers. Abstracts should include the names and affiliations of all authors. Templates can be found here. Abstracts should be sent over email to: dlymper@microsoft.com on or before February 11th 2018 with the following subject line: 2018 Microsoft Indoor Localization Competition Submission.

 

Competition Logistics

More details about the competition space and the evaluation process can be found HERE

Please read this information carefully! This year, each team is required to provide an output text file with the recorded locations at the end of the evaluation. More details are in the document linked above.

A tentative schedule for the competition is provided below:

Day                      Time                     Description
April 10th 8:30 – 17:00 Setup Day. All teams meet at the evaluation area at 8:30am to go over the rules.
April 11th 8:30 – 17:00 Evaluation Day. All teams are evaluated during their pre-assigned time slots
April 12th 11:30 – 12:30 Official Results Anouncement

Competing Teams

34 indoor location systems from 31 teams have registered to participate to the competition. Given the type and number of submissions we received, we had to classify all teams in two categories similarly to the previous years: Infrastructure-Free (12 submissions) and Infrastructure-Based (22 submissions) systems. A list of all the teams in each of the categories is provided below.

All teams that have been assigned an evaluation time slot have at least one member officially registered. Teams that have not been assigned an evaluation time slot will not be allowed to compete during the competition. The data in the table below reflects registrations as of March 20th 2018.

Infrastructure-Free (2D Localization)

Authors Affiliation Title Evaluation Time Slot
Qu et al. China – Tongji University/Naviguy Inertial Sensing Approach for Indoor Localization 8:30-8:45
Qu et al.. China – Tongji University/Naviguy Precise Indoor Localization Fusion System 8:45-9:00 
Ben-Moshe et al. Israel – Ariel University GoIn – An Accurate InDoor Navigation Framework for Mobile Devices 10-10:15
Ben-Moshe et al.. Israel – Ariel University Steps – An Accurate Relative Positioning Method for First-Responders 10:15-10:30
Martinez et al. Spain – Universidad de Castilla-La Mancha An Indoor Localization System Based on Particle Filters and Real-Time Range-Free Estimation Method N/A
Zhang et al.. China – Tianjin University Implementation of Real-Time Pedestrian Navigation System using Foot-Mounted IMU N/A
Joshi et al. India – IIT Delhi PDR-RSS Based Indoor Localization N/A
Rea et al.. Spain – IMDEA Networks Institute TWINS: Time-of-Flight Based Wireless Indoor Navigation System 9-9:15 
Xu et al. China – Nokia/Tongji university/Naviguy/Ruijie Networks Indoor Localization Based on CSI and Mobile Sensors 9:15-9:30 
Fineway et al.. China – Beijing Fineway Technology Ltd. Co. An Inertial Sensor-based Indoor Location Tracking Solution 9:30-9:45 
Ali et al. South Korea – Yeungnam University Infrastructure-free Indoor Positioning System Using Smart Phone Sensors 9:45-10
Jia et al.. China – Center for Indoor Positioning Studies Geomagnetism and WiFi Fingerprint Fusion Based Indoor Location N/A

Infrastructure-Based (3D Localization)

Authors Affiliation Title Eval. Time Slot
Lin et al. China – China Academy of Electronics and Information Tech./Beijing Institute of Technology VL-Loc: An Indoor Localization System Based on Mobile Phone and Illumination Facility 15:30-15:45
Li et al.. China – ElegenTech Co./Nanjing University of Aeronautics and Astronautics A Low Cost 3D Positioning Solution of High Accuracy and High Refresh Rate for Robot Control 10:30-10:45 
Kukovyakina et al. Russia – National Research University MPEI SKM: low-cost precise positioning based on TDoA UWB and MEMS IMU 10:45-11
Huang et al. China – Zhejiang University AALOC: An Accurate Acoustic Indoor Localization System 11-11:15
Chen et al.. China/U.K. – Zhejiang University/University of West London AIDLOC: An Accurate Acoustic 3D Indoor Localization System 11:15-11:30
Zhang et al. China/U.K. – Zhejiang University/University of West London MAIDLOC: A Modified Accurate Acoustic Indoor Localization System 11:30-11:45
Schroder et al.. Germany – TU Braunschweig InPhase: No-Cost Phase-Based Ranging and Localization 11:45-12 
Ansell et al. U.K. – RaceLogic Indoor Positioning System for Highly Dynamic Applications 12-12:15 
Zhang et al.. China/U.K. – Zhejiang University/ University of West London RA2LOC: A Robust Accurate Acoustic Indoor Localization System 12:15-12:30
Tiemann et al. Germany – TU Dortmund University ATLAS: TDOA-based UWB Localization 14-14:15 
Gunes et al.. Germany – Quantitec GmbH IntraNav – Low-cost Indoor Localization System and IoT Platform N/A
Gao et al. China – SND Smart Ltd. Com./ATE Electronics Ltd. Com. NavInThings – An Indoor Localization and Navigation System based on UWB 14:15-14:30
Collier et al.. France – INTELLIDOM UWB-Based Indoor Localization System 15:45-16:00 
Sardar et al. India/South Africa – IIT Hyderabad/University of Cape Town Indoor Localization System Based on Commensal Radar Principle N/A
Chen et al.. Saudi Arabia – King Abdullah University of Science and Technology KAUST Acoustic Positioning System N/A
De et al.. USA – UIUC Finding by Counting – A Packet Count Based Indoor Localization Technique Using BLE Sensors N/A
Li et al.. Canada – McMaster University 3D Indoor Localization with Commercial-off-the-shelf UWB Radios 14:30-14:45
Strachen et al.. USA – University of Wisconsin-Madison Accurate Indoor Navigation with Spinning Magnets N/A
Iqbal et al.. Pakistan – National University of Sciences and Technology RSSI Based Indoor Localization Using RF Beacons N/A
Lazik et al.. USA – CMU ALPS: The Acoustic Location Processing System 15:15-15:30
Miller et al.. USA – CMU Realty and Reality: Where Location Matters 14:45-15
Schmitz et al.. Germany – RWTH Aachen University/Roboter Club Aachen e.V. Ultrasound-based Cooperative Indoor Localization for Robotic Applications 15-15:15

Previous Competitions