The slide deck that was used during the official results announcement session can be found HERE.
The detailed ground truth measurements and error visualization for each team, as well as the 3D point cloud of the evaluation area can be found HERE. Each team is assigned a short acronym comprised of 2-4 letters. There is an excel file for each category (2D and 3D) that maps the full submissions to these acronyms. Important note: these results have been further refined since the official announcement of the results to better synchronize the system under test and the ground truth system. As a result, some teams might see slight differences in the reported accuracies in this data when compared to the graphs below. These differences have no impact on the ranking of the top teams. Ranking only changed for a few teams towards the tail.
Note 1: Organizers considered Naviguy and Ariel University to have the same accuracy as the difference of 10cm is well within the error of this type of technologies. Given that Naviguy had to be explicitly initialized to a known location at the beginning of the evaluation, Ariel University takes the first place, and Naviguy the second place.
Note 2: Given that we had only 7 submissions in the 2D category, only the top 2 teams are eligible for awards.
Note 3: Rea et al. had 2 of their 10 APs fail during the evaluation. In addition, they used the wrong system configuration during evaluation. After fixing some of the issues, and re-processing the data after the competition, the team reported an improved accuracy of 3.39m.
Notes: Organizers considered SND Smart Ltd and Yodel Labs to have the same accuracy.
This is the final list of winners and their prizes:
Call for Contesters
Accurate indoor localization has the potential to transform the way people navigate indoors in a similar way that GPS transformed the way people navigate outdoors. Over the last 15 years, several indoor localization technologies have been proposed and experimented by both academia and industry, but we have yet to see large scale deployments. This competition aims to bring together real-time or near real-time indoor location technologies and compare their performance in the same space.
Both academia and industry submissions are encouraged. All location techniques, such as ranging, fingerprinting, infrastructure, or device free, are welcome, except those that require end users’ manual measurements. Contesters can deploy their own infrastructure of up to 10 devices (this number is tentative and it will be finalized after the submission deadline). Normal RF interference is expected, but no jammers from other deployments are allowed. The results must be shown on a portable device, such as a phone or a tablet/laptop that a person can easily carry around.
Demo submissions that do not meet one or more of the guidelines above will be included in the poster session and will be evaluated as a regular submission, but they will not be considered for prizes.
The competition will take place if at least 5 teams respond to this preliminary call for competition.
Depending on the nature and number of submissions multiple categories might be defined based on the accuracy (i.e., point-based vs. area based), the size, the cost, or the type (i.e., software vs. hardware) of the proposed solution.
Given our past experience, this year we expect to have the following categories (note that these categories have not been finalized. The final categories will be announced shortly after the registration deadline and they will depend on the number and type of submissions received):
- Commercial off-the-shelf (COTS) Technologies: Submissions in this category should be able to work with unmodified commercial off-the-shelf devices such as laptops, phones, and tablets. In this category, the unmodified COTS device is localized. Teams in this category will not be allowed to interface any custom hardware to the COTS devices (i.e., UWB or ultrasound hardware). Only changes to the software of the devices will be allowed. Submissions in this category could be further classified to infrastructure-based and infrastructure-free depending on their requirement to deploy custom hardware (i.e., BLE beacons) in the evaluation area.
- Commercial off-the-shelf (COTS) Technologies with Initialization: Submissions in this category should meet all the criteria of the COTS Technologies category, with the additional requirement to initialize the location of the COTS device being localized to a ground truth location before the evaluation.
- Modified Commercial off-the-shelf (COTS) Technologies: Submissions in this category could interface custom hardware to the COTS devices to be localized (i.e., UWB or ultrasound hardware). Submissions in this category could be further classified to infrastructure-based and infrastructure-free depending on their requirement to deploy custom hardware (i.e., BLE beacons) in the evaluation area. Most, if not all, submissions in this category are expected to be infrastructure-based.
- WiFi fingerprinting approaches that also leverage inertial sensors will be classified as infrastructure-free COTS Technologies as they do not require to interface any custom hardware to the COTS device being localized.
- WiFi fingerprinting approaches that also deploy custom BLE beacons in the evaluation area will be classified as infrastructure-based COTS Technologies as they don’t modify the COTS device being localized, but they require the deployment of custom hardware.
- Approaches based on inertial sensing that require to be initialized to a ground truth location before being evaluated will be classified as COTS Technologies with Initialization.
- UWB approaches will be classified as infrastructure-based modified COTS Technologies as they require modifications to the COTS device being localized, and they need to deploy custom hardware in the evaluation area.
In this year’s competition, teams will be required to report 3D locations (X,Y,Z). The evaluation area we have reserved, even though it does not contain multiple floors, it includes locations with different elevation characteristics. On top of that, we plan to place the devices to be localized at different heights during the competition to evaluate the 3D localization accuracy of each team.
Not all teams will have to report 3D locations. We understand that some technologies are not suited for 3D localization (i.e., WiFi/geo-magnetic fingerprinting, dead reckoning, BLE-based etc.), but they are still excellent candidates for commercial indoor location systems. With this in mind, the organizers will determine shortly after the registration deadline which teams will be classified as 3D localization teams. It is not up to the individual teams to decide if they will report 2D or 3D locations. The organizers will ensure that all the teams in a given category will either report 2D or 3D locations. There will be no categories containing both 2D and 3D localization teams.
LiDAR-based systems will not be accepted to the competition this year. LiDAR is reserved for ground truth measurements. Also, the use of laser range finders is allowed only during the setup of each system and not during its evaluation.
Other than the LiDAR and laser range finder restrictions outlined above, there are no other technology restrictions.
Evaluation and Prizes
Results are judged based on accuracy, and an award will be given to the top 3 teams in each category. When accuracy ties, infrastructure requirements will be used for tie breaking. The winning teams in each category will be invited to present their approach at the conference, and receive a cash award. The exact accuracy metrics that will be used during evaluation will be announced shortly before the competition takes place.
Traditionally, teams were evaluated on their accuracy in estimating locations of individual points. This year, teams will be evaluated on their tracking accuracy as opposed to point accuracy. Each system under test will be taken through a pre-determined path and will output its estimated location at a predetermined rate while walking along the path. The system that will be able to approximate the ground truth path with the highest overall accuracy wins. Organizers will provide more details about the evaluation process and its logistics as we get closer to the competition.
A poster session dedicated to all competition participants will be organized during the conference. Participants will have the opportunity to explain their system to conference attendees.
Contesters must submit an abstract describing their approach and deployment requirements by the contest registration deadline. Submissions are treated as confidential until the competition. Submissions must be at most two (2) single-spaced 8.5″ x 11″ pages, including figures, tables, and references. Submission should follow the exact same format as regular, full IPSN 2018 papers. Abstracts should include the names and affiliations of all authors. Templates can be found here. Abstracts should be sent over email to: email@example.com on or before February 11th 2018 with the following subject line: 2018 Microsoft Indoor Localization Competition Submission.
More details about the competition space and the evaluation process can be found HERE
Please read this information carefully! This year, each team is required to provide an output text file with the recorded locations at the end of the evaluation. More details are in the document linked above.
A tentative schedule for the competition is provided below:
|April 10th||8:30 – 17:00||Setup Day. All teams meet at the evaluation area at 8:30am to go over the rules.|
|April 11th||8:30 – 17:00||Evaluation Day. All teams are evaluated during their pre-assigned time slots|
|April 12th||11:30 – 12:30||Official Results Anouncement|
34 indoor location systems from 31 teams have registered to participate to the competition. Given the type and number of submissions we received, we had to classify all teams in two categories similarly to the previous years: Infrastructure-Free (12 submissions) and Infrastructure-Based (22 submissions) systems. A list of all the teams in each of the categories is provided below.
All teams that have been assigned an evaluation time slot have at least one member officially registered. Teams that have not been assigned an evaluation time slot will not be allowed to compete during the competition. The data in the table below reflects registrations as of March 20th 2018.
Infrastructure-Free (2D Localization)
|Authors||Affiliation||Title||Evaluation Time Slot|
|Qu et al.||China – Tongji University/Naviguy||Inertial Sensing Approach for Indoor Localization||8:30-8:45|
|Qu et al..||China – Tongji University/Naviguy||Precise Indoor Localization Fusion System||8:45-9:00|
|Ben-Moshe et al.||Israel – Ariel University||GoIn – An Accurate InDoor Navigation Framework for Mobile Devices||10-10:15|
|Ben-Moshe et al..||Israel – Ariel University||Steps – An Accurate Relative Positioning Method for First-Responders||10:15-10:30|
|Martinez et al.||Spain – Universidad de Castilla-La Mancha||An Indoor Localization System Based on Particle Filters and Real-Time Range-Free Estimation Method||N/A|
|Zhang et al..||China – Tianjin University||Implementation of Real-Time Pedestrian Navigation System using Foot-Mounted IMU||N/A|
|Joshi et al.||India – IIT Delhi||PDR-RSS Based Indoor Localization||N/A|
|Rea et al..||Spain – IMDEA Networks Institute||TWINS: Time-of-Flight Based Wireless Indoor Navigation System||9-9:15|
|Xu et al.||China – Nokia/Tongji university/Naviguy/Ruijie Networks||Indoor Localization Based on CSI and Mobile Sensors||9:15-9:30|
|Fineway et al..||China – Beijing Fineway Technology Ltd. Co.||An Inertial Sensor-based Indoor Location Tracking Solution||9:30-9:45|
|Ali et al.||South Korea – Yeungnam University||Infrastructure-free Indoor Positioning System Using Smart Phone Sensors||9:45-10|
|Jia et al..||China – Center for Indoor Positioning Studies||Geomagnetism and WiFi Fingerprint Fusion Based Indoor Location||N/A|
Infrastructure-Based (3D Localization)
|Authors||Affiliation||Title||Eval. Time Slot|
|Lin et al.||China – China Academy of Electronics and Information Tech./Beijing Institute of Technology||VL-Loc: An Indoor Localization System Based on Mobile Phone and Illumination Facility||15:30-15:45|
|Li et al..||China – ElegenTech Co./Nanjing University of Aeronautics and Astronautics||A Low Cost 3D Positioning Solution of High Accuracy and High Refresh Rate for Robot Control||10:30-10:45|
|Kukovyakina et al.||Russia – National Research University MPEI||SKM: low-cost precise positioning based on TDoA UWB and MEMS IMU||10:45-11|
|Huang et al.||China – Zhejiang University||AALOC: An Accurate Acoustic Indoor Localization System||11-11:15|
|Chen et al..||China/U.K. – Zhejiang University/University of West London||AIDLOC: An Accurate Acoustic 3D Indoor Localization System||11:15-11:30|
|Zhang et al.||China/U.K. – Zhejiang University/University of West London||MAIDLOC: A Modified Accurate Acoustic Indoor Localization System||11:30-11:45|
|Schroder et al..||Germany – TU Braunschweig||InPhase: No-Cost Phase-Based Ranging and Localization||11:45-12|
|Ansell et al.||U.K. – RaceLogic||Indoor Positioning System for Highly Dynamic Applications||12-12:15|
|Zhang et al..||China/U.K. – Zhejiang University/ University of West London||RA2LOC: A Robust Accurate Acoustic Indoor Localization System||12:15-12:30|
|Tiemann et al.||Germany – TU Dortmund University||ATLAS: TDOA-based UWB Localization||14-14:15|
|Gunes et al..||Germany – Quantitec GmbH||IntraNav – Low-cost Indoor Localization System and IoT Platform||N/A|
|Gao et al.||China – SND Smart Ltd. Com./ATE Electronics Ltd. Com.||NavInThings – An Indoor Localization and Navigation System based on UWB||14:15-14:30|
|Collier et al..||France – INTELLIDOM||UWB-Based Indoor Localization System||15:45-16:00|
|Sardar et al.||India/South Africa – IIT Hyderabad/University of Cape Town||Indoor Localization System Based on Commensal Radar Principle||N/A|
|Chen et al..||Saudi Arabia – King Abdullah University of Science and Technology||KAUST Acoustic Positioning System||N/A|
|De et al..||USA – UIUC||Finding by Counting – A Packet Count Based Indoor Localization Technique Using BLE Sensors||N/A|
|Li et al..||Canada – McMaster University||3D Indoor Localization with Commercial-off-the-shelf UWB Radios||14:30-14:45|
|Strachen et al..||USA – University of Wisconsin-Madison||Accurate Indoor Navigation with Spinning Magnets||N/A|
|Iqbal et al..||Pakistan – National University of Sciences and Technology||RSSI Based Indoor Localization Using RF Beacons||N/A|
|Lazik et al..||USA – CMU||ALPS: The Acoustic Location Processing System||15:15-15:30|
|Miller et al..||USA – CMU||Realty and Reality: Where Location Matters||14:45-15|
|Schmitz et al..||Germany – RWTH Aachen University/Roboter Club Aachen e.V.||Ultrasound-based Cooperative Indoor Localization for Robotic Applications||15-15:15|