Microsoft Indoor Localization Competition – IPSN 2016

Summary

vienna

Accurate indoor localization has the potential to transform the way people navigate indoors in a similar way that GPS transformed the way people navigate outdoors. Over the last 15 years, several indoor localization technologies have been proposed and experimented by both academia and industry, but we have yet to see large scale deployments. This competition aims to bring together real-time or near real-time indoor location technologies and compare their performance in the same space.

The team from LINZ CENTER OF MECHATRONICS GMBH put together a video showing some of the action during the competition:

News

News

04/15/2016: The official results of the 2016 competition have been announced. An overview of the results can be found HERE. The detailed results for all teams can be found HERE. Congratulations to all the teams that participated!

04/04/2016: The competition’s detailed schedule has been announced. All registered teams have been assigned a 15min evaluation time slot on Monday April 11th, 2016.

03/17/2016: Details about registration, evaluation area, and various logistics can be found HERE.

03/16/2016: The competition will take place on Sunday April 10th, and Monday April 11th. The results will be officially announced on Tuesday April 12th (currently working to move the announcement from Wednesday April 13th as listed on the official program to Tuesday April 12th).

02/19/2016: people that want to attend the competition as observers and not contestants can do it by registering as competition observers: http://www.cpsweek.org/2016/reg.html

02/19/2016: the registration for this year’s competition is currently open: http://www.cpsweek.org/2016/reg.html. All contestants planning to attend the competition must officially register for the competition! Registration for the IPSN conference is NOT required.

02/11/2016: the teams participating in this year’s competition along with the competition categories have been announced!

02/11/2016: We have received 49 submissions this year! The competition is on!

Registration deadline: January 20th 2016

10/20/2015: The 2016 Microsoft Indoor Localization Competition has been officially announced!

Previous Competitions

Check the 2015 and 2014 competition websites above to get a better idea of the evaluation setup, the type of systems expected, and pictures of the different teams in action!

Organizers

Official Results

The detailed results can be found HERE and HERE.

The team from JRC that performed the ground truth mapping of the evaluation area has made the 3D scanning data available. For more information on the collected 3D scanning dataset you can contact Simone Ceriani and Pierluigi Taddei.

2016competitionresults

2016competitionresultsawards

Important information about the results and the final ranking of the teams

  1. Marinis et al. in the 2D category was not considered for the awards. The system required a significantly different evaluation process compared to the official one. The team required to go over specific evaluation points multiple times, and to also record the exact evaluation point they were at, at any given time. Given that no other teams followed this process, this team was not included in the awards.
  2. Elias et al. competed with two versions of the same system: one that requires explicit initialization to a known ground truth location before the evaluation, and one where the initial location was automatically inferred by the system based on WiFi and magnetic fingerprints. Both versions of the system complied to the rules of the competition, and the version of the system with initialization clearly achieved the lowest error. However, given that the system from Su et al. that achieved very similar accuracy to that of Elias et al. did not use explicit initialization, we believe that a fair comparison can only be made among the Elias et al. without initialization and the rest of the teams. Given this, Su et al. got the first place, and Elias et al. the second place. Given the technology leveraged by both teams, the achieved accuracy by these teams is practically identical.
  3. Morin et al. in the 3D category was an exhibition team from the competition’s sponsor Leica. They used a backpack equipped with 5 high-end camera modules, 2 LIDARs and a commercial-grade GPS receiver. This system was not a real-time system as they had to post-process image data and manually select the evaluation points in these images. As a result, it was not considered for the awards.
  4. Flores et al. in the 3D category: during the evaluation process, the organizers wrote down the wrong x-coordinate for evaluation point 15. Even though the team reported (13.38, 0.44, 3.65), the organizers wrote down (30.38, 0.44, 3.65). With this correction, the average location error for this team is 1.74m and not 2.86m.
  5. Kulm et al. in the 3D category reported coordinates without adjusting for their 1.5m tall tripod they used to attach their tag to. Unfortunately, they informed the organizers about this after the competition was completed. When adjusting for the 1.5m fixed offset of the tripod, the location error of this team comes down to 0.43m from the 1.43m shown in the graphs above.

Competition Logistics

We are happy to announce we have received 49 submissions this year! All 49 submissions have been classified into two categories: 3D and 2D localization. All 27 teams in the 3D localization category require to deploy custom hardware, and will have to provide 3D locations during evaluation. All 22 teams in the 2D localization category rely on already available signals such as WiFi and FM, and will have to report 2D locations during evaluation.

Teams in green fonts have at least one team member officially registered for the competition, and will be allowed to deploy their systems during the competition. Teams in red fonts do not have any of their team members officially registered, and therefore they will NOT be allowed to deploy their systems during the competition. (based on registration data as of Monday March 21st).

The detailed schedule for both days of the competition:

            Date                

        Time         

 Session Description

 Sunday:04/10/2016 (Setup Day)  9:15 – 10:00Welcome, space walk-through, rules overview and team check-in
 10:00 – 17:00Setup time: all teams deploy, calibrate and test their systems
 Monday:04/11/2016 (Evaluation Day)  8:30 – 17:30Evaluation of all teams in 15min pre-assigned time slots. Teams should be present at least 1.5 hours before their assigned evaluation time slot. The time slots assigned to each team can be found in the two tables below.
 Tuesday:04/12/2016 Official Results Announcement 14:30 – 15:30 The official results are announced. Winning teams give 5min talks describing their systems. This session will be held in the same room where the IPSN 2016 conference will take place here.
 Tuesday:04/12/2016 Poster Session  16:00 – 18:00 Poster session co-located with the official 2016 IPSN poster session

3D Localization Category

Authors AffiliationTitleEvaluation Time Slot
Lazik et alUSA – Carnegie Mellon UniversityALPS: The Acoustic Location Processing System N/A
Hoppe et al. Germany – Freiburg im Breisgau, Telocate GmbHTelocate ASSIST 11:45
Hammer et al.Austria – Linz Center of Mechatronics GmbH (LCM)Sound Based Localization System for Safety Applications in Underground Mining 12:00
Wang et al.China – Zhejiang UniversityAIDLOC: An Accurate Acoustic Indoor Localization System 12:15
Al-darowbi et al.Saudi Arabia – KFUPM3D Indoor Localization in the Presence of a Single Beacon N/A
Flores et al. Germany – Friedrich-Alexander-Universitat Erlangen-Nurnberg (FAU)A Range-Bearing based Ultrasonic Positioning System (RB-UPS) 12:30
Perez-Cruz et al.USA – Nokia Bell LabsTDoA-based Localization Using Enhanced Multilateration N/A
Dobrev et al.Germany – Friedrich-Alexander-Universitat Erlangen-Nurnberg (FAU), Symeo GmbH, Neubiberg/MunichA Wireless Localization Network for 6D Pose Estimation 12:45
Baszun et al.Poland – BiocontrolReal Time Livestock Tracking System Based on UWB Technology 13:00
He et al.China – University of Science and TechnologySystem for 3D Positioning Based on UWB 13:15
Cabuz et al.Japan – Ritsumeiken University

Germany – Jacobs University Bremen, ZIGPOS GmbH

A Distributed Indoor Localization Framework 13:30
Dardari et al.Italy – University of Bologna, Uniset s.r.l., DMM s.r.l.SEQUITUR: An Ultra-wideband Flexible 3D Locating Platform 13:45
Loreto et al.Italy – NexTechs SrlNexTechs Indoor Positioning System: An UWB Approach N/A
Krikke et al. Netherlands – OpenRTLSOpenRTLS UWB 14:00
Trybula et al.Poland – Elokon Polska Sp. z o.o.UWB Indoor Localization System based on Two-way Ranging Measurement for Industrial Applications N/A
Tabarovsky et al.Russia – RTLS Research & Development LLCHigh Accuracy Indoor Positioning System with Decawave Transceivers and Auto-calibration 14:15
Lou et al.China – Hangzhou Sunsend Info Tech Co. Ltd.UWB ToA and TDoA Hybrid Localization System 14:30
TerragridIndia/Bulgaria – TerraGridTerraGrid: Taking you where GPS cannot N/A
Kulm et al.Estonia – ElikoPrecise UWB Indoor Positioning System 14:45
Kempke et al.USA – University of MichiganPolyPoint: High-Precision Indoor Localization with UWB 15:00
Kämäräinen et al.Finland – VTT Technical Research CenterUWB Localization Complemented by Impulse Radar 15:15
Gunes et al.Germany – Quantitec GmbH IntraNav – Low-cost Indoor Localization System and IoT Platform 15:30
Pichler et al.Austria – Linz Center of Mechatronics GmbH (LCM)UWB Indoor Localization in Time-of-Arrival Configuration 15:45
Morin et al.Switzerland – Leica GeosystemsLeica Geosystems Pegasus Backpack 16:00
Vadeny et al.USA – University of AlbanyVSLAM and VLC Based Localization 16:15
Zhang et al. USA – Real Earth, IncCompact, Real-time Localization without Reliance on Infrastructure 16:30

2D Localization Category

AuthorsAffiliationTitleEvaluation Time Slot
Huang et al. China – Ubirouting Technology Corp.A Magnetic Field Based Lightweight Indoor Positioning System for Mobile Devices 8:30
Su et al.China – Tongji UniversityPrecise Indoor Localization Platform Based on Wifi-GeoMagnetic Fingerprinting and Aided IMU 8:45
Wu et al.China – Glodon Software Co., LtdParticle Filter and Support Vector Machine based Indoor Localization System 9:00
Smirnov et al.Russia – NavigineNavigine Navigation Technology for Handheld Mobile Devices N/A
Zheng et al.China – Xiamen UniversityPrecise Hand-held Indoor Location 9:15
Wang et al. China – Huawei Technologies Co., Ltd., Chinese Academy of Sciences, Beijing University of Posts and Telecommunications MagLoc: Indoor Localization using Geomagnetic, WiFi, and INS N/A
Peng et al.China – Xiamen UniversityPrecise Indoor Positioning System via Multiple Sensor Fusion9:30
Narzullaev et al.Malaysia – IQALAM DASTUR SDN. BHD.CATCH Indoor Positioning System N/A
Han et al.Republic of Korea – Korea Advanced Institute of Science and Technology KAILOS: KAIST Indoor Locating System 9:45
Ben-Moshe et al.Israel – Ariel UniversityGoIn – An Accurate InDoor Navigation Framework for Mobile DevicesN/A
Herrera et al.Mexico – Navix Indoor NavigationNavix: Smartphone Multi-sensor, Radio and Probability Map Based Indoor PositioningN/A
Elias et al. Portugal – Fraunhofer Portugal Research Center for Assistive Information and Communication SolutionsA Motion Tracking Solution for Indoor Location Using Smartphones and Initialization10:00
Elias et al.Portugal – Fraunhofer Portugal Research Center for Assistive Information and Communication SolutionsA Motion Tracking Solution for Indoor Location Using Smartphones10:15
Danaci et al.Turkey – Ege UniversityIndoor Location Positioning System with WiFi TechnologyN/A
Shu et al.China – Zhejiang UniversityR-Loc: A Robust Wifi-based Indoor Localization System11:30
Mosharafa et al.Egypt – Nile UniversityMazeIn: An Indoor Localization Engine10:30
Yoshizawa et al.Japan – Ricoh Company, Ltd. Indoor Positioning by Fusing Pedestrian Dead Reckoning and Proximity Beacon Technologies10:45
Liao et al.Taiwan – Aquaways CO. Ltd. Indoor Localization Using Broadcasting FM Signals TDOA Triangulation N/A
Barbadillo et al.Philippines – University of the PhilippinesAuto-Calibrated 2D Localization System Using RSSIN/A
Rea et al.Spain – IMDEA Networks Institute, University Carlos IIISwitzerland – ArmasuisseRobust WiFI Time-of-Flight Positioning System11:00
Rosen et al.Israel – InfuseInfuse: Indoor Localization Using Sensor FusionN/A
Schmitz et al.Real-time Indoor Localization with TDOA and Distributed Software Defined RadioReal-time Indoor Localization with TDOA and Distributed Software Defined Radio11:15
Marinis et al.Italy – DUNE s.r.l.Italy – DUNE s.r.l.11:30

Call for Contesters

Eligibility

Both academia and industry submissions are encouraged. All location techniques, such as ranging, fingerprinting, infrastructure, or device free, are welcome, except those that require end users’ manual measurements. Contesters can deploy their own infrastructure of up to 5 devices. Normal RF interference is expected, but no jammers from other deployments are allowed. The results must be shown on a portable device, such as a phone or a tablet/laptop that a person can easily carry around.

Demo submissions that do not meet one or more of the guidelines above will be included in the poster session and will be evaluated as a regular submission, but they will not be considered for prizes.

The competition will take place if at least 5 teams respond to this preliminary call for competition.

Competition Categories

Depending on the nature and number of submissions multiple categories might be defined based on the accuracy (i.e., point-based vs. area based), the size, the cost, or the type (i.e., software vs. hardware) of the proposed solution.

Given our past experience, this year we expect to have the following categories (note that these categories have not been finalized. The final categories will be announced shortly after the registration deadline and they will depend on the number and type of submissions received):

  1. Commercial off-the-shelf (COTS) Technologies: Submissions in this category should be able to work with unmodified commercial off-the-shelf devices such as laptops, phones, and tablets. In this category, the unmodified COTS device is localized. Teams in this category will not be allowed to interface any custom hardware to the COTS devices (i.e., UWB or ultrasound hardware). Only changes to the software of the devices will be allowed. Submissions in this category could be further classified to infrastructure-based and infrastructure-free depending on their requirement to deploy custom hardware (i.e., BLE beacons) in the evaluation area.
  2. Commercial off-the-shelf (COTS) Technologies with Initialization: Submissions in this category should meet all the criteria of the COTS Technologies category, with the additional requirement to initialize the location of the COTS device being localized to a ground truth location before the evaluation.
  3. Modified Commercial off-the-shelf (COTS) Technologies: Submissions in this category could interface custom hardware to the COTS devices to be localized (i.e., UWB or ultrasound hardware). Submissions in this category could be further classified to infrastructure-based and infrastructure-free depending on their requirement to deploy custom hardware (i.e., BLE beacons) in the evaluation area. Most, if not all, submissions in this category are expected to be infrastructure-based.

For instance:

  • WiFi fingerprinting approaches that also leverage inertial sensors will be classified as infrastructure-free COTS Technologies as they do not require to interface any custom hardware to the COTS device being localized.
  • WiFi fingerprinting approaches that also deploy custom BLE beacons in the evaluation area will be classified as infrastructure-based COTS Technologies as they don’t modify the COTS device being localized, but they require the deployment of custom hardware.
  • Approaches based on inertial sensing that require to be initialized to a ground truth location before being evaluated will be classified as COTS Technologies with Initialization.
  • UWB approaches will be classified as infrastructure-based modified COTS Technologies as they require modifications to the COTS device being localized, and they need to deploy custom hardware in the evaluation area.

In the past, teams were asked to report 2D locations (X,Y). In this year’s competition, teams will be required to report 3D locations (X,Y,Z). The evaluation area we have reserved, even though it does not contain multiple floors, it includes locations with different elevation characteristics. On top of that, we plan to place the devices to be localized at different heights during the competition to evaluate the 3D localization accuracy of each team.

Not all teams will have to report 3D locations. We understand that some technologies are not suited for 3D localization (i.e., WiFi/geo-magnetic fingerprinting, dead reckoning, BLE-based etc.), but they are still excellent candidates for commercial indoor location systems. With this in mind, the organizers will determine shortly after the registration deadline which teams will be classified as 3D localization teams. It is not up to the individual teams to decide if they will report 2D or 3D locations. The organizers will ensure that all the teams in a given category will either report 2D or 3D locations. There will be no categories containing both 2D and 3D localization teams.

Evaluation and Prize

Results are judged based on accuracy, and an award will be given to the top 3 teams in each category. When accuracy ties, infrastructure requirements will be used for tie breaking. The winning teams in each category will be invited to present their approach at the conference, and receive a cash award. The exact accuracy metrics that will be used during evaluation will be announced shortly before the competition takes place.

Poster Session

A poster session dedicated to all competition participants will be organized during the conference. Participants will have the opportunity to explain their system to conference attendees.

Submission Guidelines

Contesters must submit an abstract describing their approach and deployment requirements by the contest registration deadline. Submissions are treated as confidential until the competition. Submissions must be at most two (2) single-spaced 8.5″ x 11″ pages, including figures, tables, and references. Submission should follow the exact same format as regular, full IPSN 2016 papers. Abstracts should include the names and affiliations of all authors. Templates can be found here.