2017 Microsoft Indoor Localization Competition @ IPSN 2017

2017 Microsoft Indoor Localization Competition @ IPSN 2017

Official Results

Important notes on the official results:

  1. The JRC (European Commission Joint Research Center) team was NOT competing. This team helped with the organization of the event. In particular, they used 3D laser scanners to map the evaluation area and measure ground truth coordinates for all evaluation points. The accuracy reported for this team is based on their mobile lidar-based system (different than the ground-truth measuring system) that won the 2015 Microsoft Indoor Localization Competition. Even though not officially competing, we provide this system’s results for awareness.
  2. The team from Kiwii logged the estimated locations for two of the points while swapping X and Y coordinates. This resulted into a higher average localization error. When this mistake is taken into account, the average localization error for this team is within 2m.

 

2017 Microsoft Indoor Localization Competition (Slides used to announce the results during the Indoor Localization Session)

2017CompetitionOfficialResults (Raw data including the ground truth coordinates and estimated locations for every team)

The JRC team that helped with ground truth measurements has decided to make the collected point cloud available to the public. You can find the data here.

 

Call for Contesters

Accurate indoor localization has the potential to transform the way people navigate indoors in a similar way that GPS transformed the way people navigate outdoors. Over the last 15 years, several indoor localization technologies have been proposed and experimented by both academia and industry, but we have yet to see large scale deployments. This competition aims to bring together real-time or near real-time indoor location technologies and compare their performance in the same space.

Eligibility

Both academia and industry submissions are encouraged. All location techniques, such as ranging, fingerprinting, infrastructure, or device free, are welcome, except those that require end users’ manual measurements. Contesters can deploy their own infrastructure of up to 10 devices (this number is tentative and it will be finalized after the submission deadline). Normal RF interference is expected, but no jammers from other deployments are allowed. The results must be shown on a portable device, such as a phone or a tablet/laptop that a person can easily carry around.

Demo submissions that do not meet one or more of the guidelines above will be included in the poster session and will be evaluated as a regular submission, but they will not be considered for prizes.

The competition will take place if at least 5 teams respond to this preliminary call for competition.

Competition Categories

Depending on the nature and number of submissions multiple categories might be defined based on the accuracy (i.e., point-based vs. area based), the size, the cost, or the type (i.e., software vs. hardware) of the proposed solution.

Given our past experience, this year we expect to have the following categories (note that these categories have not been finalized. The final categories will be announced shortly after the registration deadline and they will depend on the number and type of submissions received):

  1. Commercial off-the-shelf (COTS) Technologies: Submissions in this category should be able to work with unmodified commercial off-the-shelf devices such as laptops, phones, and tablets. In this category, the unmodified COTS device is localized. Teams in this category will not be allowed to interface any custom hardware to the COTS devices (i.e., UWB or ultrasound hardware). Only changes to the software of the devices will be allowed. Submissions in this category could be further classified to infrastructure-based and infrastructure-free depending on their requirement to deploy custom hardware (i.e., BLE beacons) in the evaluation area.
  2. Commercial off-the-shelf (COTS) Technologies with Initialization: Submissions in this category should meet all the criteria of the COTS Technologies category, with the additional requirement to initialize the location of the COTS device being localized to a ground truth location before the evaluation.
  3. Modified Commercial off-the-shelf (COTS) Technologies: Submissions in this category could interface custom hardware to the COTS devices to be localized (i.e., UWB or ultrasound hardware). Submissions in this category could be further classified to infrastructure-based and infrastructure-free depending on their requirement to deploy custom hardware (i.e., BLE beacons) in the evaluation area. Most, if not all, submissions in this category are expected to be infrastructure-based.

For instance:

  • WiFi fingerprinting approaches that also leverage inertial sensors will be classified as infrastructure-free COTS Technologies as they do not require to interface any custom hardware to the COTS device being localized.
  • WiFi fingerprinting approaches that also deploy custom BLE beacons in the evaluation area will be classified as infrastructure-based COTS Technologies as they don’t modify the COTS device being localized, but they require the deployment of custom hardware.
  • Approaches based on inertial sensing that require to be initialized to a ground truth location before being evaluated will be classified as COTS Technologies with Initialization.
  • UWB approaches will be classified as infrastructure-based modified COTS Technologies as they require modifications to the COTS device being localized, and they need to deploy custom hardware in the evaluation area.

In this year’s competition, teams will be required to report 3D locations (X,Y,Z). The evaluation area we have reserved, even though it does not contain multiple floors, it includes locations with different elevation characteristics. On top of that, we plan to place the devices to be localized at different heights during the competition to evaluate the 3D localization accuracy of each team.

Not all teams will have to report 3D locations. We understand that some technologies are not suited for 3D localization (i.e., WiFi/geo-magnetic fingerprinting, dead reckoning, BLE-based etc.), but they are still excellent candidates for commercial indoor location systems. With this in mind, the organizers will determine shortly after the registration deadline which teams will be classified as 3D localization teams. It is not up to the individual teams to decide if they will report 2D or 3D locations. The organizers will ensure that all the teams in a given category will either report 2D or 3D locations. There will be no categories containing both 2D and 3D localization teams.

Evaluation and Prizes

Results are judged based on accuracy, and an award will be given to the top 3 teams in each category. When accuracy ties, infrastructure requirements will be used for tie breaking. The winning teams in each category will be invited to present their approach at the conference, and receive a cash award. The exact accuracy metrics that will be used during evaluation will be announced shortly before the competition takes place.

Poster Session

A poster session dedicated to all competition participants will be organized during the conference. Participants will have the opportunity to explain their system to conference attendees.

Submission Guidelines

Contesters must submit an abstract describing their approach and deployment requirements by the contest registration deadline. Submissions are treated as confidential until the competition. Submissions must be at most two (2) single-spaced 8.5″ x 11″ pages, including figures, tables, and references. Submission should follow the exact same format as regular, full IPSN 2017 papers. Abstracts should include the names and affiliations of all authors. Templates can be found here. Abstracts should be sent over email to: dlymper@microsoft.com on or before January 27th 2017 with the following subject line: 2017 Microsoft Indoor Localization Competition Submission.

 

Competition Logistics

The competition is a 2-day event. During the first day, teams setup and calibrate their systems. During the second day, each team is evaluated at a pre-assigned time slot. Please refer to the Competing Teams tab for more details on the teams and their assigned time slots.

The schedule for the competition is as follows:

Tuesday April 18th 2017

  • 8:30am – 9am: Introduction and overview of the competition rules
  • 9am – 4pm: System setup time

Wednesday April 19th 2017

  • 9am-5pm: Evaluation of individual teams at pre-assigned time slots

Thursday April 20th 2017

  • 14:20 – 15:10 : Announcement of the official competition results

Previous Competitions

Competing Teams

29 indoor location systems from 27 teams have registered to participate to the competition. Given the type and number of submissions we received, we had to classify all teams in two categories similarly to the previous years: Infrastructure-Free (14 submissions) and Infrastructure-Based (15 submissions) systems. A list of all the teams in each of the categories is provided below.

All teams that have been assigned an evaluation time slot will be allowed to participate. Teams that have not been assigned an evaluation time slot have NOT officially registered for the competition as of 03/28/2017 and will not be allowed to participate. 

Infrastructure-Free (2D Localization)

Authors Affiliation Title Evaluation Time Slot
Ben-Moshe et al. Israel – Ariel University GoIn: An Accurate Indoor Navigation Framework for Mobile Devices 11:00
Liao et al.. Taiwan – Aquaways CO. Ltd. Indoor Localization using Broadcasting FM Signals TDOA triangulation N/A
Sadhu et al. USA – Rutgers University, Department of Homeland Security CollabLoc: Infrastructure-free Privacy-preserving Localization via Collaborative Information Fusion 09:00
Castro et al.. Portugal – Fraunhofer Portugal Research Center Precise Indoor Location 09:20
Castro et al. Portugal – Fraunhofer Portugal Research Center Precise Indoor Location Through Multiple Devices 09:40
Ju et al.. Korea – Seoul National University Pedestrian Dead Reckoning System Considering Actual Condition of the Foot-mounted IMU 10:00
Cutri et al. Italy – GiPStech s.r.l A Hybrid Geomagnetic Field Indoor Localization Technology N/A
Li et al.. China – Jinkun Innovation Technology Co., Ltd Accurate Indoor Localization and Navigation System N/A
Jia et al. China – Baidu Fingerprint Location Method Based on WiFi and Geomagnetism Fusion N/A
Zheng et al.. China – Huawei Dolphin Indoor Localization System based on Hidden Markov Model N/A
Li et al. Canada – McMaster University Robot-assisted Fingerprint-based Indoor Localization 10:20
Su et al.. Taiwan – National Taiwan University of Science and Technology Implementing an iBeacon Indoor Positioning System using Ensemble Learning Algorithm 11:40
Liddel et al Iceland – Locatify Automatic Museum Guide by Locatify N/A
Kikuchi et al.. Japan – Iwate University DOD-based Indoor Localization using BLE Beacons 10:40
Wu et al China – Glodon Software Co., Ltd Indoor Positioning using WiFi Fingerprinting and Inertial Sensors 11:20

Infrastructure-Based (3D Localization)

Authors Affiliation Title Eval. Time Slot
Wang et al. Canada – McMaster University Indoor Localization System with Asynchronous Acoustic Beacon 11:40
Lin et al.. China Low-cost, High-Accuracy Indoor Positioning N/A
Chen et al. Saudi Arabia/Ireland – King Abdullah University of Science and Tech., University College Dublin KAUST Acoustic Positioning System 12:00
Acton et al.. USA – ARIN Technologies Scalable, Low-cost Indoor Localization System using TDoA and UWB 13:40
Beuchat et al. Switzerland/UK – ETH, Embotech, Hexagon Mining, Aerospace Systems UWB-based Indoor Localization 14:00
Franzel et al.. Germany – Fraunhofer Institute of Optronics, Schmalkalden University of Applied Science A Wireless Ad Hoc Localization System N/A
Cheng et al. USA – Kiwii Power Technology Corporation Real Time Indoor Locating System based on UWB and Machine Learning 14:20
Xia et al.. USA – M2Robots Inc. Low-cost, User-friendly, Indoor Localization Device 14:40
Singh et al. Switzerland – ETH IR-UWB based Indoor Localization System N/A
Gunes et al.. Germany – Quantitec GmbH IntraNav – Low-cost Indoor Localization System and IoT Platform N/A
Hemamali et al. India – Trakray Pvt. Ltd. Low Power UWB 3D Positioning System 15:00
Bjornsson et al.. Iceland – Locatify Locatify UWB Solution N/A
Zhu et al. Singapore/China – Kuncen Technology PTE. Ltd., University of Electronics Science and Technology of China Ultra Precise Location SYstem based on UWB Sparse Pulses N/A
Snyder et al.. USA – Astrobotic AstroNav: Robust, High Rate SLAM for Planetary Exploration 15:20
Zhang et al.. USA – Kaarta Compact, Real-time Localization without Reliance on Infrastructure 15:40