NIST Pilot TC4TL Challenge


Contact Us

For TC4TL Challenge information (data, scoring software, etc.) please email: tc4tl_poc@nist.gov

For the TC4TL Challenge discussion please visit our Google Group. https://groups.google.com/a/list.nist.gov/forum/#!forum/tc4tl-challenge


Summary

One of the keys to managing the current (and future) epidemic is notifying people of possible virus exposure so they can isolate and seek treatment to limit further spread of the disease. While manual contact tracing is effective for notifying those who may have been exposed, it is believed that automated exposure notification will be a necessary addition as societies open up. Current approaches to automated exposure notification rely on using Bluetooth Low Energy (BLE) signals (or chirps) from smartphones to detect if a person has been too close for too long (TC4TL) to an infected individual. However, the received signal strength indicator (RSSI) value of Bluetooth chirps sent between phones is a very noisy estimator of the actual distance between the phones and can be dramatically affected in real-world conditions by i) where the phones are carried, ii) body positions, ii) physical barriers, and iv) multi-path environments, to mention a few. To better characterize the effectiveness of range and time estimation using the BLE signal, many research organizations around the world are collecting Bluetooth handshake data as well as other phone sensor data (e.g., accelerometer, gyroscope, proximity) between various types of phones with simulated real-world variability. The best hope for a solution to this difficult and important problem is to leverage the world-wide research community with common tasks, data, and success metrics that allow for the exchange of and building on collective ideas and approaches.

Objectives

The National Institute of Standards and Technology (NIST), in coordination with the MIT PACT project, is organizing a TC4TL detector evaluation to facilitate this research effort. The evaluation serves the following goals:

  • to explore promising new ideas in TC4TL detection using BLE signals,
  • to support the development of advanced technologies incorporating these ideas,
  • to measure performance of the state-of-the-art TC4TL detectors.

It is intended to be of interest to all researchers in the machine learning community interested in the TC4TL detection problem using BLE signals augmented with IMU sensor data. To this end the evaluation is designed to be simple, to focus on core technology issues, to be fully supported, and to be accessible to those wishing to participate.

Tasks and Metrics

The basic task in the NIST TC4TL Challenge is estimating the distance and time between two phones given a series of RSSI values along with other phone sensor data. These distance and time estimates will be converted into contact event hypothesis labels, i.e., TC4TL or not-TC4TL, using two parameters, namely distance (D) and time (T) that define a TC4TL event. Reference TC4TL labels are generated using true distance and time from contact events. Hypothesized labels are compared to reference labels and probability of false negative (Pmiss) and probability of false positive (Pfa) are calculated. A normalized decision cost function (nDCF) combines these two errors into a single value using weights reflecting the relative cost of each type of error.

For more information about the challenge please visit the announcement page or send questions to tc4tl_poc@nist.gov.

June 19, 2020: Evaluation Plan Published

June 22, 2020: Challenge Registration Begins

July 1, 2020: Development/Test Data Available

July 1, 2020: Scoring Software Released

August 28, 2020: Virtual Workshop


Disclaimer:

The purpose of this pilot evaluation is to foster research in and measure performance of proximity detection technologies using a preliminary dataset containing RSSI and IMU measurements of interactions between phone pairs. Acknowledging that this preliminary dataset has limitations in terms of size and available real-world conditions, the results, analysis, and participant feedback from the pilot challenge will be used to refine data, design, and metrics for future evaluations. Participants are allowed to publish the leaderboard results unaltered, but they should not make advertising claims regarding their standing/ranking in the evaluation or winning the evaluation, or claim NIST or the U.S. Government endorsement of their system(s) or commercial product(s). See the evaluation plan for more details regarding the participation rules in the TC4TL Challenge.


TC4TL Challenge

Updated: 2020-08-13T19:56:06-04:00
RANK TEAM TIMESTAMP NDCF(D=1.2|SET=FINE) NDCF(D=1.8|SET=FINE) NDCF(D=3.0|SET=FINE) NDCF(D=1.8|SET=COARSE)
1 Contact-Tracing-Project 20200804-233604 0.730 0.900 0.950 0.530
2 LCD 20200806-173523 0.600 0.580 0.630 0.550
3 PathCheck 20200807-164314 0.820 0.800 0.780 0.690
4 MITRE 20200805-094311 0.940 0.860 0.970 0.880


Note:

The table below summarizes proximity detection results obtained by systems that use the NIST supplied development set for training. The live leaderboard above is for systems not trained using the NIST supplied development set.


TEAM NDCF(D=1.2m|SET=FINE) NDCF(D=1.8m|SET=FINE) NDCF(D=3.0m|SET=FINE) NDCF(D=1.8m|SET=COARSE)
PathCheck 0.37 0.04 0.23 0.02
LCD 0.23 0.21 0.23 0.04
MITRE 0.99 0.17 0.29 0.04
Contact-Tracing-Project 0.24 0.21 0.23 0.19