TREC 2019 Fair Ranking Track

Click here to switch to the current Fair Ranking Track webpage.

The TREC Fair Ranking track evaluates systems according to how well they fairly rank documents. The 2019 task focuses on re-ranking academic abstracts given a query. The objective is to fairly represent relevant authors from several, undisclosed group definitions. These groups can be defined in a variety of ways and the track emphasizes the development of systems which have robust performance across a variety of group definitions.

GUIDELINES

We have released the track guidelines, including a description of the dataset, experimentation protocol, and evaluation metrics here. We are also releasing simulation code to generate query sequences similar to those you will receive in August.

DATA

The corpus for this project is the Semantic Scholar (S2) Open Corpus from the Allen Institute for Artificial Intelligence, consisting of 47 1GB data files. We have an associated list of ~600 queries and relevance estimates that we plan on releasing the first week of June. For the authors appearing in candidate set, we have example group assignments. Data formats are described in the guidelines document.

TIMELINE

• June 3, 2019: guidelines, data released; system development begins
• August 13, 2019: evaluation query sequences released
• August 16, 2019: submission site open
• September 2, 2019: submissions due
• late September, 2019: evaluated submissions returned

DOWNLOADS

• TREC Fair Ranking evaluation package: scripts and data (updated October 28, 2019). Note: by downloading the data sample contained in this package, you agree to the Semantic Scholar Dataset License Agreement.
• TREC Fair Ranking Track Participant Instructions: guidelines, experimentation protocol, and data format description (updated August 13, 2019).
• query-sequence-generator.py: Python script to generate query sequences (updated June 3, 2019).
• fair-TREC-training-sample.json: sample relevance estimates. note: by downloading this sample, you agree to the Semantic Scholar Dataset License Agreement. (updated July 2, 2019).
• fair-TREC-sample-author-groups.csv: sample author group membership (updated June 3, 2019).
• fair-TREC-evaluation-sample.json: evaluation queries and documents to be re-ranked. note: by downloading this sample, you agree to the AI2 terms of service. (updated August 13, 2019).
• fair-TREC-evaluation-sequences.csv: evaluation query sequences. (updated August 13, 2019).
• validate-run.py: submission validation script and example submission. (updated August 16, 2019).

RESOURCES

• Google Group
• Twitter
• Slack channel

CITATION

@inproceedings{trec-fair-ranking-2019,
	Author = {Asia J. Biega and Fernando Diaz and Michael D. Ekstrand and Sebastian Kohlmeier},
	Booktitle = {The Twenty-Eighth Text REtrieval Conference (TREC 2019) Proceedings},
	Title = {Overview of the TREC 2019 Fair Ranking Track},
	Year = {2019}}

Organizers

Montréal

Montréal