Abstract:
There are many problems in real life that involve collecting and aggregating evaluation from people, such as conference peer review and peer grading. In this thesis, we consider multiple sources of biases that may arise in this process: (1) human bias — the data collected from people are noisy and reflect people’s calibration criteria and subjective opinions; (2) estimation bias — algorithms and estimators may yield different performance on different subgroups of the population; (3) mechanism bias — inappropriate policies may induce misaligned incentives and undesirable behaviors of the agents.

In this thesis, we aim to understand the nature and the extent of these biases, and develop methods to mitigate them. On the human side, we propose randomized algorithms that work under arbitrary miscalibration of people. On the estimation side, we analyze the bias (defined in statistics as the expected value of the estimate minus the true value) when using the maximum-likelihood estimator on pairwise comparison data, and propose a simple modification to reduce the bias significantly. On the mechanism side, we describe an outreach effort to reduce the bias caused by the alphabetical-ordering authorship in scientific publications.

Building on these results, we propose to further study (1) the human bias induced by the specific experience of people; (2) the estimation bias induced by the variance of the items of interests; (3) the mechanism bias which leads to authors’ insufficient self-selection on the quality of the paper submissions, in the context of the massive number of papers submitted to AI/ML conferences in recent years.

More Information

Thesis Committee Members:
Nihar B. Shah, Chair
Ariel Procaccia
Artur Dubrawski
Jeff Schneider
Avrim Blum, Toyota Technological Institute at Chicago