Abstract: There are many problems in real life that involve collecting and aggregating evaluation from people, such as hiring, peer grading and conference peer review. In this thesis, we focus on three sources of biases that arise in such problems, and propose methods to mitigate them. First, we study human bias, that is, the bias [...]