ABSTRACT: This talk covers some of our recent explorations on estimating the robustness of black-box machine learning models across data subpopulations. In other words, if a trained model is uniformly accurate across different types of inputs, or if there are significant performance disparities affecting the different subpopulations. Measuring such a characteristic is fairly straightforward if the subpopulations are clearly defined and provided as categorical labels for a validation dataset. However, in most scenarios and datasets, such a detailed breakdown does not exist and therefore one cannot evaluate such robustness metrics. In this talk, we go over our explorations on how non-categorical proxies of data subpopulations could be used to infer an estimate of such performance disparity metrics, and thus the robustness of a black-box model.

BIO:

Shervin Ardeshir is a Senior Research Scientist at Netflix, where he works on the intersection of media understanding and recommender systems. His research interest is in Machine Learning Safety, and with a focus on probing black-box ML models in terms of robustness.

Prior to joining Netflix, he received his MSc & PhD degree from University of Central Florida in 2016 and 2018, and his BSc degree in electrical engineering from the Sharif University of Technology in 2012.

Homepage: http://www.shervin-ardeshir.com/

Sponsored in part by:   Meta Reality Labs Pittsburgh