PhD Thesis Proposal

Abstract: A new category of computing devices is emerging: augmented and virtual reality headsets, collectively referred to as extended reality (XR). These devices can alter, augment, or even replace our reality. While these headsets have made impressive strides in audio-visual immersion over the past half-century, XR interactions remain almost completely absent of appropriately expressive tactile [...]

Abstract: In this talk, we present our work on advancing Vision-Language Models (VLMs) beyond scaling laws through improved evaluation and (post-)training strategies. Our contributions include VQAScore, a state-of-the-art alignment metric for text-to-visual generation. We show how VQAScore improves visual generation under real-world user prompts in GenAI-Bench. Additionally, we explore training methods that leverage the language [...]

Abstract: Many real-world challenges, from robotic control to resource management, can be effectively formulated as optimization problems. Recent advancements have focused on incorporating these optimization problems as layers within deep learning pipelines, enabling the explicit inclusion of auxiliary constraints or cost functions, which is crucial for applications such as enforcing physical laws, ensuring safety constraints, [...]

Abstract: Large text-to-image models learn from training data to synthesize "novel" images, but how the models use the training data remains a mystery. The problem of data attribution is to identify which training images are influential for generating a given output. Specifically, removing influential images and retraining the model would prevent it from reproducing that [...]

Abstract: This thesis explores diverse decision-making strategies for autonomous agents by examining knowledge-dependent and data-dependent approaches in stationary and dynamic data environments. We address five core research problems across three thematic areas: knowledge-dependent, stationary data-dependent, and evolving data-dependent decision-making. We first investigate knowledge-driven decision-making within robotic swarms, characterizing vulnerabilities in systems governed by consistent rule-following [...]

Abstract: Ensuring pedestrian safety in dynamic environments is a key challenge for autonomous systems, particularly in dynamic, multi-agent environments. Trajectory forecasting plays a central role in enabling these systems to anticipate pedestrian behaviors and respond appropriately. This thesis addresses three core limitations in trajectory forecasting systems which impede safe and robust trajectory forecasting: inadequate evaluation protocols [...]

Abstract: Generative models have advanced significantly, synthesizing photorealistic images, videos, and text. Building on this progress, our work explores the potential of diffusion models to bridge generative and discriminative learning, uncovering new pathways for leveraging their strengths in visual perception tasks. In the first part, we propose Diff-2-in-1, a unified framework for multi-modal data generation [...]

Abstract: Uncrewed Aerial Vehicles (UAVs) have attracted the interest of researchers, industry, and the general public in many applications. Noticing that high-altitude tasks sometimes require active interaction with the environment, there have been more and more works focusing on aerial manipulation recently. Each of them has demonstrated the ability to use a specific aerial manipulator [...]