Abstract:
The introduction of generative AI has brought about many improvements in the artistic world. It allows many individuals to create artwork via simple descriptive text prompts. This has, in particular, created an avenue for non-artistic individuals to express their thoughts through generated art. Our work focuses on how emotion can be added as an additional modality to assist individuals and guide the system better to create more user-intended media in a collaborative space. In particular, we use audio as the primary input modality to capture the emotion through the voice’s raw tone, pitch, and pace. We hypothesize that users will best benefit from active emotional feedback during art generation in an interactive space instead of a simple generative pass or an interactive system without emotion analysis. Therefore, we propose using a personalized Speech Emotion Recognition system combined with a collaborative system, and the generation of desirable artistic media is obtained. To address this, we consider (1) a personalized emotion calibration model, (2) an emotion-guided Detect and Respond (DR) system from the personalized emotion calibration model, and (3) the introduction of the combined personalized finetuned emotion recognition DR system (PIE-FRIDA) in a collaborative artistic space. Our results show and support the advantage of introducing emotion in the generative space.

Committee:
Prof. Jean Oh (advisor)
Prof. Henny Admoni
Prof. Jim McCann
Peter Schaldenbrand