Human-Centered LLM-Agent User Interface: A Position Paper
Chin, Daniel Wang, Yuxuan Xia, Gus
Chin, Daniel
Wang, Yuxuan
Xia, Gus
Author
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Large Language Model (LLM) -in-the-loop applications have been shown to effectively interpret the human user’s commands, make plans, and operate external tools/systems accordingly. Still, the operation scope of the LLM agent is limited to passively following the user, requiring the user to frame his/her needs with regard to the underlying tools/systems. We note that the potential of an LLM-Agent User Interface (LAUI) is much greater. A user mostly ignorant to the underlying tools/systems should be able to work with a LAUI to discover an emergent workflow. Contrary to the conventional way of designing an explorable GUI to teach the user a predefined set of ways to use the system, in the ideal LAUI, the LLM agent is initialized to be proficient with the system, proactively studies the user and his/her needs, and proposes new interaction schemes to the user. To illustrate LAUI, we present Flute X GPT, a concrete example using an LLM agent, a prompt manager, and a flute-tutoring multi-modal software-hardware system to facilitate the complex, real-time user experience of learning to play the flute.
Citation
D. Chin, Y. Wang, and G. Xia, “Human-Centered LLM-Agent User Interface: A Position Paper,” Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST, vol. 650 LNICST, pp. 251–268, 2025, doi: 10.1007/978-3-031-97254-6_18
Source
Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST
Conference
13th EAI International Conference on ArtsIT, Interactivity and Game Creation, ArtsIT 2024
Keywords
Human-computer interaction, llm agent, LLM-in-the-loop, user interface
Subjects
Source
13th EAI International Conference on ArtsIT, Interactivity and Game Creation, ArtsIT 2024
Publisher
Springer Nature