Domain-Specific Fine-Tuning of Large Language Models for Interactive Robot Programming

Programming robots currently requires expert knowledge and can be prohibitively expensive, especially for complex tasks involving vision and force sensing. In the paper, we explore the use of Large Language Models (LLMs) to develop a digital assistant for robot programming via interactive dialogue. The assistant will provide explanations for programming skills, usage examples, and step-by-step descriptions of expected robot behavior, helping non-experts understand how to assemble complex programs from simpler skills.

We investigated strategies for fine-tuning large language models in a data- and compute-efficient way. Given the constraints of industrial small and medium enterprises, we focused on techniques that could work within a single server-grade GPU. We experimented with fine-tuning pretrained instruction-following and streaming models using different domain-specific datasets.

Through automatic and user evaluations, we found that directly fine-tuning an instruction-following model achieved the best performance. However, all models still exhibited shortcomings like incorrect or out-of-domain responses, suggesting that fine-tuning alone is not sufficient to thoroughly impart new knowledge with limited data.

Moving forward, we plan to explore prompting strategies without fine-tuning, which may be better suited to this challenge. We also aim to conduct larger-scale user studies to better understand real-world model capabilities. Our goal is to develop an assistant that can reliably help non-experts program complex robot applications through natural language. With further refinements, we believe this research could help make advanced robotics more accessible.