Composable Semantic Frames for Grounding Language in Robot Control Primitives

Abstract

As robots become increasingly capable of taking on challenging tasks, we want robots to be commanded in intuitive ways. Non-expert users in particular should be able to communicate with robots about task goals. As a result, modes of interaction such as language have gained interest for commanding robots. We present composable semantic frames, which ground commands in robot control primitives by composing high-level commands from lower-level commands. We demonstrate that composable semantic frames allow robots to understand and execute a variety of challenging commands, such as those involving multiple verb meanings, command variations, and compound nouns. The robot quickly processes composable semantic frames and accurately grounds and executes the commanded tasks, demonstrating the power of composable semantic frames for allowing users to intuitively interact with robots.