Introduction to Reference Resolution Using Large Language Models

The paper introduces the problem of reference resolution, which is crucial for conversational AI and voice assistants to understand the context of user queries. This includes resolving references to entities on the user's screen, previous conversational turns, and background processes. The authors propose using large language models (LLMs) for this task, as they have shown great potential in various natural language processing (NLP) tasks.

• Reference resolution is essential for understanding context in different scenarios, including on-screen entities, conversational history, and background processes.
• LLMs have demonstrated powerful capabilities in various NLP tasks but are underutilized for reference resolution, particularly for non-conversational entities.
• The paper shows how to convert reference resolution into a language modeling problem by encoding entities as natural text, including on-screen entities.

Challenges and Motivation for Using LLMs

Traditional reference resolution systems have focused on conversational and visual/deictic references but have not explored on-screen references as much. The authors argue that using LLMs for this task can address several challenges:

• On-device constraints: Using a single large end-to-end model may be infeasible on devices with limited computing power and latency constraints.
• Integration with existing pipelines: LLMs can be incorporated into existing pipelines without a complete overhaul.
• Modularity and interpretability: A focused LLM model for reference resolution can be swapped with improved versions transparently.
• Handling non-conversational entities: LLMs can potentially resolve references to on-screen and background entities not part of the conversational history.

Encoding On-Screen Entities for Large Language Models

The authors propose a novel approach to encode on-screen entities as text input for large language models. This involves:

• Parsing the screen to extract entities, their bounding boxes, and surrounding text elements.
• Sorting the entities and objects based on their spatial positions (top-to-bottom, left-to-right).
• Constructing a textual representation of the screen by placing objects on different lines based on their vertical levels and separating elements on the same line with tabs.
• Injecting the encoded entities into the textual representation, tagged with their types.

Experimental Setup and Datasets

The authors used three types of datasets for training and evaluation:

• Conversational data: User queries referring to entities in synthetic lists provided by annotators.
• Synthetic data: Templates with type-based references and slot values.
• On-screen data: Web pages with phone numbers, email addresses, and physical addresses, annotated by crowdworkers.

Model Architecture and Baselines

The proposed model, ReALM, is based on the FLAN-T5 large language model fine-tuned on the reference resolution datasets. The authors compare ReALM's performance against two baselines:

• MARRS: A non-LLM approach specifically designed for reference resolution.
• ChatGPT (GPT-3.5 and GPT-4): Using in-context learning with and without screenshots for on-screen references.

Results and Analysis

The authors report that ReALM outperforms the MARRS baseline across different types of references, including on-screen entities. ReALM also achieves performance comparable to or better than GPT-4, despite being a much lighter model. Key findings include:

• ReALM performs better than MARRS and GPT-3.5 on conversational, synthetic, and on-screen datasets.
• ReALM's performance is comparable to GPT-4 for on-screen references, despite being purely text-based.
• ReALM outperforms GPT-4 on domain-specific queries, benefiting from fine-tuning on relevant data.
• Larger ReALM models show improved performance, especially for complex on-screen datasets.
• ReALM exhibits better semantic understanding, summarization, world knowledge, and commonsense reasoning compared to baselines.

Conclusion and Future Work

The paper demonstrates the effectiveness of using large language models for reference resolution, enabling context understanding for conversational AI and voice assistants. The proposed approach encodes on-screen entities as textual representations, allowing LLMs to resolve references to various entity types. The authors suggest exploring more complex spatial encoding techniques and grid-based representations for future work.