Post-Training LLMs as Better Decision-Making Agents: A Regret-Minimization Approach
Chanwoo Park, Ziyang Chen, Asuman Ozdaglar, Kaiqing Zhang
Why It Matters
What makes this one worth your time
Improving LLMs' decision-making abilities is crucial for their effective deployment in dynamic environments, making this research relevant for both practical applications and further theoretical exploration.
Iterative RMFT enhances LLM decision-making through a novel regret-minimization approach.
Summary
The paper introduces Iterative Regret-Minimization Fine-Tuning (Iterative RMFT), a post-training method for enhancing decision-making capabilities of large language models (LLMs) by distilling low-regret decision trajectories into the model.
Key contributions
- Introduction of Iterative RMFT as a novel post-training procedure for LLMs.
- Demonstration of improved decision-making performance across various LLM architectures.
- Theoretical analysis showing the potential of a single-layer Transformer as a no-regret learner.
Notable insights
- The approach leverages model-generated reasoning to avoid rigid output engineering, allowing for more flexible training signals.
- The theoretical insight that a single-layer Transformer can act as a no-regret learner under this paradigm is a notable contribution to understanding LLM capabilities.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2511.04393v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed as "agents" for decision-making (DM) in interactive and dynamic environments. Yet, since they were not originally designed for DM, recent studies show that LLMs can struggle even in basic online DM problems, failing to achieve low regret or an effective exploration-exploitation tradeoff. To address this, we introduce Iterative Regret-Minimization Fine-Tuning (Iterative RMFT), a post-training procedure that repeatedly distills low-regret decision trajectories back into the base model. At each iteration, the model rolls out multiple decision trajectories, selects the k-lowest regret ones, and fine-tunes itself on them. Unlike prior methods that (a) distill action sequences from known DM algorithms or (b) rely on manually crafted chain-of-thought templates, our approach leverages the regret metric to elicit the model's own DM ability and reasoning rationales. This reliance on model-generated reasoning avoids rigid output engineering and provides more flexible, natural-language training signals. Empirical results show that Iterative RMFT improves LLMs' DM performance across diverse models - from Transformers with numerical input/output, to open-weight LLMs, and advanced closed-weight models like GPT-4o mini. Its flexibility in output and reasoning formats enables generalization across tasks with varying horizons, action spaces, reward processes, and natural-language contexts. Finally, we provide theoretical insight showing that a single-layer Transformer under this paradigm can act as a no-regret learner in a simplified setting. Overall, Iterative RMFT offers a principled and general post-training framework for enhancing LLMs' decision-making capabilities.