2025-01-14-bougie25a.md

title	Exploring Beyond Curiosity Rewards: Language-Driven Exploration in RL
booktitle	Proceedings of the 16th Asian Conference on Machine Learning
year	2025
volume	260
series	Proceedings of Machine Learning Research
month	0
publisher	PMLR
pdf	https://raw.githubusercontent.com/mlresearch/v260/main/assets/bougie25a/bougie25a.pdf
url	https://proceedings.mlr.press/v260/bougie25a.html
openreview	qHv7qTETsw
abstract	Sparse rewards pose a significant challenge for many reinforcement learning algorithms, which struggle in the absence of a dense, well-shaped reward function. Drawing inspiration from the curiosity exhibited in animals, intrinsically-driven methods overcome this drawback by incentivizing agents to explore novel states. Yet, in the absence of domain-specific priors, sample efficiency is hindered as most discovered novelty has little relevance to the true task reward. We present iLLM, a curiosity-driven approach that leverages the inductive bias of foundation models — Large Language Models, as a source of information about plausibly useful behaviors. Two tasks are introduced for shaping exploration: 1) action generation and 2) history compression, where the language model is prompted with a description of the state-action trajectory. We further propose a technique for mapping state-action pairs to pretrained token embeddings of the language model in order to alleviate the need for explicit textual descriptions of the environment. By distilling prior knowledge from large language models, iLLM encourages agents to discover diverse and human-meaningful behaviors without requiring direct human intervention. We evaluate the proposed method on BabyAI-Text, MiniHack, Atari games, and Crafter tasks, demonstrating higher sample efficiency compared to prior curiosity-driven approaches.
layout	inproceedings
issn	2640-3498
id	bougie25a
tex_title	{Exploring Beyond Curiosity Rewards}: {L}anguage-Driven Exploration in {RL}
firstpage	127
lastpage	142
page	127-142
order	127
cycles	false
bibtex_editor	Nguyen, Vu and Lin, Hsuan-Tien
editor	given family Vu Nguyen given family Hsuan-Tien Lin
bibtex_author	Bougie, Nicolas and Watanabe, Narimasa
author	given family Nicolas Bougie given family Narimasa Watanabe
date	2025-01-14
address
container-title	Proceedings of the 16th Asian Conference on Machine Learning
genre	inproceedings
issued	date-parts 2025 1 14
extras	label link Supplementary PDF https://raw.githubusercontent.com/mlresearch/v260/main/assets/assets/bougie25a/bougie25a-supp.pdf