Vector-ICL: In-context Learning with Continuous Vector Representations (Zhuang et al., ICLR 2025).
Vector-ICL is a way of conducting in-context learning with data of any forms.
1. Encode data into embedding vector
2. Project the vector into LLMs' representation space
3. Build the context with the projected embeddings
And we only need to train a light-weight projector (a linear matrix works most of the time!).
(a) Pretraining with next token prediction (such as language modelling objective) enables Vector-ICL,
(b) task finetunes further improves LLMs' ability to conduct Vector-ICL.
We show that Vector-ICL works for a wide range of modalities and tasks, surpassing few-shot ICL and domain-specific models and tunings.
We show examples of using Vector-ICL at text-based tasks over here:
| Example | Script |
|---|---|
| Pretraining Projector | example script |
| Fituning Projector | example script |
| Evaluating text classification | example script |
| Evaluating text generation | example script |
| Evaluating ICL Baselines | example script |
For cross-modal Vector-ICL, the scripts are provided in this folder with training and inferencing pipelines.
If you have any questions related to the code or the paper, feel free to reach out to us at [email protected].
If you find our paper and code useful, please cite us:
@article{zhuang2024vector,
title={Vector-ICL: In-context Learning with Continuous Vector Representations},
author={Zhuang, Yufan and Singh, Chandan and Liu, Liyuan and Shang, Jingbo and Gao, Jianfeng},
journal={arXiv preprint arXiv:2410.05629},
year={2024}
}