Transformers-as-Algorithms

This repository contains the source code of our paper. Part of the code is borrowed from in-context-learning.

Transformers as Algorithms: Generalization and Stability in In-context Learning
Yingcong Li, M Emrullah Ildiz, Dimitris Papailiopoulos, Samet Oymak
Paper: https://arxiv.org/pdf/2301.07067.pdf

Start up

Follow in-context-learning's instruction by running

conda env create -f environment.yml
conda activate in-context-learning

Training

1. Enter into a specific directory src or src_dynamic
2. Run train.py based on the chosen config file: python train.py --config conf/[config_file].yaml

Test

[TO BE DONE]

Contact

If you have any question, please contact Yingcong Li ([email protected])