In-context learning refers to a surprising ability of very large machine learning models to solve tasks unseen during training, only after observing a short prompt. This is achieved by a long computationally heavy pre-training step, on a multitude of other diverse tasks. Following this step, the model is introduced with a short prompt, describing the new task, and asked to solve it (e.g., predict the next outcome). Despite the fact that the model only observes this short prompt describing the new task, the empirical success of transformer-based language models in such tasks is remarkable. In this project we aim to propose a simple statistical learning model for in-context learning, and to characterize learnability. Unrealistically, previous works typically show that the role of the prompt is to identify a similar task during training. The goal of this project is to go beyond this, and attempt to characterize learnability, specifically in terms of the prompt length.
