Towards strong generalization in LLMs

Abstract: In order to truly aid in solving open-ended problems, future language models must demonstrate new capabilities beyond their training data — a paradigm of strong generalization. I argue that this requires three ingredients: robust out-of-distribution generalization, continual learning without forgetting, and effective learning from limited supervision. I’ll show how curriculum design in data and supervision feedback can drive these abilities, and discuss key questions toward building LLM systems that continue to learn with limited supervision.