Building a Transformer from Scratch: A Collaborative Workshop
Ella Batty, Harvard University, with support from Kempner and MIND Institute community members
Join us for an interactive, cross-continental online workshop bringing together students from the Kempner Institute at Harvard University and the MIND Institute at the University of the Witwatersrand in Johannesburg, South Africa! This workshop aims to foster collaboration and learning primarily among advanced undergraduates and early-stage graduate students. Participants will dive into a language model’s components, training pipeline, and ingredients of a typical end-to-end language model set-up. During the session, participants will work through hands-on exercises and have the opportunity to connect with peers and experts from both institutions.
What will attendees learn from this workshop?
- The components of a standard decoder-only transformer.
- How to code and train a language model from scratch.
- Practical applications and exercises for hands-on learning.
- Opportunities for collaborative problem solving with peers from another institution.
- Potential for continued content learning and experience with global collaborators.
Prerequisites
- Basic familiarity with PyTorch and matrix multiplication.
- Familiarity with machine learning (feedforward network and training)
- Commitment to completing short prework assignment prior to workshop.
Who can attend?
- Primarily aimed at advanced undergraduates and early-stage graduate students from Harvard and Wits. Other community members are also welcome to apply.
Contact Information:
For any questions about the workshop, please contact kempnereducation@harvard.edu