Learn AI by
Building It
“What I cannot create, I do not understand.”— Richard Feynman
text
→tokens
→vectors
→attention
→GPT
transformer.py
input
"The cat sat on the mat"|
↓
tokenize
Thecatsatonthemat
embed
[ 0.82, -0.31, 0.67, ...]
[-0.45, 0.91, 0.23, ...]
self-attention
generate
"...and purred softly."
Learn by Doing
Coding Challenges
RunSubmit
def gelu(x):
"""GELU activation."""
return 0.5 * x * (1 + tanh(
sqrt(2/pi) * (x + 0.044715
* x**3)))
✓ All tests passed
Interactive Visuals
d₀
d₂
d₄
d₆
Each dimension oscillates at a different frequency.
Progress Tracking
✓04 Positional Enc.3/3
✓05 Attention2/2
✓06 Multi-Head1/1
07 Feed-Forward1/2
08 Residuals0/2
Courses
Choose a path and start building.
Build GPT from Scratch
From raw text to a working GPT. You'll implement every component of a transformer-based language model from the ground up.
TransformersPyTorchGPT
11 chapters · 20 problems
Start Building
Coming Soon
Mechanistic Interpretability
Reverse-engineer neural networks. Dissect transformers layer by layer, neuron by neuron.
InterpretabilityTransformersResearch
Locked