CLIP: Two towers, one embedding space: image ↔ text by cosine similarity
Click any block in the architecture to see what it does, the math behind it, and the exact source lines. Then walk one sample end-to-end and stress-test yourself with the debug lab.
TowersImage ViT/ResNet + Text transformer
Text context77 tokens (BPE), causal
Shared dimembed_dim (e.g. 512)
Match rulecosine · learned temperature
Click a stageMath cardSample stepperDebug trace
Upstream: https://arxiv.org/abs/2103.00020 · code snapshot openai/CLIP @ d05afc4 · MIT
Mental model
The ideas that make this code readable
Layer-by-layer math
Every formula links to the source that implements it
Follow one sample
Tensor flow
Shapes must agree across every stage
| Stage | Input | Output | Op | Note |
|---|
Debug lab
Pick the symptom, work one boundary at a time
Code excerpts
Only the lines these notes point at, from the upstream repository at the pinned commit. Gaps are marked.
Select a file
Knowledge check
Click a question to reveal the answer