LLaVA: CLIP patches → MLP projector → spliced into a Llama token stream
Click any block in the architecture to see what it does, the math behind it, and the exact source lines. Then walk one sample end-to-end and stress-test yourself with the debug lab.
VisionFrozen CLIP ViT-L/14
Projector2-layer GELU MLP
FusionIMAGE_TOKEN_INDEX = −200
LLMVicuna / Llama
Click a stageMath cardSample stepperDebug trace
Upstream: https://arxiv.org/abs/2304.08485 · code snapshot haotian-liu/LLaVA @ c121f04 · Apache-2.0
Mental model
The ideas that make this code readable
Layer-by-layer math
Every formula links to the source that implements it
Follow one sample
Tensor flow
Shapes must agree across every stage
| Stage | Input | Output | Op | Note |
|---|
Debug lab
Pick the symptom, work one boundary at a time
Code excerpts
Only the lines these notes point at, from the upstream repository at the pinned commit. Gaps are marked.
Select a file
Knowledge check
Click a question to reveal the answer