Read six foundation models like you debug them live

Each course opens a clickable architecture: click a stage to see what it does and the exact source lines, read the layer-by-layer math, walk one sample through every shape, and drill the debug lab. All pages are fully offline and cite pinned upstream snapshots.

Vision encodersViT · CLIP · DINOv2

Multimodal & VLALLaVA · OpenVLA · π₀.₅

Per pagePipeline · math · stepper · code

Network callsNone

Courses

Pick a model to start

Vision encoder

ViT

An image is 16×16 words: pure-transformer image classification

google-research/vision_transformer64801f1Apache-2.0

Vision-language dual encoder

CLIP

Two towers, one embedding space: image ↔ text by cosine similarity

openai/CLIPd05afc4MIT

Self-supervised vision encoder

DINOv2

Self-supervised features: no labels, register tokens, teacher–student EMA

facebookresearch/dinov27764ea0Apache-2.0

Multimodal LLM (image + text)

LLaVA

CLIP patches → MLP projector → spliced into a Llama token stream

haotian-liu/LLaVAc121f04Apache-2.0

Vision-language-action (discrete)

OpenVLA

Robot actions as language: fused DINO+SigLIP → Llama → 256-bin action tokens

openvla/openvlac8f03f4MIT

Vision-language-action (continuous flow)

π₀.₅

PaliGemma prefix + action-expert suffix + flow-matching denoising

Physical-Intelligence/openpi215abfbApache-2.0

Cross-model comparison

Same questions, six answers: how each model tokenizes, projects, trains, and decodes

Aspect	ViT	CLIP	DINOv2	LLaVA	OpenVLA	π₀.₅
Patch / token embedding	Conv patch → token	Conv patch (img) + BPE (text)	Conv patch + register/mask	CLIP patches (CLS dropped)	Fused DINO+SigLIP patches	SigLIP patches + prompt tokens
Special tokens	CLS	CLS (img) + EOT (text)	CLS + register + mask	IMAGE_TOKEN_INDEX (−200)	action-bin tokens (last vocab)	state + action tokens
Projection layer	patch conv → D	linear proj to shared dim	none (raw features)	MLP projector (2×GELU)	MLP projector (3×GELU, fused)	linear action/state proj
Training objective	supervised cross-entropy	symmetric contrastive (InfoNCE)	DINO + iBOT + KoLeo (SSL)	next-token cross-entropy	next-token CE on action tokens	flow-matching MSE
State representation	— (no state)	— (no state)	— (no state)	— (no state)	none (explicit)	discrete state tokens / state_proj
Output decoding	class logits	cosine similarity	feature vectors / maps	text tokens	discrete action bins → unnorm	continuous actions (Euler)

How to read any of these

Start at the architecture map and click the stage you care about.
Open the exact code from the drawer; it highlights the lines in place.
Check the math card for the formula behind that layer.
Follow one sample to watch shapes transform end-to-end.
Break it on purpose in the debug lab, then confirm your fix.

Attribution, licenses, and per-file hashes: see ATTRIBUTION.md and source-manifest.json.