One encoder, many ways to feel
Robot touch comes in incompatible shapes: 1D pressure arrays, 3-axis force taxels, and camera-based sensors that see their own gel. These notes walk through one way to learn a shared representation from all of them without labels: per-sensor stems into a shared transformer, trained with the DINOv2 recipe (DINO, iBOT, KoLeo) and an EMA teacher. Click any block for what happens and a short snippet.
Inputspressure arrays · 3-axis taxels · tactile images
Backboneshared transformer + register tokens
ObjectiveDINO + iBOT + KoLeo, EMA teacher
Readoutsdense patches or a fixed token budget
Click stage → detail + codeDebug checkpointArtifact boundary
Mental model
The ideas that make a self-supervised tactile encoder readable
Math & flow
From raw taxels to tokens, targets and losses: each formula links to a snippet
Data contracts
What each stage hands to the next: get these right and training composes
| Artifact | Produced by | Consumed by | Format | Notes |
|---|
The one rule that shapes everything: a batch holds exactly one sensor type. Stems, token counts, normalization and positional tables all depend on it, and every DDP rank must make the same choice on every step.
Debug lab
Pick the symptom you see, then work one boundary at a time
Code snippets
Short snippets written for these notes. They show the method, not a specific codebase.
Select a file
Knowledge check
Click a question to reveal the answer
Debug order
- Check the batch contract: one sensor type per batch, the same on every rank.
- Check normalization: per-sensor statistics, computed on training data only.
- Check the token count: frames × taxels after chunking, plus registers.
- Watch for collapse: teacher entropy, KoLeo term, probe accuracy, before the total loss.
- Only then tune: mask ratio, crop sizes, temperatures, loss weights, momentum.