These models are trained and benchmarked almost entirely on bare hands. But the places where you’d most want to collect this kind of data, factories, warehouses, kitchens, are exactly the places where operators wear gloves. I wanted to know two things: what gloves actually break, and which choices at capture time (the glove, the cameras, a quality check) fix it.

How I tested it

The plan was a two-camera rig at home with real gloves. I didn’t have a second camera, so the experiment moved to DexYCB: 19 held-out grasping sequences, each filmed by 8 synchronised cameras, with ground-truth 3D hand joints. Most results use two of those cameras, about a metre apart at 70°.

The gloves are painted on. Inside each frame’s hand mask I replace the skin colour with a glove texture, but keep the original brightness as shading, and the ground-truth skeleton never moves. That last part matters: a real thick glove makes the hand look bigger without moving the bones, and that’s the effect I wanted to isolate.

I ran three hand-pose models: MediaPipe Hands, WiLoR and HaMeR. Because painted gloves are only an approximation, I also checked detection on 83 real photos of bare and gloved hands. Every number below is a mean over sequences, with a 95% bootstrap interval over sequences behind it.

Real gloves break detection, and painted ones undersell it

On the real photos, both models find every bare hand. With worn gloves, WiLoR finds 52–65% of them and MediaPipe 7–13%. The range comes from how strictly a predicted box has to match the annotated one, and glove boxes often include the cuff, which makes strict matching harsher than it should be.

Comparing that with the painted gloves was the first useful surprise.

For MediaPipe the synthetic and real drops agree: it simply stops seeing hands. For WiLoR they don’t. Painted gloves cost it almost nothing, while real ones cost 35–48 points. Box size doesn’t explain it away (the glove photos are mostly close-ups, which should be the easy case), and cropping doesn’t rescue MediaPipe either. So the rest of this post measures pose on frames where a hand was found, and WiLoR’s synthetic detection rates should be read as optimistic.

A bigger-looking hand is a closer hand

A single camera has no direct way to know how far away a hand is. It has to infer depth from how big the hand looks. A glove changes how big the hand looks.

The dark insulated glove puts the hand 45 mm too close. Grow its outline by four pixels and that becomes 124 mm, roughly 20 mm per pixel of extra silhouette. A bright colour removes the bias for HaMeR but not for WiLoR, so it’s a property of the model rather than something you can rely on. The rule I’d write into any capture spec: when operators wear gloves, never take hand position from a single RGB camera.

Cameras fix where the hand is, not what the fingers do

The obvious fix is more cameras. With two calibrated views you can triangulate the hand instead of guessing its depth. To compare fixes I use the share of the glove-induced error that each one removes: 100% means the glove no longer hurts at all.

A second camera removes 72–88% of the position error, and very little of the finger-pose error: 3–49%, depending on the glove. Going to eight cameras pushes position a bit further (83–93%) and does nothing new for the fingers. Swapping the work glove for thin nitrile, on the other hand, removes 73–92% of the finger error.

I first assumed this was about colour, since the nitrile glove is bright and the work glove is dark. That’s why G4 exists: the same insulated glove, painted bright. It detects 3 points better than the dark version and its finger pose doesn’t improve at all. Holding colour fixed instead, thin nitrile beats the insulated glove by 12.8 mm of finger error. It’s thinness, and the finger shading and creases it preserves, that does the work.

One more result I didn’t expect: for finger pose, a single camera’s learned hand model beats triangulating two views (8.8 mm vs 12.3 mm bare). Triangulation is great at where, and a model with a strong prior on hand shape is better at how.

A quality gate flags batches, not episodes

In a real pipeline you’d want an automatic check that throws out bad data. A natural one is two-view reprojection error: a frame is valid if the two cameras agree to within 3.1 px (the bare-hand 95th percentile), and an episode passes if at least 80% of its frames are valid.

At the level of a glove type, it works: 95% of bare episodes pass and only 5% of dark insulated ones. But within one glove type, the fraction of valid frames doesn’t predict which episodes have worse pose (|ρ| ≤ 0.36, p > 0.1). So it’s a good alarm for “this batch of data is out of the model’s domain” and a poor tool for picking individual episodes.

What I’d put in a capture spec

• Use a ViT/MANO-class model like WiLoR or HaMeR, not MediaPipe. MediaPipe’s detection goes from 93% to 6% or less with any glove.
• Choose thin, form-fitting nitrile gloves. Colour matters much less than thickness. Each extra pixel of outline costs about 0.4 mm of pose error.
• Capture with two or more calibrated views, roughly a metre apart at around 70°, or use RGB-D, for anything that depends on hand position.
• Gate per batch: reprojection under 3.1 px on at least 80% of frames, and investigate a batch when fewer than about 75% of its episodes pass.

What I got wrong along the way

Some of the findings above are corrections of earlier ones, and I think that’s worth saying out loud.

My notes originally credited glove colour as the lever. The G4 control was added specifically to test that, and it showed the opposite. The real-glove drop for WiLoR was first reported as 48 points, until looking at example images showed skeletons sitting on hands that failed the match only because the annotated boxes include the cuff. It’s now a range. My first sequence selection came entirely from the split HaMeR was trained on, which I caught before running inference. And the first render re-seeded the glove texture every frame, which would have made every gloved hand look jittery.

Limits

The gloves are 2D image edits: no real 3D thickness, no folds or seams. For WiLoR they clearly underestimate the real detection drop. It’s one subject handling lab objects on a table, not a warehouse floor. The models may have seen this subject in training, so trust the relative drop from bare to gloved more than the absolute numbers. And the real-glove check is detection only, on 83 photos.

What I’m fairly confident about is the shape of it: gloves hurt detection more than synthetic tests suggest, they bias single-camera depth in a predictable direction, extra cameras fix position but not fingers, and if you control the glove, choose thin.

DexYCB (Chao et al., CVPR 2021) is CC BY-NC 4.0; frames shown here are for non-commercial research illustration. Real glove photos are from the Roboflow Universe “Gloves and bare hands detection” dataset, CC BY 4.0. All code, figures and numbers are in the repository, and make all reproduces every number from cached predictions.

That makes it a good test of whether a vision-language-action model actually reads. So I took SmolVLM2-500M, a small video-capable VLM, and fine-tuned it into a policy that looks at two camera views, reads one instruction, and outputs a 7-number action: six for moving the hand, one for the gripper.

The first version never completed a single task. This post is about why, because the why was more interesting than the fix.

The setup

I followed the OpenVLA recipe, the same pairing VLA-0-Smol uses. Each of the 7 action numbers is cut into 256 buckets, each bucket gets its own token, and predicting an action becomes ordinary next-token prediction. The language model is the policy. There is no separate controller bolted onto it.

A few choices mattered more than they look:

• Bucket edges at the 1st and 99th percentiles, so a handful of extreme actions don’t stretch every bucket.
• Reusing existing vocabulary (token ids 48896 to 49151) instead of resizing the embedding matrix.
• LoRA at rank 16, which trains 56.9M of the 564M weights.
• No image splitting. With it, each prompt was 2,268 tokens. Without it, 156. That was the difference between fitting on an 8 GB card and not.
• bf16 weights with the loss in fp32. VLA-0-Smol reports fp16 diverging on this setup, and I didn’t want to find out.

Training ran in plain PyTorch at batch 4 with 4 accumulation steps, peaking at 5.8 GB.

It learned the average demonstration

The policy trained cleanly and scored 0%. Watching it, the failure was almost funny: it emitted the same action at every step, whatever it saw and whatever it was told. That action turned out to be the most common bucket in the training set on all six movement dimensions. The model had learned the average demonstration.

The loss said the same thing. A model that ignores its inputs entirely and only learns how often each bucket appears would reach a cross-entropy of 3.44. Run 1 plateaued at 3.40 by step 375 of 2,000. It had captured the bucket frequencies and very little else, so training longer was not going to help.

It wasn’t exactly nothing, though, and that corrected an earlier conclusion of mine. Taking the probability-weighted average over buckets instead of the top bucket gave 8 different outputs across 8 scenes. The input dependence existed. It was just too weak to move the most likely bucket, so greedy decoding turned a nearly flat model into a perfectly flat one.

Before going further I ruled out the boring explanations. The model can memorise a single batch down to near-zero loss. The adapter loads. Training and evaluation share bucket edges. The simulator renders on the GPU. Nothing was broken in the plumbing.

Asking the model what it uses

At 0% success, the usual test (give the policy a wrong instruction and see if it behaves differently) tells you nothing, because 0% looks the same either way. So I measured on the training loss instead. Hold the correct action fixed, corrupt one input, and see how much the loss goes up. Every condition sees the same examples in the same order, so batch difficulty cancels out.

Corrupting the camera images hurt about six times more than corrupting the instruction.

On this benchmark that is exactly backwards. The images cannot tell the tasks apart at the start, and only the sentence can. It’s also not a quirk of small models: robustness studies of LIBERO find much larger VLAs replaying memorised trajectories when the scene or wording changes.¹

Three fixes that made it worse

I tried three reasonable things: more scene variety from two other LIBERO suites, predicting five actions at a time, and classifier-free guidance on the instruction. All three made the imbalance worse. The worst run ended at 42-to-1 in favour of vision, against 6.4-to-1 before.

The scene-variety failure ended up being the most useful result of the project. I wanted to know whether the new scenes really were “harder”, so I built about the crudest task classifier possible: nearest-neighbour matching on 24 × 24 pixel thumbnails of the first frame.

On LIBERO-Goal the probe is below chance, as it should be. On the two suites I’d added, it names the task almost every time. So “more variety” had handed the model 70% more data in which the image answers the question outright. I had been feeding the shortcut.

When does the picture give the task away?

If the first frame is uninformative on LIBERO-Goal, the next question is how long that lasts. I ran the same probe at every point in the episode.

It sits at chance at the start and is well clear of it by about frame eight, once the arm has started moving and its direction says where it’s going. Those first eight frames are roughly 7% of a typical episode.

Every run so far had sampled training frames uniformly. Which means about 93% of every batch was made of frames where ignoring the instruction was the correct answer. The model wasn’t failing to learn. It was learning the data I gave it.

Changing only the sampler

Run 4 changes one thing. Half of each batch is drawn from the first eight frames of an episode and half is drawn uniformly. Same model, same data, same loss, same schedule.

Inside that early window, corrupting the instruction now costs more than corrupting both camera views, the first configuration in the project where language is the dominant input. Vision didn’t get worse. Both grew, and language grew faster, which is what climbing out of a degenerate solution looks like. Outside the window the ordering is unchanged, because there the image really does answer the question. In the simulator, 60-step rollouts from the same start under the true and a wrong instruction no longer share a single identical step.

0% to 20%

Closed-loop on LIBERO-Goal with the provided harness: 10 tasks × 5 episodes, 300-step cap.

Twenty percent is not a good LIBERO-Goal score, and I don’t want to dress it up. VLA-0-Smol reaches 95.6% with the same 500M backbone, but with 100k steps at batch 192, full fine-tuning, action masking and 8-step chunks, against my 20k steps at batch 16 with LoRA and one action at a time. SmolVLA’s own ablation puts a single-action policy at 54%,² which is the fairer comparison.

What run 4 can do: turn on the stove, and put the bowl on the stove or on top of the cabinet. What it can’t: any drawer task, or anything that needs a precise grasp. The failures run out the full 300 steps instead of going wrong early. The policy never finishes, which is the signature of small errors accumulating with nothing to correct them. Five episodes per task is also a small sample (one success reads as 20%), so I trust the pooled number and not the per-task ones.

What it cost the language model

The token ids I borrowed for action buckets were not spare slots. They are real English word pieces, including “Britain” and “selectivity”, and training on actions overwrote them. Worse, the action head is the language model’s own output layer, so every training step also reshaped how it predicts ordinary words.

The interesting part is that multiple-choice accuracy barely moves even when perplexity explodes. Perplexity is absolute, so losing half the probability mass to action tokens destroys it. Multiple choice only ranks four options, and the leaked mass shifts all four about equally. Fine-tuning damaged the model as a writer and largely spared it as a reader. (With 20 questions the error bar is about 11 points, so “no damage I can detect” is the honest version.)

Run 6 avoids the problem by using a separate small head and never touching the output layer, and comes back indistinguishable from the base model. A cheaper fix I didn’t get to is training only the 256 action rows of the output layer.

Two smaller results

State tokens did nothing. I added the robot’s own position to the prompt as numbers. Corrupting them barely changes the loss, the signature of an input the model has learned to skip. The wrist camera is bolted to the gripper, so the image already encodes arm position more precisely than my numbers did.

A continuous head is faster and closer on average, but less reliable. Run 6 replaces the 256-bucket classifier with a small network that outputs the 7 numbers directly. It is 1.7× faster and has lower average error, yet scores 12% against 20%. The pooled gap isn’t significant, so I re-ran the strongest task with three times the episodes, where it is (p = 0.0022).

The reason isn’t accuracy. When two actions are both valid, say passing left or right around an object, their average is the path straight through it. A classifier can commit to either bucket. A regressor has to pick the middle. Tokens buy reliability, not accuracy or speed.

What I’d do next

Action chunking is the biggest gap to published results and the thing most likely to fix the long-horizon failures. The catch is that chunking dilutes the language signal, which is part of what broke run 3b. So the experiment worth running is chunking together with run 4’s frame weighting, not either on its own.

The number I care about from this project was never the 20%. It was finding out why the model ignored the instruction, and then watching the same model listen once it was trained on the moments where listening pays off.