DeepMind ships SL2T: sign language finally gets a real translation model, not a glove

Google DeepMind announced that its sign-language-to-text model, SL2T, is now live in two consumer products: sign-to-text dictation in Gboard and conversational transcription in Live Transcribe, both on Pixel 11, starting with American Sign Language to English. This is worth pausing on, because most "AI for accessibility" announcements stop at a demo. This one shipped into a keyboard.

The problem that gloves couldn't solve

There are more than 200 sign languages in active use, serving an estimated 70 million Deaf and hard-of-hearing people. Spoken-language AI — dictation, translation, conversational interfaces — has had a fifteen-year head start built on abundant paired audio-text data and a relatively forgiving problem structure: transcription is mostly a sequential mapping from sound to text within one language.

Sign language translation doesn't get either advantage. First, it's genuine machine translation, not transcription — ASL and English have different grammars, different word orders, and no token-for-token correspondence. Second, the input isn't audio, it's simultaneous whole-body motion: hands, arms, torso, head, and face all carry meaning at once, and a system has to track that at high frame rates before it can even begin translating.

That second constraint is why earlier attempts, like instrumented signing gloves, were structurally limited from the start. A glove can capture hand shape. It cannot capture facial grammar, eyebrow raises that mark questions, or the space around the signer's body that ASL uses to track referents. Sign languages aren't English rendered on the hands, so any approach that only reads the hands is solving the wrong problem.

What SL2T actually does differently

Two design choices in SL2T stand out as the real engineering content behind the announcement.

The first is the input representation. Rather than sending raw video to a server, an on-device model (MediaPipe Holistic) extracts pose landmarks — geometric coordinates of the signer's body — and only those coordinates leave the device. The original video is discarded immediately. That's a privacy decision, but it's also a modeling decision: it forces the translation model to work from a compact, structured signal instead of raw pixels.

The second, more consequential choice is skipping glosses. Prior sign language translation work has typically used glosses — an intermediate written notation for individual signs — as a bridge between the visual input and the target-language output. Glosses are appealing because they look like a natural intermediate representation, similar to how phonemes bridge audio and text. But DeepMind's team notes that glosses lose the non-manual and spatial information that carries real grammatical weight in sign languages, and they impose a fixed vocabulary that caps translation quality. SL2T instead translates the landmark sequence directly into text, end to end. That removes an artificial ceiling: quality can keep scaling with data rather than being bounded by how complete a gloss vocabulary is.

The infographic below lays out the pipeline and contrasts it with the gloss-bottlenecked approach it replaces.

The model is trained on more than 100,000 hours of data across 50+ sign languages, with roughly a quarter of that in ASL. Training jointly across languages rather than in isolation reportedly helps: shared structure across sign languages improves individual-language performance versus single-language models, which is a familiar transfer-learning result but a nontrivial one to prove out in a modality this data-scarce.

On the FLEURS-ASL benchmark, SL2T reports a zero-shot BLEURT score of 70, which DeepMind describes as well above any previously reported score for ASL-to-English translation. Benchmark numbers from the model's own team are worth the usual grain of salt, but the deployment is the more convincing evidence here — you don't ship a keyboard feature to production Pixel devices on a model that only works on a leaderboard.

Where it still breaks

DeepMind is reasonably direct about failure modes, and they're the ones you'd expect from a system still early in its real-world life: errors on rare signs, rapid fingerspelling getting mistranslated (their example: "prey" read as "grey"), passive constructions, classifier depictions (a claw shape can get dropped entirely), and tense inferred without enough context. The team also called out three practical constraints that rarely show up in an academic benchmark but matter enormously for a shipped product: streaming latency, hallucination on non-signing input, fairness for the roughly 10% of signers who are left-handed, and usable performance for one-handed signing — which is common in practice, since a lot of people are holding a phone in the other hand.

That last point is a good example of the gap between a benchmark result and a product. A model that only performs well on canonical two-handed signing from a fixed camera angle isn't actually usable by someone standing on a train platform signing one-handed while holding their phone.

The governance layer

The other notable piece of this announcement is process, not model architecture: DeepMind describes building SL2T with Deaf community involvement from conceptualization onward, including a formally established AI Sign Language Advisory Committee bringing in Deaf organizations and subject-matter experts. For a technology whose entire value proposition depends on getting a specific community's language right, that's not a nice-to-have — a translation model trained and evaluated without native signers in the loop would be very likely to encode exactly the kind of errors described above (classifier drops, non-manual markers) without anyone on the build team noticing.

Why this is worth tracking

The interesting signal isn't the BLEURT number — it's that a genuinely hard multimodal translation problem (simultaneous whole-body input, no aligned intermediate representation, chronic data scarcity) went from research paper to shipped keyboard feature. That's the harder step for most accessibility AI work, and it's the one most announcements never actually take. Whether SL2T generalizes cleanly to the other 49+ sign languages in its training mix, and beyond ASL to actual bidirectional conversation, is the thing to watch next.