Google’s DiffusionGemma AI Hits 1,000 Tokens Per Second—And It’s Free

Google has dropped DiffusionGemma Todayan open model AI that generates text the way image generators create images: start with noise, refine until it makes sense. It reaches 1,000 tokens per second on an NVIDIA H100. (Tokens are the basic unit of information that an AI model processes.) That means it’s four times faster than regular Gemma. It’s also free, Apache 2.0, with weights on Hugging Face.

The catch, as always, is in the small print. Per Google’s announcementthe model achieves “700+ tokens per second on NVIDIA GeForce RTX 5090.” It also lags behind the standard Gemma 4 in terms of output quality.

Google says so itself. This is a speed model, not a quality upgrade.

What this actually does

Every LLM you have used is a typewriter. One token at a time, with each word dependent on the last. That’s how autoregressive architectures work.

DiffusionGemma doesn’t do that. Instead of generating tokens sequentially, it starts with parallel refined chunks of unreadable text. According to Google developer guideit “starts with a canvas of random placeholder tokens” and iteratively locks confident tokens until the entire block comes into focus. Two hundred fifty-six tokens per forward pass. The GPU remains busy.

The side effect is bidirectional attention: each token can see every other token as it is being generated, which is impossible in autoregressive models (they cannot see the future, what is going to be encoded). That makes it extremely good at tasks where the end of the answer limits the beginning: code completion, structured output, problems with heavy constraints, etc. Google has refined a version to solve Sudoku as a demo. The basic model got about 0% of the puzzles correct.

The refined version reached 80%.

Text distribution has been a research project for years. MDLM, SEDD, LLaDA, Dream—academic models that proved the approach worked on a small scale and mostly survived as a proof of concept. Inception Labs dispatched Mercury 2 in February 2026 as the first commercial diffusion reasoning model, claiming speeds five times faster than speed-optimized competitors.

But none of it had open weight, and none of it came with day-zero support in vLLM, Hugging Face Transformers, and Unsloth. DiffusionGemma is the first major open release from a tier-one lab.

There is also a historical irony worth mentioning. Image generators started as diffusion models (hence the name Stable Diffusion) and are now evolving towards autoregressive architectures for better quality. Language models started out as autoregressive and now experiment with diffusion for speed.

Why it’s hard to run… for now

Running DiffusionGemma efficiently requires a drawer: a lightweight module that represents token blocks in parallel, which the main model then verifies in one forward pass. This is called speculative decoding. DFlash is a framework published in early 2026 that uses a small diffusion model as a drawer, making some tasks more than six times faster. It is the engine that makes this model class practical.

The problem: DiffusionGemma requires a specific drawing program that can be run locally via MLX, Apple’s machine learning framework for Apple Silicon. That module does not exist in any public version of mlx-lm, in any open pull request, or in the bundled LM Studio runtime.

We tried running DiffusionGemma with Hermes via NVIDIA NIM. The model loaded, but then: “agent init failed: model google/diffusiongemma-26b-a4b-it has a context window of 8,192 tokens, which is lower than the minimum of 64,000 required by Hermes Agent.”

To be precise, DiffusionGemma’s actual context window is 256K tokens. The 8,192 number was Nvidia confusing things by default, not the model’s architectural limit.

In practice, properly configuring it for agentic use requires manual work that most regular users have not yet figured out, and Hermes Agent simply cannot initialize without it. Parallel speed means nothing if the agent can’t boot.

Hopefully, in the coming days, the community will produce better resources to run these models.

Who is this actually intended for?

Developers with NVIDIA RTX 4090 or 5090 hardware building real-time tools: inline editors, autocomplete, code completion, structured generation. That’s the goal. As Decrypt pointed out in May, Google has been continuously working to make local inference faster without any new hardware.

For researchers, bidirectional generation opens up an area that autoregressive models simply cannot reach: protein sequences, mathematical graphs, anything where position N depends on position N+50. That is no small matter.

Google launched Gemma 4 under Apache 2.0 in April, and DiffusionGemma continues that strategy. A draft llama.cpp PR has already been opened as of today. When the toolchain catches up, it will reach a much wider audience.

On a machine with a capable discrete GPU, 1,000 tokens per second is real.