Can NCM run on a MacBook without its GPU? I tested it against GPT-2 and Gemma 4

Most language models, GPT-2 and Gemma included, write one token at a time, and every token costs a full pass through the model. Diffusion language models start from a blank, masked-out text and fill in all of it over many passes. NCM takes a third path. A learned tokenizer squeezes text into a short list of codes, which the paper calls concepts, at several levels of detail. The model writes the coarsest level first, then each finer level in one pass, and a small decoder adds all the levels together and turns them into words.

The news summary model has 9 levels, from 4 codes to 48, so every summary takes 9 passes. The paper counts 8, leaving out the pass that picks the first level. It reports it matching the strongest diffusion model's summary score while generating 31 times faster, on that H100.

One 14-inch MacBook Pro from 2021, with an Apple M1 Pro and 16 GB of memory, CPU only with its GPU unused, one request at a time. Every model got one untimed warm-up run before the clock started.

NCM ran its released news summary model on XSum, a set of BBC articles with one-sentence summaries, and its released story model, trained on TinyStories, a set of simple children's stories. GPT-2 small is about the same size and writes one token at a time, so it shows what NCM's design changes. It was never trained to summarize, so it is a speed reference only, made to write as many tokens as NCM did. Gemma 4 12B, Google's open model, is about 80 times bigger and shows what a much larger model does on the same CPU.

NCM's code is written for NVIDIA GPUs and loads its checkpoints as Python pickles with weights_only=False, which lets a file run code as it loads. I checked with pickletools that the files hold only tensors, loaded them with weights_only=True, and took out every GPU call.

Gemma was harder to keep off the GPU. Ollama's Gemma 4 builds still put all or part of the model on the Mac's GPU even when told not to, so Gemma ran in llama.cpp with no GPU device at all.

NCM's 9 steps against GPT-2's 20 to 28 should have made it faster. It was slower, 1.06 s against 0.61 s, so I timed each part of one summary. Almost all of it was the 9 passes. The decoder and the word picking took 0.03 s together.

The reason is the article. Each NCM pass ran all 512 article tokens through the model again, along with the summary codes. GPT-2 reads the article once and keeps the attention keys and values it computed, its KV cache, so each new token costs one small pass.

Inside NCM, the attention mask blocks the article from looking at the summary, so the article's keys and values come out the same in all 9 passes. The fix keeps them from the first pass and runs the next 8 passes on the summary's codes alone. It is about 20 lines in the test script, behind a --cache flag, and NCM's own code stays unchanged.

NCM's code has a cache option of its own, but it is turned off for this model, and it would skip the mask that hides the article's padding. The test patch keeps that mask, which is why the output did not change.

Summaries went from 1.06 s to 0.42 s, faster than GPT-2, and the 100 summaries came out word for word the same, so quality is untouched. On one thread the gain is bigger, from 1.77 s to 0.49 s.

The story model writes from nothing, so there is no article to reread. It wrote a 128-token story in 11 passes and 0.47 s. The paper counts 12 steps here, since it includes the first level, one code, which the released model draws from a stored frequency table with no pass. GPT-2 needed 128 passes and 3.01 s, so NCM wrote about 218 words per second against 31. On one thread it was 0.53 s against 2.64 s.

Gemma 4 12B took 16.4 s to write a 128-token story, 7.7 tokens per second. In llama.cpp's own benchmark on this CPU it reads about 27 tokens per second and writes about 11, because every token goes through almost 12 billion parameters.

Gemma 4 12B is about 80 times bigger than NCM, 11.9 billion parameters against 149 million, and it shows. On the first 20 articles its summaries scored 39.0 on ROUGE-L, a standard score for how close a summary comes to one written by a person, against NCM's 27.1. It took about 25 seconds per summary, most of it spent reading the article.

NCM scored 27.5 on 100 articles, close to the paper's 28.0, so the laptop did not change its quality. But that quality is rough. One story it wrote reads "One day, a little dog named Max went for a walk. He had a big big cat named Lily. Max liked to take walks to Lily. Lily would put itch on. leg to bite Max's leg feel and and."

Here is one BBC article, summarized three ways.

A person wrote "A man has appeared in court after firearms, ammunition and cash were seized by police in Edinburgh."

NCM wrote "A man has been arrested in connection with a number of firearms ammunition and ammunition were seized in Edinburgh."

Gemma 4 12B wrote "A 26-year-old man has appeared in court after police seized firearms, ammunition, and a five-figure sum of money during searches in Edinburgh."

It is not an old laptop. The M1 Pro's memory and matrix units are much faster, so an older machine would likely be slower for every model. The one-thread runs show how each model scales with less CPU, not how an old machine performs.

It is not NCM against diffusion models, which is what the paper's speedups compare, and no diffusion model ran here. It is not NCM's long-text model either. The 1,024-token OpenWebText model is listed in NCM's repo but not released yet, so 128-token stories are the longest test possible.

And Gemma ran in different software. llama.cpp is a common way to run it at home, while NCM and GPT-2 both ran in plain PyTorch, so their comparison is like for like.

The repo has the scripts, the logs behind every number in this post, and a setup script that downloads NCM's code at the tested commit, its two models and the test articles. It runs on macOS and Linux, and on Windows through WSL.