Memory bandwidth, not memory size, decides which model runs well on a DGX Spark
· Alexandr Pavlenko · 493 words
A DGX Spark has 128 GB of unified memory. A 72-billion-parameter model quantised to four bits weighs 47 GB. It fits with room to spare — and it is close to unusable. The reason is a number most spec sheets bury.
The number that actually matters
Generating one token requires reading every active weight out of memory. Not some of them — all of them, once per token. So the ceiling on generation speed is not compute, it is how fast memory can be read.
A DGX Spark reads at roughly 273 GB/s. Divide that by the size of the model and you get the theoretical maximum token rate. For a 47 GB model that is about 5.8 tokens per second before any overhead. Real output lands lower.
Five tokens per second is a sentence every twenty seconds. For a chatbot that is painful. For an agent that makes several passes over the same question before answering, it is unusable.
Why the same machine runs a 30B model faster than a 14B one
This is where mixture-of-experts architectures stop being a curiosity and start being the obvious choice. A MoE model holds many parameters but activates only a fraction of them per token.
A 30B model with 3B active parameters reads roughly 2 GB per token instead of 18. On the same hardware that is an order of magnitude more headroom. It behaves like a 3B model and answers like something far larger.
We measured this directly while building the tutor for our own programme. On an identical agentic task — execute code, read the real output, report it — a dense 14B model took 42 seconds. A 30B MoE took 31, using fewer tokens to get there.
The trap of judging a model by its parameter count
A widespread belief holds that anything under 27B is not worth running. It is wrong in both directions.
Small dense models fail at agentic work not because they are small but because tool calling is a trained behaviour, and many of them do not have it. We watched a 14B coder model emit a tool call as plain text in the middle of its answer, and an 8B general model invent a plausible-looking result rather than run the code. Neither failure was about size.
Meanwhile a well-trained MoE at the same memory cost handled the same task correctly on the first attempt. Architecture and training beat parameter count on hardware where bandwidth is the constraint.
How to choose in practice
Take the memory bandwidth of your machine. Divide by the on-disk size of the quantised model. If the result is under 15 tokens per second, the model is too heavy for interactive use no matter how much memory you have left over.
Then check that the model actually supports tool calling as a trained capability, not as a flag you can switch on in a configuration file. Ask it to run code and print something it cannot possibly know — a hostname, a directory listing. If the answer looks plausible but generic, it invented it.
Memory size tells you what will load. Bandwidth tells you what will work.