Can a chip built around SRAM serve a model with trillions of parameters? It is one of the questions Cerebras hears most, and on October 28, 2026 the company posted a nine-minute answer. In the video, Sarah Chieng, who leads developer experience, and co-founder and chief technology officer Sean Lie stand in the company's hardware labs and work through the arithmetic on a whiteboard. Their example is Kimi K3, an open-weights model from Moonshot AI that is marketed as a three-trillion-parameter model.

The answer they reach is specific: 59 wafers to hold the model, and room for about 190 simultaneous requests of 256,000 tokens each. The sums are simple enough to check with a calculator, which is part of the appeal. They also reveal why the design works, where its limits are, and which questions the whiteboard leaves open.

Why SRAM raises eyebrows

SRAM is the fast memory that sits right beside a processor's logic. It is quick, but it takes up silicon area, so chips that rely on it hold very little. Lie concedes the worry is real for conventional designs: “on traditional chips that are smaller that use SRAM, that is in fact true.” His argument is that a wafer-scale chip changes the scale of the problem. A single Cerebras WSE-3T wafer carries 44 GB of on-chip memory, and in the video he says that much SRAM “maps really naturally” onto large models. A trillion-parameter model still needs terabytes for its weights alone, so no single wafer can hold it. The question is how many wafers it takes and what it costs to link them.

The reason this matters is how text generation works. To produce each new token, a model reads the weights of every layer it activates, and for a single user that is mostly a memory problem rather than an arithmetic one. A GPU keeps its weights in separate high-bandwidth memory stacks beside the processor, so each token means moving data across that gap. Cerebras's argument is that if the weights already sit in SRAM next to the compute, the trip disappears. Its CS-4 materials claim 43.2 petabytes per second of memory bandwidth per wafer, though The Register suspects that peak figure is purely theoretical and notes that the previous chip lacked the compute to saturate its SRAM during inference.

The model on the whiteboard

As the video describes it, Kimi K3 has 93 layers: 69 “linear” layers and 24 “dense” ones. They are arranged in groups of four, each with three linear layers and one dense layer, repeated 23 times, plus one final dense layer. (Check: 23 times three is 69, and 23 plus one is 24.) The difference between the two types is how they handle attention. A linear layer uses a form of attention that does not need a KV cache, the per-request memory of everything the model has already read. A dense layer uses global attention and holds all of that cache.

The cache matters because it grows with every token of context. For each token it has read, a full-attention layer stores a compact record so the model does not have to recompute the past; multiply that by a long context and many simultaneous users and the cache can rival the weights in size. Because 69 of the 93 layers need no cache, only 24 do. Had all 93 kept one of the same size, the cache per sequence would be roughly four times larger, by our arithmetic from the video's figures.

The exercise rests on three assumptions: a 256K-token context, weights quantized to four bits, and an eight-bit KV cache. Chieng warns that the figures are illustrative and that real production configurations “may vary.”

Outside reporting adds context. Moonshot released the model through its API on July 16, 2026 and published the weights on July 27, according to Vast.ai. Reports put it at about 2.8 trillion total parameters, around 104 billion of them active for any token, with 896 experts of which 16 fire per token, and a context window near one million tokens. In other words, “three trillion” is a round number, and the video's 256K is a working assumption rather than the model's limit. The companion post on Cerebras's website also cites the model's reported results of 88.3% on Terminal-Bench 2.1 and 91.2% on BrowseComp.

Step one: how big is a layer?

“This is just simple arithmetic,” Lie says. Three trillion parameters at four bits each, eight bits to the byte, come to 1.5 terabytes. Divide by 93 layers and each layer weighs about 16 GB. As a cross-check, Vast.ai puts the published weights at roughly 1.4 TB in a four-bit format, which squares with 2.8 trillion parameters at half a byte each.

Step two: packing layers onto wafers

With 44 GB per wafer and 16 GB per layer, the layout follows from the layer types. Linear layers need no KV cache, so two fit on one wafer, using about 32 GB. A dense layer gets a wafer to itself, because its cache has to live beside its weights: the 16 GB of weights plus whatever remains for the context of every concurrent user.

Between wafers, only activations travel. They follow the logical pipeline of the model: the output of one layer becomes the input of the next, carried over what Lie calls a high-performance RDMA-over-Ethernet network. The companion post puts that at roughly two microseconds per hop and stresses that weights never move. The picture is a pipeline in which each wafer is a stage. While one stage works on a request, the stage before it can already be working on another, so many requests are in flight at once.

Step three: how many requests fit?

This is the heart of the video. A dense wafer has 44 GB, 16 GB of which holds weights, leaving 28 GB for the cache. In this model each token on each dense layer takes 576 bytes of cache. Multiply by 256,000 tokens and one sequence needs about 147 megabytes per dense layer. Divide 28 GB by 147 MB and you get roughly 190 concurrent requests.

Lie calls that “the equivalent of a batch size of 190, which is very, very high,” and adds that “in most production cases, we wouldn't even need such a high batch size to reach high performance.” So the figure is a ceiling set by memory, not a target.

Two readings of the number are worth spelling out. First, the 190 is system-wide, not per wafer in addition to the others. Each of the 24 dense wafers holds one layer's cache for the same set of sequences, so a full 256K-token sequence occupies about 24 times 147 MB, or 3.5 GB, spread across the dense wafers. Second, 190 sequences amount to roughly 670 GB of cache across the system. (These two totals are our arithmetic from the video's figures.) On the origin of 576: the number equals 512 plus 64, the latent-plus-positional layout used by multi-head latent attention, which is probably where it comes from, though the video does not say so.

Step four: counting wafers

The tally is quick. Sixty-nine linear layers at two per wafer is 34.5, rounded up to 35 wafers. Twenty-four dense layers at one per wafer is 24 wafers. Together, 59 wafers hold the whole model. Because 69 is odd, the last linear wafer holds a single layer and has 28 GB to spare. Those wafers carry about 2.6 TB of SRAM in total, of which the weights take 1.5 TB, and the rest is for cache and working state. Packed three to a CS-4 system, which is how the newest Cerebras hardware is built, 59 wafers would come to about 20 systems; that figure, too, is our arithmetic, not the company's.

Scaling to 10 and 100 trillion

Chieng asks what happens with a 10-trillion or 100-trillion-parameter model. Lie's answer is that “all of this scales linearly as a pipeline”: double or triple the layers, add wafers, and the same logic applies, which he again attributes to the amount of SRAM that wafer-scale integration provides. The companion post says each added wafer contributes compute and on-chip memory.

As a back-of-envelope extension, keeping the same wafer mix would put a 10-trillion-parameter model near 200 wafers and a 100-trillion-parameter model near 2,000. Capacity does scale that way. Speed per user is a different matter, because a token must cross every stage in turn. At roughly two microseconds a hop, 59 wafers mean 58 hops, or about 0.12 milliseconds of transport per token, small beside the one millisecond a user at 1,000 tokens a second allows for each token. At 2,000 wafers the hops alone would take about four milliseconds, which on its own would cap a single user near 250 tokens a second before any compute is counted. This is our extrapolation, and the company's answer is finer-grained stages and smaller micro-batches; its post notes that smaller micro-batches lower per-request latency, while larger ones raise total throughput.

What the whiteboard leaves out

  • Headroom. The 28 GB is treated as all cache, leaving nothing for activation buffers on a dense wafer. Linear layers keep a small fixed-size state per request, which the video sets aside “for this exercise.” A production layout would trim the 190.
  • Layer order. Pairing 69 linear layers into 35 wafers assumes any two can share a wafer. In the model's rhythm of three linear layers followed by one dense layer, a wafer holding two neighbors leaves one linear layer per group stranded between dense layers. If wafers could hold only adjacent layers, the count would be 46 linear wafers plus 24 dense, or 70 (our arithmetic). Cerebras says its compiler maps layers across wafer regions and that production systems use finer-grained stages, so the 59 is best read as the whiteboard's memory-driven count rather than a layout.
  • Uniform layers. Dividing the weights evenly across 93 layers is a simplification; real layers differ in size.
  • Prompt processing. The video is about holding the model and serving decode. The Register reports that CS-4 systems act mainly as decode accelerators, with prompt processing offloaded to other hardware.
  • Measured speed. The video shows no benchmark. The companion post's summary cites 1,000 to 2,000-plus tokens per second, a figure that comes from the company.

Why the exercise matters

Kimi K3 is a mixture-of-experts model: only about 104 billion parameters are active for a given token, yet all 2.8 trillion must stay resident. That makes capacity, not arithmetic, the binding constraint, which is why a capacity-first walkthrough is the right way to test a design. The Register makes a related comparison: Cerebras would need only a few dozen chips to hold a trillion-parameter model, where designs with far less SRAM per chip, such as the Groq LPUs in Nvidia's LPX racks, would need about 2,000, depending on weight precision.

It is still a vendor explaining its own architecture, with round numbers and the caveat Chieng states up front. What the video does well is show its work. Anyone can redo the sums, and anyone can see where the assumptions sit: four-bit weights, an eight-bit cache, a 256K context and a clean split between layer types. Change one and the wafer count moves.

For anyone weighing hardware for a large model, the practical lesson is the order of the questions. Start with the weights, then the cache per user at the context length you need, then the concurrency you want to sell, and only then ask how many chips that adds up to. A vendor that shows those steps, as this one does, is easier to hold to account than one that quotes a single speed number.

Sources and notes

Video: Cerebras CTO Sean Lie: Deploying Trillion-parameter Frontier Models, Cerebras, October 28, 2026 (9:12). Companion post: How we deploy trillion-parameter frontier models on Cerebras. Model details: Vast.ai. Hardware context: The Register. The video's auto-generated captions render the model as “Chinchilla 3”; the video description and Cerebras post name it Kimi K3. Related: Cerebras CS-4 and WSE-3 Turbo. Thumbnail: still from the video, Cerebras.