TL;DR: An AI model needs enough HBM to hold the model itself, plus a large enough KV cache to handle long conversations. Researchers are finding ways to shrink the KV cache, from Google’s TurboQuant to EdgeRock-4B, which caps its attention cache at 72 MiB per conversation. Can that lower demand for memory, and will RAM prices go down as a result? Not on their own. Saved memory lets the same hardware serve more users or moves to other kinds of memory, and AI memory demand has kept rising. Prices depend on supply, which grows slowly: the new fabs cited here start adding output from mid-2027, and TrendForce still expects conventional DRAM contract prices to rise 10% to 15% in the fourth quarter.
In October, the AI lab Rock Matrix released EdgeRock-4B, an open model. In an ordinary AI model, the memory that holds the conversation, called the KV cache, grows as the conversation gets longer; EdgeRock caps it. By the team’s calculation from the model’s configuration, its attention cache stays at 72 MiB per conversation, against 8 GiB for Qwen3.5-4B, the model it was built from, at 256K tokens (about 262,000 words or pieces of words). That is a 99.1% reduction. Everything EdgeRock stores about a conversation comes to about 98 MiB, also flat, but the model’s weights and working buffers come on top, so total GPU memory still grows with the number of conversations running at once.
Memory-chip stocks have reacted to this kind of research before. In March, Google Research said TurboQuant, its KV cache compression method, shrank that memory by at least six times with no loss of accuracy in its tests on open models. Micron’s stock fell 7% two days later, and Samsung and SK hynix each slid more than 5% in Seoul. We looked at the method itself at the time in Will Google’s TurboQuant demolish the AI memory wall?
Share prices are not memory prices, though. The question for anyone buying or selling hardware is whether RAM prices go down as AI models use less memory. Here RAM means DRAM, the memory in servers and PCs. HBM, the stacked memory on AI GPUs, belongs in that question because it is built from DRAM made in the same fabs.
Why does AI use so much memory?
Two major things fill memory when an AI model runs. One is the model itself, its weights, which take a fixed amount of space. The other is the KV cache, which works as the model’s working memory: every token the model reads or writes leaves an entry so the model can keep track of what came earlier. Longer conversations and more users at once both make the cache grow.
That is why the 8 GiB figure matters. Working from it, ten such conversations on a 4-billion-parameter model would fill an 80GB H100 before the model itself is loaded, which is why inference servers often run out of memory before they run out of compute.
What are the algorithms doing about it?
The approaches fall into five broad groups to solve the memory issue of AI model inference.
| Approach | How it works | Example |
|---|---|---|
| Store each entry in fewer bits | Compress cached values from 16 bits to 3 or 4 | Google’s TurboQuant |
| Discard some entries | Keep recent tokens and drop older ones judged less important | Sliding windows, token eviction |
| Design attention to store less | An architecture choice: attention heads share one set of cached values, or store a compressed version | GQA, DeepSeek’s MLA |
| Swap part of the cache for fixed-size memory | Some layers keep a running summary instead of every token; only some hybrid models end up with a fully capped cache | Hybrid models, EdgeRock |
| Move the cache to a cheaper tier | Park cache that isn’t needed right now in CPU memory or on SSDs | Nvidia’s inference context storage |
Part of the motivation is that longer context has limits. When the database company Chroma tested 18 leading models last year, their performance grew less reliable as inputs got longer, even on simple tasks. Human memory offers a loose analogy: working memory is small, and long-term memory stays out of the way until something calls it up. Several of the methods above follow the same pattern.
Capping the cache has costs. By the team’s own tests, EdgeRock kept most of Qwen3.5-4B’s math score, an 82.9% average pass rate on AIME 2024–2026 against 86.1%. It scored lower on GPQA-Diamond, a graduate-level science test (64.6 against 77.3), and on LiveCodeBench v6, a coding test (61.6 against 75.7). None of these results has been reproduced independently.
If AI inference uses less memory, does AI memory demand fall?
EdgeRock’s throughput test shows what saved memory buys. On one 80GB Hopper GPU, with 32K-token prompts and 32K-token answers, the team measured 12,124 output tokens per second for EdgeRock at 512 conversations at once, against 1,889 for Qwen3.5-4B at 64, each at its best measured batch size. That is 6.42 times the total output, counted over the full run including prompt processing. EdgeRock ran on the team’s own optimized runtime and Qwen on the standard vLLM server, so the comparison doesn’t isolate the effect of the memory savings. A separate test with one 128K-token request and a 128K-token answer, which timed answer generation only, found EdgeRock about 1.54 times faster.
What happens next can go two ways. In one outcome, cheaper serving brings more use. Economists call this the Jevons paradox: when using something gets cheaper, total use can rise by more than the savings. The record so far is consistent with that. DeepSeek’s V2 model cut its KV cache by 93.3% against the company’s previous model in 2024, and over the same years the HBM on each Nvidia GPU still grew from 192GB on the B200 to 288GB on Rubin, while HBM stayed in short supply. That shows demand growing alongside efficiency gains. It does not show the gains caused the growth.
In the other outcome, a provider serves more work on the GPUs it already owns and postpones buying new ones. AI usage and installed HBM can both keep growing while new memory orders grow more slowly than they would have without the savings. That is the real test of the bearish case.
Moving cache to a cheaper tier splits the effect again. Cache parked in CPU memory adds to demand for server DRAM. Nvidia’s Inference Context Memory Storage platform, announced at CES in January, offloads less frequently used KV cache to NVMe SSDs, and that demand lands on NAND flash, a separate market from DRAM.
Saved cache memory can push memory demand up, slow new orders, or shift demand to other kinds of memory. Prices still depend on how much the fabs produce and how fast total demand grows.
What really decides RAM prices?
The algorithms change how much work each chip can do. DRAM prices are set mainly by how much memory the fabs can produce, and that output rises slowly.
Fabs take years. SK hynix announced in April 2024 a plan to start building its M15X fab that month and finish the building in November 2025, about 19 months for the shell alone before production could begin.
Makers also remember the last downturn. In fiscal 2023, Micron took in $15.54 billion and lost $5.83 billion. This cycle Micron is lining up customer commitments as it expands. It says it has signed 26 multi-year agreements with take-or-pay volumes, expected to make up more than 35% of its revenue through 2030, and it plans about $25 billion of capital spending in the first half of fiscal 2027 alone. Its first new Idaho fab is due to start wafer output in mid-2027.
HBM takes a large share of what the fabs make. Micron’s CEO said in 2024 that, industrywide, HBM3E consumes approximately three times the wafer supply of DDR5 to produce the same number of bits on the same process. Every wafer that goes to HBM is capacity that ordinary server memory doesn’t get.
Supply and demand do balance eventually, often by overshooting. A September CSIS report warned that “the investment needed to relieve today’s shortage may eventually recreate the conditions for the industry’s next glut.” The forecasts we gathered on Samsung’s and SK hynix’s new fabs don’t expect them to ease the shortage before late 2027 or 2028.
The latest outlooks, each from a different source and measuring a different thing:
- Micron, September 30: industry DRAM bit shipments growing only in the low-20s percent range in 2027 and 2028, with supply staying tight in both years.
- TrendForce, September 30: conventional DRAM contract prices up 10% to 15% in the fourth quarter, a slower rise than earlier this year, with server DRAM still undersupplied.
- Citi, August: memory prices peaking in the second quarter of 2027.
A price peak and the end of a shortage are different events, so Citi and Micron can both be right, and a forecast peak says nothing about how soon or how far prices fall afterward. These are contract prices; retail kits and used modules follow their own paths. New suppliers don’t necessarily undercut, either. CXMT, a newer DRAM maker, has charged more than Samsung’s roughly $1,240 for comparable 64GB DDR5 server modules. As we wrote in our look at CXMT’s new DDR5, its supply could slow price increases once qualified modules ship in steady volume.
Should anyone worry about an AI bubble?
With Micron’s Idaho fab and the Samsung and SK hynix projects above due to add output from mid-2027 onward, demand is what could move prices sooner. In a VentureBeat survey published in October, 79% of the 218 companies that reported GPU utilization said their GPUs are busy half the time or less. The survey did not ask how those GPUs were obtained, and it is a self-selected sample. Other signals point the other way: the hourly rental rate for an H100 rose 21.9% in a month to $3.28 in September. A rising rental price shows strong demand for GPU time, but it doesn’t measure how much of that capacity is actually used.
The same survey found that 70% of companies with idle GPUs still plan to add or replace an AI compute platform within a year, and that 91% of September respondents planning one had no finance-reviewed return figure. That is the setup for a correction if finance teams start cutting compute nobody uses.
Even then, the path to DRAM prices is slow. Take-or-pay contracts like Micron’s hold volumes in place for a time. Makers could then move DRAM wafer starts from HBM back to conventional memory, though HBM’s stacking and packaging lines can’t be converted, and any shift needs production planning and customer qualification. In that scenario, which is our reading rather than a forecast, prices could fall faster than current outlooks suggest.
What to watch next
Three sets of numbers will show whether demand for memory is cooling: TrendForce’s monthly contract prices, along with the inventories and lead times memory makers report; the cloud providers’ spending plans for 2027 in their quarterly results; and GPU rental rates. If contract increases shrink toward zero while spending flattens and rental rates fall, that would be consistent with cooling demand. It would not show that cache savings caused it; that would take providers tying their efficiency gains to what they buy. For companies holding decommissioned servers or planning to sell spare memory, those readings say more about whether RAM prices go down than any model release.