The memory that does not fit
Sandisk reported a 78.4% GAAP gross margin in its fiscal third quarter of 2026, before HBF was shipping. The quarter says much more about NAND scarcity than about a product category that barely exists.
AI servers are consuming more enterprise flash while suppliers have been slow to add capacity. Contract prices have risen sharply. The longer-term claim is architectural: NAND flash may be moving closer to the processor. Sandisk and SK hynix describe High Bandwidth Flash, or HBF, as a new layer between high bandwidth memory and SSDs. NVIDIA, using a different design, now calls its flash-based CMX product a context memory tier.
Both products exist because large AI models need more working memory than accelerators can economically carry, yet ordinary storage is too distant for data that an inference request may need again. Servers clearly need more capacity between HBM and storage. Public evidence does not yet show which architecture will supply it or how much of the value will remain with flash manufacturers.
Research cut-off: August 17, 2026. HBF remains a precommercial technology, and quoted performance figures include company targets and analytical models rather than production measurements.
Why inference changes the hierarchy
Training and inference put different pressure on a system. Training moves large batches through a model and rewards maximum compute throughput. Inference has to serve many users with different prompts, context lengths, and latency requirements. Each new token depends on model weights and on information saved from previous tokens.
That saved information is the key-value cache, usually shortened to KV cache. It prevents a model from recalculating the same attention state at every step. The trade is straightforward: the system saves computation by keeping more data in memory.
The cache grows with context length and the number of simultaneous requests. NVIDIA estimates that a single 128,000-token context for Llama 3 70B can use about 40 GB of KV cache. Ten concurrent users with similarly large contexts would require roughly ten times as much before allowing for model weights or other working data. The exact number changes with model architecture and precision. In every case, longer conversations and greater concurrency make memory capacity a limit on useful accelerator throughput.
Software can shrink the cache or avoid moving it. Quantization stores each entry with fewer bits, while paged allocation reduces empty space. A scheduler can also route a request back to the GPU that already holds its cache. These techniques matter because moving data is often more expensive than doing the arithmetic.
A gain from compression may disappear as context or concurrency doubles. AI infrastructure therefore needs to use fast memory more efficiently and find a useful place for data that no longer fits.
HBM and SSD solve different problems
HBM sits beside an accelerator and supplies enormous bandwidth with low latency. It is where active model state, hot weights, and latency-sensitive cache belong. Its limits are cost, capacity, packaging complexity, and power. Adding more HBM is possible, but each stack occupies scarce package area and requires a wide connection to the processor.
An enterprise SSD makes the opposite trade. NAND provides far more capacity at a much lower cost per bit, retains data without power, and fits established storage systems. It reaches the processor through controllers, block protocols, PCIe links, and sometimes a network. Those layers are useful for durability and sharing. They add latency and make the drive a poor substitute for working memory.
The resulting hierarchy has an awkward discontinuity. Data can be hot and expensive in HBM, or abundant and comparatively slow in storage. Host DRAM sits between them in many servers, but it has its own capacity and bandwidth limits. Moving data across PCIe can leave an expensive GPU waiting.
No single specification defines what belongs in this gap. Any useful addition needs enough capacity to relieve HBM and enough delivered bandwidth to keep the accelerator busy. The software must also know what to place there. Different workloads put those requirements in different order. Static model weights are read repeatedly and changed infrequently. KV cache is created continuously, read according to the attention pattern, and often discarded after a session. A medium well suited to one can be poor for the other.
The industry is filling the gap in different places
HBF is the most direct attempt to make NAND behave more like memory. Instead of placing conventional flash behind an SSD controller, it stacks specialized NAND dies over a base die and connects the package through a wide local interface. The first specification announced by Sandisk and SK hynix describes 8-die and 16-die stacks with capacity up to 512 GB. It defines performance grades from roughly 0.4 TB/s to 3.0 TB/s and uses UCIe to connect to a host processor.
The first products have not shipped, so those numbers remain roadmap targets. They also cover a wide performance range, and the public material does not yet establish sustained bandwidth across production workloads. Sandisk's 2025 technical brief based its strongest comparison on an internal simulation that streamed 8-bit weights for Llama 3.1 405B. The simulated HBF system finished within 2.2% of a hypothetical HBM system whose capacity was assumed to be unlimited. The result suggests that a read-heavy workload can tolerate flash latency under favorable placement, but it does not measure a complete server.
NVIDIA is addressing another part of the gap with CMX. Its design uses BlueField processors, NVMe flash, RDMA networking, and the Dynamo software stack to create a shared context tier at the GPU-pod level. NVIDIA calls this G3.5, positioned between local server storage and durable shared storage. KV blocks are staged into host memory or HBM before decode rather than accessed as if flash were local DRAM.
CMX keeps context elsewhere in the pod and moves it back before decode. DRAM expansion pays more per bit but handles writes better. Compression and aggressive eviction reduce the hardware capacity required, although they can add recomputation, accuracy loss, or scheduling complexity.
An inference server could end up with several intermediate classes: HBM for its hottest state, DRAM for write-heavy cache, HBF for cold experts, and pod-level flash for reusable context.
HBF fits weights better than transient cache
The economic appeal of HBF comes from NAND density. A 512 GB package could hold far more of a large model near an accelerator than one HBM stack. For mixture-of-experts models, the system might retain many inactive experts in HBF and fetch only the selected weights. Fewer replicas and fewer GPUs devoted solely to weight capacity could lower the cost of inference.
This workload also accommodates NAND's weaknesses. Model weights are loaded occasionally and read many times. Writes are limited, access can be anticipated, and large transfers make bandwidth more important than the latency of one small read. HBF does not need to become DRAM to be useful here.
KV cache is a harder test. During generation, the system writes new key and value entries for every token. Many entries may never be reused after being evicted, and attention can request small, scattered regions. NAND has slower programming, finite write endurance, larger access granularity, and thermal constraints that a headline read-bandwidth number does not capture.
An August 2026 preprint tested HBF as a replacement for the flash tier in an SSD-style KV offload stack. The authors used four two-hour production traces, five dense and mixture-of-experts models, and H100 and B200 profiles. In their model, average end-to-end latency increased by 2 to 5.5 times and maximum service-level goodput fell by 1.1 to 2.7 times. Writes outnumbered reads in every trace, pushing the modeled HBF stack toward its thermal limit before it delivered its advertised bandwidth.
The paper is early research on hardware that has yet to ship. Its result identifies a condition missing from the easy HBF story. Faster flash helps only when a workload generates enough reusable reads, stays within the write budget, and does not give up too much HBM capacity or package bandwidth to install the far tier. Poor placement can turn additional capacity into another source of delay.
Today's margins come from scarcity
Sandisk does not need HBF to explain its present earnings. In its fiscal third quarter of 2026, data-center revenue was up 645% from a year earlier. HBF was not shipping, so it cannot explain the margin or revenue growth in that period.
Higher NAND prices and tighter supply drove much of the change, helped by a richer mix of data-center products. Sandisk has also signed multi-year customer agreements with financial commitments. These contracts may reduce volume risk, although their private pricing and cancellation terms determine how much of the cycle they actually remove.
Tight supply supports high earnings for now, but it does not remove the cycle. Supply recovery can take longer than expected. Higher prices still reward process transitions and additional capacity, which eventually push more bits toward customers.
HBF could improve the quality of part of Sandisk's revenue. A component qualified inside an accelerator package and tied to a software stack has more switching friction than commodity NAND wafers. It may earn a premium for delivered system performance rather than raw capacity.
That outcome requires commercial adoption. An open specification lowers the risk of dependence on one supplier, but it also makes it harder for one vendor to keep all the economics. Advanced packaging adds cost and new yield risks. Accelerator companies control the package, interfaces, and software. Hyperscalers can sponsor competing designs. If HBF becomes standardized capacity sold by several NAND manufacturers, its volume may be structural while its margin remains cyclical.
What HBF still has to prove
The standardization agreement between Sandisk and SK hynix makes multivendor support possible, but it is not a purchase order. HBF has commercial demand only when processor designers put it on qualification schedules, reserve package area and interface bandwidth, and expose it to software. Broader commitments from accelerator vendors have not been disclosed.
Production benchmarks need to report accelerator idle time, tail latency under a service target, mixed read and write behavior, and power for the complete tier. The runtime must recognize which tensors or cache blocks will be reused and move them before the accelerator stalls.
NVIDIA is building that placement logic into Dynamo and CMX. HBF will need comparable support in compilers, inference engines, and memory controllers. Workload tests should still separate model weights, prefill, decode, and reusable KV because each pattern stresses the device differently.
For investors, the financial evidence arrives after ordinary NAND supply catches up. They can then separate shortage pricing and conventional enterprise SSD revenue from any premium attached to HBF. Contracted volume and packaging cost will affect how much of that premium reaches the memory supplier; customer concentration will affect who has the bargaining power. Returns through a normal NAND cycle will show whether HBF is a differentiated category or simply a large new outlet for bits.
AI servers will use several intermediate tiers
AI inference needs more capacity between scarce HBM and conventional storage. NVIDIA is building a context tier, flash manufacturers are redesigning packages, and software teams are changing how KV cache is compressed and moved because the existing hierarchy wastes accelerator time.
HBF has a credible role, especially for read-heavy model weights and warm data whose capacity matters more than single-access latency. The case for transient KV cache is weaker. That workload changes quickly, and early system studies show how writes, heat, and poor placement can erase the benefit of high nominal bandwidth.
AI has increased the quantity and strategic importance of NAND, supporting a larger enterprise market. Current margins were produced by shortage before HBF generated commercial revenue. Treating those margins as permanent assumes that supply discipline will survive and that flash will capture a more valuable place in the computing stack.
When NAND supply loosens, Sandisk's margins will show how much of today's earnings came from scarcity. HBF becomes a separate investment case only when processor roadmaps and production workloads give flash a durable role closer to compute. For now, the rally shows that NAND has become strategically important while its old commodity cycle still sets the downside.
Sources
This report distinguishes shipping products from announced specifications and analytical models. Financial data are GAAP unless stated otherwise. The research cut-off is August 17, 2026.
- NVIDIA on KV cache size and offload
- NVIDIA, Introducing the BlueField-4 CMX context memory storage platform
- Sandisk and SK hynix on HBF standardization and Sandisk's HBF technical brief
- Sandisk, fiscal third-quarter 2026 results
- Li and coauthors, full-stack characterization of HBF for KV-centric serving