Local AI Hardware: Capacity, Bandwidth and the Cost of an Upgrade
HARDWARE & DESIGN
Local AI Hardware: Capacity, Bandwidth and the Cost of an Upgrade
Before buying more memory, choose the model and job that will use it. A small fast text model, a large MoE and a video pipeline put pressure on different parts of a computer. A complete quote and a matched workload test are more useful than a ranking built from peak specifications.
In this article
Correction, October 17, 2026: an earlier version ranked computers using forecast token speeds for an unconfirmed Qwen3.8-35B-A3B checkpoint and simplified large-model fit assumptions. Those rankings had already been withdrawn in an October 3 note. The unsupported tables and five derived charts have now been removed as well. This replacement is a purchase-planning guide, with no measured hardware ranking.
Define the workload before comparing computers
Record the model repository, revision, quantized artifact and required components. Add the context length, expected output length and number of simultaneous requests. “A 35B model” is too broad a target: different quantizations and architectures can have different file sizes, cache behavior and runtime support.
Total parameters and active parameters answer different questions. A mixture-of-experts model may select a fraction of its experts for a token, but the remaining weights still require storage or an explicit offload path. The linked memory guide explains raw weight arithmetic and why it is only the start of a budget.
Keep the memory resources separate
Dedicated VRAM, system RAM and unified memory are not interchangeable labels. A runtime may support splitting layers or offloading components, but that does not mean every byte is available to every operation at the same speed. Two GPUs do not automatically behave like one GPU with their combined capacity. Confirm how the chosen software distributes the actual model.
Leave space for the operating system, normal applications, runtime buffers and context cache. Use the same unit on both sides of the calculation: decimal GB and binary GiB differ. A model that loads into an otherwise empty machine may fail once a browser, editor and longer conversation are added.
| Resource | What to verify |
|---|---|
| Dedicated VRAM | Peak allocation for the selected quantization and context |
| System RAM | Offloaded weights, host buffers and other applications |
| Unified memory | Usable process budget and competing system allocations |
| Multiple accelerators | Runtime support, placement and interconnect overhead |
| SSD | Required files, revisions, scratch space and reload time |
Ask for a matched performance test
Peak memory bandwidth is a hardware specification. It cannot, by itself, supply a reliable tokens-per-second figure. The model architecture, precision, batch size, kernels and serving configuration all affect how much of that bandwidth is useful. A fixed efficiency multiplier does not turn a specification sheet into a benchmark.
Separate prompt processing from output generation. A long-document task may spend most of its wait on the prompt; an interactive conversation may be more sensitive to time to first token and steady generation speed. Record both phases, peak memory and failures. Compare the same prompt, output limit and model revision on each candidate.
For a video workflow, replace the language-model test with the actual resolution, frame count, precision and conditioning setup. A machine that handles a text model well need not be a good match for a video pipeline. The MiniMax H3 article linked below is a separate workload example, not a transferable LLM speed result.
Compare the configuration you would actually buy
Obtain a quote for the required memory configuration rather than the advertised starting price. Include storage, tax, delivery, adapters and any networking needed for the setup. Check which components can be upgraded later. A low entry price offers little guidance if the target model requires a much more expensive memory option.
Use dated manufacturer specifications to establish capacity and connectivity, then a current quote for price. The Apple specification link below belongs to the launch discussed in the earlier article; it is not a current storefront quotation. The newer DGX Spark article also concerns a particular configuration, not a universal price for every Spark system.
Calculate ownership cost against the same work
A break-even calculation needs an equivalent alternative. Add the purchase cost and expected electricity, maintenance and paid software costs; account for any resale assumption separately. Compare that with the hosted cost of the same successful workload. A flat chat subscription and metered model API can have very different limits and capabilities.
For example, a hypothetical $3,000 purchase would take 30 months to offset $100 of avoided monthly spending, before running costs or resale. The arithmetic is simple; proving that the spending is actually avoidable is the difficult part. Do not count a subscription you would keep anyway as savings from the machine.
The useful outcome is a short record: a released artifact, a reproducible test, a complete quote and an acceptable wait time. If one is missing, retain the uncertainty in the purchase decision. Trying a smaller confirmed model on existing hardware is often the cheapest way to discover which constraint matters for your own work.
Comments
Post a Comment