Running a 120B hybrid MoE model at scale means most teams either cap concurrent users or burn through node budgets before lunch.

Serving large context at scale has been a hardware tax, not a model problem

Large hybrid Mamba-Transformer MoE models are accurate but punishing to serve. Active parameters, KV cache, and Mamba state combine to strangle concurrency the moment user token rates climb.

The weight drop is what changes the economics

Meet Nemotron Labs 3 Puzzle 75B A9B is a compressed variant of Nemotron-3-Super, shrinking total parameters from 120.7B to 75.3B and active parameters from 12.8B to 9.3B, while keeping the parent’s 88-block hybrid layout intact. You deploy it from one of three Hugging Face checkpoints: BF16, FP8, or NVFP4. The output is the same hybrid Mamba-Transformer inference pipeline your team already runs, now with weights that drop from 70 GB to 44.5 GB.

Infrastructure teams feel this before anyone else does

• ML infrastructure engineers who need to double concurrent user capacity without adding nodes get a 2.03x throughput lift at matched NVFP4 precision.
• Applied AI teams running long-context workloads who have been limited to one 1M-token session per H100 can now run eight simultaneously on the same hardware.
• Model deployment leads evaluating compression tradeoffs get a concrete benchmark cost: Arena-Hard-V2 drops 4.2 points, SWE-Bench drops 2.6, but RULER and long-context retrieval barely move.

The compression method matters as much as the result. NVIDIA used an iterative version of their Puzzle framework rather than a single-step pass, and that choice alone accounts for a 0.57 average-point quality gain at the same compression target.

The inference cost curve just shifted for hybrid MoE deployments

On 8xB200, total throughput over the parent model rises between 1.60x and 2.14x depending on configuration, which puts Nemotron-Labs-3-Puzzle-75B-A9B ahead of what most teams achieve by simply quantizing a dense model. As long-context workloads become a baseline expectation rather than an edge case, the gap between teams who can serve them concurrently and those who cannot will widen fast.

What you can run with this today

• Deploy eight simultaneous 1M-token sessions on a single H100 node.
• Swap in the NVFP4 checkpoint to hit 2x throughput at 100 tokens per second per user.
• Benchmark long-context retrieval with RULER before committing to a production swap.
• Compare iterative Puzzle compression against your current pruning pipeline on the same target.

Pricing not listed — check our directory.

The real cost is a 4-point drop on hard reasoning benchmarks, not infrastructure

Nemotron-Labs-3-Puzzle-75B-A9B trades measurable performance on code and hard reasoning tasks for the throughput gains, so teams running SWE-Bench-class workloads should validate before replacing the parent model in production.

For dense model compression, NVIDIA’s own pruning pipelines and tools like SparseGPT target similar weight reduction goals but without the hybrid Mamba-Transformer architecture support. Mistral’s MoE variants offer a different active-parameter tradeoff but are not designed around long-context concurrency as a primary deployment target.

Inference economics for long-context MoE are being rewritten right now

The shift from single-session to multi-session long-context serving on a single node is not incremental — it changes what a deployment is worth per dollar. We cover tools like this every Friday — subscribe here and we’ll send the best ones straight to you.