Every extra token your agent generates in a long-running task costs money, and most open models make you pay that cost whether they’re fast or not.

Long-context agents have been bleeding compute budgets quietly

Multi-turn agents that plan, call tools, and reason across sessions balloon token counts with every step. The result is inference costs that compound faster than most teams budget for.

A 550B model that activates only 55B parameters per token

NVIDIA AI Releases Nemotron 3 Ultra is a 550 billion parameter Mixture-of-Experts model where only 55 billion parameters are active per token, built on a hybrid Mamba-Attention architecture that replaces most transformer layers with Mamba layers for sub-quadratic scaling on long sequences. You deploy it via NVIDIA’s open weights release, feed it long-context agent prompts up to one million tokens, and get outputs trained across 20 trillion tokens with post-training via reinforcement learning and multi-teacher distillation. The architectural payoff is approximately 6x higher inference throughput than comparable open LLMs at on-par accuracy, according to NVIDIA’s technical report.

Infrastructure teams feel this first, but they are not alone

• ML engineers running production agent pipelines who need to cut per-query inference spend without sacrificing reasoning quality on multi-step tasks
• AI platform architects at enterprises who are choosing between closed APIs and open deployments and need a model that justifies the hosting overhead
• Research teams stress-testing retrieval and long-context recall who keep hitting context limits or latency walls with current open alternatives

The 108-layer design uses 512 experts per MoE layer with only the top 22 activated per token, and just two key-value heads to keep the KV cache lean across those million-token windows.

The open long-context race just got a serious new entry

Mistral’s Mixtral and Meta’s Llama architecture have dominated open MoE conversation for the past year, but neither was designed with the sub-quadratic memory demands of persistent agents in mind. If inference cost is the primary reason teams are still reaching for closed APIs, a 6x throughput gap is exactly the number that changes that calculation.

What you can actually do with it

• Deploy long-running tool-calling agents without exponential compute cost growth
• Run retrieval tasks over million-token legal, financial, or codebase documents
• Replace closed API calls for multi-turn reasoning chains in cost-sensitive pipelines
• Benchmark against GPT-4o class models on agentic tasks using open weights

Weights are released openly under NVIDIA’s model license, making self-hosted deployment the primary path.

The honest catch

At 550B total parameters, even with efficient activation, you need serious GPU infrastructure to self-host this at any meaningful throughput.

The alternatives worth knowing

Mistral’s Mixtral 8x22B is lighter to host and battle-tested in production, though it tops out well below one million tokens. For closed-API long-context work, Google’s Gemini 1.5 Pro offers comparable context length but no open weights and no infrastructure control.

Open model inference costs are dropping faster than most teams have priced in

The gap between closed and open model performance on agentic tasks is closing in ways that will force real infrastructure decisions this year. We cover tools like this every Friday — subscribe here and we’ll send the best ones straight to you.