Most teams benchmarking GPT-5.6 are leaving significant performance on the table because two API settings ship disabled by default.
Benchmark scores were flatlining for a fixable reason
Running repeated reasoning evaluations without retaining context forces the model to reconstruct its logic from scratch on every call, burning tokens and degrading consistency. The result is scores that look mediocre on ARC-AGI-3 not because the model is weak, but because the eval setup is.
The reasoning context was getting thrown away after every call
How enabling two settings tripled our scores on the ARC exposes two toggles in the API: reasoning retention, which carries the model’s internal chain-of-thought forward across calls, and context compaction, which compresses prior turns without truncating them. You set both flags in your API configuration, point them at your existing eval harness, and the output is a model that builds on its own reasoning rather than restarting it. The score jump on ARC-AGI-3 was not incremental.
Eval engineers are the first to close this gap
- ML engineers running ARC-AGI-3 evals who need reproducible score improvements without retraining
- AI product teams benchmarking GPT-5.6 against competitors who want to stop misreporting capability floors
- Research leads who track token efficiency and need to show cost-per-correct-answer improvements to stakeholders
The fix costs nothing extra. That is the part that stings when you realize how long the settings were sitting there unused.
ARC-AGI-3 just became the benchmark that separates configuration from capability
ARC-AGI-3 is the current standard for measuring abstract reasoning, and with GPT-5.6 now in wide API access, teams racing to publish comparative evals are doing so with misconfigured defaults. If reasoning retention becomes a standard API expectation rather than an opt-in flag, every published benchmark without it becomes a floor, not a ceiling.
What the two settings let you do
- Retain reasoning chains across multi-step ARC-AGI-3 task sequences
- Compress prior context without losing logical continuity between calls
- Cut redundant token usage on repeated eval runs by compacting history
- Compare retained versus non-retained scores to audit past eval accuracy
Pricing is usage-based through the OpenAI API at standard GPT-5.6 rates.
The main tradeoff is that reasoning retention increases latency per call, which matters if your eval pipeline is time-sensitive. The two strongest alternatives are Anthropic’s extended thinking mode on Claude, which does something structurally similar, and Google’s Gemini 2.5 Pro, which handles long-context reasoning natively without a toggle.
The default API settings benchmark is ending
Published scores that do not disclose configuration flags are about to mean less than they did six months ago. We cover tools like this every Friday — subscribe here and we’ll send the best ones straight to you.