If an AI agent writes a broken Row Level Security policy into your production database, you will not find out until a user touches data they should never have seen.

AI agents writing Supabase code have had no objective scorecard until now

Teams evaluating which coding agent to trust with their backend have been flying on vibes, cherry-picked demos, and internal gut checks. There has been no public, reproducible test that measures agent performance against the actual tasks that break in production.

The benchmark boots a real stack and runs real scenarios against it

Supabase Releases Evals lets you clone the repo, run pnpm locally, and watch coding agents attempt tasks drawn from real support tickets and GitHub issues — building a schema, debugging a failed Edge Function, fixing a broken RLS policy — then scores each result against a grounded expected output. Input is your chosen agent and provider API key. Output is a score that feeds either the public leaderboard at supabase.com/evals or your own internal regression suite.

Security engineers are paying the highest price for bad agent output

Three roles have the most to gain from a reproducible agent benchmark right now:

• Backend engineers who gate SDK releases and need proof an agent did not regress a migration or break a security rule before shipping
• Platform security leads in fintech or healthcare who require evidence that AI-generated RLS policies are correct before they touch regulated data
• Developer tooling teams who maintain Supabase docs and need daily regression signals whenever content or SDK behavior changes

The benchmark covers nine Supabase product areas and ten topic dimensions including security, observability, and declarative schema, which means coverage goes well beyond the obvious happy paths.

The agent evaluation market just got a public reference point

Claude Code, Codex, and OpenCode now have published, comparable scores on a shared task set, which is the first time any infrastructure vendor has released this kind of head-to-head data in the open. As more vendors follow, the expectation that agents ship with benchmark evidence will become the baseline, not the exception.

What you can do with it today

• Run Claude Code and Codex against identical Supabase tasks and compare scores directly
• Gate a release by wiring the regression suite into your CI pipeline
• Test whether a docs edit broke agent performance before it reaches users
• Reproduce any public leaderboard result locally with your own API keys

Supabase Evals is open source under Apache-2.0. Free to run locally; provider API costs apply.

Local runs require Docker, free ports 54321 through 54329, and valid provider keys, which adds meaningful setup friction for teams without a dedicated infrastructure environment.

For agent benchmarking without infrastructure setup, SWE-bench offers a code-repair leaderboard that runs fully hosted. If your concern is narrower than Supabase specifically, HumanEval covers general code generation scoring without any backend stack requirement.

Vendors are starting to publish agent scorecards and buyers should expect it everywhere

The shift from “our agent is good” to “here is a reproducible score on real tasks” is accelerating, and Supabase Evals is now a reference implementation any infrastructure team can copy. We cover tools like this every Friday — subscribe here and we’ll send the best ones straight to you.