SDKs graded today04runs logged10+real bugs found03built by1 + AI
The problem
Onboarding dies quietly.
Every SDK team believes their quickstart works. Almost nobody re-tests it from a truly clean machine.
The next developer to hit your docs is increasingly an AI agent in a sandbox — no tribal knowledge, no second tab, no patience.
Today you find out from lost signups. That is the most expensive way to learn.
How it works
Clean room. Real agent. Scored evidence.
target.yamlquickstart URL + promised outcome→Daytonapristine sandbox per run→agentFireworks · Kimi K2.6→judgecounts only real output→Braintrustscored eval, every run→score + badge0–100, live on Pages
A checkpoint passes only on real command output. Deviations from the docs
must be declared. Secrets are redacted. The sandbox is destroyed after every run.
Results — today, live
Same test. Very different scores.
100
elevenlabs
Perfect, twice. Copy-paste-true samples + an /llms.txt agent index. Agent-ready by design.
75
fireworks
Documented example model no longer exists → first call 404s. Agent recovered by writing its own model-discovery script. Both runs.
70
daytona
Quickstart samples call renamed SDK methods (Process.execute → exec). The meta-run still created a sandbox inside a sandbox.
30
braintrust
API unreachable from inside agent sandboxes; the offline flag is undocumented. Score volatility (75→0→30) is itself the finding.
Why history matters
Single runs lie. Tracked history doesn't.
daytona · 70 → 45 → 70
dip = our account quota, not their docs. Caught by history · attributed by transcript · reproduced after cleanup.
braintrust · 75 → 0 → 30
volatility = a hard dependency on network egress + an undocumented workaround. The instability is the finding.
Every point is a scored Braintrust eval. A nightly GitHub Actions sweep
re-grades every SDK — the board updates itself. Docs-regression CI, shipped.
Sponsor integrations — all load-bearing
Remove any one, and the lab stops.
Daytona
The clean room
25+ sandboxes created & destroyed today — one pristine box per measurement
Meta-run: the agent created a Daytona sandbox inside a Daytona sandbox via their SDK
Sandbox hygiene shipped: lab cleanup + a cleanup step in nightly CI
Fireworks AI
The brain — agent AND judge
~300 tool-calling completions on serverless kimi-k2p6
Evidence-strict judge runs on Fireworks too — every checkpoint verdict
Found a real docs bug in Fireworks' own quickstart; the agent self-recovered via their models API
Braintrust
The ledger
11 scored evals · 55 checkpoint scores — every run, no exceptions
Every history point carries its span ID; read back via the Braintrust REST API to verify
Score history lives where evals belong — the sparklines mirror it
Plus: ElevenLabs' API exercised to a verified MP3 in
all three of its 100/100 runs — the graded SDKs are integrations too.
The ask
Your next user might not be human.
Live: zhannur.github.io/agent-ready — scoreboard, readable logs, badges with one-click embed
Shipped today: nightly re-tests via CI · on-demand runs · full score history · we catch our own bad measurements
Next: model matrix · environment matrix · signed vendor-neutral reports