Best Models: Worst Tools
Show notes
Armin Ronacher noticed an intriguing behavior with the latest Anthropic models (Opus 4.8, Sonnet 5). Despite being more advanced, they are getting worse at tool calling, specifically inventing extra and invalid fields in nested arrays, such as in the edit tool of Pi. The main content of the call is correct, but these "invented" fields cause the invocation to be rejected, requiring retries. The hypothesis is that this regression is a training artifact, where the models were heavily optimized for Anthropic's Claude Code, which may have simpler tool schemas, generating noise when dealing with more complex structures.
Why it matters: This exposes a fragility in SOTA models: over-training for a specific environment can degrade the reliability of tool calls in other contexts, requiring greater attention to schema validation.
Sources: