Zapier's New AutomationBench-AA Reveals Compliance Guardrails Cut Top AI Model Scores in Half

According to Beating's monitoring, Zapier and Artificial Analysis launched AutomationBench-AA, an independent SaaS workflow automation benchmark based on real anonymized data from Zapier's platform. The benchmark comprises 657 multi-step tasks and 40 simulated SaaS environments including Gmail, Slack, Salesforce, and Jira.

Crucially, the benchmark enforces strict compliance guardrails: any violation of preset enterprise business rules automatically zeros out the final task score. In the initial evaluation, all models saw scores plummet due to compliance penalties. Claude Fable 5 completed 73% of tasks but achieved only a 48.6% final score; Gemini 3.5 Flash and GPT-5.5 both completed approximately 68% and 67% of tasks respectively but scored 42.6% and 42.1%. Open-source GLM-5.2 scored 27.8%.