Huawei’s New Benchmark Gives AI Agents Months of Your Life—Then Watches Them Fail

The field for AI personal assistants has always been the same: give the agent access to your digital life and it will take care of the rest. Your emails, your calendar, your notes, your devices: everything. Your AI knows. Your AI acts. You sleep.

Researchers from Huawei Technologies, the Beijing Institute of Technology, Beijing University and the Chinese Academy of Sciences just built a benchmark to see if that’s actually true. Spoiler: it’s not.

Claw-everything evaluates AI agents in three dimensions simultaneously: long-horizon event streams spanning more than three months of simulated user activity, interdependent backend services averaging 10.1 per task, and multi-device interactions in both CLI Linux environments and GUI Android environments.

The average context window per task is 191,700 words. Most existing benchmarks are somewhere between 1,700 and 12,000. That is not a small gap, but a completely different problem. It’s also what real life feels like, as opposed to standardized, ultra-specific benchmarks.

Your AI has no idea what’s going on

The benchmark is scored on pass@1: the probability that the agent will correctly complete a task on the first attempt, without any repetitions. A task might ask the agent to reference a price alert for a product he found weeks ago, check the user’s calendar for a relevant appointment, and respond to both from a phone. Another might ask to pull recent work from notes, email threads, and Slack and then create a presentation from scratch.

These are things people actually ask assistants. It turns out that AI isn’t very good at it. GPT-5.5 is, according to Decrypt’s previous reporting, OpenAI’s best model, built with long-horizon agentic tasks in mind. It scored 34.5%.

“Current models remain unreliable even as they gain broader access to the user’s digital world,” the paper said Claw-all-paper reads. Several models that look impressive on other benchmarks have fallen further.

The benchmark also assesses proactive assistance separately, that is, cases where the agent identifies a need and acts without being prompted. Most benchmarks don’t test this. Claw-Anything does that, and the gap is wide: agents scored 25.9% on reactive tasks and only 6.7% on proactive tasks.

Why most benchmarks don’t tell you this

The researchers make a pointed argument: existing benchmarks treat AI agents as task solvers given an empty desk. Claw-Anything treats them like personal assistants thrust into a truly messy life: irrelevant events, conflicting signals, months of built-up noise. The agent must figure out what is relevant before he can do anything useful.

The ablation results make the dependency on multiple services particularly clear. When the tools needed for cross-service tasks were removed, the success rate dropped to near zero because most tasks require agents to retrieve information and act across multiple backends instead of within a single backend.

This is not a new kind of problem in AI evaluation. OpenAI declared earlier this year that the SWE bench was tainted after scores dropped from around 70% to 23% on a version that leaks less. That was about data hygiene. This is about something more fundamental: whether the benchmarks are asking the right questions.

On the constructive side, the team released the pipeline that generated the benchmark alongside 2,000 training environments. Refining Qwen3.5-27B on 1,500 successful agent paths improved pass@1 by 23.7% – enough to beat several closed-source models on the leaderboard, including Claude Sonnet.

The researchers identify coordination between services as the benchmark’s key remaining challenge for the field. The dataset is on Hugging face and the code is enabled GitHub.