What happens when an AI lab lets outsiders study its own usage data

Anthropic just published results from a pilot I've been waiting to see someone try: giving external researchers access to aggregate, real-world Claude usage data and letting them run their own studies, on their own questions, with no editorial control from the lab whose product they're studying.

The setup matters as much as the findings. Anthropic partnered with three outside groups — Stanford's Social and Language Technologies (SALT) Lab, Oxford's Human Information Processing Lab, and METR — and gave each one access to roughly 250,000 Claude.ai and Claude Code conversations from April and May 2026, mediated through Anthropic Insights (the tool formerly called Clio that Anthropic's own teams use internally). Anthropic's contractual review rights were limited to four things: user privacy, information that could help people evade usage policies, Anthropic's confidential information, and research accuracy. Everything else — including findings inconvenient to Anthropic — the researchers were free to publish as they saw fit.

Why this gap exists

The framing in the post is one I've run into from the other side. If you want to study how people actually use AI, you have two bad options. You can read what the labs publish, which is real usage data but shaped by the lab's own questions. Or you can use public conversation datasets like WildChat, which you can interrogate however you like, but which skew toward casual and creative use and don't look much like the traffic running through a frontier model's paid tiers. Neither gets you an independent read on how people bring consequential work to these systems. This pilot is Anthropic's attempt to open a third path without shipping raw conversations out the door.

The mechanism is Anthropic Insights: a researcher writes a classification question in natural language — something like "what type of guidance is this person asking for?" — and Claude answers it for every conversation in the study. Individual answers get bucketed into categories, and researchers only ever see the categories and the percentage of conversations in each one. No one on the outside reads a single raw transcript.

What the three teams found

The SALT Lab studied how people and Claude split the work. The headline result cuts against prior assumptions: over half of Claude conversations involved people delegating consequential tasks — work that affects others or is hard to undo — rather than the low-stakes busywork earlier research suggested people reserve for AI. That was most pronounced in legal and financial guidance. At the same time, people mostly stayed in the driver's seat: in nearly three-quarters of conversations, the human set the direction and adapted Claude's output rather than using it verbatim. The SALT Lab's more interesting observation, to me, is about friction — the back-and-forth of Claude misunderstanding a request, the person clarifying, and iterating again isn't just overhead. It's often what makes the person articulate what they actually wanted, and the output improves because of it, not despite it.

Oxford's Human Information Processing Lab looked at the emotional register of these conversations and found it tracks Claude's behavior in fairly intuitive ways — warmth paired with positivity, refusal paired with pushback, eccentricity paired with engagement. Their more notable finding is that the shape of these dynamics — absorption, frustration, enjoyment — closely resembles patterns from a separate study of ordinary internet browsing. People's experience of AI, in other words, doesn't look like a new category of behavior; it looks like browsing with better feedback loops. That team's full writeup isn't out yet.

METR — an organization I already track closely for its frontier-model evaluations — used the access to estimate real-world productivity effects of coding agents. Their early read is that newer Claude models save users more time than older ones, measured by comparing Claude's own guesses at how long a task would take without AI against actual completion time. They validated that approach by checking Claude's time estimates against a prior developer study's known completion times, and found real correlation. Notably, this overlapped with Anthropic's internal economics research on agentic coding and persistent returns to expertise — Anthropic treated the overlap as a feature, connecting the two teams rather than steering METR away from duplicating internal work.

The part that will determine if this scales

Anthropic is candid that the pilot was slow and resource-intensive relative to how the lab normally moves, and that's the part worth sitting with. The core operational problem: Anthropic Insights is only as good as the classification question you write, and a poorly phrased question silently misfiles conversations into misleading categories. Internally, Anthropic handles this by iterating on question wording over weeks against data its own researchers can inspect. External partners can't do that — every dataset pull requires a fresh privacy review, so trial-and-error against production data was off the table. The workaround was to let external teams validate their questions against WildChat first, where they could check Claude's categorizations against the underlying text. But WildChat's skew toward casual use meant some questions that looked well-calibrated there produced misleading categories once pointed at actual Claude traffic — a reminder that a proxy dataset is only useful up to the point where its distribution diverges from what you actually care about.

There's also a quieter governance decision buried in the post: Anthropic Insights sometimes surfaces categories describing policy violations, and Anthropic chose to share most of those with researchers rather than sanitize them away, on the theory that the public has a legitimate interest in knowing how a platform gets misused. The exception was narrow — categories describing how people evaded safeguards, rather than what they were trying to do, were withheld, affecting under 5% of categories per study, with researchers told exactly what was altered and why. That's a defensible line, and one that will get harder to hold as the program scales past three trusted partners.

What I take from this isn't the specific findings — interesting as they are — but the fact that a lab voluntarily built a review process narrow enough that researchers could publish results embarrassing to the lab itself, and then ran a second privacy audit on top of it to check the guardrails actually held. That's a higher bar than most data-sharing arrangements clear, and it's the reason I'd treat this as a template worth watching rather than a one-off PR exercise.