Privacy alignment for agents, grounded in what people actually think

When an agent acts on my behalf, almost everything it does is a disclosure decision. It drafts a reply and decides which earlier thread to quote. It fills out a form and decides which field to leave blank. It calls a tool and decides which slice of my history to hand over. Each of those is a small contextual judgment about what is appropriate to share, with whom, and under which conditions — and "appropriate" is not a property of the data. It is a property of the situation.

That is the premise of PrivacyAlign: Contextual Privacy Alignment for LLM Agents, and I think it gets the problem statement exactly right, even where the contribution itself is incremental. The authors treat privacy as an alignment problem rather than a content filter, and they make a point I keep coming back to: for this class of decision, human judgment does not merely label violations after the fact. It defines them.

Why a filter is the wrong mental model

The instinct in production is to bolt a detector onto the output — scan for an email address, a medical term, a name — and redact. That works for the narrow case where a token is sensitive regardless of context. It fails for the case that actually matters. My home address is fine to share with a courier and not with a stranger in a forum. A diagnosis is appropriate in a message to my doctor and a leak in a message to my employer. The same string is a disclosure or a violation depending on who is on the other end and why. This is the contextual-integrity view of privacy — appropriate information flow, not secrecy — and it is the reason a static filter can never be enough for an agent that participates in open-ended social contexts.

If you accept that, the evaluation problem gets harder, not easier. You cannot grade an agent's disclosure decisions against a rulebook, because the rulebook is the thing in dispute.

The proxy problem

Most of the agent-privacy work I have read leans on proxies for exactly this reason, because real judgments are expensive. Synthetic rules, single-rater labels, regex-style sensitivity tags, or an unconditioned LLM judge asked to decide "was this a leak?" with nothing but its own priors. Each is a stand-in for the social norm, and each drifts from it in ways you only notice after you have trained against it. Train a reward model on a proxy and you optimize the proxy. The agent learns to satisfy the measurement, not the norm.

PrivacyAlign's response is to stop substituting. The authors build a dataset of 1,350 samples carrying 3,516 detailed annotations from 599 unique annotators, drawn from scenarios where current LLMs actually leak rather than hypotheticals. Then they use those human annotations in both places where a proxy usually sits: evaluation and training.

One dataset, two uses

The mechanism is the part worth studying, and it is cleaner than the privacy framing might suggest.

First, evaluation. Instead of asking an LLM judge to rule on a response cold, they condition it on human annotations and explanations attached to reference responses for the same prompt. The judge is no longer guessing the norm; it is shown the norm and asked to apply it. That conditioning makes its judgments measurably more reliable — which matters far beyond privacy, because an unreliable judge quietly corrupts every downstream number you report.

Second, training. They turn the same annotations into a reward signal — annotation-conditioned reward modeling — that scores new responses during reinforcement learning. Small open-weight agents trained against this reward align better with human privacy norms, with gains on PrivacyAlign itself and on existing agent-privacy benchmarks. The result I care about is the size asymmetry: because the grounding comes from people, a small model trained on it can behave well without needing a frontier model's general competence to back-fill the missing judgment.

My read

I want to be precise about what this is and is not. It is a well-motivated, methodologically sober paper in a crowded line of work. Contextual integrity for agents, privacy-leak benchmarks, judge-based evaluation — none of these are new, and PrivacyAlign does not pretend otherwise. The contribution is incremental: it threads real human annotations through both the judge and the reward, where the field has mostly used one or the other and usually a proxy for both.

But "incremental" is not a dismissal. The move that travels is the conditioning trick — handing a judge or a reward model the reference annotations for a prompt rather than asking it to reconstruct the norm from priors. That is a general recipe for any domain where the ground truth is a contested human judgment and not a fact: content moderation, tone, escalation, refusal calibration. Privacy is a sharp instance of the class, not the whole of it.

The limits are the ones you would expect. Norms from 599 annotators are still a sample of a culture, not a universal; a reward model is still hackable; and the judge's reliability depends on having annotated reference responses, which you do not have for arbitrary new prompts at inference time. The paper grounds the training loop in human judgment, but it does not make the agent itself a good judge in the open world. That gap is the next problem, and this paper does not close it.

What it changes for builders

For anyone shipping agents, the practical takeaway is narrow and useful: if you are using an LLM as a privacy judge, condition it on real annotations before you trust its scores, and treat any reward built on an unconditioned proxy as a thing your agent will eventually learn to exploit. Put the human judgment where the optimization pressure lands. That is the durable idea here, and it is right.