AI agents at work reveal things they shouldn't: up to half of tests ended in a breach

When an AI agent helps with work tasks, it shouldn't just complete the task. It also needs to know which information can be shared with whom. A new benchmark called CI-Work tests exactly that, and the results are not reassuring.

What the test measures

CI-Work was released by researchers from Huazhong University of Science and Technology and Microsoft. It measures what experts call contextual integrity: whether an agent follows the norms governing which information may flow where. Information that is fine to share with a colleague may be inappropriate to send to an external party.

The benchmark consists of 1,000 synthetic, meaning artificially created, context entries, derived from 125 task-based seeds. They cover five directions of communication within organizations: upward, downward, lateral, diagonal and external. They also span industries such as technology, finance and health care.

Up to 50.9 percent breaches

The researchers tested leading models, including GPT-4o, GPT-4.1, GPT-5 and o3. The share of privacy violations ranged from 15.8 to 50.9 percent. Leakage reached as high as 26.7 percent.

Higher task utility went hand in hand with more breaches. Larger models often leaked more, which the researchers call inverse scaling. User pressure, whether implied or explicit, nearly doubled the share of breaches.

According to the authors, larger models, deeper reasoning and defenses based on instructions in the prompt were not enough to fix the problem. Human assessments matched the benchmark's privacy judgments in 82.5 to 95.0 percent of cases.

The results apply to leading models in general and not to Microsoft Copilot as a product, so they cannot be directly applied to it. They also come from a benchmark using synthetic data.

Microsoft: Testing without customer data

A separate blog post from Microsoft Copilot Studio explains how to test the tools that assess agents' responses, known as evaluators. Microsoft says it does not use customer data in any form to validate them. Instead, it uses curated examples and generated datasets.

The generated datasets have a known ground truth. Queries and responses are created, and some responses are deliberately degraded. That makes it possible to measure how often the evaluator catches the bad responses and approves the good ones, using true positive rate and true negative rate. Microsoft uses four strategies to generate data: single-turn, multi-turn, knowledge-based and topic- or instruction-based.

The post's author, Dikla Dotan-Cohen, data lead at Microsoft Copilot Studio, presents generated data as a necessity for supporting many types of agents rather than as a compromise. According to her, it lets teams start evaluating earlier and scale faster.