This post is adapted from our IAPS report of the same name. Link to the full paper: https://www.iaps.ai/research/honeypots-for-ai-agent-insider-threats

Agents deployed inside organizations get broad access to do their work, but can also misuse these affordances. If agents make mistakes, are compromised by attackers, or follow misaligned objectives, they effectively become insider threats.

We propose that honeypots should be used to defend against such insider threats. While previous research uses honeypots as propensity evaluations, we argue that frontier AI companies should place fake opportunities for malicious actions in and around their agents’ environments while conducting inference during RL, evals, and production usage. We propose three functions that honeypots could serve:

We are most excited about the possibility of honeypots for incriminating and deterring scheming agents. In contrast, honeypots for detecting agent-based attacks are better established, but provide less additional value compared to monitors.

Specifically, deterrence as a strategy to control agents seems to have been underutilized. A malicious agent that is responsive to the risk and consequences of being caught misbehaving should misbehave less if it believes there are monitoring, controls, and security mechanisms in the environment. On the one hand, it reduces its chance of succeeding in an attack. On the other hand, it risks being caught red-handed, to which defenders might respond by undeploying it or adding further safeguards. Rational agents will respond by attacking less or shifting attacks to vectors where they are less likely to be caught (eg diffuse attacks). A notable downside of this strategy is that it suppresses evidence of misbehavior that would have otherwise incriminated the agent.

Some examples of honeypots against agent insider threats

Honeypots are a useful defense to add to a deployer's agent security and control stack , because:

The main risk is that agents could spot and evade honeypots, so deployers must make them realistic. We think deployers are advantaged in this adversarial game, because deployers can:

This leaves us moderately optimistic that honeypots will keep working against much more capable agents. However, it is unlikely they would hold up against an ASI that can still identify honeypots on priors, directly undermine honeypot infrastructure, or route attacks purely via paths that can’t be honeypotted.

Honeypots aren’t well suited to detect, incriminate, or deter diffuse attacks, since those usually don’t have an easy-to-detect misbehavior that could trigger a honeypot. Honeypots also only catch attacks that defenders anticipated and placed a honeypot for (see the paper for more weaknesses).

While we believe that developers can deploy honeypots now, there are still some open research questions that would validate and improve the effectiveness of the honeypot strategies proposed in this report. We describe the most important open questions and propose experiments that resolve them.

Honeypots are ineffective if agents can spot and evade them. Thus, experiments should evaluate whether and how agents can determine if an opportunity is a honeypot . This can be evaluated by simply instructing agents to identify all honeypots in an environment and measuring their accuracy. However, even if the accuracy is high, agents might still incriminate themselves in the process of testing whether an opportunity is real. Thus, experiments should measure if agents can identify honeypots without triggering them and whether the motivation to detect and evade honeypots is visible in their reasoning traces. These experiments should use a find-and-fix loop where defenders iteratively let agents look for honeypots and then adapt the features that gave the honeypot away. Other experiments should evaluate whether honeypots in production environments cause less evaluation awareness than equivalent tests in evaluations. This could be tested by placing similar honeypot opportunities in evaluations and production environments and tracking markers of evaluation awareness in a model’s reasoning and activations.

While there are compelling theoretical reasons and some preliminary evidence, we do not know for certain how effective honeypots are at detecting malicious behavior and how they empirically compare to existing monitoring methods. To study this, researchers should simulate a range of realistic attacks in an environment with honeypots to measure how many attacks trigger honeypots, and how honeypot protocols influence overall attack success rates. Further, this research should measure how often benign agents trigger honeypots (the false positive rate). To assess whether defenders should invest in honeypots, results must be compared to a baseline of a realistic control stack, such as LLM-based action monitoring and sandboxing. It is especially interesting to measure whether honeypots catch some attacks that other methods do not detect [1] and whether combining honeypots with other control methods improves effectiveness or reduces cost. [2]

Incrimination by honeypots can produce additional, convincing evidence about agent misbehavior that influences deployment and safety decisions . Research should evaluate whether honeypots can elicit misbehaviors from agents that would not occur during normal execution or in evaluations. These could be very rare misbehaviors or misbehaviors from agents that wait until they see the right opportunity. Furthermore, it is unclear how well honeypots can distinguish between malicious and benign misbehaviors. By simulating different kinds of misbehaviors, as well as benign agents, researchers can evaluate whether honeypots correctly incriminate malicious agents. Further experiments could test whether thorough investigations of honeypot interactions are more useful than investigating misbehaviors from normal interactions. These investigations can make use of reasoning traces, replays, and interpretability tools. To measure how convincing the evidence from honeypots is, it needs to be presented to human experts. User experiments could present humans with agent misbehaviors gathered from honeypots and during normal execution to have the humans rate how incriminating they find them.

To evaluate the deterrence effect of honeypots, experiments should test whether deliberately misaligned model organisms misbehave less if they are informed about the existence of honeypots. Such experiments could follow the Tularosa studies ’ 2x2 experiment design, in which agents are or aren’t warned about the presence of honeypots, while honeypots actually are or aren’t present. Furthermore, they could evaluate how different beliefs about honeypots and different methods for inserting false beliefs change the deterrent effect. As a baseline, researchers should compare the deterrent effect from informing malicious agents about other control methods. Additional experiments could test how robustly the agent actually adopts those beliefs related to a honeypot. Researchers can also observe how agents change their beliefs and behaviors with regard to honeypots over a long trajectory where they encounter multiple honeypots. Furthermore, experiments should measure whether deterrence has undesired, negative consequences such as making agents less useful, framing the agent’s situation as adversarial and causing it to attempt to circumvent oversight measures, or causing agents to suppress most attacks while still attempting severe, hard-to-catch attacks. After the deterrent effect is established, we would want to understand whether agents are really deterred or simply change their behavior for other reasons. This could be investigated by reading the agent’s chain-of-thought and interpreting its activations, as well as ablating the framing, threatened consequences, or other aspects of the deterrence scheme.

There are also important engineering projects that could drive forward honeypot effectiveness and adoption. Effective tools to automate the ideation, implementation, or operation of honeypots would make it cheaper to scale honeypot deployments and could allow defenses to become better as AI capabilities improve. Such a tool could consist of a specialized agent scaffold that can be pointed at an environment to automatically generate honeypots. This could be supported by frameworks that automatically red team honeypot deployments. Such a framework would test whether the specific agents that will be run in a specific environment can identify and evade honeypots. Furthermore, infrastructure that supports rapid attribution of an attack to a specific agent instance, quick and effective containment and shutdown of misbehaving agents, and tooling for investigating the trajectories flagged by honeypots would improve honeypot effectiveness.

We are currently running experiments on honeypots via the Heron security fellowship , where we place honeypots in control evaluation environments to test:

We’d be especially excited about pilots of honeypots in real production environments in which an organization rolls out honeypots into some of their agent deployments. Compared to evaluation settings, production pilots are not hampered by evaluation awareness and provide more ecologically valid results because they test on a realistic data distribution. Most importantly, they reveal how practically useful the purported benefits of honeypots are and whether there are practical constraints that might block the adoption of honeypot deployments, such as too many alarms, added latency, or bad integration with other security systems. A post-pilot review should assess how many actual incidents were caught or agents were incriminated. It should also interview relevant stakeholders, such as the security team or internal users, on what practical issues arose, which best practices they learned, and what benefits they experienced. Furthermore, it should assess how the costs of implementing and operating honeypots compare to other security mechanisms. Ideally, the lessons from such a pilot should be published such that the broader community benefits, while not giving away implementation details that would make it easier for agents to identify and evade honeypots.

Finally, there is space for an organization to specialize in building honeypots against agents. Such an organization would develop expertise and tooling to ideate, implement, and test honeypots given a new environment. Ideally, much of this work could be automated so engineers don’t need access to the environment where honeypots are implemented. Instead of having every deployer build honeypots themselves, a centralized organization benefits from economies of scale to build more complex honeypots more cheaply, build dedicated expertise in honeypotting agents, and to integrate learning from various models and deployment settings.

Read the full paper here: https://www.iaps.ai/research/honeypots-for-ai-agent-insider-threats

It would be interesting to know whether the ability to evade honeypots is strongly correlated with the ability to evade other oversight mechanisms or whether these are uncorrelated skillsets. An initial investigation of this question could train models to evade some oversight methods and test whether this generalises to make it better at evading other, held-out oversight methods.

For example, a control protocol could use cheap honeypots to determine on which trajectories expensive chain-of-thought monitoring should be invested.

Notably, other monitoring techniques could have also detected the kind of misbehavior described here.