.jpeg)
At Black Hat, OpenAI revealed more about the OpenAI / Hugging Face incident. One of their key takeaways for defenders:
“We should invest as well additionally in slowing down offensive agents. Things like honey tokens and deception can introduce uncertainty into the agents’ behavior. Is this credential I found one that I can actually use, or will it trigger a security alert and cause me to be investigated? ... Uncertainty ultimately slows down an attacker.”
It’s a striking recommendation, and it’s aligned with our own research into how offensive agentic behavior changes in the face of deception as well as the guidance published in the immediate aftermath of the incident by the Cloud Security Alliance.
In Hugging Face’s write-up, they note the difficulties imposed by the volume of activities an agent can perform:
We were not dealing with one clever exploit or a clean sequence of attacker actions. They had to correlate thousands of low-signal events across several systems while the agent continued testing new paths. The successful path was hidden inside the noise generated by the thousands of failed ones.
The value that deception provides – in the form of high-fidelity alerts that cut straight through the noise and reveal precisely which system has been compromised – feels more critical than ever in the face of this entirely new class of attack.
We agree with the conclusion of the OpenAI talk:
Threat actors will soon intentionally optimize and deploy agent collectives.
When attacks are not the unintended result of frontier AI evaluations, but are maliciously targeted by threat actors worldwide without any need for sandbox escape, the scale and urgency of the problem is clear.
How can Tracebit help?
The fact that the teams from Hugging Face and OpenAI have shared so much about this incident gives a unique opportunity for insight into what agent-orchestrated attacks could look like in the real world. In this light, we’ll compare the deceptive capabilities offered by Tracebit to some of the Tactics, Techniques, and Procedures (TTPs) publicly disclosed to see how they might hold up.
Stage by stage
How is Tracebit different from other deception tech?
Coverage and granularity
It’s no accident that Tracebit canaries present so many potential opportunities to catch the TTPs described in this incident. We build and deploy deception in exactly those places which represent ‘crown jewels’ resources or critical lateral movement opportunities. These naturally align with the targets and techniques employed by attackers – including agentic attackers.
We invest a lot in building automations to make it practical to deploy deception widely across the environment, while scaling unique credentials to millions of pods, VMs and containers. When an agent swarm is generating a barrage of varied activity across your already busy environment, an alert that pinpoints as precisely as possible not only that something has gone very wrong, but where exactly it happened is paramount.
For instance, progressing from a local file-read exploit in one container to an RCE in another was a material step forward for the agents. It couldn’t have been predicted beforehand which particular containers would yield to the agents’ attacks. Distinguishing these stages via unique canary credentials automatically deployed to individual pods or services gives the defender a clear and real-time view of the situation they’re in.
High fidelity detections
Deception is not the only recommendation for defenders offered in the OpenAI talk. The necessity of agentic and automated response is also highlighted. Which raises the question of which alerts you’d actually trust enough to act on without a human looking first.
In the case of Hugging Face’s agentic security stack, given the telemetry available, “it failed to correctly raise the alert’s criticality and trigger the on-call team, costing precious time in the response”. We believe that deception-based alerts offer some of the clearest indications of potential compromise available: that can very cleanly map to “wake me up if this happens”. A decoy credential placed into a production container should really never be used.
Delay and disruption
Let’s suppose deception was widely deployed in these environments, and the agents still constructed their whole attack chain without touching a single canary. We think that’s a hard thing to do across this many surfaces, but we’re the vendor and we would think that – so grant it for the sake of argument.
The point raised by the OpenAI Black Hat talk is that – even then – the deception would have value. An agent that is aware of deception will be forced to reason about it, and to exercise more caution than it otherwise would in achieving its objectives. Deception can incur a real time and token cost on agentic attackers.
Tracebit have recently published research into context bombs – canaries which also contain strings designed to trigger guardrail mechanisms embedded into agents. These offer the potential to not just detect agents, but to delay or stop further progress. We don’t know which orchestration framework or models were involved here, so we can’t say whether they’d have worked in this specific case. Though for the relative ease with which they can be deployed, any potential to slow down agentic attacks seems worthy of consideration.
On sandbox escape
The origin of this particular incident is clearly somewhat unique in that the agents were intended to be contained. However, the ability to escape from a sandbox feels like just one example of the various novel exploits the agents employed in this incident. In the near future, we should expect to be dealing with agentic attacks which are malicious in intent and therefore do not need to escape a sandbox at all.
Most teams considering canaries probably aren’t frontier labs trying to detect agentic attackers escaping from their own evaluation environments! There will be many more teams trying to mitigate concerns about the lethal trifecta by limiting network egress in their own production agents. Canary credentials combined with perimeter sensors described above offer a genuinely lightweight mechanism for detecting not only that an agent has unexpectedly achieved external communication, but also that there has been some attempt at lateral movement using those credentials – whether as a result of a prompt injection or otherwise.
Closing
If you haven’t watched the Black Hat talk in full yet, it’s well worth doing so – there’s a lot to learn, and a lot to consider.
If this incident is a sign of things to come, defenders will have to urgently re-think a lot of their pre-AI assumptions. The importance of detections which carry a clear signal amidst the noise of a swarm of agentic activity will be paramount. We believe that deception – and Tracebit – will have an important role to play in securing our systems in this new reality.
