Keeping canaries fresh
A canary is only believable relative to the environment around it, and that environment never stops moving. Your teams might decide on a new naming convention, a new standard for tagging assets, changes their security posture of credential expiration time, adds a new regions, deprecates a service, or changes the scheme of production tables. Every one of those changes can turn a once-convincing decoy into an obvious red flag. A canary that was perfect eighteen months ago and has not been touched since is now the one resource in the account still following last year's convention, and that inconsistency is exactly the tell an attentive attacker looks for. And with the advancement of AI-assisted attacks, an LLM can easily identify assets that seem “out of place” and side step your tripwires completely.
Freshness is continuous work. Your decoys have to evolve in step with your real environment, which means someone or something has to notice when the environment has moved and update the canaries to match. This includes the boring mechanics like rotating canary credentials on a cadence that matches how real credentials rotate, so that a key which has never once been rotated does not stand out in a fleet where everything else rotates every ninety days. It includes refreshing seeded contents, updating tags to match new standards, retiring canaries for deprecated services, and adding canaries for new asset types as the estate grows.
A canary that has never been touched can itself be a tell, because in a live environment almost everything gets read, listed, or described by some legitimate automation eventually. A resource with a completely blank access history can look more like bait than a resource with a plausible, quiet background of legitimate-looking access. Freshness is partly about keeping the metadata alive as well as current.
Set a cadence for each maintenance activity and assign an owner. The point of writing it down is that freshness fails silently: nothing breaks when a canary goes stale, it just quietly stops working, so it needs a schedule with a name against it.
Exercise: Age the fleet
Pull the creation date and last-modified date of every canary you have deployed, and the same for a sample of your real resources of the same types. Plot them side by side. If your canaries cluster at one date and your real resources spread across a range, your decoys are visibly a cohort, and a cohort is a fingerprint. This exercise tells you how much drift has already accumulated and whether your freshness cadence is real or aspirational.
Generating the signal
With a stronger baseline of canary design for your strategy, now it’s time for the critical part 2 of generating the signal when a canary is touched. Afterall, a canary that doesn’t tell you when it was accessed is nothing more than a fun decoration. The value of the whole system is the signal, and the signal has to be both high-fidelity and rich in forensic context, because the entire premise of deception is that a touch is meaningful. Nobody has a legitimate reason to read a decoy secret or list a decoy bucket, so when it happens you want to know quickly, and you want to know everything the logs can tell you about who did it. Quickly has a number attached to it, and you should know that number before you design the rest of the pipeline. AWS documents that CloudTrail "typically delivers logs within an average of about 5 minutes of an API call", and that "this time is not guaranteed". Our own measurements against Tracebit’s organization trail put the average lower, at around two and a half minutes, with the 99th percentile just over five minutes and rare outliers running to hours. Treat delivery as the floor rather than the answer. Your detection rule, your enrichment step, your routing hop and your paging tool all add latency on top of it, and it is that total you should be recording, measured in your own account and in every region you deploy into, rather than the figure AWS publishes.
Two things follow from delivery being best effort rather than guaranteed. Keep CloudTrail itself as the durable record and treat the event-driven path on top of it as the fast lane rather than the source of truth, so that if a rule misfires the evidence is still sitting in the trail. And monitor the pipeline, not only the canaries. An EventBridge rule that fails to invoke its target, a dead-letter queue quietly filling up, or a notification channel that has stopped delivering all produce exactly the same experience as an environment with no intruders in it. Put an alarm on failed invocations and on anything arriving in a dead-letter queue, because a detection pipeline that has stopped working looks identical to one that has nothing to report. Knowing a canary was tripped is a great step, but if you don’t know who tripped it, or where they currently are in your system, shutting off access will be incredibly difficult. Additionally, many attackers leave backdoors into your environment, so getting a trail of their activity is critical to re-securing your environment.
In AWS the primary mechanism is CloudTrail. A GetObject on a canary bucket, a GetSecretValue on a canary secret, an AssumeRole on a canary role, a use of a canary access key all of land in CloudTrail and record the forensic context you need: which principal made the call, from which source IP, with which user agent, at what time, from which account, and against which resource.
The engineering work is wiring the detection so that a canary touch reliably produces that enriched signal and nothing else does. That means identifying the exact API calls that constitute a touch for each canary type, filtering CloudTrail (or EventBridge patterns over CloudTrail, or a metric filter, or a query in your log platform) to those calls against those specific resources, and enriching the resulting event with the context an analyst needs. An important thing to note is not every interaction with every resource type generates an equally prompt or equally detailed log record, and knowing where those gaps are will be important to how you design the signaling. Some data-plane events need to be explicitly enabled: S3 GetObject and DynamoDB reads are data events, which no trail logs by default and which are billed per event.
There is a second gap in the same place, and it is the one most likely to leave you with a canary that never fires. GetSecretValue and AssumeRole are recorded by CloudTrail with readOnly set to true, and an EventBridge rule in the default ENABLED state does not match read-only management events at all. To receive them you have to do two things. Set the rule state to ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS, which AWS supports only through the CLI and CloudFormation, not the console. And make sure a trail exists in that account whose event selector logs management events with a ReadWriteType of ReadOnly or All, so the events reach EventBridge in the first place. AWS only added this support in November 2023, so any event pattern you copy from an older post will be silently wrong.
Put the two gaps together and the failure mode is worth stating outright. Follow the standard advice with a default trail and a default EventBridge rule, and all three of the canary types this guide leads with produce nothing: the bucket because data events are off, the secret and the role because the rule does not match read-only calls. No error appears anywhere. You simply never get paged, and you find out the first time it mattered. Some cross-account or cross-service interactions log differently than you would expect. The credibility of your signal rests on knowing exactly what fires, how fast, and with what detail, for each canary type you run.
The fidelity of cloud canary signals is unusually good compared with most detection sources, and that is much of their appeal. A touch on a well-designed decoy is close to a true positive by construction, because there is no legitimate reason to touch it.
For each canary type, pin down what a touch is, where it logs, how fast, and what context you can extract. Blanks in this table are blind spots in your detection.
Exercise: Fire every wire
For each canary type in your plan, actually touch a test instance and confirm the alert arrives with full context, and time it. Record the latency and the fields you got. This is the single most valuable thing you can do to trust your deployment. Repeat it on a schedule and continue to test your signals.
Routing and response
The other half of signaling is getting that signal to someone that can do something about it. The signal has to reach a human who is expected to act on it with enough context attached that the first responder can triage without too much hunting.
The design question is when a canary fires who finds out, how do they find out, and what do they do next. For most teams the answer runs the enriched event into the SIEM for correlation and record, and simultaneously into the places people actually look, Slack or Teams for visibility, PagerDuty or Opsgenie for anything that should page, ServiceNow or Jira for anything that needs a tracked ticket. Because canary signals are so high-fidelity, many teams route them at a higher severity than a typical detection, and rightly so: a genuine canary touch is one of the strongest single indicators of intrusion you can get, so it should not sit at the bottom of a triage backlog.
Attach a runbook to the alert. A canary touch should arrive with, or link to, the answers a responder needs in the first minute: what this canary is and where it lives, what a touch means, the forensic context already extracted, and the first three actions (verify the principal, check what else that principal did, contain if confirmed). The response plan is part of the design and should be rehearsed so that the first time a canary fires for real is not the first time anyone has thought about what to do.
Response time here is going to be really important, again especially as attackers have the ability to leverage AI to shorten their attack times. We’re talking about minutes between detection and data exfiltration.
Suppressing self-inflicted noise
The fastest way to kill a deception program is to have it cry wolf. If your own vulnerability scanner lists every bucket in the account nightly, and one of those buckets is a canary, you get a canary alert every night from your own tooling. A week of that and people start ignoring canary alerts, and an ignored alert is worthless. Suppressing self-inflicted noise keeps the signal trustworthy.
The environment is full of legitimate things that will touch your canaries if you let them, from security scanners to backup managers. Each of these can generate a canary touch that means nothing.
The design response is to enumerate your known-benign actors and build precise suppression around them, then keep that suppression current. You want to suppress the specific principal doing the specific benign thing, not blanket-mute a whole category, because an over-broad suppression rule is a hole an attacker can hide in. If you suppress all access from a scanner's role, and an attacker compromises that role, they can now read your canaries invisibly. The better pattern is to suppress the known automation identity from the known source doing the known read pattern, and to treat anything outside that tight envelope as real. This is ongoing work, because your tooling changes: a new scanner, a new backup product, a new CSPM trial, and the suppression list needs updating or the noise creeps back.
Another consideration in your strategy is not to ever fully suppress anything, but instead just lower the level of notification and routing. It may still be worth capturing all of these touches and reviewing them on a weekly basis for anomalies, only again because human error is often the reason environments are compromised.
Exercise: Two weeks of nothing
Before you route canary alerts to anyone who can be paged, run them in monitor-only mode and log every touch.
Monitor-only means "do not page yet". It does not mean the event is benign. Triage each hit as though it were real, because the premise of this entire guide is that you assume you are already breached, and a monitor-only window is precisely when a genuine touch is cheapest to miss, since nobody is being woken up to look at it. A team that spends a fortnight writing every hit off as noise has taught itself to write off the first real one too.
Work each hit back to a named principal, a named tool and a named reason, and downgrade it only when it lines up with a record you did not have to take on trust: a scheduled job that actually ran, a change ticket, a pipeline execution with its own authenticated log. Matching on timing and principal alone is not the same thing, because both are available to anyone holding the credential. If you cannot produce all three, you do not have noise, you have an incident.
When you do accept an exception, write down who owns it, why it exists, exactly what it covers and when it expires. Keep the raw event either way, and restore paging the moment anything about the pattern shifts: a different identity, a different source, a different action, a different resource, a different time of day. An exception with no expiry date turns into a permanent hole that nobody remembers opening.
Two weeks is a minimum, not an answer. It covers your daily and weekly automation and misses everything on a longer cycle: monthly patch windows, quarterly access reviews and DR drills, half-yearly certificate rotation, the annual pentest. Those hits will arrive after you have promoted to paging, which is the worst possible time to meet them. Either run the window for a full quarter, or run a fortnight and then work through your change calendar, your scheduled CI jobs and your compliance calendar to enumerate what has not yet had a chance to run, and add those actors to the register in advance.
Promote to paging when the monitor-only stream is quiet and every hit in it has a name against it. What you must not do is treat "it happened during the window" as a reason on its own to allow-list something. That quiet, earned by naming every hit, is your evidence that a real touch will stand out.
Ownership
Everything above is a continuous effort.
- Coverage has to expand as the estate grows
- freshness and believability has to be maintained as the environment changes
- signals have to be verified as plumbing changes
- suppression has to be updated as tooling comes and goes
- routing and runbooks have to keep pace with the team.
None of that is a project with a finish line never to be touched again. It is an ongoing program, and programs need an owner, or they decay.
The failure mode is often quite predictable. A motivated engineer stands up a strong deception deployment as a project, it works, everyone is pleased, and then that engineer moves teams. Six months later half the canaries reference a naming convention that has been retired, three new account types have no coverage, the suppression list is out of date so alerts are noisy, and nobody is quite sure who owns it. The deception strategy decays over time, which is worse, because the org still believes it has coverage it no longer really has.
So the design has to answer the ownership question explicitly: who runs this next quarter, and the quarter after. Which team holds it, who is accountable when it drifts, who does the routine freshness work, who gets consulted on major environment changes that would affect canary believability, who is informed when a canary fires. Writing that down, and keeping it written down as people move, is as much a part of the strategy as any technical choice.
This is the type of thing a RACI is perfect for. The irony is not lost on us though that the RACI itself, becomes another thing to be properly maintained…
Exercise: The bus test
For each row in your RACI, ask what happens if the named person leaves next month. If the answer for any row is that the work simply stops, you have found where the program will decay first. Fill those gaps before you consider the strategy done, because a strategy that depends on one person staying is not a strategy, it is a liability with a countdown on it.
The build-versus-buy decision
Every single item in this section is doable. None of it is secret knowledge or proprietary technique. A capable cloud security engineer can design believable canaries, map coverage across three rings, deploy through IaC, wire CloudTrail, route to Slack, suppress the scanners, and write a RACI. If you have worked through the exercises here, you have most of a real strategy already, and you should trust that.
The first canary is genuinely easy. It is an afternoon, and it will give you real value. What this section has tried to show is the shape of the thing beyond that first afternoon: coverage that spans every resource type, hundreds of accounts, more than one cloud, from endpoints to SaaS; freshness that has to be maintained continuously against an environment that never holds still; signal, routing, and suppression that all need verifying and re-verifying; and an owner who is still there next quarter to keep all of it true. The hard part is doing the design, and then doing the upkeep of the design, continuously, forever.