Deception Technology: Building Your Own Canary

We sell deception technology, and we think you should understand how it works. One of the best ways to do that is to build a canary yourself and watch it fire.

That is the purpose of this chapter. You will create one fake secret in AWS, retrieve its value, and receive an email telling you that the canary has been tripped. By the end, you will have followed the whole chain from an interaction with a resource to a notification in your inbox.

The exercise has a deliberately small finish line. Once that email arrives and you have checked that it matches your test, the build is complete. The later Levels explain what it would take to make this useful across a real environment. They are technical considerations to understand, rather than more components to install.

Building the first canary should feel achievable. The question that follows is what it takes to keep the same promise across all your accounts, as the environment around it changes.

Level 0 — The mental model

Our canary will be an entry in AWS Secrets Manager, the service used to store things such as application passwords and database credentials. It will contain fictional credentials that cannot connect to a real database.

For this exercise, “tripped” means that someone successfully retrieves this secret’s value using GetSecretValue. Seeing its name in a list does not trip it. Opening a page that only shows its metadata does not trip it. Reading its stored value does.

Here is the path we will build:

Read the fake secret → CloudTrail records the call → EventBridge matches it → SNS sends an email.

CloudTrail supplies the record of the interaction. EventBridge supplies the rule that recognises our particular secret being read. Amazon Simple Notification Service, or SNS, delivers the notification. Secrets Manager records GetSecretValue calls through CloudTrail. AWS: Secrets Manager logging

The secret itself contains no special code. Its contents do not phone home. The detection comes from watching access to the AWS resource that holds it. If someone copies the returned text and reads that copy later, this canary will not generate another event.

Level 1 — Build one working canary

Before you start

Use an AWS sandbox account where you are allowed to create and remove exercise resources. You will need a role with permission to use CloudShell and manage CloudFormation stacks, Secrets Manager secrets, CloudTrail trails, S3 buckets and bucket policies, SNS topics and subscriptions, and EventBridge rules and targets. Your role also needs permission to retrieve the secret. If your organization manages these permissions, ask its AWS administrator for a suitable sandbox role.

You also need an email inbox you can access. Use the US East (N. Virginia), us-east-1, Region throughout this exercise.

The setup creates a dedicated lab trail and an S3 bucket for its audit logs. That bucket supports the detector; the secret is the single canary. AWS charges can apply for the secret, logging, storage, and notifications. An additional trail can incur charges for another copy of management events already being recorded. Use a quiet sandbox and remove the exercise resources when finished. AWS: CloudTrail pricing

Step 1 — Open the terminal in AWS

Sign in to the AWS console, select US East (N. Virginia), then search for and open CloudShell. Use its Bash shell. CloudShell includes the AWS command-line tool and uses your signed-in AWS identity, so you do not need to install software or create an access key on your computer. AWS: CloudShell

Paste this command and press Enter:

aws sts get-caller-identity --region us-east-1 --output table

The output shows the account and identity you are using. Check that this is your intended sandbox before continuing. Keep this CloudShell tab open and run the following blocks in order. If a command reports an error, resolve it before moving to the next step.

This establishes who will create the exercise and later read the canary. For production, the deployment identity and the identities able to discover or read decoys would need deliberate permission design. The lab uses your existing sandbox role to make the first test straightforward.

Step 2 — Save the canary definition

We will use CloudFormation, AWS’s service for creating a group of resources from a text file. AWS calls that group a stack. It lets us create the secret and its detection path together, without manually copying resource identifiers between services.

Copy this entire block into CloudShell, including the first line and the final YAML line, then press Enter. It saves a file called canary-lab.yaml; it does not deploy anything yet.

cat > canary-lab.yaml <<'YAML'
AWSTemplateFormatVersion: '2010-09-09'
Description: One inert secret, a read detector, and an email notification.

Parameters:
  NotificationEmail:
    Type: String
    Description: Email address that will confirm the SNS subscription.

Resources:
  CanarySecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub '${AWS::StackName}/orders/db'
      Description: Learning exercise containing only fictional credentials.
      SecretString: '{"engine":"postgres","host":"orders-db.example.invalid","username":"orders_app","password":"fictional-password-for-this-exercise"}'

  AlertTopic:
    Type: AWS::SNS::Topic
    Properties:
      Subscription:
        - Protocol: email
          Endpoint: !Ref NotificationEmail

  AlertTopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties:
      Topics: [!Ref AlertTopic]
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: sns:Publish
            Resource: !Ref AlertTopic

  TrailLogs:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain

  TrailLogsPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref TrailLogs
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: s3:GetBucketAcl
            Resource: !GetAtt TrailLogs.Arn
            Condition:
              StringEquals:
                aws:SourceArn: !Sub 'arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/${AWS::StackName}-trail'
          - Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: s3:PutObject
            Resource: !Sub '${TrailLogs.Arn}/AWSLogs/${AWS::AccountId}/*'
            Condition:
              StringEquals:
                s3:x-amz-acl: bucket-owner-full-control
                aws:SourceArn: !Sub 'arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/${AWS::StackName}-trail'

  LabTrail:
    Type: AWS::CloudTrail::Trail
    DependsOn: TrailLogsPolicy
    Properties:
      TrailName: !Sub '${AWS::StackName}-trail'
      S3BucketName: !Ref TrailLogs
      IsLogging: true
      IsMultiRegionTrail: false
      IncludeGlobalServiceEvents: false
      EventSelectors:
        - IncludeManagementEvents: true
          ReadWriteType: ReadOnly

  CanaryReadRule:
    Type: AWS::Events::Rule
    DependsOn: [LabTrail, AlertTopicPolicy]
    Properties:
      Description: Notify when this lab secret is successfully retrieved.
      EventBusName: default
      State: ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS
      EventPattern:
        source: [aws.secretsmanager]
        detail-type: [AWS API Call via CloudTrail]
        detail:
          eventSource: [secretsmanager.amazonaws.com]
          eventName: [GetSecretValue]
          errorCode:
            - exists: false
          requestParameters:
            secretId:
              - !Ref CanarySecret
              - !Sub '${AWS::StackName}/orders/db'
      Targets:
        - Id: EmailAlert
          Arn: !Ref AlertTopic
          InputTransformer:
            InputTemplate: |
              {"message":"CANARY TRIPPED: the lab secret was read.","event":<aws.events.event.json>}

Outputs:
  SecretName:
    Value: !Sub '${AWS::StackName}/orders/db'
  SecretArn:
    Value: !Ref CanarySecret
  LogBucketName:
    Value: !Ref TrailLogs
  RuleName:
    Value: !Ref CanaryReadRule
YAML

The file describes four parts of the exercise:

Two details make the read detector work. A logging trail must exist for these CloudTrail events to reach EventBridge; seeing events in CloudTrail’s default Event history is not sufficient. The rule also uses ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS so that it can match read-only management events. An ordinary enabled rule would miss this read. AWS: CloudTrail events in EventBridge

The errorCode condition limits this exercise to successful retrievals. The notification adds a clear message and includes the original event, which you can use to check who made the call and when. The full event is inserted as a JSON object by EventBridge’s input transformer. AWS: Input transformation

For production, you would fit the detector into your established logging, access controls, retention rules, and deployment process. This lab creates its own trail to make the dependency explicit and the exercise self-contained. Its trail records read management events across this Region, while the alert rule selects only the canary read. Copying the lab trail into every deployment without considering existing logging would create unnecessary duplication.

Saving the file prints nothing, so confirm it arrived:

ls -lh canary-lab.yaml

head -n 5 canary-lab.yaml

You should see a file with a non-zero size, and a first line reading AWSTemplateFormatVersion: '2010-09-09'. If the file is missing or empty, paste the block above again before continuing.

Nothing has been created in AWS yet. This is a text file sitting in your CloudShell home directory, and AWS has not read it. The next step is what hands it to CloudFormation.

Step 3 — Create the resources

Replace <INSERT_EMAIL_HERE> with the address that should receive the notification, then run:

aws cloudformation create-stack --region us-east-1 --stack-name canary-lab --template-body file://canary-lab.yaml --parameters ParameterKey=NotificationEmail,ParameterValue=<INSERT_EMAIL_HERE> --query StackId --output text

A successful response contains the stack’s identifier. If AWS says canary-lab already exists, inspect that stack before continuing; this tutorial assumes a fresh lab stack.

After creation has been accepted, run:

aws cloudformation wait stack-create-complete \
  --region us-east-1 \
  --stack-name canary-lab

The command waits while AWS creates the resources, then returns to the prompt without output on success. If it reports a failure, open CloudFormation → canary-lab → Events and read the failed resource’s reason before proceeding.

You now have one secret and its detection path. CloudFormation has connected the resource identifiers and permissions described in the file. The example has not created a database or issued a usable database credential.

It is worth going to look at what was just built rather than taking the command's word for it. Everything is in us-east-1, and the CloudFormation stack's Resources tab links to each piece directly:

• Secrets Manager → Secrets holds canary-lab/orders/db. Open it and choose Retrieve secret value to see the fictional credentials. Be aware that doing so is itself a successful read, so it will trip the canary once the subscription is confirmed.
• CloudTrail → Trails lists canary-lab-trail, the record that makes the read visible.
• EventBridge → Rules, on the default bus, holds the rule that recognises this particular secret being read. Its Event pattern is the matching logic in full.
• SNS → Topics holds the topic and its email subscription, which will read Pending confirmation until the next step.
• S3 holds the bucket receiving the trail's log files. It will be empty for the first few minutes.

Clicking through these is the fastest way to see that the diagram at Level 0 is made of ordinary AWS resources, each of which you could have created by hand.

Keeping a definition in code is also the starting point for repeatable deployment. Across a real environment, you would need to decide which accounts and Regions receive it, which settings vary, who can change it, and how a failed deployment is detected. Creating this one stack establishes the mechanism; it does not make those decisions for you.

Step 4 — Confirm your email subscription

Open the email from AWS Notifications with a subject containing Subscription Confirmation, and select Confirm subscription. If it is not in your inbox within a minute or two, check your spam or junk folder — this one is filtered often enough that it is worth looking there before assuming something has gone wrong. SNS will not send canary alerts to the address until the subscription is confirmed. AWS: SNS email subscriptions

That confirmation email is part of setup. It is not evidence that the canary works yet. The next step tests the complete path.

An inbox is a convenient destination for this exercise. In production, the destination needs an owner, an expected response time, and a way to handle delivery failures. We will return to those requirements at Level 3.

Step 5 — Read the secret and watch it fire

After the stack is complete and the subscription is confirmed, give the new rule a minute to take effect. Then paste:

aws secretsmanager get-secret-value \
  --region us-east-1 \
  --secret-id canary-lab/orders/db \
  --query SecretString --output text

CloudShell should print the fictional database credentials. That successful retrieval is the interaction our rule watches. The API can identify the secret by its name or full ARN, both of which are included in our rule. AWS: GetSecretValue

Next, look for a separate notification whose body includes:

CANARY TRIPPED: the lab secret was read.

The body also contains the event. Check that eventName is GetSecretValue, requestParameters.secretId identifies your lab secret, and eventTime corresponds to the read you just performed. The userIdentity information should correspond to your test identity.

Delivery is asynchronous, so the email arrives separately from the command’s response rather than with it. In a quiet sandbox it is often quick: our own run produced the notification within seconds of the read. Do not read that as a service level. AWS describes this event delivery as best effort, and a busier or more complex environment can be slower. Record how long it actually takes in your own account, because that measurement is worth more to you than any published figure. AWS: Secrets Manager events

Once the matching email arrives, you have built and tripped your first canary. You read a resource, AWS recorded the interaction, your rule recognized it, and a notification reached you. That is the complete hands-on objective.

The same rule can match another identity’s successful read, provided that identity already has the necessary access. The template does not make the secret public or grant other identities permission to retrieve it. It also does not treat a denied request, a metadata lookup, or every possible retrieval method as the same event. A production detector needs its intended access paths and failure cases defined and tested explicitly.

If the email does not arrive

If there is no notification after about ten minutes, start with the checks below. Ten minutes is a troubleshooting checkpoint, not an AWS delivery guarantee. If you test again, note the new time so you can distinguish a delayed earlier alert from the latest attempt.

This follows the same path as the detector: resource interaction, event record, matching rule, notification delivery. It is also how you would localise a failure in a larger deployment.

Step 6 — Remove the exercise resources

When you have finished testing, first display the name of the bucket holding this exercise’s logs:

aws cloudformation describe-stacks \
  --region us-east-1 \
  --stack-name canary-lab \
  --query "Stacks[0].Outputs[?OutputKey=='LogBucketName'].OutputValue" \
  --output text

Save that bucket name. Then delete the lab stack:

aws cloudformation delete-stack \
  --region us-east-1 \
  --stack-name canary-lab

aws cloudformation wait stack-delete-complete \
  --region us-east-1 \
  --stack-name canary-lab

This removes the fictional secret, trail, alert rule, and notification resources. The template retains the log bucket so that stack deletion does not fail because it contains log files. After stack deletion completes, open S3, select the exact bucket name you saved, and empty and delete that exercise bucket if you no longer need its logs. The template file in CloudShell is just a local file and does not run anything by itself.

Two things outlive the stack, and both matter if you intend to run the exercise more than once. Secrets Manager does not delete a secret on request; it schedules the deletion and holds the name in reserve through a recovery window that is measured in days. A second stack of the same name will therefore fail to create, reporting that a secret with that name is already scheduled for deletion. The log bucket is the other, retained deliberately for the reason above. The least troublesome way to run the exercise again is to give the new stack a different name, since the secret's name is derived from the stack's, and to clean up each run's bucket as you go.

That is a fair description of retirement in general. Removing a detector is rarely a single command, and the parts that linger are the ones nobody wrote down.

You have now completed the exercise and its cleanup. In production, retirement needs the same care as deployment: stop the detector intentionally, account for any references to the canary, retain evidence according to policy, and update the coverage inventory so nobody still believes that resource is being watched.

Level 2 — Coverage: put the canary on a path that matters

The lab proved that a read could reach your inbox. It did not prove that an intruder would encounter this particular secret.

Imagine a compromised application role that can read one specific production secret. If that role cannot even discover your decoy, placing the decoy elsewhere in the account adds little coverage for that intrusion path. Placement and permissions have to make sense together. The canary needs to be discoverable through a relevant path while remaining unused by real applications.

This is where the coverage map from the strategy chapters becomes useful. For each important path, identify what an attacker would be able to enumerate or retrieve, where a decoy would fit, and which interaction would produce a useful signal. A decoy should never need genuine production credentials or access to a real database to be convincing.

Across accounts and Regions, that also becomes a deployment problem. New accounts need appropriate coverage, retired accounts need cleanup, and an inventory needs to distinguish resources that merely exist from detectors that have passed a test. Different resource types can require different logging: for example, reading an S3 object requires CloudTrail data-event configuration, unlike the secret read used here. AWS: CloudTrail event categories

The next task is to make coverage follow the environment. Repeating the same secret everywhere would increase the count without necessarily covering the paths that matter.

Level 3 — Forensic context and real routing

Our notification includes the raw event. A responder could inspect it to find the caller, source address, API action, account, Region, and time. That is useful evidence, but a raw JSON email still leaves work for the person receiving it.

A production alert should identify the canary clearly, explain what the observed action means, and present the relevant event fields consistently. For an assumed role, the session and role information can help connect the read to surrounding activity. Source IP and user agent provide additional clues, but they are not proof of who was behind the action; user-agent strings in particular can be chosen by the caller.

Routing needs to fit your existing response process. The event may belong in your SIEM for investigation and retention, with a notification sent to the team expected to act. Repeated reads may need grouping so that one probing session does not produce dozens of separate incidents. Delivery failures need to be visible, and the alert should link to a runbook that starts with verifying the caller and checking its other activity.

The email proved that a message could arrive. Operational readiness means the right person receives enough information to investigate and has an agreed next action. A canary hit deserves attention; it still needs interpretation.

Level 4 — Believability and freshness

Our secret is easy to recognize as an exercise. Its name starts with canary-lab, its description says what it is, and its value contains an obviously fictional password. Those choices make the lab easy to understand and clean up.

A deployed decoy would need to fit its surroundings. Its name, description, tags, value structure, and placement should make sense beside resources used by the team it appears to belong to. If nearby database secrets contain a port, database name, and application-specific fields, a canary containing only a password may stand out. Visible deployment metadata can also reveal that otherwise unrelated decoys came from the same template.

Synthetic data can help with the content. An LLM could generate a plausible credential structure from a schema you provide, but the result still needs validation: correct fields, consistent values, and no real credentials or unintended live destinations. Convincing text alone does not make the resource convincing.

Then the environment changes. Applications adopt a different schema, teams rename services, and genuine secrets follow a rotation process. The canary needs to remain consistent with those changes. Updating its value may be only a small part of that work; its metadata, placement, and detection rule may also need attention.

At one resource, someone can review those details by hand. Across hundreds of accounts, believability becomes a continuing process of observing changes, generating appropriate variations, and checking that updates have not broken detection.

Level 5 — Active monitoring

A quiet canary might be healthy and untouched. It might also have disappeared, or be sitting behind a disabled rule. Your inbox cannot tell you which.

Monitoring needs to check both the resources and the path between them. The secret can exist while the trail has stopped logging. The trail can work while a publish permission prevents EventBridge from reaching SNS. SNS can accept the message while the intended recipient never receives it.

Checking that a resource exists is useful, but it does not prove that its detector works. Stronger monitoring performs a controlled read and confirms that the corresponding notification reaches the expected destination within an agreed window. It also notices when that scheduled check fails to run at all.

Health failures should be distinguishable from possible intrusion alerts. They also need a notification path that can still reach an operator when the canary’s ordinary alert path is broken. Otherwise, the system can fail in exactly the way it is supposed to report and remain silent.

That is the requirement to carry forward: know whether you have working detection, rather than inferring it from an absence of alerts.

Level 6 — Tuning without creating blind spots

You just caused a canary alert yourself. That illustrates a useful distinction: the detector reports an interaction, while the surrounding context tells you why it happened.

In a real account, inventory tools, security scanners, investigations, or an engineer exploring an unfamiliar environment may interact with decoys. The first response should be to understand the behavior and whether a change in placement would avoid unnecessary touches. Every new hit is evidence to examine, not automatically an actor to ignore.

Where suppression is appropriate, make it specific and reviewable. A known identity performing a known action from an expected context is more informative than a blanket exclusion for an entire role. Even that narrower combination is not proof of safety: an attacker may compromise the same automation or use its normal access path.

You can retain the underlying event while reducing its notification priority, so later investigations still have the evidence. Suppression decisions need an owner and a review point as tools and access patterns change.

The goal is to protect trust in the signal while retaining visibility into the activity that produced it. A permanently muted scanner role could otherwise become the route an intruder uses to avoid your canaries.

Level 7 — Testing the detection you intend to rely on

The manual read in this exercise answered a precise question: could this identity retrieve this secret and receive the corresponding notification through this configuration?

Production testing needs to preserve that precision. Define the interaction being tested, the identity performing it, the expected event and destination, and the maximum acceptable delay. A test passes when the matching notification arrives, not merely when the read command returns successfully.

Test the access paths you actually intend to cover. Retrieving a secret by name, using its full ARN, attempting a denied read, and using a batch retrieval API are separate cases to verify against the detector’s stated scope. The lab handles successful GetSecretValue calls using the name or full ARN; extending coverage means checking the resulting events and adapting the detection deliberately.

Routine monitoring asks whether the known path still works. Broader testing asks whether the design catches the activity you care about. A controlled red-team exercise can examine whether an intruder would discover the decoy, recognize it as artificial, or take a different route to the real asset. Measure where the canary fires relative to the actions you wanted to detect early.

Repeat those checks after relevant changes to permissions, logging, deployment, and routing. A successful test is evidence about a configuration at a point in time. Keeping that evidence current is part of operating the control.

From one canary to a program

The practical exercise ended at Level 1. Everything after it describes work you would need to consider before relying on canaries across a changing cloud environment.

You have now seen why the first one is achievable. A fake secret, a recorded read, a matching rule, and an email are enough to demonstrate the mechanism. You can point to each part and explain what it does.

Now apply that understanding to your own coverage map. Which accounts and access paths need a decoy? What would make each one believable there? How will it stay current? Who will know when its detection stops working, and who will respond when it fires?

None of those questions makes the first canary less useful as a learning exercise. They explain the distance between building a tripwire once and operating dependable coverage over time.

That is the useful basis for a build-versus-buy decision: you understand the mechanism, you can demonstrate it yourself, and you can assess the ongoing work against the team and priorities you actually have.