We’re excited to share our latest canary module: Kubernetes - this is now available for all Tracebit customers; with native support for AWS EKS, full support for Azure AKS and GCP GKE coming soon.
Why there’s an urgent need for security canaries in Kubernetes
We talk to a lot of security teams about where they might have detection gaps. What’s so great about building a canary platform like Tracebit is we’re not constrained to a particular surface area - cloud, identity, SaaS, workstations - we really want to know it all when understanding pain points that security teams are facing.
In these fairly wide ranging conversations, Kubernetes is by far one of the biggest areas of concern we hear from security teams. It’s a complex beast that defers a good deal of operational responsibility to engineering teams, it enables a great deal but what’s going on inside the cluster from a security perspective can sometimes be a bit of a black box. This is backed by some real world compromises we dive into below as well as a Red Hat Study report where 45% of respondents had encountered Kubernetes runtime security incidents.
Our approach to security canaries in Kubernetes
As usual our approach is to consider attack paths inside Kubernetes clusters and opportunities for canaries to detect and delay attacks.
The flexibility of Kubernetes, which is a source for concern for security teams, also provides a powerful platform for Tracebit to leverage for the automated deployment of canaries.
Here we won’t delve into the secret sauce that enables our security canaries, but we will reveal the threats we have focused on and the security canary outcomes the platform enables.
Pod Compromise: Canary Credentials in Kubernetes Pods
Container compromise is not unique to Kubernetes, but is naturally front of mind for teams deploying at scale. This could be an exploit in vulnerable code, a supply chain attack or an over-priveleged Engineer’s access being exploited.
It’s important to be able to detect this early stage of an attack. Teams may or may not deploy agents to help with this, but their detections can only go so far.
The Tracebit Kubernetes module can automatically inject canary credentials into Pods. The value in this detection is that the credentials are 1) unique to the Pod 2) refreshed regularly. If these credentials trigger it indicates a threat actor has compromised the pod, taken credentials and used those credentials. A very strong indication, that could also be considered for an automated response.
Cluster Compromise: Canary Resources in Kubernetes Clusters
The Kubernetes control plane is a natural cause for concern and a common attack path. Compromise may have occurred via lateral movement from a privileged Pod or compromise of a privileged Engineer's identity.
In such cases, it’s important to be able to quickly detect which principal is compromised and ideally shut down their access.
The Tracebit Kubernetes module takes the same approach as we take to public cloud environments - we’re able to scan and profile the Kubernetes clusters and deploy Kubernetes native resources into them that should never be interacted with. One example would be Kubernetes Secrets - there purely to deceive and detect accounts.
The value in this approach is its dynamism and simplicity - our canary resources are able to change as dynamically as the other resources in the cluster, and the precision of the detections is clear - a high fidelity detection for an event that should not be occurring.
How does it work?
We’ve been quite thoughtful about ensuring a simple, safe deployment - especially for teams with sensitive production workloads. We’ll keep the exact details out of the public eye for now - but the deployment can be up and running in minutes via Helm chart deployment, with source code available for customers who wish to verify independently the actions that Tracebit can and cannot take in their clusters.
Can you share some real-world examples?
This can be really wide ranging - a phish of an engineer can lead to cluster compromise, a supply chain attack can lead to Pod compromise.
For a recent and quite critical example, we'd point to Wiz’s research on IngressNightmare allowing Cluster Compromise.
This attack enabled both Pod and Cluster compromise - providing two opportunities where Tracebit could detect the attack.
Learn more
If security visibility in Kubernetes is a challenge that resonates with you and you’d like to learn more about what we’re doing, we’d love to talk!
The easiest way to get in touch is to book a demo - we can walk you through what we’re doing and dive into the technical details.
