Why flying blind in early security programs needs to change
Priority 0: Prod is down
Picture the scene: we’re at a 30 person startup and the Head of Customer Success has just run over to the engineering team, sweating:
“You know Major Client? The one we’re this close to closing? They just tried to login to the platform on a call with me and got a 504 Gateway Timeout error! I tried logging in myself right after - same thing, complete timeout!"
Panic mode. The engineering team confirms what everyone fears: prod is down.
They load up dashboards, search through logs, pore over source code history, check recent deployments. Five minutes later, the scramble subsides. A recent change is reverted, and the app comes back online. Embarrassing, sure. But no-one was harmed. Phew.
The next day, a post-incident review takes place. Out of curiosity, the Head of Customer Success checks the writeup.
One finding jumps out:
Coffee is spat. Brows are furrowed. Sorry, what? "Won't Fix"?
They dig through the meeting notes:
Engineer 1: We didn't detect the outage, the Head of Customer Success did. We'd actually been down for 15 minutes before anyone noticed.
Engineer 2: At my previous company, we had detections for this. Pretty simple to set up, honestly. Seems like a valuable action item.
Tech Lead: Detections! I wish! We’d love to have them. But we still have the observability project with the DevOps team planned for Q2. Unfortunately, they’re still spinning up our new improved logging platform, that’s still months out.
Engineer 2: Even without a fancy observability system, my last company had simple checks for critical events - like the entire app going down.
Tech Lead: No, sorry. That's planned for later. We also need to build out playbooks and plans for what will happen when the app does go down, processes, who to notify, etc. etc. That’s more a Q3 thing, let’s revisit when all these pieces are in place.
Sound familiar? It probably doesn’t sound familiar at all, you’re probably even thinking that I made this story up.
What’s so fanciful about this conversation is the idea that an engineering team would consider it acceptable to lack visibility for a significant platform issue that had actually occurred and impacted the business.
No matter if they’re drowning in technical debt, even if they don't have all the shiniest tools or the time to think about their incident response procedures - the idea of not just doing something right now to detect an outage better next time, would be untenable for most engineering teams.
The security equivalent of Priority 0 - prod is down
What’s interesting about this conversation is that whilst in an engineering team this would amount to quite significant negligence, in a security team the story often can look a little different.
We know this because we hear this relatively often when talking about what we do - security canaries and deception.
Security canaries are lightweight, simple detective measures for the “should never happen” events (like the P0, app going down). It could be some credentials buried in a container that act like a tripwire for an attack, or some files in a storage bucket with apparently sensitive data that should never be downloaded.
If those credentials get used or those files get accessed, it's a very strong indicator that something is wrong. Perhaps an attacker has compromised your system and is trying to steal data. Or maybe there's malware on an engineer's laptop attempting lateral movement.
Let’s play out a similar conversation - we’re back at an early stage startup, a sole security engineer owns all of security and is building out their security program.
Security Canary Champion: I don’t know if you saw, but one of our competitors was breached recently through their CI/CD, it wasn’t a great look. I know we’re early in the program but I was wondering, have we considered deploying security canaries into our application builds?
Security Engineer: I wish! No, we’ve just started talking to centralised logging (SIEM) vendors, the next two quarters are going to be spent collecting all of our logs and then some initial tuning - the build system logs won’t make it into there until Q2 the earliest.
Security Canary Champion: But we've seen at least one public example of a company like us getting hit right now. Couldn't we offset this specific risk pretty quickly by deploying canaries into CI/CD today?
Security Engineer: Sounds great, but if those canaries trigger, what would I do? I don’t have the logging in place to investigate properly. I don’t even have any playbooks ready to deal with a breach. We’ve pencilled in Q3 for our breach response and notification plans.
Security Canary Champion: The right canary would probably point pretty clearly to the issue. Sure, the response would need to be figured out on the fly, but wouldn't it be better to know now than to find out in six months when your logging project is complete and the attacker has been in your environment the entire time?
Security Engineer: Unfortunately, we just don't have the capacity. I'm hoping to get another headcount toward the end of the year. I'll need them in place to help handle incidents like that.
Coming at this from an engineering mindset, with my earlier made up story in mind, what I find fascinating is that the Security Engineer’s stance is not by any means a controversial one.
Our view is that in a world of ever-increasing complexity and threat actors that are genuinely getting better and faster due to AI, it’s probably time to change this perspective.
Security Canaries on Day 1
We think this story illustrates an important point: early on in security programs, we’d strongly encourage security teams or individuals to ask themselves - what’s my equivalent of “P0 - prod is down?”.
There will likely be many answers. But what if we also ask “within that set, what are the quick detection wins I can achieve right now”?
The equivalent of the simple, external health check the fictional Tech Lead pushed back on.
Our view is that security canaries can play a key role in that Day 1 detection work. In fact, you could probably turn it around in a day.
They’re quick to setup, low-maintenance alerts that will give you that critical detection exactly when you need it most.
We’re not suggesting that a SIEM or security data lake shouldn’t be high on the list of a security program - but we are challenging you: if that is priority number 1, and the lead time is non-trivial, does that not imply that there is an urgent need to bring some visibility right now?
It won’t be perfect. If a canary triggers, there will be work to do, and that work won’t be as neat as you’d like. But consider the alternative: you spend 6 months implementing a SIEM (or other work) only to discover the P0 breach already happened and could have been detected.
Is that the right outcome for the business? I don’t think so.
And if you implement security canaries and they never trigger during that time - would that have given you some assurances during that time? Every time you read about a new exploit ripping through the internet, a competitor being breached, or an engineer accidentally exposes credentials? It probably would, probably quite significantly.
That's not nothing. That's actually quite valuable.
Our call to action
We truly believe that security canaries are a hugely valuable detective control that should come at the very beginning of a security program. This is why we’re building Tracebit.
Ask yourself - what are my security equivalents of “P0 - prod is down?”
Could they be:
- Someone using AWS credentials on your CFO’s laptop?
- Credentials accessible by a CI/CD build being used to read data?
- An application in production reading files from an S3 bucket it should not?
Is there a simple way you could detect these using canaries?
If the answer is yes - why not just do it, today?
In terms of how we can help - this view of the world is why we launched Tracebit Community Edition, a completely free forever subset of our platform that you could get started with right now to protect your estate. Why not sign up and see which canaries could make sense for your ‘P0’?
