Why Tracebit is written in C#

One of the first decisions I faced when we started building Tracebit was a common one - what tech stack were we going to use? Some of these were quite easy choices (AWS, Postgres). One that was less clear to me at the time was which language to write Tracebit in - and the decision I took here probably wouldn’t be one of your first guesses! It certainly wasn’t what I expected to pick when I first started considering the options.

Tracebit is a B2B SaaS Security Product. If you did a straw poll of Engineers - especially readers of a certain orange website - about the ‘best’ language for such a system, I think the common responses would include Python, TypeScript, Golang, Rust, something JVM-y, maybe Elixir. There’s obviously no ‘right answer’ here, but after a bit of reflection, we took a different path: Tracebit is written in C#.

Why C#?

The advice often given when choosing a tech stack for a startup is to stick to what you know - especially if you know it works well. That certainly rings true, and it led to most decisions being pretty easy to make.

Professionally, I’ve worked in large and critical codebases in Python and TypeScript. I’ve seen the value they can bring (ecosystem, hiring, Engineer ramp times) but I’ve also experienced what felt like real drawbacks (dynamic typing, packaging & dependencies, maintaining large codebases).

So when it came to the language, I wanted to explore options including those which I hadn’t previously used.

Criteria

Productivity

If I had to define the key criterion I was looking for in a programming language, it would be Productivity. In general, I think a stack that lets you feel like you’re spending your time on what matters is the essence of a good developer experience.

There are many facets to productivity, and they’ll be of different relative importance at each stage in a project or company's lifecycle.

In the earliest days of a startup, it’s critical that you can iterate and refine your ideas quickly; because there’s a very good chance that you’re building the wrong thing. That should begin before you start writing code, but when you do, you’ll want to be using a language that allows you to express your ideas and realise a product quickly.

Another aspect of productivity is maintenance. All software requires maintenance - even from the earliest days. You’ll want to refactor effectively and safely as both your understanding of the problem domain and your product’s feature set increase. You also want to minimise the amount of time spent on maintenance tasks which don’t benefit your product or your Engineers.

As your product, code base and Engineering department grow in size, will the language enable you to collectively maintain your productivity and your software itself?

We've found .NET to be an incredibly productive stack so far and are confident that it's a very solid foundation for future growth.

Free & open source

I wanted to be building on a platform that’s open source (and free as in beer). It just makes things more difficult if you can’t see how something’s implemented - and at the earliest stages we didn’t want to be spending money where we didn’t have to.

Pretty much every stack I considered met these requirements^[1], but I don’t think I knew before looking into it that .NET is now free, open source and MIT-licensed. I guess Microsoft realised the real money was in selling CPU time in Azure, and the more developers using .NET the more likely they are to choose to deploy on Azure. Of course, you don’t have to deploy .NET on Azure - and we currently don’t! We pay more to Microsoft for a single Teams licence than we do for our whole stack.

Cross-platform

I haven’t used Windows for over 15 years and I had absolutely no intention of trying to deploy production systems on it! I mention this only because I think I was surprised to learn that C# / .NET is genuinely cross-platform - the very first lines of Tracebit were written on an Arch Linux box with no difficulty at all. We now develop on MacBooks and deploy on Linux running on ARM cores.

Microsoft have even worked with Ubuntu to provide ‘chiseled’ (distroless) base Docker images to minimise attack surface and patch management, while consuming patches directly from a trusted and widely-compatible upstream.

Popularity

A language being popular clearly isn’t necessarily an indicator of quality, but I think it matters a lot. If things go well, you’ll want to quickly accelerate your hiring of Engineers. If there isn’t a big enough pool of Engineers from which to hire, you risk slowing the business’s ability to grow quickly. At times in a startup’s life, scaling the team is arguably the most significant bottleneck Engineering will face. Equally, it’s important that your Engineers will be gaining transferable skills which will serve them well in their future careers!

There are many other reasons why it’s beneficial to be using a popular language:

• availability of SDKs and libraries
• quantity of documentation and examples online - and in LLM training sets!
• well-trodden paths means you’re less likely to be the one who first encounters bugs or limitations of language features and key libraries

All these things help you to be productive as an Engineer.

I really only considered languages that were on the TIOBE Top 20 and the Stack Overflow Developer Survey, where C# is fairly consistently at number 5 (for general purpose languages).

I was somewhat surprised to see it so high, but I had to remind myself that popularity is not the same as being fashionable - C# is anything but fashionable. It doesn’t regularly make the front page of Hacker News (I think it’s probably the least discussed language relative to the user base). At the same time, you don’t often see people extolling the virtues of deploying software on Linux hosts or using relational databases - but that doesn’t mean those aren’t popular choices, it means there are a lot of people just quietly getting on with it.

Memory-safety

Memory-safety was completely non-negotiable - this feels so uncontroversial that I don’t think it’s worth discussing much. Honestly, good luck to any security startup building in C while CISA are advising on “The Urgent Need for Memory Safety in Software Products”.

Garbage collection

Why garbage collection? Or I suppose, “why not Rust”? Honestly, we’re not doing real-time/systems/embedded programming; we can afford a GC pause. I didn’t want to take the risk that by choosing Rust I’d be spending more time trying to wrestle with a borrow checker than iterating on the product.

Statically-typed

It was important to me that the language was statically-typed. Having worked with both statically and dynamically-typed languages before, I believe that any ‘overhead’ that static typing introduces when writing code is repaid many times over. There are some - debatable - arguments that static typing reduces bugs. It can also improve performance.

I think the key benefit for me is what it enables in terms of reading and maintaining code. I find that static types help me understand the intent and implementation of half-remembered or unfamiliar code much more quickly. Not to mention the advantages they provide in an IDE to be able to “Find all usages” / “Jump to definition” more precisely, and refactor code with more confidence.

If type systems fall on a spectrum between ‘true + true == “2”’ and ‘a monoid is a monad in the category of endofunctors’, then C# feels like it falls nicely in the middle. It has useful things I like (generics, record types, runtime support, decent inference). The only thing I really miss is discriminated unions, which are currently only a proposal.

What about Gradual Typing? I’ve built significant production systems using such languages. I found my mindset was "we're incrementally transitioning dynamically typed code over to statically typed” rather than “oh great, let’s not bother with types in this new code we’re writing”. So why not just start from the desired goal? And in an ecosystem where all your colleagues and dependencies are doing the same?^[2]

Stability

Here C# and .NET really shines. A major version is released at the same time every year, alternating between LTS (Long term support) and STS (Short term support) releases. LTS releases get three years of patches; while STS get 18 months. Changes seem to be well-documented and tested and generally backwards compatible. It’s nice that so much of the ecosystem marches to the beat of this release schedule; so you’ll find that many of your core dependencies will actually immediately benefit from new language features, data types and performance improvements, and that they’ll have been tested together.

If you’re coming from a language where access modifiers (public, private, internal etc) aren’t available or widely used, they can feel a bit verbose or restrictive at first. These really pay dividends when upgrading major versions, because you will have stuck to API surfaces that your dependencies intended to make public.

Since starting Tracebit, we’ve done three major version upgrades and each has been very uneventful (taking less than a day each).

Batteries included

The .NET platform contains a huge number of well-supported and high-quality libraries, APIs and frameworks. If there’s something you want to achieve, there’s a good chance you’ll find something to support your use case in a library maintained by the .NET team / Microsoft^[3]. Whether that’s generic collections, date/time functions, streaming deserialization of huge JSON files, cryptographic functions, structured logging, testing frameworks, a world class HTTP server, OpenIdConnect implementations, an ORM, a WASM compiler, OpenAPI spec generation, a distributed actor framework - the list goes on (and on).

I can’t overstate how valuable this has been for us. In some ecosystems, I feel like you can spend an awful lot of time in a sort of analysis paralysis around which of the n dependencies you’re going to take on to solve a problem:

• how is it licensed and will that change?
• who is maintaining this and how trustworthy are they?
• how many transitive dependencies are coming with this?
• is it performant / well tested / well documented?
• how many Dependabot alerts is this going to result in and how soon will patch releases be available?
• should we just write the code ourselves?

Tracebit is a security product with some degree of privileged access to our customers’ environments - we have to have a very high bar here. Having a clear answer to the question “how are we going to do this?” saves a lot of time, both during initial implementation and ongoing maintenance. Some fairly trivial CSS build steps using node have more dependencies than our entire C# product!

Tooling

The variety and quality of tooling available is another benefit to working in such an established ecosystem. We use Rider, which is a brilliant IDE and is free for non-commercial use - I would thoroughly recommend it. It does everything you’d expect (debugging, tests, refactoring etc) and probably a lot more too - e.g. via Dynamic Program Analysis it can automatically detect things like inefficient database access patterns, excessive memory allocation issues, slow HTTP handlers. Interactive debugging is seamless and very powerful. There’s great support for runtime memory profiling, comparing snapshots, inspecting the heap, detecting leaks.

There are a huge number of static analyzers available which serve as great guardrails, often with automatic fixes built-in.

There are tools which can easily be built into docker images to allow for runtime diagnostics, gathering a huge amount of diagnostic data to get to the heart of an issue. For instance, we’ve collected traces and heap dumps from staging environments to quickly address performance issues which didn’t manifest locally; and would otherwise require a very tedious and iterative process of speculatively adding additional logging, spans and metrics to get the granularity of data needed to identify the culprit.

Performance

Performance is rarely the primary reason to choose a language, but it’s certainly a nice bonus. C# has been great from this perspective. Taking a look at what feels to me like the most representative TechEmpower benchmark ‘fortunes’ (using ORMs & a ‘full’ framework), you’ll find the mainstream ‘paved road’ .NET stack at 10th (out of 80). It’s probably fair to say that many of the frameworks which beat it are a little more niche and aren’t the kind of things you’d just happen to find yourself using.

I don’t put too much stock in benchmarks, but it’s clear that performance is a significant priority for the .NET team. Every release, I look forward to the incredibly thorough articles describing the latest performance improvements, along with the noticeable improvements we get just for upgrading. There are some problems that aren’t trivial to scale horizontally, and any time saved in not having to optimise code or troubleshoot performance is time that can be used elsewhere.

‍

Features

There are too many things to expand upon here that I’ve found both technically impressive or incredibly productive about .NET: LINQ, Entity Framework, TPL Dataflow, ASP.NET, F# to name just a few. These really are worth taking a closer look at - I genuinely think some of these are unrivalled in other stacks.

How’s it going?

We had a lot of decisions to make - and quite quickly - when we started Tracebit. This was one among many, and honestly it felt like a bit of a risk - but I’m happy with the decision to use C#.  We've made 1000s of commits and releases across >100k lines of code since then, including onboarding Engineers who have never touched C# before. We've all been able to become productive with the language very quickly (or we'd have ditched it just as quickly!). C# has been a key part in getting us to where we are today, and there have been no nasty surprises. In many areas, it surpassed my expectations.

When it comes to choice of language, there’s no panacea, and ultimately it’s still quite a subjective decision. There’s a good chance we would have been equally happy with several other choices too.

I suppose this article is partially aimed at a past version of myself (who due to some preconceptions wouldn’t have considered C# at all): it’s worth a second look. And to anyone who is seriously considering C#: I hope you enjoy it too!

Footnotes

‍