Cover Photo by Yan from Pexels

Yes - this post started as a tweet. One that went semi-viral. It struck a real, naked, buzzing nerve. A nerve that most of us prefer not to touch.

Some of us pretend it's not a real pain, others are just too busy fixing production issues. Still others - and that would probably be the majority of the industry - are yet to discover how much of a distress it is when they meet with it face to face.

the gap between the elite and everybody else is only growing wider

But the truth is out there. Mere mortals can't have Continuous Delivery and even less so - Continuous Deployment. CD has become the privilege of that group that the folks at DORA quite non-accidentally call "elite performers". And the gap between the elite and everybody else is only growing wider.

Fooling Ourselves

Most organizations we work with say: "of course we have CI/CD pipelines!"
But when one digs deeper - there's usually some CI - and no CD in sight. Or, as @itaysk noted "it's not even CI, but continuous build..."

When asked what stops them from safely and regularly deploying every change into production environments - everybody seems to have their own reasons. Organizational, cultural, historical, technical, contractual.. Some go as far into denial as saying : "Oh, we don't need continuous delivery. In fact most companies out there don't really need it." But the underlying reason is of course the lack of confidence. Nobody wants to be the culprit for a system outage. According to a number of industry surveys the average cost of one hour of downtime is around 75000 USD. There's a lot at stake!
So instead we choose to move slower, to add controlled handoffs and build home-grown guardrails. To hire more Ops engineers and call them SRE to feel more secure. Rarely discussing the price of establishing and maintaining all of these over time.

But why can't we have CD?

Continuous Delivery is a sociotechnical practice. And as many Twitter commenters correctly noted - the barriers on the way to having it are two-fold. As with anything in DevOps it starts with culture and shared understanding that continuously delivering in small increments makes everything better. Engineers who've experienced true CD can't really fathom any other way of delivering software. As @giltayar puts it "CD ... is a total game changer. It changes how you perceive software development and delivering features... I did CD and EVERYTHING about how I developed changed. It was magical."

The Social Dilemma

But we humans are scared of change. The new mode of delivery challenges our perceptions: of ownership, of reliability, of hierarchy. If your SRE team is responsible for production site uptime - then what's their incentive for enabling the constant flow of change that continuously threatens the very thing they are responsible for? If you have folks whose job it is to control what gets released when - what will they do when this control is made obsolete? The existing organizational barriers make the blame game easier - thus providing us with a false sense of confidence. Because the tools we currently have can't promise true confidence - and this bring us to...

The Technical Dilemma

The socio-cultural obstacles are truly the hardest to remove. But as Archimedes used to say: "Give me a lever long enough and a fulcrum on which to place it, and I shall move the world." Technology, while meaningless on its own can become a great enabler for societal innovation.
Trouble is - the tools for continuous delivery/deployment are still lacking. And this is especially true for the new brave cloud/edge-native world we see rapidly unfolding before our eyes.

But Aren't CI Tools Enough?

This is where some readers might say: "Why are you saying there are no tools for CD? We already have Jenkins/CircleCI/Github Actions... Why can't we use those? and then there's Spinnaker, isn't there?"

That, of course, is a grave mistake. Yes - any CI server or even generic workflow automation tool can theoretically orchestrate your deployments - the mechanics of deployment are trivial. But deploying like this is the same thing as the proverbial "throwing changes over the wall" practice that brought on the DevOps revolution.
Because CI tools ignore the semantics of change. The only kind of feedback they provide is deterministic one - verifying a pre-defined functionality under pre-defined conditions. While the production environment has inherent uncertainty leading it to behave in often unpredictable manner. Therefore - in modern complex systems no change is verified until it reaches production. As they say - until the wheels hit the road.

And that is exactly why most orgs out there can't have CD. Because blindly pushing into production is scary, stressful and in the end falls on the shoulders of the undermanned SRE team.

And that is exactly why most orgs out there can't have CD. Because blindly pushing into production is scary, stressful and in the end falls on the shoulders of the undermanned SRE team.

Cloud Native CD is Possible

It's not all bad, of course. Some teams we talk to succeed to establish true cloud native CD by investing multiple man-months in home-grown solutions. This is costly, most orgs can't allow this, but those who do are very proud of their achievement - until the platform changes under their feet and they need to reinvent the home-grown solution.

Some very interesting OSS projects have emerged in the last couple of years in an attempt to tackle the problem. ArgoCD with Argo Rollouts, Flux and Flagger, Shipper and Keptn are all definitely worth looking at.

Still no one comprehensive, reliable, usable platform exists that can help us deploy to production continuously with confidence and without complex unsustainable in-house hackery.

That's why we at Canarian decided to step up to the challenge.

We're building a platform that will allow you to deploy continuously with confidence, full observability and automated recovery.

In the next post I'll describe the feature set that we see as the minimal viable proposition for such a platform and how we're building it.

Sounds interesting? Send us an email, sign up for our beta version on the site or just follow this blog.

We'll keep you continuously updated ;)

Keep delivering!