This blog post is excerpted from the white paper Terraform v0.12.x versus Pulumi.
Over the last eight years in DevOps I have used everything from homegrown scripts, to CloudFormation on OpenStack using Heat templates, to multi-cloud deployments using Terraform.
With Terraform we have deployed over 2 million Docker containers in the last 12 months, tens of thousands of EC2 instances, and have over 2 billion Lambda invocations. It goes without saying that Terraform has played a significant part in the success of many of our projects.
But as both DevOps and the tools that support it evolve, as a leader my job is to look at what’s coming down the pipeline and what tools might be available to help our clients be even more successful.
Terraform has been a key tool in our arsenal as we leveraged the cloud. But v0.12.x brought about some changes that have caused me to question whether Terraform continues to be the best tool.
Originally Terraform was simple and clean, with few primitives. Programmatic control was simply enforced through the relationship of the resources being deployed. Creativity often occurred outside of Terraform, with wrappers, scripts, and programmatic control coming through other mechanisms. For example, if I wanted to choose a CIDR for a new VPC that didn’t overlap with other VPCs I could query my IPDB or list all the VPCs across my AWS accounts and look for subnets that weren’t being used and feed the result into Terraform as a variable for the VPC CIDR. This ensured that I could always establish peering relationships without having to tear down and rebuild.
The simplicity with earlier versions of Terraform was that I could handle programmatic control in a language that was suited for it – typically Python – and leave the resource dependency management to Terraform.
At HATech, we created what we termed ‘dynamic Terraform,’ which involved looking at what a developer wanted to release, identifying the pattern that was involved, and then using a Terraform module written in a templating language like Jinja to create a new instance of that from a templated Terraform module. If the developer no longer wanted to deploy that resource, he simply removed the package that he no longer wanted to deploy, and we would then remove the instance of the Terraform module that we created.
‘Dynamic Terraform’ enabled us to serve hundreds of developers through a simple model-driven DevOps strategy.
But with Terraform v0.12.x things are changing. Originally there was a pledge from Hashicorp to never add control structures such as ‘while’, ‘for’, and ‘if’ to the Terraform and HCL language – as such structures muddy the waters between resource management and programmatic control.
However, with v0.12.x we are starting to see HCL and especially Terraform becoming more of a programming language in its own right. The problem here is we already have lots of programming languages, and they all have their place. Some are better than others, but most programming languages can be coerced into doing much the same thing as each other, albeit in strange and often more convoluted ways.
This is where I fear Terraform v0.12.x has digressed into its own little world. With more and more primitives and business logic being crammed into Terraform, and the lack of an SDK to incorporate that capability into business-driven applications, there is a widening gap between developers and infrastructure automation teams. That’s a regression back toward siloed development – the opposite direction than we should be moving.
The underpinning of DevOps is that it creates awareness between developers and operators. Developers create new things in new ways to meet the product teams’ requirements. Operators keep things stable, consistent, and predictable. Without DevOps, when developers and operators are incentivized and measured in diametrically opposed ways – even though they’re both trying to do their best for the organization – the outcome is chaos.
With DevOps, success is measured simply as “the lack of inconvenience experienced by your customers while innovating your product.” Seems like a goal everyone in the organization can get behind. And yet, tools keep popping up that push development and operations teams apart again. That is where I fear Terraform v0.12.x will take us.
Take for example the teams that are trying to collaborate and drive reliably deployed and maintained product innovations. Is it better to have languages that support each individual team’s needs, or is it better to have one language to support the entire workflow/ pipeline?
How about this thought experiment: Imagine your developers only spoke English, your QA team only spoke French, and the operations team only spoke Spanish? It would be carnage!
You’d never hire that way. So why, when choosing DevOps tools, do people find it acceptable to have different languages, DSLs that only key individuals understand?
That was the question that drove me to seek out an alternative to Terraform, given the anti- DevOps features of v0.12.x. Now, let me be clear: I believe Hashicorp has done more for the cloud and operations automation space than almost all other vendors added together. But my desire is a common language and framework that provides consistency and also reduces the risk footprint associated with introducing large complicated toolchains.
With large scale, innovative DevOps as my driver, I spent a week exploring Pulumi, and documented the experience. To see what I learned, and how it has informed my conclusion about Pulumi as the best tool for DevOps, check out the white paper Terraform v0.12.x versus Pulumi. It’s a quick overview of my experience exploring Pulumi as an alternative to Terraform, and why I think Pulumi provides a great tool for a DevOps team that wants to keep its runtimes consistent.