Declarative specifications are all the rage in infrastructure today, the more declarative the better. In the article linked here, I talk about why this approach falls short. In a nutshell, declarative approaches are really good when we don't care about the details of how something happens, just that it does. For many DevOps operations, this is true. We don't really care about how a container makes it to a particular server and starts accepting traffic. We only care that it does. But in many other cases, we care a great deal about how things happen. Consider some examples:
- Deploy this new code. Check that it’s running successfully first. Then stop running the old code.
- Deploy a single instance of this new code and wait for manual approval to go further (Canary deploy)
- Deploy the new code to some of the instances, and slowly age out old ones. Automatically roll back if problems are detected. (blue/green deploy)
- Bring up a new version of the database, sync the data, then tear down the old version.
- Start the new availability zone. Once all services in the zone are up, then update DNS.
It turns out that the web development community has been dealing with many of the problems that modern declarative infrastructure systems face, and they've been doing it for 25 years. In this article, I want to talk a bit about how web developers have addressed these problems, and how we can apply these lessons to infrastructure. As it turns out, many of the lessons and solutions apply quite directly.