The operational playbook growing teams eventually build

We have all seen the internal "playbooks" companies often keep, a play-by-play on how to fix certain problems on the app, or how to apply a patch to a server, or deploy an additional instance. They're more common than we'd like to admit, and most of the time they are out of date, and the last person that maintained them has left. We always aim to keep them updated, the road to efficiency is paved with good intentions. But, let's be honest, we rarely do update them.

Seventy percent infrastructure#

Go through the typical playbook and you will likely find about 70% of the content covers infrastructure, and 30% covers use cases in the application itself. You don't want to delete any of it, just "in case".

You will likely find that the application content goes out of date a lot less than the infrastructure, as the app is designed around business rules, and they change a lot less often than infrastructure requirements and setups.

Every project has its own chapter#

Open the playbook from a team that is running more than one app, and each one will have its own chapter, and no two chapters tend to agree.

The main product likely deploys through a pipeline that someone built several years ago, with a manual step for database migrations. The client app scales when someone resizes the server because the client complained that it feels slow. The internal tool has a queue worker that has to be restarted by hand every few weeks, and only the person who actually wrote it knows why. One app keeps its logs on the server, another one uses a hosted service, and the third's are nowhere anyone can remember.

None of this is actually deliberate. Each project was set up by the person that started it, with whatever they knew at the time, and nobody went back to make them all match. The playbook isn't your problem. It's a record of how many different ways that your team has ended up running the same kinds of applications.

Nobody should need tribal knowledge to ship#

A mature engineering team shouldn't need some sort of tribal knowledge passed down engineer to engineer to deploy, scale or simply just keep an application running. If you have to remember which project has the manual migration step when shipping a change, or who you have to ask about why the queue worker falls over, then you have a gap in how the team operates, whatever the playbook might say.

Deploying or scaling a web app is already a solved problem. Every hour you and your team are spending on project-specific operational know-how is another hour spent re-solving it, maybe slightly differently, for each and every app you run. The knowledge worth keeping in your heads is the domain specifics: how the business actually works, why the code is shaped and architected the way it is. How the app gets from local to production should be the same solution for every single project. It should be intentionally boring.

Who keeps it up to date?#

In general, these playbooks are a team effort to keep up to date. If you have an infrastructure team they will likely manage this themselves, and this article likely isn't for you right now. But if you don't have the infrastructure team, then this is costing you quite a bit of mental overhead to maintain, not to mention the time it takes to keep it up to date.

What you will usually find is that if the person who keeps this up to date leaves, or they're away on annual leave, it doesn't get touched until someone notices nobody is still maintaining it. It's left to chance, or "he left ages ago and we haven't needed to update it".

When someone new joins, they are given this playbook full of previous developers' opinions, developers who have since likely left the company, and they have nobody to turn to.

What's actually in those pages#

If you're using one of the usual hyperscaler platforms like AWS, GCP, Azure, etc, then your playbook will likely contain details about the services that are running, which services are running as daemons, how to patch language versions, how the deployment process works, and more specifics detailing how to maintain and look after your infrastructure. Maybe you have several pages about the CRON tasks you have running and why, if you are lucky!

I've had to go through and build this up from an infrastructure setup with no documentation, using cPanel with just a list of CRON tasks, and having to fill the gaps across several projects from the source code.

Playbooks in certain circumstances are useful, especially when using one of the named services above. If you have an infrastructure team, they will maintain this alongside your infrastructure, so you don't have to think about it.

Six steps into a 24-step process#

When someone follows an outdated playbook, typically they will only notice it's out of date once they've run a few commands and are partway through patching something or running something. This could leave your infrastructure in a broken state, as not all actions can be undone easily.

I remember having a game of tennis between PHP and Node sharing the same dependency. Node wanted to update the dependency, which broke PHP, which meant I had to update PHP, which my app wasn't ready for yet. You end up playing chase your own tail, pulling at a thread that you have to keep pulling at, getting more and more worried as you go.

But who do you turn to when you notice it is out of date? Is the engineer who set it up still in the company? Do they have time to answer you? Are you on a time crunch, you needed it done yesterday, and Google is not helping at all? An outdated playbook can cause devastation without you even realising until you are six steps into a 24-step process.

The playbook on Sevalla#

Let's take a moment to imagine these same projects are running on Sevalla. Take that 70% infrastructure from the playbook, and shorten it to maybe 2-5%.

OK, so it isn't getting rid of everything, and it shouldn't. You still want a bit about how things are set up on Sevalla, and why. But it's less troubleshooting content, and more informative information that will rarely change. You will keep track of instance sizes you found useful for certain types of applications, best database sizes, how many instances of a background process you want for queues and CRON. Nothing to do with "restart X service every third Thursday of a month, unless it's a blood moon and you've made a sacrifice to the Terraform gods".

Your team deploys from Git. Sevalla handles runtime orchestration, networking, scaling, failover, observability, and deployment workflows behind the platform boundary.

What a new hire gets instead#

Imagine for a minute you are a new hire for your company, being given your company playbook for the first time. Imagine how much of an easier onboarding experience it would be if the 70% infrastructure was replaced with maybe 2-5% about how you're utilising Sevalla.

No more trying to figure out the nuance of how a previous engineer set up a deployment process, no more having to reverse engineer decisions that you weren't around for, and have no idea of the context that went into them. Instead you can read the specifics about how you are using Sevalla, skip any deployment processes as it's handled for you, and if you have further questions, simply go to the Sevalla docs and backfill any knowledge about Sevalla you might need to brush up on.

You don't need to pester anyone on the team with infrastructure onboarding, just very specific domain knowledge questions to get working on the code quicker.

"But then we don't really know how our infrastructure works"#

I've heard these comments before. The simple answer is, yes you do. You just don't need to worry about it anymore. It's still running on Kubernetes, it's still nodes and pods.

The key difference is that you don't need to maintain the overwhelming amount of YAML it takes to handle your infrastructure, or the Terraform maintenance. The bit you worry about, the code, lives inside the containers you define.

You aren't losing anything other than stress and worry, and that's replaced by simplicity and an interface you can explore in seconds, over a YAML file that takes several strong cups of coffee to get through. No more misconfiguration nightmares in your Terraform causing your bill to 10x in a month.

Put a real workload on it#

So a QA app won't actually tell you that much. Pick a production application your team actually runs and uses, ideally the one that has the biggest section in your playbook, and move that to Sevalla.

Before you start though, write down absolutely everything it takes to run the app today: the deployment steps, the scripts, the scaling rules, the way you patch, the pages in the playbook, and the things that only one person actually knows. After it has been running on Sevalla for a few weeks or so, go through the same list and cross off all the things that no longer apply.

There are three main things you will want to compare. How much deployment logic is left in the playbook. How many pages in the playbook still need to exist. And how much of that one person's knowledge you can remove and still keep the app running day to day.

Put this list in front of your team and ask them if they want to keep maintaining that chapter for every other app as well.