Dev productivity or standardization?

Do we need to choose between developer productivity and standardization? If we optimize one, do we have to sacrifice the other?

I'm working on a project with a "standard" local setup - common to all developers. It's supposed to be the "perfect setup", easy to install, and great to onboard new developers. But in reality, it's heavy and slooooow.

And what happens when developers are slowed down by their tools? They optimize for their own productivity! This is what I did, and I wanted to share some of the thoughts and decisions I made along the way.

If you're wondering if you should run Kubernetes locally if monoliths are making a comeback to replace microservices, or if you totally forgot what a debugger is, keep reading!

Or if you prefer to listen instead of reading, here's an AI-generated audio podcast - 7 minutes long - generated by NotebookLM.

Context

Let's start with a bit of context. We are a small team of rather experienced developers, working on an internal service to provision and manage Kubernetes clusters - using Talos Linux on both on-prem and public cloud. So we're 100% "cloud native" and buzzword compliant.

The code is written in Go, in a single git repository. And this monorepo is split into different Go modules - aka microservices. Why? Because we have:

• different concerns: cluster management, machine management, etc.
• different kinds of components (technically): API, Temporal workers, CLI, authorization plugin, and a common library.

Oh, and of course everything is deployed on Kubernetes. So it's easy to scale the microservices based on their load - for example adding more "cluster worker" instances when we need to manage more clusters.

Standard dev setup with Devspace

Our initial setup was based on Devspace - defined as an open-source developer tool for Kubernetes that lets you develop and deploy cloud-native software faster. It has a few interesting features:

• only 1 tool to install locally
• same config to deploy to different environments: local or remote (prod)
• hot reloading: it watches for changes in the code and automatically rebuilds and deploys the app

Sounds great, right? We should have both developer productivity and standardization. All the microservices and their dependencies (Temporal, database, ...) can be installed directly in the local Kubernetes cluster. It's easy to onboard new developers, and it's easy to maintain.

The issue with Devspace

Let's be honest, all these nice things came with a cost:

• you have to run a local Kubernetes cluster. But anyway, you're a cloud-native developer, right? And everybody knows that real cloud-native developers run Kubernetes on their laptops ;-)
• when your code is running in a container, inside a Kubernetes cluster which is itself a container - kind - inside a VM because you're on a Mac, do you know how to configure your debugger? Even with great tools such as Telepresence or mirrord, most people will just avoid the complexity and add more logs instead. And redeploy. And test. And add more logs. And redeploy. And... you get the idea.

So it's heavy, it's slow, and often you lose the knowledge of "how it works on my machine", treating it as a black box. At this point, using a remote Docker daemon wouldn't make it worse, right?

If you're following, you might tell me that Devspace is made for that exactly: to make it easy to deploy on a remote cluster, with the same tools and config as locally. But it's not a very standard way to deploy applications on Kubernetes. So even if we started deploying our POC that way, soon we moved away from Devspace to deploy to a remote cluster, because we added more environments, and wanted to align with common industry/enterprise practices - such as producing Helm charts and using GitOps and ArgoCD.

At the same time, our local Kubernetes cluster - still managed with Devspace - grew bigger and bigger. Because it's easy and convenient, we added more and more components:

• OpenTelemetry-collector and Uptrace for observability
• Envoy through Emissay for ingress, using an Envoy's External Authorization filter
• and other small utilities, such as cert-manager.

Sure, it takes a bit of time for the initial bootstrap - more than 10 minutes - but that's still ok as long as it doesn't happen too often...

But the real problem was the hot-reload feature. Each of our 6 microservices would be compiled in isolation in its own container, not sharing the build cache with the others. So the initial compilation is slow, but every time you change something in the common library, it would affect all microservices, and recompile everything in parallel - and in isolation. At this point, it's not slow, it's very slow. Reloading takes several minutes. This is when devs disable the hot-reload feature and start to lose productivity.

Optimizing for developer productivity

Let's see if we can fix this issue, by optimizing for developer productivity. For the exercise, let's forget about the standardization aspect, and focus only on the "productivity" part. By that, I mean that I don't care if it takes 1 day to set up the local environment the first time, as long as I have the shortest feedback loop possible when I'm coding. Think seconds instead of minutes. It's time to transition to 10x engineer mode ;-)

Getting rid of Kubernetes

First, do we really need to run Kubernetes locally? Even if our application creates and manages Kubernetes clusters, we don't need to run a Kubernetes cluster to develop it. So let's get rid of it!

Now we have to find how to run our microservices and their dependencies. Of course, my first thought was for Docker Compose, immediately followed by Dagger. And yes, I felt guilty for not thinking about Dagger first ;-)

But it still requires containers, which means building container images and running Docker. I want something faster - no more containers. No more extra layers of abstraction, let me run my code directly on the host.

Running microservices as a monolith

The easiest way to remove the difficulty of running microservices locally is to NOT run microservices locally. Let's run a monolith instead! Monoliths are great for local development: you can run them in your IDE, with the debugger, and you can easily change the code and see the result.

And in our specific context, it makes sense for a few reasons:

• they are all part of the same project, and share the same codebase - git repository
• they are all written in Go and can be compiled together
• and more importantly: we can't easily use the API without the Temporal worker, or the opposite. So we need to run them together anyway.

So I created a new Go module, which is just a main file importing all the other modules, and starting all the servers. It required a bit of code refactoring, to make sure that all Go modules properly exposed their servers - instead of an ugly main function. And it's a good practice anyway.

The result? Faster compilation time, and very easy to debug! This is an important point because unfortunately, it seems like fewer and fewer developers use a debugger...

IDE first: VS Code integration

So far so good. But we don't only need to run our code, we also need to run the dependencies:

• an SQL database. The good news is that we support SQLite - that's how the project started - so we can avoid running a database server. One less dependency to run!
• Temporal. But it supports a "dev server" mode - which is already what we're deploying in our local Kubernetes cluster. It's just a single binary to run.
• Envoy, for our External Authorization filter - which loads its configuration from a Kubernetes CRD - through the Emissay ingress. No Kubernetes, no CRD, no config. Well, we'll just have to write it manually. It can't be that hard ;-)

It turns out that writing this Envoy configuration manually was indeed not that hard - and that was good news. I was expecting it to be a pain because I've always configured Envoy through an external control plane - such as Emissary. And looking at the 3k lines of (indented) JSON generated by Emissary, I was not looking forward to writing it manually. 100 lines of YAML later, it was done.

And we're left with 2 dependencies: Temporal and Envoy. Both are easy to install - thank you Homebrew - and easy to run. But I'm lazy and I want to avoid running them manually. Maybe I could use a quick target in a Makefile? Or a Taskfile? Or just... a justfile? ;-)

What about neither? I want to optimize for productivity, and as a developer, am I not more productive when I'm using my IDE? So let's use it!

I'm a VS Code user, so let's see what we can do with it.

First, I can configure a Launch Configuration to start my application with the debugger. This is pretty nice because if it's always running in debug mode, it means I can inspect it at any time. No need to stop the process, remember how to start it in debug mode, replay the same scenario, etc.

{
	"version": "0.2.0",
	"configurations": [
		{
			"name": "Polymer all-in-one",
			"type": "go",
			"request": "launch",
			"mode": "auto",
			"cwd": "${workspaceFolder}/services/all-in-one",
			"program": "${workspaceFolder}/services/all-in-one/main.go",
			"envFile": "${workspaceFolder}/.secrets/.env",
			"env": {
				"OTEL_COLLECTOR_OTLP_ENDPOINT": "localhost:4317"
			},
			"args": [...],
			"preLaunchTask": "Create .secrets dir"
		}
	]
}

Then, I can configure Tasks to run the dependencies. I can easily configure them to use different Terminal views in my IDE, making it easy to see the logs of either Temporal or Envoy. I can even run otel-tui, a terminal UI for OpenTelemetry data, to see the traces and metrics of my application. Directly from my IDE.

Oh, and the best thing about VS Code tasks? I can configure them to run when I open my IDE, using the runOn=folderOpen option. That sounds like a small feature, but it means that I don't have to think about them anymore.

{
	"version": "2.0.0",
	"tasks": [
		{
			"label": "Run otel-tui",
			"type": "process",
			"command": "otel-tui",
			"args": [
				"--host",
				"localhost"
			],
			"presentation": {...},
			"runOptions": {
				"runOn": "folderOpen"
			}
		},
		{
			"label": "Run Temporal",
			"type": "process",
			"command": "temporal",
			"args": [
				"server",
				"start-dev",
				...
			],
			"options": {
				"cwd": "${workspaceFolder}/services/all-in-one"
			},
			"presentation": {...},
			"runOptions": {
				"runOn": "folderOpen"
			}
		},
		{
			"label": "Run Envoy",
			"type": "process",
			"command": "envoy",
			"args": [
				"--config-path",
				"envoy.yaml"
			],
			"options": {
				"cwd": "${workspaceFolder}/services/all-in-one"
			},
			"presentation": {...},
			"runOptions": {
				"runOn": "folderOpen"
			}
		}
	]
}

Note that it's not perfect, because while Temporal stops gracefully when I close VS Code, Envoy doesn't. There are a few opened issues here and here. A quick workaround to avoid port conflicts is to ensure that the previous process is killed before starting a new one. This can be done by adding a "stop" task and introducing dependencies between the tasks.

{
	"version": "2.0.0",
	"tasks": [
		{
			"label": "Run Envoy",
			"type": "process",
            ...,
			"dependsOn": [
				"Stop Envoy"
			]
		},
		{
			"label": "Stop Envoy",
			"type": "shell",
			"command": "killall -v envoy || true"
		}
	]
}

The last final touch is the preLaunchTask option, to retrieve some secrets from Vault before starting a debug session. The secrets are just stored in a local .env file, ready to be consumed by the launch configuration to configure the environment variables of the application.

{
	"version": "2.0.0",
	"tasks": [
		{
			"label": "Create .secrets dir",
			"type": "shell",
			"command": "make .secrets",
			"options": {
				"cwd": "${workspaceFolder}"
			}
		}
	]
}

The final result

So what do we have now?

• all microservices re-packaged as a monolith
• running "natively" on the host - no VM, no container
• native dependencies running on the host too
• everything orchestrated by the IDE

Are we back to the 2000s? That's how I started, running a monolith Java app in its Application Server, from the IDE. It's simple, it's effective, fast, and easy to debug.

I can deploy locally in seconds, make some changes, redeploy when I decide, and it's up and running again in seconds. Notice something strange? I can add a breakpoint directly.

Key takeaways

• Don't run Kubernetes locally if you don't need to. Unless you're developing a Kubernetes Operator, you usually don't need to run a Kubernetes cluster to develop your application.
• When microservices are too micro, consider running them as a monolith. It's easier to run, easier to debug, and easier to change. Google Service Weaver anyone? ;-)
• Know your IDE. It's a powerful tool, and it can do a lot of things for you. This is where you spend most of your time, and staying there will reduce context switching.
• Don't forget about the power of the debugger. Over the years, we've added layers and layers of abstractions, and we almost forgot the basics.
• If you want to optimize something, iterate. Go step by step, and see what works for you.
• Everyone will optimize in different ways, based on their context. Know your context, and optimize for it.

Conclusion

In the end, this is a rather well-optimized setup. I'm productive again! Or at least I feel productive ;-)

But this is a very personal point of view. Other people might not agree. And they might not be wrong, because there are a lot of flaws in this setup:

• It's pretty far from what's running in prod: we run a monolith instead of microservices, we don't run in containers, etc. I know and I accept the risks ;-)
• It's not CI-friendly at all. This is why we're also keeping our existing Devspace-based setup - at least for the moment. But now local dev, CI, and prod are different...

And the worst of course is on the standardization side. If other devs want to do the same, they'll need:

• to install the dependencies manually, and if we have devs on different OS, they'll need to figure out how to install them
• to configure their IDE to run the tasks. I can share the VS Code settings, but for other IDEs, they'll have to figure it out themselves
• to ensure they follow the version upgrades of the dependencies - in a synchronized way

There are a lot of things we can do to make it easier. And doing so, we'll start optimizing for standardization. And we'll lose some of the productivity we gained...

People might argue about the importance of developer productivity vs standardization. In a different context, I might have favored standardization. But in this specific context, I'm confident that developer productivity is the right choice. That said, nothing is set in stone, and I have more ideas to improve this setup. Stay tuned for part 2 ;-)

Do we need to choose between developer productivity and standardization? If we optimize one, do we have to sacrifice the other? Obviously, my answer is yes. But we can make trade-offs, and find a good balance. It all depends on the context... as always!