Mason: a declarative build tool on top of Dagger

I wrote something. A build tool. A declarative build tool on top of Dagger.

Why? Because I really like Dagger, but as I tried to spread its usage at work, I found that in its current form, it was difficult to scale its deployment.

My main concerns are:

• It has a significant impact on the projects where you use it. If you create a specific module for the project, even if you reuse other modules for most of the work, you'll still need to write a lot of boilerplate code to make it work.
• Its usage is non-trivial, meaning that if you were used to running make build, you won't be able to replace it with just dagger run build. You'll need to give it a few more arguments, because it won't automatically read your environment variables or host filesystem. And even worse, you'll need multiple calls if you want to export files, such as test results, and exit with an error code in case of failure.

What I'd like is:

• a simple CLI UX, common across all projects. Maven got it right years ago: you see a pom.xml file, you know that running mvn package will produce the artifacts.
• a declarative and easy way to configure projects that are mostly similar, and could share the same Dagger module, but with different parameters. GoReleaser is a good example of this: a single configuration file to configure how to produce artifacts, where to publish them, etc. Kubernetes is another good example: a generic envelope common to all kinds of resources, but a specific spec, making it easy to filter resources and implement custom logic for each of them.
• an extensible system, making it easy to implement custom logic in any language. Dagger is a great tool for this, with its powerful modules.

I had this idea in my mind for a while, thinking about how to "orchestrate" Dagger modules in a nice and declarative way. And when the Dagger team released the Interactive Shell, with the ability to run Dagger scripts, I quickly saw how it could solve some of my concerns. I just needed something to generate the "right" script based on a configuration file.

We can solve any problem by introducing an extra level of indirection.

What about using Dagger to generate the Dagger script? A 2 phases plan/apply design, where the first phase generates the script, and the second phase runs it. The configuration file would be split into multiple independent resources, each with a kind/group... or rather, kind/module. This way, we have an easy way to call any module, to process its own resources, and generate a script.

Our build tool can now be very simple: a coordinator that calls Dagger twice, and in the middle merges the scripts to produce a single one. Need to use the output of one module in another? Just define a variable in the first module and use it in the second one.

We just need to define a few conventions, such as phases - test, lint, package, publish - to handle the UX topic. Running the test phase will filter the scripts to only include the relevant ones. No need to guess which module to run or which arguments to pass.

Writing a build tool on top of Dagger really highlights the power of Dagger:

• Caching? Don't care, Dagger handles it.
• plugin system? Dagger modules.
• sandboxed execution? Dagger handles it.
• Nice TUI? Of course.

Replacing Dagger with Mason

What does that change? Everything and not that much ;-)

Everything, because:

• I need to replace all the "local" Dagger modules with YAML configuration files
• I need to replace the multiple Dagger calls from my CI config with a single call to Mason
• I need to write new Dagger modules to process the new configuration files

And not that much, because:

• I can still re-use my existing Dagger modules as-is, if I write new modules to "only" generate the scripts
• Or I can extend my existing Dagger modules to also generate a script, in addition to their original purpose
• Under the hood, it's still Dagger, so same UIs: TUI, Web UI, etc.

Is it worth it?

In a corporate environment, where you might have specific ways to connect to systems, publish artifacts, etc, chances are that you already have custom Dagger modules. Some kind of private Daggerverse. With support for Go projects, Python projects, etc.

In this case, "daggerizing" a new project by just adding a few YAML files is simple, fast, and easy to maintain. On top of that, the ability for the developers to switch between projects and still have the same UX is a big plus.

But if you have a single project, it might seem overkill to add this extra layer...

Next steps

This is only the beginning. Release early, release often.

There are still a lot of things to do, such as some kind of "reporting" feature, to allow any module to analyse the result. A pluggable system that could easily report to other systems, or... someone said LLM? ;-)

But ideally, this should be part of Dagger itself. I like to think of Mason as Dagger's "fig" - the small project that became Docker Compose ;-) It started as a way to define Docker containers in a declarative way. Simple UX too: docker compose up...