Although uncommon, there are valid reasons to have a monorepo - a single git repository for multiple projects, for example migration projects. Until yesterday, I thought this was not possible in Azure DevOps.
A colleague informed me it’s possible to rename the file something other than
/azure-pipelines.yml. From there I figured out how to accomplish create multiple Azure DevOps YAML pipelines in a monorepo.
In this tutorial, you will learn how to:
- setup a root pipeline
- setup 2 pipelines in subfolders and triggered by changes in those folders.
- rename pipelines in DevOps UI
- use triggers
- change working directories
The full example is available here
Note: these commits were pushed separately to generate distinct “Last run”s.
I. Project Structure and YAML files
Let’s imagine we have the following setup:
. ├── README.md ├── azure-pipelines.yml ├── service-a |── azure-pipelines-a.yml │ └── … └── service-b |── azure-pipelines-b.yml └── …
In a standard Azure DevOps project, you have a single
azure-pipelines.yml file in your project root folder. In our project, we will have 3 different pipeline files:
These files will be very similar to your standard YAML pipelines, with two small exceptions: triggers and working directories. We’ll cover those later. First we will add the pipelines.
Step 1 - Add the Pipelines
When you create a new DevOps pipeline, select the repository and on the “Configure your pipeline” page, select “Existing Azure Pipelines YAML file”, which will open up this overlay on the right:
You want to go through this process 3 times, each time selecting a different YAML file. In the image above, I have chosen
/a/azure-pipelines.yml, which is original filename before I renamed it later.
Step 2 - Rename your pipelines
By default, Azure DevOps names your pipelines per GitHub user/org and repository name, so you will end up with 3 pipelines named similar to this:
- julie-ng.azure-devops-monorepo (1)
- julie-ng.azure-devops-monorepo (2)
Not very helpful. Find more options button and select “Rename/move”.
I’ve chosen the following names:
- azure-devops-monorepo (root)
- azure-devops-monorepo (Service A)
- azure-devops-monorepo (Service B)
II. Triggers and how this works
Normally pipeline runs when anything changes. These three pipelines are defined so they only build when their respective files change. We accomplish this with trigger path definitions.
Root Project must
Note the root project excludes our subdirectories. This means that a change to
service-a/readme.md will not trigger our root pipeline.
trigger: paths: exclude: # Exclude! - 'service-a/*' - 'service-b/*'
We have two sub-projects with their own pipelines. We have to adjust each appropriately so it only runs when the sub-project’s code changes:
trigger: paths: include: # Include! - 'service-a/*' # or 'service-b/*'
Now you have your multi-pipeline monorepo setup! But you are not finished. There are reasons why the monorepo setup is not common. While it is acceptable to choose this path, you should understand the disadvantages and caveats.
Be Aware of Other Triggers
There are many reasons for a pipeline to be built. In fact, the official docs name four different types of events that can trigger build pipelines:
|CI triggers||Git Push|
|PR triggers||Pull Requests|
|Scheduled triggers||Schedules defined in Cron format|
|Pipeline triggers||Pipelines can call each other|
|Manual||A human clicks a button|
I add manual runs to make it 5. Although we are limiting path triggers to our subfolder, the when is partially determined by external factors. This means that a pull request that conceptually only affects service B may unintentionally trigger a build of service A.
If you are used to committing and pushing incomplete changes, you may have an unusual number of broken builds. A common symptom is seeing multiple commits in a row that start with “update….” This danger also applies to the separate repo use case. But in a monorepo case it is made worse. The danger here is that a developer or team gets used to red or broken builds and stop reacting to them. So it’s important to be disciplined across your entire team when committing and pushing your changes.
I haven’t tried it. But theoretically you should be able to separate schedules for the pipelines.
Building A or B or both?
Let’s say you have a commit history that looks like this:
0883cf8 b: change number 3 (47 minutes ago) <=== git push 2896d9c a: change number 5 (49 minutes ago) 3fa6757 root: add newlines to readme (49 minutes ago)
First off, both pipeline A and pipeline B will run and they will run with the files from the working tree at
In this example, a developer first made changes to service A and then later to service B. Because the changes were pushed together, the
azure-pipelines-a.yml pipeline runs with files not from
2896d9c but from the future 🤯.
This means if you actually have dependencies outside of that include path in the
triggers: property, you may experience unexpected build results. It seems unlikely in our example. But what if you had such a project structure?
. ├── service-a | |── pipeline-a.yml │ └── … ├── service-b | |── pipeline-b.yml │ └── … └── common-components |── pipeline-c.yml └── …
Then you would be more concerned. This is a trade-off that comes with monorepos. Builds may be accidentally triggered and you should prepare for that. If you’re working in teams, make sure it’s very transparent what everyone is working on.
Keep your Working Directory in Mind
To illustrate this caveat, service B is a Node.js project. Although our YAML file for service B sits in the correct subfolder, the working directory will still be the root. If you try to run
npm install without changing directories, it will fail because there is no
package.json in the root.
We can change this by using the
workingDirectory key in the YAML:
- script: npm install workingDirectory: service-b/
workingDirectory is only available under
steps:, which means you cannot set it once on the whole pipeline, but rather for every task, script, etc. You can make this less painful by using a variable like in my code sample. See the official docs: YAML Reference for details and further limitations in the YAML syntax.
If you have good reason to use a monorepo and want to setup multiple Azure DevOps pipelines, you can. But remember that you lose some sense of control over when and what you are building in your CI pipeline. So if you march down this path, over-communicate within your team, keep your commits squeaky clean, and carry on.