Fun with Azure DevOps Multistage Pipeline

Folkert Azure

I’ve been a huge fan of Octopus Deploy and Teamcity. However, the downside of these great tools is the lack of storing the configurations in versioncontrol. Configurations are done in the web interface and requires additional knowledge of the tool itself. Adding new projects resulted in cloning existing projects and adjust it where it was needed.

Last year I have been working with Gitlab which is an awesome tool for CI purposes. We combined Gitlab with Octopus Deploy, although Gitlab is capable of handling CD too. Octopus Deploy is and will always be an awesome workhorse for managing deployments. I really dig this tool.

But the idea of using pipelines as Gitlab has implemented is hard to beat. Using pipelines this way, it is possible to put CI/CD in version control. And this is the place where is should be, not inside a tool itself, but as part of your version-controlled solution.

For some new projects, I had the change to move from Teamcity/Octopus Deploy to Azure Devops. Having some experience in using Gitlab, I was curious about the possibilities of Azure Devops.

Let the madness begin!

Off topic: For the ones who do not know this guitarist: this is Randy Rhoads, I think one the greatest guitar players of all time. Unfortunately he died at a age of 26 in a place crash. He has left us with two amazing albums together with Ozzy who always started his live shows shouting: ‘Let the madness begin!”

In Azure DevOps, I’ve started with a build pipeline to define your build steps. This is great to manage your CI in code but the CD part still required additional steps on the ‘release’ section. This was sort of an Octopus Deploy implementation inside Azure DevOps. Not what I was looking for.

thankfully, the development team of Azure Devops added a preview version of Multistage pipelines:

you can activate this feature in the settings.

This article is based upon multistage pipeline and is not using the releases.

Let’s start with a single basic pipeline, from here, we’ll move on to the fun parts:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'

- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo See https://aka.ms/yaml
  displayName: 'Run a multi-line script'

This is just a basic pipeline, let’s transform it to a multistage pipeline:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

stages:

- stage: Build
  
- stage: tst_deploy

- stage: uat_deploy

- stage: prd_deploy

As you can see, there is a stages section added with some defined stages, for example a build stage, and some deployment stages. Let’s add some jobs to the stages:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Build
  jobs:
  - job: Build
    pool:
      vmImage: 'Ubuntu-16.04'
    continueOnError: true
    steps:
    - script: echo Restore NuGet Packages
      displayName: 'NuGet - Restore Packages'
    - script: echo Build Solution
      displayName: 'DotNet - Build Solution'
    - script: echo Publish Solution
      displayName: 'DotNet - Publish Solution'
    - script: echo Publish Build Artifacts
      displayName: 'Publish - Artifcts'

- stage: tst_deploy

- stage: uat_deploy

- stage: prd_deploy

Results in Azure DevOps:

Important to understand is the types of jobs, there are 2 different kind of jobs you can use:

  1. A regular Job type, which says: hey, let’s do some stuff!
  2. A Deployment type, which says, let’s move some files!

A deployment job has some additional features. The most important feature is the possibility to link a deployment to an ‘environment’.

Environments

Environments can be created in Azure Devops:

An environment is just a label, you can label them whatever you want. For example: Seed/Flower/Tree. Being part of a CI/CD pipeline, naming these like TST/UAT/PRD is more appropriate 😉
Having an environment defined, we can add approvals to these environments:

Here you can add details for who is allowed to approve the deployment job.

So, let’s create a deployment job first and link the deployment to an environment:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Build
  jobs:
  - job: Build
    pool:
      vmImage: 'Ubuntu-16.04'
    continueOnError: true
    steps:
    - script: echo Restore NuGet Packages
      displayName: 'NuGet - Restore Packages'
    - script: echo Build Solution
      displayName: 'DotNet - Build Solution'
    - script: echo Publish Solution
      displayName: 'DotNet - Publish Solution'
    - script: echo Publish Build Artifacts
      displayName: 'Publish - Artifcts'

- stage: tst_deploy
  jobs:
    - deployment: tst_deployment
      environment: TST
      strategy: 
        runOnce:
          deploy:
            steps:
              - script: echo Deploy Artifact from Build to environment
                displayName: 'Deploy - Deploy Artifact'

- stage: uat_deploy
  jobs:
    - deployment: uat_deployment
      environment: UAT
      strategy: 
        runOnce:
          deploy:
            steps:
              - script: echo Deploy Artifact from Build to environment
                displayName: 'Deploy - Deploy Artifact'

- stage: prd_deploy
  jobs:
    - deployment: prd_deployment
      environment: PRD
      strategy: 
        runOnce:
          deploy:
            steps:
              - script: echo Deploy Artifact from Build to environment
                displayName: 'Deploy - Deploy Artifact'

And the result:

As you can see, the pipeline is paused when the UAT stage is reached. This is because I’ve added an approval on the UAT environment. There is no approval set on TST, so the build deploys directly to the TST and stops when UAT is reached:

We can approve the pipeline, then it will continue to UAT and PRD:

Ok, so far so good.

Adding steps

Let’s add some more ‘real world’ steps to the deployment, for example, when we are deploying to an Azure environment, we can design our CD like:

  1. Create a new Azure Webapp
  2. Set App settings
  3. Add some IP whitelisting stuff
  4. Add hostnames
  5. Deploy artifact
  6. Run E2E Tests
  7. Switch Trafficmanager

(just an example)

I won’t create these actual jobs, but I’ll add placeholders as a replacement to get an idea.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Build
  jobs:
  - job: Build
    pool:
      vmImage: 'Ubuntu-16.04'
    continueOnError: true
    steps:
    - script: echo Restore NuGet Packages
      displayName: 'NuGet - Restore Packages'
    - script: echo Build Solution
      displayName: 'DotNet - Build Solution'
    - script: echo Publish Solution
      displayName: 'DotNet - Publish Solution'
    - script: echo Publish Build Artifacts
      displayName: 'Publish - Artifcts'

- stage: tst_deploy
  jobs:
    - deployment: tst_deployment
      environment: TST
      strategy: 
        runOnce:
          deploy:
            steps:
              - script: echo Create an Azure Webapp
                displayName: 'Azure CLI - Create webapp'
              - script: echo Add App Settings
                displayName: 'Azure CLI - Add app settings'
              - script: echo Add IP Whitelisting
                displayName: 'Azure CLI - Add IP Whitelisting'
              - script: echo Add hostnames
                displayName: 'Azure CLI - Add hostnames'
              - script: echo Deploy artifact
                displayName: 'Azure CLI - Deploy artifact'
              - script: echo Start E2E test
                displayName: 'Powershell - Start E2E test'
              - script: echo Switch Trafficmanager
                displayName: 'Azure CLI - Switch Trafficmanager'

- stage: uat_deploy
  jobs:
    - deployment: uat_deployment
      environment: UAT
      strategy: 
        runOnce:
          deploy:
            steps:
              - script: echo Create an Azure Webapp
                displayName: 'Azure CLI - Create webapp'
              - script: echo Add App Settings
                displayName: 'Azure CLI - Add app settings'
              - script: echo Add IP Whitelisting
                displayName: 'Azure CLI - Add IP Whitelisting'
              - script: echo Add hostnames
                displayName: 'Azure CLI - Add hostnames'
              - script: echo Deploy artifact
                displayName: 'Azure CLI - Deploy artifact'
              - script: echo Start E2E test
                displayName: 'Powershell - Start E2E test'
              - script: echo Switch Trafficmanager
                displayName: 'Azure CLI - Switch Trafficmanager'

- stage: prd_deploy
  jobs:
    - deployment: prd_deployment
      environment: PRD
      strategy: 
        runOnce:
          deploy:
            steps:
              - script: echo Create an Azure Webapp
                displayName: 'Azure CLI - Create webapp'
              - script: echo Add App Settings
                displayName: 'Azure CLI - Add app settings'
              - script: echo Add IP Whitelisting
                displayName: 'Azure CLI - Add IP Whitelisting'
              - script: echo Add hostnames
                displayName: 'Azure CLI - Add hostnames'
              - script: echo Deploy artifact
                displayName: 'Azure CLI - Deploy artifact'
              - script: echo Start E2E test
                displayName: 'Powershell - Start E2E test'
              - script: echo Switch Trafficmanager
                displayName: 'Azure CLI - Switch Trafficmanager'

And the result:

Yes, that’s more like it! However, our pipeline is becoming very large if we are adding more steps. Thankfully we can move parts into a template. Using templates keeps our pipeline clean and is preventing code duplication.

Templates

More information regarding templates can be found here:
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops

First, let’s move the steps to a template file, called steps.yaml:

steps:
    - script: echo Create an Azure Webapp
    displayName: 'Azure CLI - Create webapp'
    - script: echo Add App Settings
    displayName: 'Azure CLI - Add app settings'
    - script: echo Add IP Whitelisting
    displayName: 'Azure CLI - Add IP Whitelisting'
    - script: echo Add hostnames
    displayName: 'Azure CLI - Add hostnames'
    - script: echo Deploy artifact
    displayName: 'Azure CLI - Deploy artifact'
    - script: echo Start E2E test
    displayName: 'Powershell - Start E2E test'
    - script: echo Switch Trafficmanager
    displayName: 'Azure CLI - Switch Trafficmanager'

now we have moved our steps to a template, we need to call the template to the pipeline:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Build
  jobs:
  - job: Build
    pool:
      vmImage: 'Ubuntu-16.04'
    continueOnError: true
    steps:
    - script: echo Restore NuGet Packages
      displayName: 'NuGet - Restore Packages'
    - script: echo Build Solution
      displayName: 'DotNet - Build Solution'
    - script: echo Publish Solution
      displayName: 'DotNet - Publish Solution'
    - script: echo Publish Build Artifacts
      displayName: 'Publish - Artifcts'

- stage: tst_deploy
  jobs:
    - deployment: tst_deployment
      environment: TST
      strategy: 
        runOnce:
          deploy:
            steps:
              - template: steps.yaml

- stage: uat_deploy
  jobs:
    - deployment: uat_deployment
      environment: UAT
      strategy: 
        runOnce:
          deploy:
            steps:
              - template: steps.yaml

- stage: prd_deploy
  jobs:
    - deployment: prd_deployment
      environment: PRD
      strategy: 
        runOnce:
          deploy:
            steps:
              - template: steps.yaml

The result will be the same as my previous example, but my pipeline is way cleaner. Let’s move more to the template, the full jobs part:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Build
  jobs:
  - job: Build
    pool:
      vmImage: 'Ubuntu-16.04'
    continueOnError: true
    steps:
    - script: echo Restore NuGet Packages
      displayName: 'NuGet - Restore Packages'
    - script: echo Build Solution
      displayName: 'DotNet - Build Solution'
    - script: echo Publish Solution
      displayName: 'DotNet - Publish Solution'
    - script: echo Publish Build Artifacts
      displayName: 'Publish - Artifcts'

- stage: tst_deploy
  jobs:
    - template: jobs.yaml

- stage: uat_deploy
  jobs:
    - template: jobs.yaml

- stage: prd_deploy
  jobs:
    - template: jobs.yaml

and our jobs template:

jobs:
  - deployment: tst_deployment
    environment: TST
    strategy: 
      runOnce:
        deploy:
          steps:
          - script: echo Create an Azure Webapp
            displayName: 'Azure CLI - Create webapp'
          - script: echo Add App Settings
            displayName: 'Azure CLI - Add app settings'
          - script: echo Add IP Whitelisting
            displayName: 'Azure CLI - Add IP Whitelisting'
          - script: echo Add hostnames
            displayName: 'Azure CLI - Add hostnames'
          - script: echo Deploy artifact
            displayName: 'Azure CLI - Deploy artifact'
          - script: echo Start E2E test
            displayName: 'Powershell - Start E2E test'
          - script: echo Switch Trafficmanager
            displayName: 'Azure CLI - Switch Trafficmanager'

Yes, this is going to work:

oopsie… we just deployed our code 3 times to the same environment, although the stage says differently. The job is mentioning the TST environment.

Variables to the rescue!

Okay, back in business, let’s add some variables and use parameters in the template to pass the correct values. Let’s start by adding an environment parameter to the template file and use it in the steps:

parameters:
  environment: ''

jobs:
  - deployment: ${{ parameters.environment }}_deployment
    environment: ${{ parameters.environment }}
    strategy: 
      runOnce:
        deploy:
          steps:
          - script: echo Create an Azure Webapp
            displayName: 'Azure CLI - Create webapp'
          - script: echo Add App Settings
            displayName: 'Azure CLI - Add app settings'
          - script: echo Add IP Whitelisting
            displayName: 'Azure CLI - Add IP Whitelisting'
          - script: echo Add hostnames
            displayName: 'Azure CLI - Add hostnames'
          - script: echo Deploy artifact
            displayName: 'Azure CLI - Deploy artifact'
          - script: echo Start E2E test
            displayName: 'Powershell - Start E2E test'
          - script: echo Switch Trafficmanager
            displayName: 'Azure CLI - Switch Trafficmanager'

Now, we only need to send the correct value to the template in our pipeline:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Build
  jobs:
  - job: Build
    pool:
      vmImage: 'Ubuntu-16.04'
    continueOnError: true
    steps:
    - script: echo Restore NuGet Packages
      displayName: 'NuGet - Restore Packages'
    - script: echo Build Solution
      displayName: 'DotNet - Build Solution'
    - script: echo Publish Solution
      displayName: 'DotNet - Publish Solution'
    - script: echo Publish Build Artifacts
      displayName: 'Publish - Artifcts'

- stage: tst_deploy
  jobs:
    - template: jobs.yaml
      parameters:
        environment: TST

- stage: uat_deploy
  jobs:
    - template: jobs.yaml
      parameters:
        environment: UAT

- stage: prd_deploy
  jobs:
    - template: jobs.yaml
      parameters:
        environment: PRD

And the result:

Multistage deployment working

And it’s working again!

More fun…

Working for a single project is fun but in the real world, we’re often not creating single apps for a single brand but we’re doing multi brand sites, based upon Sitecore for example. These deployments require more complex build pipelines. Let’s have a look at some of these challenges.

Adding more brands to our pipeline

So, let’s assume, we need to adjust the pipeline to work with multiple brands. As always, we’re keeping the principle: build once, deploy everywhere. Our build is some sort of a ‘white label’ artifact where the app settings will be responsible for rendering the correct application.
Let’s assume we’ll need to deploy multiple webapps for some car brands:

  1. Toyota
  2. Volkswagen
  3. Fiat
  4. Chrysler
  5. Ford

This means we need to run all the steps for all these brands, that are a lot of steps! And we do not want to use duplicate code. In this case we can use pipeline loops (each function):

parameters:
  environment: ''
  brands: []

jobs:
  - deployment: ${{ parameters.environment }}_deployment
    environment: ${{ parameters.environment }}
    strategy: 
      runOnce:
        deploy:
          steps:
          - ${{ each brand in parameters.brands }}:
            - script: echo Create an Azure Webapp for ${{brand}}
              displayName: 'Azure CLI - Create webapp for ${{brand}}'
            - script: echo Add App Settings for ${{brand}}
              displayName: 'Azure CLI - Add app settings for ${{brand}}'
            - script: echo Add IP Whitelisting for ${{brand}}
              displayName: 'Azure CLI - Add IP Whitelisting for ${{brand}}'
            - script: echo Add hostnames for ${{brand}}
              displayName: 'Azure CLI - Add hostnames for ${{brand}}'
            - script: echo Deploy artifact for ${{brand}}
              displayName: 'Azure CLI - Deploy artifact for ${{brand}}'
            - script: echo Start E2E test for ${{brand}}
              displayName: 'Powershell - Start E2E test for ${{brand}}'
            - script: echo Switch Trafficmanager for ${{brand}}
              displayName: 'Azure CLI - Switch Trafficmanager for ${{brand}}'

As you can see, I’ve added an array parameter to the parameters section. Then I’ve added a line which iterates all brands:

And our pipeline:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Build
  jobs:
  - job: Build
    pool:
      vmImage: 'Ubuntu-16.04'
    continueOnError: true
    steps:
    - script: echo Restore NuGet Packages
      displayName: 'NuGet - Restore Packages'
    - script: echo Build Solution
      displayName: 'DotNet - Build Solution'
    - script: echo Publish Solution
      displayName: 'DotNet - Publish Solution'
    - script: echo Publish Build Artifacts
      displayName: 'Publish - Artifcts'

- stage: tst_deploy
  jobs:
    - template: jobs.yaml
      parameters:
        environment: TST
        brands: ["Toyota", "Volkswagen", "Fiat", "Chrysler", "Ford"]


- stage: uat_deploy
  jobs:
    - template: jobs.yaml
      parameters:
        environment: UAT
        brands: ["Toyota", "Volkswagen", "Fiat", "Chrysler", "Ford"]

- stage: prd_deploy
  jobs:
    - template: jobs.yaml
      parameters:
        environment: PRD
        brands: ["Toyota", "Volkswagen", "Fiat", "Chrysler", "Ford"]

Yes, there is some duplication. I can’t set the brands as a variable at the moment due to a bug in Azure Devops.

At this moment we’re able to use multiple brands in our pipeline with a minimum amount of code!

But…. we can have even more fun!

Looking at our CD pipeline, we need to create an Azure WebApp. So all brands will create an Azure webapp when the pipeline is triggered. Looking at the command of creating an Azure webapp:

az group create `
    --location "West Europe" `
    --name "rg-pipeline-demo-tst"

az appservice plan create `
    --name "rg-pipeline-demo-tst-sp" `
    --resource-group "rg-pipeline-demo-tst"

az webapp create `
    --name "rg-pipeline-demo-tst-[brand]-app" `
    --plan "rg-pipeline-demo-tst-sp" `
    --resource-group "rg-pipeline-demo-tst" `
    --sku B1

These lines will create a new resourcegroup, an app serviceplan and a webapp for each brand, using the B1 sku. Where [brand] will be the actual car brand.

But what if the ‘sku’ is different between the brands, let’s say: Toyota requires a way heavier resources then Ford. Then Toyota requires a P2V2 while Ford requires an S1. To accomplish this in our template, we need some sort of a multidimensional array to pass in to our template:

Our pipeline:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Build
  jobs:
  - job: Build
    pool:
      vmImage: 'Ubuntu-16.04'
    continueOnError: true
    steps:
    - script: echo Restore NuGet Packages
      displayName: 'NuGet - Restore Packages'
    - script: echo Build Solution
      displayName: 'DotNet - Build Solution'
    - script: echo Publish Solution
      displayName: 'DotNet - Publish Solution'
    - script: echo Publish Build Artifacts
      displayName: 'Publish - Artifcts'

- stage: tst_deploy
  jobs:
    - template: jobs.yaml
      parameters:
        environment: TST
        brands:
          - brand:
            name: 'Toyota'
            sku: 'P2V2'
          - brand:
            name: 'Volkswagen'
            sku: 'S1'
          - brand:
            name: 'Fiat'
            sku: 'S3'
          - brand:
            name: 'Chrysler'
            sku: 'B1'
          - brand:
            name: 'Ford'
            sku: 'P1V2'

- stage: uat_deploy
  jobs:
    - template: jobs.yaml
      parameters:
        environment: UAT
        brands:
          - brand:
            name: 'Toyota'
            sku: 'P2V2'
          - brand:
            name: 'Volkswagen'
            sku: 'S1'
          - brand:
            name: 'Fiat'
            sku: 'S3'
          - brand:
            name: 'Chrysler'
            sku: 'B1'
          - brand:
            name: 'Ford'
            sku: 'P1V2'

- stage: prd_deploy
  jobs:
    - template: jobs.yaml
      parameters:
        environment: PRD
        brands:
          - brand:
            name: 'Toyota'
            sku: 'P2V2'
          - brand:
            name: 'Volkswagen'
            sku: 'S1'
          - brand:
            name: 'Fiat'
            sku: 'S3'
          - brand:
            name: 'Chrysler'
            sku: 'B1'
          - brand:
            name: 'Ford'
            sku: 'P1V2'

And the template:

parameters:
  environment: ''
  brands: []

jobs:
  - deployment: ${{ parameters.environment }}_deployment
    environment: ${{ parameters.environment }}
    strategy: 
      runOnce:
        deploy:
          steps:
          - ${{ each brand in parameters.brands }}:
            - script: echo Create an Azure Webapp for ${{brand.name}}
              displayName: 'Azure CLI - Create webapp for ${{brand.name}} using sku: ${{brand.sku}}'
            - script: echo Add App Settings for ${{brand.name}}
              displayName: 'Azure CLI - Add app settings for ${{brand.name}}'
            - script: echo Add IP Whitelisting for ${{brand.name}}
              displayName: 'Azure CLI - Add IP Whitelisting for ${{brand.name}}'
            - script: echo Add hostnames for ${{brand.name}}
              displayName: 'Azure CLI - Add hostnames for ${{brand.name}}'
            - script: echo Deploy artifact for ${{brand.name}}
              displayName: 'Azure CLI - Deploy artifact for ${{brand.name}}'
            - script: echo Start E2E test for ${{brand.name}}
              displayName: 'Powershell - Start E2E test for ${{brand.name}}'
            - script: echo Switch Trafficmanager for ${{brand.name}}
              displayName: 'Azure CLI - Switch Trafficmanager for ${{brand.name}}'

The result:

Wow! really impressive!

For now, I think this post is long enough but there is so much more to discover. I’ll write new posts when new cool stuff is there!

Folkert

I'm a webdeveloper, looking for the best experience, working between development and design. Just a creative programmer. When I'm getting tired of programming C#, i'd love to create 3D images in 3D Studio Max, play the guitar, create an app for Android or crush some plastics on a climbing wall or try to stay alive when i´m descending some nice white powdered snowy mountains on my snowboard.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.