Azure DevOps Pipeline Tutorial — Part 2: How to make an Azure DevOps template

In part two of the Azure DevOps Pipeline Tutorial, Penny Xuran Qian, explains how to convert this pipeline into a reusable template and add more features. Read part one here and read more below.

By: Penny Xuran Qian Machine Learning Engineer at ABN AMRO
Source: ABN AMRO Developer news

In the first article, we covered basic terminology to start a simple CI pipeline in Azure DevOps using YAML definition. We are satisfied with it and now we want more people to be able to reuse our CI pipeline.

Surely the copy+paste can be a solution, but it is also error-prone and will add difficulty for maintenance. One solution from Azure DevOps is to make the reusable content and logic into a template.

Templates function in two ways: insert reusable content with a template or you can use a template to control what is allowed in a pipeline. The second way is more useful for increasing security, this article is mainly around the first way.

Make into a template

In Part 1, we created a simple one job one step CI pipeline named azure-pipeline.yml, to use it as a template, we can simply make a few changes to it:

  • remove name
  • remove trigger
  • remove stages and stage

The new code should look like this, name it as ci_template.yml

# Repo: MySpace/TemplateRepo
# File: ci_template.yml

  jobs:
  - job:
    displayName: yamllint_checks
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - script: |
        python -m pip install --upgrade pip
        pip install yamllint
        yamllint -d "{extends: relaxed, rules: {line-length: {max: 200}, new-line-at-end-of-file: disable, new-lines: disable}}" .
      displayName: 'YAML Lint Checks'

ci_template on Github

We made our CI pipeline into a template on jobslevel, the reference level should also be the same as the template level, in this case, jobs. It is flexible to make templates at other levels, depending on your needs.

Now we can already refer to this template in the pipeline definition, create another azure-pipeline2.yml , in this YAML file, we can use our template by simply using keyword: template: template_path, this should give the same results as the pipeline in blog Part 1.

# Repo: MySpace/UserRepo
# File: azure-pipeline2.yml

name: cicd_ci

trigger:
  branches:
    include:
      - master

stages:
- stage: CI_Checks
  jobs:
  - template: ci_template.yml

azure-pipeline2 hosted by GitHub

The template paths should be relative to the file that does the including. The nested hierarchy works the same way as the file system. For example:

steps: 
- template: dir1/fileB.yml 
- template: dir1/dir2/fileC.yml

Use in other repositories

The templates can also be put in a repo and then referred to it from different repos. One template can be used in multiple pipelines. Let’s take our above example YAML file, which is located in repo1 , now we create another repo named repo2 , we create a pipeline YAML file azure-pipeline3.yml :

# Repo: MySpace/UserRepo
# File: azure-pipeline3.yml

name: cicd_ci

trigger:
  branches:
    include:
      - master

resources:
  repositories:
  - repository: ci_template
    type: git
    name: MySpace/Repo1
    ref: 'refs/heads/master'

stages:
- stage: CI_Checks
  jobs:
  - template: ci_template.yml@ci_template

azure-pipeline3 hosted by GitHub

Resources

A resource is anything used by a pipeline that lives outside the pipeline. To use templates in another repository, the system should know about that repository first.

  • Repository: repository identifier string, this is used when the template is referred.
  • Type: the git type refers to Azure Repos Git repos. There are other types supported for example github
  • Name: repository name, the format depends on type .
  • Ref: ref name to use

Use the resources specification to provide the location of the core/original repo. When you refer to the core repo, use @ and the name you gave it in resources. In our case, we used template: ci_template.yml@ci_template

Parameters and Variables

Let’s extend our template further, add one more step for Pylint check (a code analysis for Python), we name it as pylint_checks in this step, we include three commands to install Pylint packages, set up the system PATH, and execute the Pylint command.

# Repo: MySpace/TemplateRepo
# File: ci_template.yml

parameters:
- name: pylint_threshold
  type: string
  default: 8.0
- name: lint_folder
  type: string
  default: "mockcase-on-test"
  
jobs:
- job:
  displayName: yamllint_checks
  pool:
    vmImage: 'ubuntu-latest'
  steps:
  - script: |
      python -m pip install --upgrade pip
      pip install yamllint
      yamllint -d "{extends: relaxed, rules: {line-length: {max: 200}, new-line-at-end-of-file: disable, new-lines: disable}}" .
    displayName: 'YAML Lint Checks'
    
- job:
  displayName: pylint_checks
  pool:
    vmImage: 'ubuntu-latest'
  steps:
  - script: |
      pip install pylint-fail-under pylint-junit
      export PATH="/home/pipeuser/.local/bin":$PATH
      pylint-fail-under --fail_under ${{parameters.pylint_threshold}} --output-format=colorized ${{ parameters.lint_folder}}
    displayName: 'Pylint fail under ${{ parameters.pylint_threshold }}'

ci_template2 hosted by GitHub

To enable users to pass values to the pipeline and have more control over the steps, we specified two runtime parameters in the above pipeline template, pylint_threshold and lint_folder. They are used to set the Pylint pass threshold and which folder to run the Pylint scan individually.

Parameters can be viewed as placeholders, they are only available at template parsing time, and they can’t be changed by a pipeline while it’s running. Pipeline parameters are expanded just before the pipeline runs so the values surrounded by ${{ }} are replaced with parameter values.

From the user upfront, the parameters’ values can be rewritten with key-value pairs, as shown in the example below:

# Repo: MySpace/UserRepo
# File: azure-pipeline3.yml

name: cicd_ci

variables:
  - name: PylintThreshold
    value: 8.0
  - name: LintFolder
    value: "folder_dir"

trigger:
  branches:
    include:
      - master

resources:
  repositories:
  - repository: ci_template
    type: git
    name: MySpace/Repo1
    ref: 'refs/heads/master'

stages:
- stage: CI_Checks
  jobs:
  - template: ci_template.yml@ci_template
    parameters:
      pylint_threshold: $(PylintThreshold)
      lint_folder: $(LintFolder)

azure-pipeline4 hosted by GitHub

variables can be a convenient way to collect information. Unlike parameters that have data types such as numbers and strings, all variables are stored as strings and are mutable. This means if you define a parameter type as number and pass the value from a variable, the pipeline would fail to parse because of unmatched data type.

The variable syntax varies depending on the usage, there are three different ways to reference variables: macro, template expression, and runtime expression. Macro is used to provide input for a task.

Source

Variable security

The secret variables should be avoided being set in the YAML file, as operating systems often log commands for the processes that they run, and you wouldn’t want the log to include a secret that passed in as an input.

There are several ways to set secret variables in the pipeline:

Create a pipeline variable and the usage of it

Wrap-up:

  • From this tutorial we turned our pipeline definition into a template, that can be reusable in either the same repositories or other repositories.
  • We were able to extend our one job pipeline into a multiple job pipeline and used parameters and variables to provide more pipeline flexibility.
  • We also covered the security of using variables and introduced several solutions to create secret variables.

 

Read more articles like this here!
Lees meer van dit soort artikelen hier!