Duration 15 minutes
Level Beginner
Type Demo + Configuration

Learning Objectives

After completing this lab, you will be able to:

  • Understand CI/CD pipeline concepts for test automation
  • Read and modify an Azure DevOps YAML pipeline
  • Run functional and accessibility suites as independent matrix jobs
  • Create a pipeline in Azure DevOps from an existing YAML file
  • View test results and artifacts in Azure DevOps
  • Decide when to run tests (push, PR, schedule)

Prerequisites

  • Lab 03 completed
  • An Azure DevOps organization and project
  • Push access to the Azure DevOps (or mirrored GitHub) repository

[!NOTE] Choose GitHub Actions for the alternative CI lab. Azure Pipelines can also build a GitHub repository.

Exercises

Exercise 1: Review the Pipeline

Open .azuredevops/pipelines/playwright-tests.yml in VS Code. Focus on this matrix from the full pipeline:

jobs:
  - job: Playwright
    displayName: 'Playwright tests'
    strategy:
      maxParallel: 2
      matrix:
        Functional:
          suite: 'functional'
          testSuiteId: '3203'
          expectedTestCount: '9'
          junitPath: 'test-results/junit.xml'
          reportPath: 'playwright-report'
          resultsPath: 'test-results'
        Accessibility:
          suite: 'accessibility'
          testSuiteId: '3204'
          expectedTestCount: '3'
          junitPath: 'test-results/accessibility/junit.xml'
          reportPath: 'playwright-report/accessibility'
          resultsPath: 'test-results/accessibility'

Each matrix entry runs independently on an Ubuntu agent. maxParallel: 2 allows both suites to run at once, subject to your organization’s parallel-job capacity. With one available slot, the jobs queue instead of running simultaneously.

Read the steps below the matrix in the full pipeline:

  • UseNode@1 installs Node.js 20; npm install installs test dependencies.
  • Playwright installs Chromium and its system dependencies.
  • npm run test:$(suite) selects the functional or accessibility config.
  • PLAYWRIGHT_SCREENSHOT: 'on' enables screenshots for every test, and the shared config records traces on the first retry.
  • PowerShell@2 runs scripts/publish-test-plan-results.ps1 to publish $(junitPath) into a build-linked automated run in Test Plan 3201. Failed tests, missing results, unmatched automation identities, or publication errors fail this step. It runs after test failures with succeededOrFailed().
  • PublishPipelineArtifact@1 publishes suite-specific reports and raw results with condition: succeededOrFailed(), preserving generated evidence after failures.

[!IMPORTANT] The YAML pr trigger applies to GitHub repositories. For Azure Repos Git, configure a build validation branch policy on main to run this pipeline for PRs. The trigger section runs builds for pushes to main in either case.

Test Plan Publishing Setup

This project’s pipeline targets Plan 3201, with static suites 3203 (Functionality, nine cases) and 3204 (Accessibility, three cases). Each case must have its native automation association: exact test name, source filename, and automation ID. Each case currently has one test configuration and test point. Change the plan ID, suite IDs, and expected counts when using another project.

The publisher matches JUnit name and classname to those associations, creates a run bound to the suite’s test points and current build, records actual outcomes, attaches the original JUnit report, and completes the run. It then verifies each point’s last run and outcome. It does not change test-case workflow states or mark User Stories complete. Failed tests stay failed; skipped tests are not passed.

Native automation association alone does not update test point outcomes when using PublishTestResults@2. This pipeline replaces that task with plan-aware publishing, so each suite has one build-linked run per job attempt. Retrying a job creates another execution attempt. Do not add the old publisher alongside this step, or the build will contain duplicate result sets. Screenshots, traces, and HTML reports remain pipeline artifacts; result comments link to them.

The YAML passes $(System.AccessToken) only to the publishing step. In Azure DevOps, ensure the job’s Build Service identity can read the plan, suites, and work items and has View test runs, Create test runs, and Manage test runs permissions. For a project-scoped job token this is normally OnSIS Build Service (MngEnvMCAP675646); collection-scoped jobs use the collection Build Service identity. A 401 or 403 fails publication and requires checking token availability and these permissions. Do not substitute a PAT in source code. Only trusted pipeline code should receive this write-capable token.

Run the publisher’s isolated checks from the repository root:

pwsh -File scripts/publish-test-plan-results.Tests.ps1

These checks use mocked API responses and do not write to Azure DevOps. After publishing the YAML, verify a real CI run using the Build Service identity. Concurrent executions targeting the same test points compete for the latest outcome; a verification mismatch fails the step rather than claiming success. On-demand execution from the Test Plans UI is not configured by this publisher.

Exercise 2: Create the Pipeline in Azure DevOps

If the pipeline has not been created yet in your Azure DevOps project:

  1. Open your project in Azure DevOps and select Pipelines > New pipeline.
  2. Choose the location of your code (Azure Repos Git or GitHub).
  3. Select your repository.
  4. Choose Existing Azure Pipelines YAML file.
  5. Select the branch (main) and path /.azuredevops/pipelines/playwright-tests.yml.
  6. Review the YAML preview, then select Run to save and queue the first run.

Exercise 3: Push a Change

Make a small modification to verify the pipeline triggers correctly. Add a console.log statement to any test:

test('navigate to Ontario.ca search page', async ({ page }) => {
  console.log('CI pipeline verification');
  await page.goto('/search?query=driver+licence');
  await expect(page).toHaveTitle(/ontario/i);
});

Commit and push from your work-item feature branch. Replace 1234 with your User Story or Bug ID, linked to a Feature and Epic, then open a PR targeting main:

git add playwright-tests/tests/ontario-search.spec.ts
git commit -m "test: verify parallel CI suites AB#1234"
git push -u origin HEAD

Exercise 4: Watch the Pipeline

  1. Open your project in Azure DevOps and select Pipelines.
  2. Find the run triggered by your PR (with the required trigger or branch policy).
  3. Check both Functional and Accessibility matrix jobs and their logs.

The pipeline progresses through Node.js setup, test execution, test results publishing, and artifact upload. Each step shows its own log output.

Exercise 5: View Results

After the pipeline completes:

  1. Check the overall status: green checkmark (passed) or red X (failed).
  2. Select the Tests tab to see pass/fail counts, durations, and failure details sourced from the JUnit report.
  3. Select Artifacts and find both suites’ outputs:
Suite HTML report JUnit, screenshots, and traces
Functional playwright-report-functional test-results-functional
Accessibility playwright-report-accessibility test-results-accessibility

Open Test Plans > Plan 3201 > Execute and inspect each static suite. Confirm the nine functional and three accessibility points have the same outcomes as the build-linked runs, with populated last-run and result links.

Exercise 6: View HTML Report

  1. Download and extract one of the HTML report artifacts.
  2. From playwright-tests, run npx playwright show-report <extracted-report-directory>.
  3. Explore the interactive report: test names, durations, pass/fail status, and screenshots for every test (enabled by PLAYWRIGHT_SCREENSHOT=on in the pipeline).

These reports match local runs with npm run test:functional and npm run test:accessibility. Each job’s report contains only its own suite.

Exercise 7: Discussion

Consider these pipeline strategies with your team:

  • Branch policy: Add a build validation branch policy on main that requires this pipeline to pass before a pull request can be completed. This prevents regressions from reaching the main branch.
  • Nightly schedule: Add a schedules trigger (cron: '0 2 * * *') to catch issues from external dependencies or site changes overnight.
  • Environment matrix: Run tests across multiple browsers (Chromium, Firefox, WebKit) using a pipeline matrix strategy to verify cross-browser compatibility.

Verification Checkpoint

Both matrix jobs finish, the Tests tab contains separate suite runs, and all four artifacts are available when tests execute. Diagnose failures from the live site using the corresponding suite’s evidence; do not assume all scans will pass.

The functional suite includes intentional @failure-demo tests for practicing failure investigation. These can make the functional job red even when normal scenarios pass; the accessibility job should still complete independently.

Summary

CI/CD pipelines ensure tests run consistently on every code change without manual intervention. The pipeline automates browser installation, test execution, and report generation in a clean environment, catching regressions before they reach production.

Next Steps

Continue with Lab 05: Accessibility Testing to explore the axe scans, diagnose violations, and practice manual checks. This optional 20-minute extension follows the one-hour core workshop.

Resources for Further Learning

This site uses Just the Docs, a documentation theme for Jekyll.