Nitya Narasimhan, Ph.DWelcome to Day 20 of #30DaysOfPWA! New to the series? Three things you can do to catch up:
- Read the kickoff post below for context.
- Watch the GitHub Repository for updates.
- Bookmark #30DaysOfPWA Blog for resources.
This is a shortened version of this canonical post for the #30DaysOfPWA.
About The Author
Today's post is authored by Nitya Narasimhan (๐๐ฝ) - a Senior Cloud Advocate on the Developer Relations team at Microsoft. She is also the visual storyteller behind @sketchthedocs and created the illustrations used in this blog series. Follow Nitya at @nitya or here on dev.to.
Welcome to day week 3 day 6 of the 30 Days of PWA series! In today's post we'll talk about Test Automation - what it is, why it matters, and developer tools to help us build reliable end-to-end testing workflows for our PWA.
Test Automation
Test Automation is the practice of executing tests automatically to validate software product functionality, then using testing insights to iteratively improve product quality. Test Automation is key to continuous delivery in modern DevOps culture: build-deploy pipelines keep software products in a continuous state of readiness for production release.
Why explore Test Automation for PWA? Two reasons: growing test complexity & better tooling capability.
Testing Complexity
Progressive Web Apps combine the reach of traditional web apps with the richer capability of platform-native apps. The first needs cross-browser testing to ensure usable experiences across all targeted browsers. The second needs progressive enhancement testing strategies to validate the enhanced experiences available to more capable devices.
Increased frequency of testing (e.g., on every feature commit or bug fix) coupled with many testing granularities (unit, integration, end-to-end) and configurations (browsers, device platforms) makes manual testing time-consuming and error-prone. This is where test automation excels, reducing time to detect issues, eliminating operator errors, and scaling effectively with cloud resources.
Tooling Capability
Adopting automated testing strategies today is a no-brainer thanks to the rich ecosystem of tools and technologies available to web developers. Modern web automation software can execute user actions and other operations on a web browser without human intervention. Check out the Microsoft Edge docs for more on relevant testing tools and technologies for web development, including:
- WebDriver - a W3C standard wire protocol for out-of-process clients to remotely instruct browsers.
- DevTools Protocol - directly inspect, debug, and profile, Chromium-based browsers like Edge.
- Puppeteer - a Node.js library for browser automation using DevTools Protocol.
- Playwright - framework for reliable end-to-end testing of modern web apps, across browsers.
- webhint - linter for accessibility, performance, security, PWA and cross-browser compatibility.
We'll focus on Playwright in this post - but do explore the other resources to learn how they fit into your testing toolkit.
Test Pyramid
๐ฃ ETYMOLOGY: Note that the term "Test Pyramid" had its origins in the 2009 book "Succeeding with Agile" by Mike Cohn. Read more about history and evolution of the term in this 2012 post from Martin Fowler.
To craft an Agile testing strategy, it helps to use visual metaphors like this practical test pyramid that organizes our thinking around two guiding principles: (1) Write tests at various granularities, and (2) Write fewer tests as you go higher up the pyramid.
What does testing granularity mean? My version of the pyramid has unit testing at the base, with exploratory testing at the top. Test automation helps with the bottom four layers, while manual testing is key to uncovering "gaps" that automated testing might overlook at the top.
In implementing the strategy, we want many small, fast tests at the base, and fewer more coarse-grained tests as we go higher. There are numerous unit testing libraries and tools that make the first objective achievable. But end-to-end test automation can be "notoriously flaky" given the many "browser quirks, timing issues, animations, and unexpected UI interactions" that we need to deal with.
This is where having a reliable testing framework like Playwright really helps. Its test hierarchies make it easy to compose high-level test suites from low-level test cases - and use test configurations with multiple test projects to create maximum flexibility. The test runner can focus on execution, running tests in parallel for efficiency, using multiple projects for cross-browser coverage, and capturing traces for post-execution analysis.
Testing PWA
Next, let's think about the functional specification of these tests - what kinds of features (unit testing) and interaction workflows (end-to-end testing) should we consider for Progressive Web Apps? Here's one way to approach this:
- PWAs are web apps. Start with existing "strategies for carrying out testing" for web apps, and identify the optimal testing configurations (target devices, browsers) for your intended audience.
- PWAs have additional desirable characteristics to validate. We know from the previous post that auditing tools are ideal here and can be integrated into a test automation workflow it they expose a programmable interface. Ex: playwright-lighthouse makes it easy to run Lighthouse audits as part of your end-to-end testing workflow.
- PWAs may have functional testing requirements for core components, that are automation-friendly. Ex: we can emulate offline mode to test service workers and caching strategies. Read this testing service workers post for useful guidance.
- PWAs may use new or experimental web capabilities that are unevenly supported across browsers and devices. We can use feature detection to test for their existence on target devices, and execute conditional test actions based on the outcome.
But how can we put these specifications into implementation for reliable end-to-end testing? Let's talk about Playwright!
Hello, Playwright!
Playwright is an open-source testing framework that enables reliable end-to-end testing and automation for modern web applications. It comes with a built-in Playwright Test Runner for test automation, and a Playwright Library to simplify integration with third-party solutions.
Playwright has a number of benefits that differentiate it from other test automation frameworks. Scan the visual guide for more details on each:
- It has a unified API for use across browsers and device platforms.
- It enables mobile web testing with native emulation and rich device profiles.
- It has multi-language support - including TypeScript, JavaScript, Java, Python and .NET.
- No flaky testing - with auto-waiting and web-first assertions eliminating messy timeouts.
- No limits - aligns with modern web architectures, runs out-of-process, and supports multi-everything.
- Powerful tools - for authoring, debugging, profiling, and reporting tests and execution.
- Full isolation - with browser contexts ready in ms, fast execution and parallelization options.
Want to get hands-on experience with the various capabilities? Check out the demo.playwright repository for code examples showcasing test scenarios for accessibility, android, authentication, performance (DevTools & Lighthouse), and visual-comparison. Plus, learn how Continuous Integration options like GitHub Actions work seamlessly with Playwright, right out of the box.
Using Playwright
Playwright currently has two options to kickstart your end-to-end testing journey:
- Use the Getting Started guide to use Playwright from the command line.
- Use the Playwright Test for VSCode extension (in preview) to use Playwright from the IDE.
For now, we'll use the first option since the extension is still in preview and works only with the latest version of Playwright Test. In an earlier post, I PWA-enabled an existing web app. Today, I'll revisit that project and instrument it for end-to-end testing with Playwright. Ready? Let's go.
1. Install Playwright
Since I'm scaffolding Playwright tests for an existing project, I'll use the command below. If you are creating a new project, just add the project name as the final argument.
$ npm init playwright
The output looks as shown (compressed for clarity). Note that you can choose to get a GitHub Actions workflow instrumented for you during setup. The scaffolding process may take a while the first time since it installs Playwright (Test runner and library) and all supported browsers (Chromium, Firefox and Webkit) by default.
Need to install the following packages:
create-playwright
Ok to proceed? (y) y
Getting started with writing end-to-end tests with Playwright:
Initializing project in '.'
โ Do you want to use TypeScript or JavaScript? ยท JavaScript
โ Where to put your end-to-end tests? ยท e2e-tests
โ Add a GitHub Actions workflow? (Y/n) ยท true
Installing Playwright Test ...
Downloading browsers (npx playwright install) ...
..
Writing playwright.config.js.
Writing .github/workflows/playwright.yml.
Writing e2e-tests/example.spec.js.
Writing package.json.
โ Success! Created a Playwright Test project at <....>
The end result: three key files - a test specification, a test configuration, and a GitHub Actions workflow for continuous integration. We'll review these later.
2. Run Playwright test
The process ends by suggesting we run npx playwright test so let's do that. We see that the default test specification apparently defines 75 tests and uses 3 workers (one for each browser project), completing the entire run in 35 seconds.
$ npx playwright test
Running 75 tests using 3 workers
...
75 passed (35s)
To open last HTML report run:
npx playwright show-report
3. View the HTML Report
Excellent! The default tests ran and passed! And hey - we have a report so let's check that out!
$ npx playwright show-report
Serving HTML report at http://127.0.0.1:9323. Press Ctrl+C to quit.
The report looks something like this. Note how it provides execution time stats on a per-test (case) + per-browser (project) basis, and for the spec (test suite) as a whole.
The HTML reporting format allows you to drill down further into test steps, and break down the time taken for executing each test action. This is what the report looks like when you click on the first line item, and drill down into details. For instance, we can see the test step involved, along with the actual code that was executed from the script.
4. Validate GitHub Actions workflow
Great! We validated the test script locally and checked that it ran against all browser targets and generated a report that we were able to view locally. The setup step also created a playwright.yml file with the GitHub actions workflows for running Playwright tests and uploading reporting artifacts for post-run analysis.
Let's validate that this works by committing our code to GitHub to trigger the workflow. You can see the initial scaffolded code in my add-playwright branch with these key files:
- playwright.yml = GitHub Actions workflow
- e2e-tests/example.spec.js = Test specification
- playwright.config.js = Test configuration
Merging that branch into main triggers the GitHub action as anticipated, leading to the entire suite of tests being run in the cloud. The test run took 5m 40s with the bulk of the time (~4m) used for installing Playwright and browsers. We can also see the HTML Report Artifact available for download and local viewing.
๐ Time to celebrate! We have an end-to-end testing harness setup with continuous integration using GitHub Actions for our project. It's time to start customizing the test script and configuration to be more aligned to your project needs. We don't have time to dive into details, but let's take a quick look at the two key files to know.
5. Inspect the Configuration File
The Playwright configuration file lets you specify test options for running multiple test projects concurrently, and execution options to control how those tests are executed by Playwright. The playwright.config.js file in the default setup looks something like this (cleaned up for clarity):
// @ts-check
const { devices } = require('@playwright/test');
/**
* @see https://playwright.dev/docs/test-configuration
* @type {import('@playwright/test').PlaywrightTestConfig}
*/
const config = {
testDir: './e2e-tests',
timeout: 30 * 1000,
expect: {
timeout: 5000
},
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
workers: process.env.CI ? 1 : undefined,
reporter: 'html',
use: {
actionTimeout: 0,
trace: 'on-first-retry',
},
/* Configure projects for major browsers */
projects: [
{
name: 'chromium',
use: {
...devices['Desktop Chrome'],
},
},
{
name: 'firefox',
use: {
...devices['Desktop Firefox'],
},
},
{
name: 'webkit',
use: {
...devices['Desktop Safari'],
},
},
};
module.exports = config;
Some observations:
- The
projectsproperty defines target browsers to run the test against - The
workersproperty decides the degree of parallelization of the runs - The
reporterproperty selects the type of report (HTML) generated for the run - The
traceproperty requests that traces be recorded only on first retry
We can now customize this in many ways. Some options to try:
- Set trace to
onto trigger recording, then view the data with Trace Viewer. Read this post for more details. - Add emulator targets to
projects- validate responsiveness of app experiences. - Set screenshot to
on- make Playwright Test capture screenshots after every test for analysis. - Set video to
on- make Playwright Test record videos after every test for analysis.
Note: Every such action has a related cost and complexity. It's good to experiment with various configuration options in your local test environment - then use the reports to iterate for efficiency in time and resource usage.
6. Inspect the Test Script
Start by looking at the quickstart guide's first test script to understand key aspects of the test specification. Here is what that looks like:
const { test, expect } = require('@playwright/test');
test('basic test', async ({ page }) => {
await page.goto('https://playwright.dev/');
const title = page.locator('.navbar__inner .navbar__title');
await expect(title).toHaveText('Playwright');
});
Some observations:
-
pageis the most common fixture used in tests, creating an isolated context for running that test. -
page.locatorcreates a Locator, a view into page based on an associated selector.-
expect(..).toHaveText(..)is an example of a web-first Assertion with convenience async matchers that enforce auto-waiting for reliable testing.
-
You can learn more about these and other core concepts like test hooks, browser contexts, and explore the Playwright API, by scanning the documentation. Now, look at the scaffolded e2e-tests/example.spec.js and see if you can get an understanding of how these concepts translate to more complex end-to-end testing specifications.
7. Next Steps
The default scaffolded test provides a comprehensive end-to-end testing example for a typical "TODO" list application showcasing various features of the Playwright Test API. However, this may be overwhelming for a beginner. Instead, try these steps:
- Complete the Quickstart - learn core concepts like fixtures, hooks, and assertions.
- Get more familiar with the Command-Line Tools - I have a Tool Talk post on the topic if useful.
- Try using Codegen to author tests, and Inspector to debug execution.
- Dive into the Playwright API to start figuring out selectors and actions you can use in crafting your own test specifications.
You can now go back into the scaffolded test specification file, delete the contents, and start writing your own test actions, test cases, and test suites! Test automation for the win!
Exercise
We covered a lot today. Now it's your turn!
- Pick a PWA - for instance, a sample PWA - to experiment on.
- Instrument it to setup your first end-to-end test with Playwright.
- Validate setup locally by running tests and inspecting the reports.
- Validate the GitHub Actions workflow by committing this to a GitHub repo.
- Try out the next steps above - and start customizing your testing scripts!
And set the stage for your end-to-end testing strategy with Playwright!
Resources
- The Practical Test Pyramid - Martin Fowler, Feb 2018
- Cross Browser Testing - MDN Web Docs
- Playwright Docs - Reliable E2E Testing Framework for modern web apps
- MS Edge DevTools Protocol - Instrument, inspect, debug & profile browsers.
- MS Edge: Automate with Playwright - run Microsoft Edge in headed mode, with Playwright
- Using Origin Trials in Microsoft Edge - early testing with experimental APIs
- PWA Checklist - What makes a good PWA?
- Testing Service Workers - Chromium Dev Team, Mar 2017
- Implementing Feature Detection - writing feature detection tests
Want to learn more about Playwright? Follow @playwrightweb on Twitter and #playwrightweb here on dev.to
#playwright
And check out this earlier series of posts from me covering core Playwright concepts and tools in more detail. It is part of a planned #30DaysOfPlaywright series that is currently paused, with plans to resume with new content soon.
Want to read more content from Microsoft technologists? Don't forget to follow Azure right here on dev.to:
