Visual Regression Testing with Playwright

Instructor: [0:00] Common requirement in web application testing is to make sure that your web application looks visually exactly as you expect.

[0:06] For example, we want to make sure that this button is this color of blue with this text of white and this row is in there. We can do that quite easily with Playwright as it supports visual comparisons.

[0:17] Within an example spec file, we bring in the usual suspects, which are test and expect and create a before each to just navigate to the localhost:3000.

[0:26] We create a new test, and within this test, we will be doing the visual regression testing for this particular button.

[0:32] Doing the test is quite easy as we can get a screen shot of the page by using awaitpage.screenshot.

[0:38] Then, we can expect it to match a given screenshot that we have saved in the past under a given snapshot name, which we are calling buttonbase.png.

[0:48] Now, of course, when we start off, this particular image will not exist and the test will fail. However, when it fails because of a lack of an image, it actually creates that image for us. You can see that in the error details.

[1:02] Now this image is created in a folder right next to where our spec file is. We can see that within the file explorer. This is a PNG file, and this is what it looks like. With this image file in place, the next time that we run the test, it will pass because of course, the screenshot will match what already exists.

[1:22] Now, if you were to modify the page for example, remove the rows from the button component and then run the test again, this time we will expect the test to fail because the new image does not match what we have saved previously.

[1:37] For these cases, it will also generate unexpected actual and a diff file and we can open them up externally, or if you just scroll down, you can see them in line with the test results. This is the screenshot that we received in this test run.

[1:52] This is the screenshot that already existed, and this is the difference. The grayed-out portion of the diff image is what matched and what had a difference has been highlighted in red. In our case, the rows is missing and the button has shrunk a bit.

[2:07] Now, you could move the actual file to be the snapshot, but there's an easy way to do that by simply adding --updatesnapshots to NPX Playwright test. With this command, the tests will pass and the snapshots will be updated. You can see the updated image in the snapshots folder right next to the spec file.

[2:28] A common issue with doing visual regression testing, is that it can be a bit too aggressive. Fortunately, you can soften that up by providing a custom threshold. You can either provide it to your individual tomatchsnapshot calls as an extra options object, or alternatively, you can do it globally within your playwright.config.ts file.

[2:49] Within the expect section, you can provide a default for tomatchsnapshot. The threshold value is actually just passed in to the underlying Library called pixelmatch.

[2:59] It takes values from zero being extremely strict to one being completely lenient, and the default is at .1.