You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Be sure the ./snapshots directory is empty or deleted.
Run npm run playwright (runs the test and fails).
Expected behavior
I have two versions of what to expect.
A. (preferred)
Reference snapshots are generated with the same name as for test result attachments (without the -reference suffix, as it is now).
B.
Reporters can access the expected reference snapshot file name through testResult or testCase. Right now, we can only parse the value from testCase.errors which is inconvenient and unstable as the error message can change and is not meant for further processing.
Actual behavior
After the step 4:
✅ The test failed because the reference snapshot is missing.
See the console output, it contains the testResult and testCase objects from a custom reporter ./reporters/screenComparisonReporter.ts.
What PW gives to our custom reporter through testResult.attachments:
✅ Test result attachment name is shortened ("dac89" part), which is correct because it exceeded 60 characters name length limit (covered in #29719, released in v1.43).
❌ PW creates different snapshot reference name (Agenda-view-displays-empty-state-picture-instead-of-the-content-when-inventory-is-empty-1-chromium.png in this example, run npm run playwright-update-snapshots and see the ./snapshots content - the name is not shortened). Because of this, we can't know what reference snapshot is missing.
The only mention of the expected reference image is in testCase.errors:
message: "Error: A snapshot doesn't exist at /<your path>/pw-snapshot-name/snapshots/Agenda-view-displays-empty-state-picture-instead-of-the-content-when-inventory-is-empty-1-chromium.png, writing actual.",
stack: "Error: A snapshot doesn't exist at /<your path>/pw-snapshot-name/snapshots/Agenda-view-displays-empty-state-picture-instead-of-the-content-when-inventory-is-empty-1-chromium.png, writing actual.\n" +
' at /<your path>/pw-snapshot-name/src/AgendaView.uitest.tsx:9:7',
But in previous PW versions...
For this example, changing the PW version back to 1.42 results in a testResult.attachment names we can work with. We know, that we can take "Agenda-view-displays-empty-state-picture-instead-of-the-content-when-inventory-is-empty-1" part, add a platform name and we have recreated the reference snapshot name PW expects in ./snapshots directory (Agenda-view-displays-empty-state-picture-instead-of-the-content-when-inventory-is-empty-1-chromium.png):
There is a chance that our approach is not right. But the test result attachments from PW@1.43+ are barely usable. Are there other ways to achieve it?
Additional context
PW snapshots taken in macOS and Windows do not match. We have mixed developer environments, so we decided to have a central point for generating PW snapshots that run only on Windows.
Then we have a custom PW reporter for processing snapshot errors. It maps the PW test result output to our screen comparison service, in which developers can see the difference in a browser and resolve all issues (apply the -actual snapshot as a new source of truth or fix it in the code). But because PW test result attachment names do not match the PW snapshot generator, we cannot apply the new snapshot version if the reference is not yet created because we don't know what name PW expects in the ./snapshot directory. We need the information for renaming the -actual attachment to the snapshot reference and committing it to the ./snapshots in the repository.
The text was updated successfully, but these errors were encountered:
chekrd
changed the title
[Bug]: From within a reporter there is no easy way to know what reference snapshot is missing
[Bug]: Cannot access info about missing reference snapshot from a custom reporter
May 7, 2024
chekrd
changed the title
[Bug]: Cannot access info about missing reference snapshot from a custom reporter
[Bug]: Missing expected reference snapshot file name (PW does not pass the info to a reporter).
May 7, 2024
It is close, but not really. I don't want any change in UI mode. I want PW to provide the reporter with the name/filepath of the expected reference snapshot when PW informs about an error of missing reference snapshot. PW knows how it should be named, he is looking for it, that's why PW can say it is missing. You already have the name and everything, but it is not sent with the test result.
Version
Started in 1.43.0
Steps to reproduce
npm ci
../snapshots
directory is empty or deleted.npm run playwright
(runs the test and fails).Expected behavior
I have two versions of what to expect.
A. (preferred)
Reference snapshots are generated with the same name as for test result attachments (without the
-reference
suffix, as it is now).B.
Reporters can access the expected reference snapshot file name through
testResult
ortestCase
. Right now, we can only parse the value fromtestCase.errors
which is inconvenient and unstable as the error message can change and is not meant for further processing.Actual behavior
After the step 4:
✅ The test failed because the reference snapshot is missing.
See the console output, it contains the
testResult
andtestCase
objects from a custom reporter./reporters/screenComparisonReporter.ts
.What PW gives to our custom reporter through
testResult.attachments
:✅ Test result attachment name is shortened ("dac89" part), which is correct because it exceeded 60 characters name length limit (covered in #29719, released in v1.43).
❌ PW creates different snapshot reference name (Agenda-view-displays-empty-state-picture-instead-of-the-content-when-inventory-is-empty-1-chromium.png in this example, run
npm run playwright-update-snapshots
and see the./snapshots
content - the name is not shortened). Because of this, we can't know what reference snapshot is missing.The only mention of the expected reference image is in testCase.errors:
But in previous PW versions...
For this example, changing the PW version back to 1.42 results in a
testResult.attachment
names we can work with. We know, that we can take "Agenda-view-displays-empty-state-picture-instead-of-the-content-when-inventory-is-empty-1" part, add a platform name and we have recreated the reference snapshot name PW expects in./snapshots
directory (Agenda-view-displays-empty-state-picture-instead-of-the-content-when-inventory-is-empty-1-chromium.png):There is a chance that our approach is not right. But the test result attachments from PW@1.43+ are barely usable. Are there other ways to achieve it?
Additional context
PW snapshots taken in macOS and Windows do not match. We have mixed developer environments, so we decided to have a central point for generating PW snapshots that run only on Windows.
Then we have a custom PW reporter for processing snapshot errors. It maps the PW test result output to our screen comparison service, in which developers can see the difference in a browser and resolve all issues (apply the
-actual
snapshot as a new source of truth or fix it in the code). But because PW test result attachment names do not match the PW snapshot generator, we cannot apply the new snapshot version if the reference is not yet created because we don't know what name PW expects in the./snapshot
directory. We need the information for renaming the-actual
attachment to the snapshot reference and committing it to the./snapshots
in the repository.Environment
The text was updated successfully, but these errors were encountered: