[Bug]: Playwright version needed changes without dependency versions having changed

[sw-tracker]

Describe the bug

We have these storybook dependencies, and we are running storybook tests in our pipeline.

    "@storybook/addon-actions": "7.0.2",
    "@storybook/addon-coverage": "0.0.8",
    "@storybook/addon-docs": "7.0.2",
    "@storybook/addon-essentials": "7.0.2",
    "@storybook/addon-interactions": "7.0.2",
    "@storybook/addon-links": "7.0.2",
    "@storybook/jest": "0.1.0",
    "@storybook/react": "7.0.2",
    "@storybook/react-webpack5": "7.0.2",
    "@storybook/test-runner": "0.10.0",
    "@storybook/testing-library": "0.1.0",
    "storybook": "7.0.2",
    "storybook-addon-manual-mocks": "0.1.1",
    "storybook-addon-mock": "4.0.0",

We have a custom image that contains the following versions of playwright:

RUN npx playwright@1.32.3 install --with-deps msedge chromium
RUN npx playwright@1.41.2 install --with-deps --force msedge chromium
RUN npx playwright@1.42.1 install --with-deps --force msedge chromium
RUN npx playwright@1.43.1 install --with-deps --force msedge chromium

Everything was working normally until yesterday, when storybook started complaining that it doesnt have the playwright dependencies it needs: Error: Executable doesn't exist at /home/ciuser/.cache/ms-playwright/chromium-1117/chrome-linux/chrome

These are the ones we had:

du -hs ~/.cache/ms-playwright/*

461M	/home/ciuser/.cache/ms-playwright/chromium-1055
499M	/home/ciuser/.cache/ms-playwright/chromium-1097
509M	/home/ciuser/.cache/ms-playwright/chromium-1105
513M	/home/ciuser/.cache/ms-playwright/chromium-1112
5.6M	/home/ciuser/.cache/ms-playwright/ffmpeg-1008
5.6M	/home/ciuser/.cache/ms-playwright/ffmpeg-1009
221M	/home/ciuser/.cache/ms-playwright/firefox-1391
227M	/home/ciuser/.cache/ms-playwright/firefox-1438
234M	/home/ciuser/.cache/ms-playwright/firefox-1440
236M	/home/ciuser/.cache/ms-playwright/firefox-1447
247M	/home/ciuser/.cache/ms-playwright/webkit-1811
244M	/home/ciuser/.cache/ms-playwright/webkit-1967
248M	/home/ciuser/.cache/ms-playwright/webkit-1983
249M	/home/ciuser/.cache/ms-playwright/webkit-1992

We had to add this to our image to get it to work again: RUN npx playwright@1.44.0 install --with-deps --force msedge chromium.

It looks like you must have latest in one of your dependencies from playwright, which is not recommended at all. We would like to be able to have reproducible builds and pipelines, and right now this happens to us quiet often lately.

To Reproduce

This will be reproduced if you have an image with very specific versions of playwright, rather than having the latest installed.

System

Environment Info:

  System:
    OS: macOS 14.3
    CPU: (10) arm64 Apple M2 Pro
  Binaries:
    Node: 18.19.0 - ~/.nvm/versions/node/v18.19.0/bin/node
    Yarn: 1.22.22 - /opt/homebrew/bin/yarn
    npm: 10.2.3 - ~/.nvm/versions/node/v18.19.0/bin/npm
  Browsers:
    Chrome: 124.0.6367.208
    Edge: 124.0.2478.105
    Safari: 17.3

Additional context

No response

[yannbf]

Hey there @sw-tracker, thanks for reporting this issue.

As you can see, the test-runner has a dependency range for Playwright that's quite low, ^1.14.0. It should conform with whatever you have installed in your lock file. You're the first person to report this kind of issue, and I'm not sure how to reproduce it easily (a repro would be appreciated!). I also don't know if this is the behavior related to the test-runner's execution of Playwright, or if it's from Playwright itself (like if it would happen when not using test-runner, but just Playwright directly).

Could you maybe do a bit more investigation on your side? Thanks!

[sw-tracker]

Thank you! You found the problem!

If you add ^ to a dependency (which in my opinion you should not do), this will allow the dependency to be updated to minor and patch updates.

We have seen playwright updates due to storybook like this: 1.32.3/1.41.2/1.42.1/1.43.1 and now the dependency is playwright@1.44.0 due to @storybook/test-runner@0.10.0 after executing npm ls --depth=2 on my project.

Playwright happens to change browser versions when it updates its version, even minor version updates. This means we have to install new versions of the browsers every time this happens due to your dependencies. We also use playwright for E2E testing, and we never have this problem with that dependency. In our projects we want to have reproducible builds as close as possible, which means we lock our dependency versions (which I think everyone should do).

Could you please lock your playwright dependency? (remove the ^)

[sw-tracker]

I would also recommend that you create an .npmrc file and add save-exact = true to it. This will save you a lot of headaches in the future.

[sw-tracker]

Hi there, we have locked the playwright version on our package.json. Thank you!

[yannbf]

Could you please lock your playwright dependency? (remove the ^)
@sw-tracker unfortunately that would mean that anyone using the test-runner would have to install a specific version of playwright. The test-runner aims to be flexible so that you can use whatever version of playwright you'd like. Regarding which version you end up using, it's up to you, to how you manage your lock files and dedupe your dependencies. You can always use resolutions/overrides to make sure you get a specific version in your project though!