[bug] upgrading from 1.22.3 to 1.22.4 introduces - "No such file or directory" (only on Gitlab CI)

[KatSick]

Describe the bug

When we upgraded moon from 1.22.3 -> 1.22.4 error: "No such file or directory" started to appear only on Gitlab CI.

If you need more logs or some specific ones, let me know. It also does not depend on the task. e.g. it fails to run Jest, tsc, packemon, webpack

Steps to reproduce

I can't find the way to reproduce it locally (on macOS arm) :(

Expected behavior

No error

Screenshots

[DEBUG 11:07:11.313] log  Creating std-ts:typecheck command (in working directory /builds/org1/clients/project1/packages/std-ts)  log.target="moon:runner" log.module_path="moon_runner::runner" log.file="crates/core/runner/src/runner.rs" log.line=270
[DEBUG 11:07:11.313] log  Creating std-ts:build command (in working directory /builds/org1/clients/project1/packages/std-ts)  log.target="moon:runner" log.module_path="moon_runner::runner" log.file="crates/core/runner/src/runner.rs" log.line=270
[DEBUG 11:07:11.313] moon_process::command_inspector  Running command tsc --build --pretty --verbose  env_vars={"MOON_PROJECT_SOURCE": "packages/std-ts", "CLICOLOR_FORCE": "0", "MOON_TARGET": "std-ts:typecheck", "PROTO_IGNORE_MIGRATE_WARNING": "true", "FORCE_COLOR": "0", "PROTO_GO_VERSION": "*", "MOON_WORKSPACE_ROOT": "/builds/org1/clients/project1", "PROTO_RUST_VERSION": "*", "MOON_PROJECT_ROOT": "/builds/org1/clients/project1/packages/std-ts", "MOON_CACHE_DIR": "/builds/org1/clients/project1/.moon/cache", "MOON_WORKING_DIR": "/builds/org1/clients/project1", "PROTO_NPM_VERSION": "10.2.4", "PWD": "/builds/org1/clients/project1/packages/std-ts", "PROTO_INTERNAL_SCHEMA_VERSION": "*", "MOON_PROJECT_SNAPSHOT": "/builds/org1/clients/project1/.moon/cache/states/std-ts/snapshot.json", "PROTO_PNPM_VERSION": "*", "PROTO_NO_PROGRESS": "true", "PROTO_BUN_VERSION": "*", "PROTO_NODE_VERSION": "20.9.0", "PATH": "/builds/org1/clients/project1/packages/std-ts/node_modules/.bin:/builds/org1/clients/project1/packages/node_modules/.bin:/builds/org1/clients/project1/node_modules/.bin:/root/.proto/tools/proto/0.31.4:/root/.proto/shims:/root/.proto/bin:/root/.moon/bin:/root/.proto/tools/node/globals/bin:/builds/org1/clients/project1/node_modules/.bin:/builds/org1/clients/node_modules/.bin:/builds/org1/node_modules/.bin:/builds/node_modules/.bin:/node_modules/.bin:/usr/local/lib/node_modules/npm/node_modules/@npmcli/run-script/lib/node-gyp-bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/gradle/bin", "MOON_PROJECT_ID": "std-ts", "PROTO_DENO_VERSION": "*", "PROTO_YARN_VERSION": "*", "PROTO_VERSION": "0.31.4", "COLUMNS": "80", "PROTO_PYTHON_VERSION": "*", "LINES": "24"} working_dir="/builds/org1/clients/project1/packages/std-ts"
[DEBUG 11:07:11.313] moon_process::command_inspector  Running command packemon build --addEngines --addExports --declaration  env_vars={"MOON_PROJECT_SOURCE": "packages/std-ts", "CLICOLOR_FORCE": "0", "MOON_TARGET": "std-ts:build", "PROTO_IGNORE_MIGRATE_WARNING": "true", "FORCE_COLOR": "0", "PROTO_GO_VERSION": "*", "MOON_WORKSPACE_ROOT": "/builds/org1/clients/project1", "PROTO_RUST_VERSION": "*", "MOON_PROJECT_ROOT": "/builds/org1/clients/project1/packages/std-ts", "MOON_CACHE_DIR": "/builds/org1/clients/project1/.moon/cache", "MOON_WORKING_DIR": "/builds/org1/clients/project1", "PROTO_NPM_VERSION": "10.2.4", "PWD": "/builds/org1/clients/project1/packages/std-ts", "PROTO_INTERNAL_SCHEMA_VERSION": "*", "MOON_PROJECT_SNAPSHOT": "/builds/org1/clients/project1/.moon/cache/states/std-ts/snapshot.json", "PROTO_PNPM_VERSION": "*", "PROTO_NO_PROGRESS": "true", "PROTO_BUN_VERSION": "*", "PROTO_NODE_VERSION": "20.9.0", "PATH": "/builds/org1/clients/project1/packages/std-ts/node_modules/.bin:/builds/org1/clients/project1/packages/node_modules/.bin:/builds/org1/clients/project1/node_modules/.bin:/root/.proto/tools/proto/0.31.4:/root/.proto/shims:/root/.proto/bin:/root/.moon/bin:/root/.proto/tools/node/globals/bin:/builds/org1/clients/project1/node_modules/.bin:/builds/org1/clients/node_modules/.bin:/builds/org1/node_modules/.bin:/builds/node_modules/.bin:/node_modules/.bin:/usr/local/lib/node_modules/npm/node_modules/@npmcli/run-script/lib/node-gyp-bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/gradle/bin", "MOON_PROJECT_ID": "std-ts", "PROTO_DENO_VERSION": "*", "PROTO_YARN_VERSION": "*", "PROTO_VERSION": "0.31.4", "COLUMNS": "80", "PROTO_PYTHON_VERSION": "*", "LINES": "24"} working_dir="/builds/org1/clients/project1/packages/std-ts"
Error:   × No such file or directory (os error 2)

Environment

Linux x64 runners on CI

[milesj]

It kind of seems like tsc or packemon bins can't be found. If you list all files in node_modules/.bin, do they exist, and are they writable?

[milesj]

Is this still happening?

[KatSick]

yeah :( even with 1.23.4

the interesting point, that MR is just an moon upgrade without any other changes, but it fails. (locally works)
also, before executing moon, we are running npm ci. but it works perfectly on 1.22.3

cilog.txt

[milesj]

@KatSick Hrmm this is weird. Based on the error, and the lack of error code/header, this error is happening outside of moons official error layer. Something is getting bubbled up.

Can you run CI with --concurrency 1 --log trace and see if that pinpoints where exactly it's happening.