Random test cases were failing when we tried to generate integration code coverage report with cypress-parallel having multiple threads

[rk011219]

I am able to generate report when there is no cypress-parallel but when I have used cypress-parallel command then code-coverage report is generating but random test cases were failing due to below errors:-

CypressError: cy.task('coverageReport') failed with the following error:

Unexpected end of JSON input

https://on.cypress.io/api/task

Because this error occurred during a after all hook we are skipping all of the remaining tests.�[0m�[90m
at (http://localhost:4210/__cypress/runner/cypress_runner.js:145679:78)
at tryCatcher (http://localhost:4210/__cypress/runner/cypress_runner.js:11318:23)
at Promise._settlePromiseFromHandler (http://localhost:4210/__cypress/runner/cypress_runner.js:9253:31)
at Promise._settlePromise (http://localhost:4210/__cypress/runner/cypress_runner.js:9310:18)
at Promise._settlePromise0 (http://localhost:4210/__cypress/runner/cypress_runner.js:9355:10)
at Promise._settlePromises (http://localhost:4210/__cypress/runner/cypress_runner.js:9431:18)
at _drainQueueStep (http://localhost:4210/__cypress/runner/cypress_runner.js:6025:12)
at _drainQueue (http://localhost:4210/__cypress/runner/cypress_runner.js:6018:9)
at ../../node_modules/bluebird/js/release/async.js.Async._drainQueues (http://localhost:4210/__cypress/runner/cypress_runner.js:6034:5)
at Async.drainQueues (http://localhost:4210/__cypress/runner/cypress_runner.js:5904:14)
From Your Spec Code:
at Context.generateReport (webpack:///./node_modules/@cypress/code-coverage/support.js:200:0)

From Node.js Internals:
SyntaxError: Unexpected end of JSON input
at JSON.parse ()
at includeAllFiles (/var/jenkins/workspace/sample_project/node_modules/@cypress/code-coverage/task-utils.js:348:28)
at coverageReport (/var/jenkins/workspace/sample_project/node_modules/@cypress/code-coverage/task.js:204:7)
at invoke (/root/.cache/Cypress/10.8.0/Cypress/resources/app/node_modules/@packages/server/lib/plugins/child/run_plugins.js:234:16)
at (/root/.cache/Cypress/10.8.0/Cypress/resources/app/node_modules/@packages/server/lib/plugins/util.js:59:14)
at tryCatcher (/root/.cache/Cypress/10.8.0/Cypress/resources/app/node_modules/@packages/server/node_modules/bluebird/js/release/util.js:16:23)
at Function.Promise.attempt.Promise.try (/root/.cache/Cypress/10.8.0/Cypress/resources/app/node_modules/@packages/server/node_modules/bluebird/js/release/method.js:39:29)
at Object.wrapChildPromise (/root/.cache/Cypress/10.8.0/Cypress/resources/app/node_modules/@packages/server/lib/plugins/util.js:58:23)
at RunPlugins.taskExecute (/root/.cache/Cypress/10.8.0/Cypress/resources/app/node_modules/@packages/server/lib/plugins/child/run_plugins.js:240:10)
at RunPlugins.execute (/root/.cache/Cypress/10.8.0/Cypress/resources/app/node_modules/@packages/server/lib/plugins/child/run_plugins.js:160:21)
at EventEmitter. (/root/.cache/Cypress/10.8.0/Cypress/resources/app/node_modules/@packages/server/lib/plugins/child/run_plugins.js:257:12)
at EventEmitter.emit (events.js:314:20)
at EventEmitter.emit (domain.js:483:12)
at process. (/root/.cache/Cypress/10.8.0/Cypress/resources/app/node_modules/@packages/server/lib/plugins/util.js:33:22)
at process.emit (events.js:314:20)
at process.EventEmitter.emit (domain.js:483:12)
at process.emit.sharedData.processEmitHook.installedValue [as emit] (/root/.cache/Cypress/10.8.0/Cypress/resources/app/node_modules/@cspotcode/source-map-support/source-map-support.js:745:40)
at emit (internal/child_process.js:877:12)
at processTicksAndRejections (internal/process/task_queues.js:85:21)

Or Sometime getting this error

CypressError: cy.task('coverageReport') failed with the following error:

Unexpected end of JSON input

https://on.cypress.io/api/task

Because this error occurred during a after all hook we are skipping all of the remaining tests.

FYI , I have followed below link in order to generate cypress integration code coverage report and we need parallelism in order to speed up test execution so I have used below command in order to run test cases in parallel:-

"cypress-parallel -s cypress:run -t 8 -d cypress/e2e/testcases --verbose --reporter cypress-mochawesome-reporter -a '"--config baseUrl=http://localhost:4210\"' --strictMode false",

https://lukas-klement.medium.com/implementing-code-coverage-with-angular-and-cypress-6ed08ed7e617

Cypress version : 10.8.0
cypress-parallel : 0.9.1

Please note we have one CI machine so we have to use cypress-parallel with multi-threading

Total test cases around 50.
Update: Seems to be related to parallel Cypress runs. Fairly consistently fails for me when running with parallelization, but succeeds when using a single (non-parallel) job.

[isstabb]

I hit this same issue and the reason is because cypress-coverage plugin merges the coverage data in a way that is not thread or multiprocessing-safe. This is fine for typical cypress runs but clashes with the way cypress-parallel works. The core problem is in the way the .nyc_output/out.json file is maintained. Under cypress-parallel multiple processes hit this code at the same time and create race conditions where data loss or corruption can occur due to concurrent read and writing to the same file.

My workaround is to add a lock around the above file access as follows:

// cypress.config.js or as described in 
// https://docs.cypress.io/guides/tooling/code-coverage#Install-the-plugin
const lockfile = require('proper-lockfile');
module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      require('@cypress/code-coverage/task')((ignored, tasks) => {
        // we use our own locking here to prevent a race condition with cypress-coverage and
        // cypress-parallel
        const myTasks = {
          ...tasks,
          combineCoverage: async (sentCoverage) => {
            const release = await lockfile.lock('/tmp/cypressCombineCoverage.lock', {
              realpath: false, // allows following symlinks and creating the file
              retries: {
                retries: 10,
                factor: 2,
                minTimeout: 100,
                maxTimeout: 1000,
                randomize: true,
              },
            });
            const ret = await tasks.combineCoverage(sentCoverage);
            await release();
            return ret;
          },
          coverageReport: async () => {
            const release = await lockfile.lock('/tmp/cypressCoverageReport.lock', {
              realpath: false, // allows following symlinks and creating the file
              retries: {
                retries: 10,
                factor: 2,
                minTimeout: 100,
                maxTimeout: 1000,
                randomize: true,
              },
            });
            const ret = await tasks.coverageReport();
            await release();
            return ret;
          },
        };
        on('task', myTasks);
      }, config);

edit: also serialized coverageReport task

[rk011219]

Thank you so much @isstabb for reply. I will try this out and will let you know the results.

[isstabb]

I made a couple changes (and updated the above comment), including also putting a lock around the coverageReport task. What I originally posted was working but I still had 1 similar failure so I added the lock to that task as well. I've not seen any failures since, but it only happens in CICD environment so hard to know for sure.

[jung-kim]

There are syntax errors in the proposed workaround so I'm not confident but below solution will fail with The task 'resetCoverage' was not handled in the setupNodeEvents method. The following tasks are registered: combineCoverage, coverageReport

This bug seriously degrades our CI pipeline and we will have to disable parallel testing

    setupNodeEvents(on, config) {
      require('@cypress/code-coverage/task')((ignored, tasks) => {
        const myTasks = {
          ...on,
          combineCoverage: async sentCoverage => {
            const release = await lockfile.lock('/tmp/cypressCombineCoverage.lock', {
              realpath: false, // allows following symlinks and creating the file
              retries: {
                retries: 10,
                factor: 2,
                minTimeout: 100,
                maxTimeout: 1000,
                randomize: true
              }
            })
            const ret = await tasks.combineCoverage(sentCoverage)
            await release()
            return ret
          },
          coverageReport: async () => {
            const release = await lockfile.lock('/tmp/cypressCoverageReport.lock', {
              realpath: false, // allows following symlinks and creating the file
              retries: {
                retries: 10,
                factor: 2,
                minTimeout: 100,
                maxTimeout: 1000,
                randomize: true
              }
            })
            const ret = await tasks.coverageReport()
            await release()
            return ret
          }
        }
        on('task', myTasks)
      }, config)
      return config
    }

[isstabb]

I accidentally left part out when adapting it to post here. I updated my comment above. Namely this was missing:

        const myTasks = {
          ...tasks,

[jung-kim]

Ah I was including the wrong object, and it seems to be working now, thanks!

[EugeneYiren]

I am getting ENOENT: no such file or directory, mkdir 'C:\tmp\cypressCombineCoverage.lock.lock'\n",

where is this temp file located? Should i be locking .nyc_output/out.json instead?

@jung-kim @isstabb

[ihorhordd]

I am getting ENOENT: no such file or directory, mkdir 'C:\tmp\cypressCombineCoverage.lock.lock'\n",

where is this temp file located? Should i be locking .nyc_output/out.json instead?

@jung-kim @isstabb

Hi, did you figure out how to solve it?

[isstabb]

@ihorhordd You just need to create the C:\tmp dir I think. You're on Windows apparently. I'm sure there's a cross-platform way to grab a tmp dir, but I didn't use it.

[ihorhordd]

@isstabb Got it. Much appreciated

[ShubhamPatilDH]

Hello @isstabb I have tried the above solution, I'm getting below error for random test.

error: "CypressError: cy.task('coverageReport') failed with the following error:\n" + '\n' + '> Lock file is already being held\n' + '\n' + 'https://on.cypress.io/api/task\n' +

Is there any solution on this?

[isstabb]

@ShubhamPatilDH The way it is written won't work if you are running this in a shared environment (e.g. like a CI/CD agent that runs multiple jobs on the same host.) You would need to change it to get a unique temp directory at the start and use it instead of /tmp. Another thought is you just need to clean up the /tmp/cypressCombineCoverage.lock file before your test run.

[ShubhamPatilDH]

@isstabb I've tried to clean /tmp/cypressCombineCoverage.lock, But it is not getting generated during execution .
No such file or directory

[isstabb]

I realized I pasted the wrong lock file, your failure was from the report stage: /tmp/cypressCoverageReport.lock

[ShubhamPatilDH]

@isstabb I've tried clearing out file '/tmp/cypressCoverageReport.lock' before test but still I'm getting 'Lock file is already being held'

[lvashistha]

@isstabb I am facing this error when using this implementation..

Unexpected end of JSON input
error	13-May-2024 13:41:32	[2] 
error	13-May-2024 13:41:32	[2] https://on.cypress.io/api/task
error	13-May-2024 13:41:32	[2] 
error	13-May-2024 13:41:32	[2] Because this error occurred during a `after all` hook we are skipping all of the remaining tests.
error	13-May-2024 13:41:32	[2] 
error	13-May-2024 13:41:32	[2] Although you have test retries enabled, we do not retry tests when `before all` or `after all` hooks fail" type="CypressError"><![CDATA[CypressError: `cy.task('coverageReport')` failed with the following error:
error	13-May-2024 13:41:32	[2] 
error	13-May-2024 13:41:32	[2] > Unexpected end of JSON input

Any idea how to resolve this??