Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cleanup of cron_schedule database table doesn't cleanup non-existing jobs #38217

Open
5 tasks
hostep opened this issue Nov 24, 2023 · 9 comments · May be fixed by #38693
Open
5 tasks

Cleanup of cron_schedule database table doesn't cleanup non-existing jobs #38217

hostep opened this issue Nov 24, 2023 · 9 comments · May be fixed by #38693
Assignees
@fredden @ananth-iyer
Labels
Area: Catalog Component: Cron Issue: Confirmed Gate 3 Passed. Manual verification of the issue completed. Issue is confirmed Priority: P2 A defect with this priority could have functionality issues which are not to expectations. Progress: PR in progress Reported on 2.4.6-p3 Indicates original Magento version for the Issue report. Reproduced on 2.4.x The issue has been reproduced on latest 2.4-develop branch
Projects
High Priority Backlog

Comments

@hostep
Copy link
Contributor

hostep commented Nov 24, 2023 •
edited

Preconditions and environment

  • Magento version 2.4.6-p3 (but most likely all versions will be affected)

Steps to reproduce

Explanation below how to trigger this with a cronjob from inside core Magento. But the most likely cases will come from 3rd party modules containing cronjobs, that were installed for a while and then removed again.

  1. Have a Magento shop with MSI modules enabled
  2. Have a working crontab and have it running for a couple of hours
  3. Inspect the cron_schedule table and notice that the job inventory_in_store_pickup_sales_send_order_notified_emails is mentioned many times with status success
  4. Now, disable all MSI modules in app/etc/config.php and run bin/magento setup:upgrade
  5. Keep the crontab running for a few more hours
  6. Inspect your cron_schedule table again

Expected result

It's expected that all jobs that were successful and having run more than 60 minutes ago are removed from the cron_schedule table.

Actual result

See that there are still jobs inventory_in_store_pickup_sales_send_order_notified_emails in the table with status success and being older then 60 minutes. Those don't get cleaned up because the MSI modules are no longer active and calling getJobs() over here doesn't return jobs from inactive (or removed) modules.

Additional information

Ideally, the cronjob cleanup should also figure out which jobs are in the cron_schedule table that are no longer found in the system and clean those up as well.
Or maybe this can be added in the existing Recurring data script that exists in the Magento_Cron module, as then it doesn't need to be checked every minute when the cron actually runs. Since removing and disabling modules requires a bin/magento setup:upgrade call anyway, it might be a good idea to only then check for outdated entries in that table and remove them...

And since the cron_schedule table receives a lot of queries the entire day, it's beneficial to performance that it contains as few rows as possible. Old data should be cleaned up automatically.

I found this in a shop where we have hundreds of entries from 2021 in the cron_schedule table for jobs that for a long time no longer existed.

Release note

No response

Triage and priority

  • Severity: S0 - Affects critical data or functionality and leaves users without workaround.
  • Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
  • Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
  • Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
  • Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.
@m2-assistant M2 Assistant
Copy link

m2-assistant bot commented Nov 24, 2023

Hi @hostep. Thank you for your report.
To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

Join Magento Community Engineering Slack and ask your questions in #github channel.
⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.
🕙 You can find the schedule on the Magento Community Calendar page.
📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

@m2-community-project m2-community-project bot added this to Ready for Confirmation in Issue Confirmation and Triage Board Nov 24, 2023
@hostep hostep changed the title cron_schedule database table doesn't cleanup inexisting jobs cron_schedule database table doesn't cleanup non-existing jobs Nov 24, 2023
@hostep hostep changed the title cron_schedule database table doesn't cleanup non-existing jobs Cleanup of cron_schedule database table doesn't cleanup non-existing jobs Nov 24, 2023
@engcom-Bravo engcom-Bravo added the Reported on 2.4.6-p3 Indicates original Magento version for the Issue report. label Nov 24, 2023
@ananth-iyer
Copy link
Member

@magento I am working on this

@engcom-Dash engcom-Dash self-assigned this Jan 12, 2024
@m2-assistant M2 Assistant
Copy link

m2-assistant bot commented Jan 12, 2024

Hi @engcom-Dash. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

    1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
    1. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
    1. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
    1. Verify that the issue is reproducible on 2.4-develop branch
      Details- Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
      - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
      - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

@engcom-Dash
Copy link

Hi @hostep

Thanks for reporting and collaboration.

It is not advisable to disable MSI modules.

As per the adobe dev docs, Disabling the Inventory Management modules can cause an unstable system and result in various issues.

Please refer the below devdocs for more information :

https://experienceleague.adobe.com/docs/commerce-admin/inventory/get-started/install-update.html

Thanks

@engcom-Dash engcom-Dash added the Issue: needs update Additional information is require, waiting for response label Jan 12, 2024
@m2-community-project m2-community-project bot moved this from Ready for Confirmation to Needs Update in Issue Confirmation and Triage Board Jan 12, 2024
@hostep
Copy link
Contributor Author

hostep commented Jan 12, 2024

@engcom-Dash: that's not the point, this was just the easiest way I could find to demonstrate the problem 😉

This will most likely happen in the real world where you had a third party module installed (which contains a cronjob) for a couple of months and then decide to remove it again. Afterwards, you'll still find entries from that no longer existing module in your cron_schedule table and that is polluting the database.

Please re-read the description and if it's not clear, try to pull in one of your colleagues and see if they understand it better. And if it's still not clear, then let me know what's not clear so I can try to describe it better.

@engcom-Dash
Copy link

Hi @hostep

Thanks for reporting and collaboration.

Verified the issue in magento 2.4 dev instance and the issue is reproducible.

After disabling MSI modules , seeing the inventory_in_store_pickup_sales_send_order_notified_emails in the table with status success when we run the cron job and checked in the cron_schedule table.

Please refer the attached screenshots.
Screenshot 2024-01-23 at 11 21 50 AM
38217_Confiremd

@engcom-Dash engcom-Dash added Component: Cron Issue: Confirmed Gate 3 Passed. Manual verification of the issue completed. Issue is confirmed Reproduced on 2.4.x The issue has been reproduced on latest 2.4-develop branch Area: Catalog labels Jan 23, 2024
@m2-community-project m2-community-project bot removed the Issue: Confirmed Gate 3 Passed. Manual verification of the issue completed. Issue is confirmed label Jan 23, 2024
@m2-community-project m2-community-project bot moved this from Needs Update to Confirmed in Issue Confirmation and Triage Board Jan 23, 2024
@m2-community-project m2-community-project bot added Issue: Confirmed Gate 3 Passed. Manual verification of the issue completed. Issue is confirmed and removed Issue: needs update Additional information is require, waiting for response labels Jan 23, 2024
@github-jira-sync-bot
Copy link

✅ Jira issue https://jira.corp.adobe.com/browse/AC-10910 is successfully created for this GitHub issue.

@m2-assistant M2 Assistant
Copy link

m2-assistant bot commented Jan 23, 2024

✅ Confirmed by @engcom-Dash. Thank you for verifying the issue.
Issue Available: @engcom-Dash, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

@github-jira-sync-bot
Copy link

❌ You don't have permission to export this issue.

@engcom-Dash engcom-Dash added the Priority: P2 A defect with this priority could have functionality issues which are not to expectations. label Jan 24, 2024
@m2-community-project m2-community-project bot added this to Dev In Progress in High Priority Backlog Jan 24, 2024
@hostep hostep linked a pull request May 3, 2024 that will close this issue
Open
5 tasks
@m2-community-project m2-community-project bot moved this from Dev In Progress to Pull Request In Progress in High Priority Backlog May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fredden fredden

@ananth-iyer ananth-iyer

Labels
Area: Catalog Component: Cron Issue: Confirmed Gate 3 Passed. Manual verification of the issue completed. Issue is confirmed Priority: P2 A defect with this priority could have functionality issues which are not to expectations. Progress: PR in progress Reported on 2.4.6-p3 Indicates original Magento version for the Issue report. Reproduced on 2.4.x The issue has been reproduced on latest 2.4-develop branch
Projects
High Priority Backlog
  
Pull Request In Progress
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove jobs no longer referenced in the codebase fredden/magento2
6 participants
@hostep @fredden @ananth-iyer @engcom-Bravo @engcom-Dash @github-jira-sync-bot