How to purge scheduled tasks?

[ErikKalkoken]

In my project we are running Celery 5.2.1 with Django 3.2, Redis 5.4 as broker and have a single queue.

We are sometimes retrying tasks with a countdown, e.g. when they encounter an API error. This can sometimes lead to thousands of task being retried.

To reset the system and start from scratch I want to be able to purge all those scheduled tasks. However, the usual purge command does not seam to work and I could not find any other way to do that.

I was trying to purge the scheduled tasks with this command:

$ celery -A project purge -f`. 
No messages purged from 1 queue.

However, after that I can still see the tasks with celery -A project inspect scheduled and they are still being executed.

I also tried to purge the redis DB:

$ redis-cli flushall;
OK

But that also did not work. After I ran the command the scheduled tasks are still there.

Here are some example tasks to reproduce this issue:

import random
from celery import shared_task


@shared_task
def retry_experiment():
    for _ in range(10):
        retry_experiment_me.delay()


@shared_task(bind=True)
def retry_experiment_me(self):
    random_duration = random.randint(15, 60)
    raise self.retry(countdown=random_duration)

Any help with how to purge these scheduled tasks would be very much appreciated.

[hulk66]

Have the same problem. Unable to achieve this when using Redis as a Broker. However, using RabbitMQ it works

[cgcgames]

Having the same problem, has anyone figured out what is causing this?

[MohamedKhattat]

however it depends the mode you are using cause celery doesnt use http requests by default it works as asynch tasks for that you need to kill the signal instead of deleting the task before.
generally it needs some efforts to do that

[thedrow]

The retried tasks are probably reserved tasks so purging them won't purge them from Celery's internal queue. I suggest that you revoke them all to avoid executing those tasks.

[ErikKalkoken]

Thanks. How would one revoke a task from celery's internal queue? Could you please share a concrete command example?

[marisn]

Stop celery before purging:
sudo systemctl stop celery && celery -A project purge -f && rabbitmqctl purge_queue celery && sudo systemctl start celery

[ErikKalkoken]

Thank you for your reply. However, the question is about redis as broker, not rabismq. So your solution does not apply.

[jessielw]

This was the answer

[Navifra-Chris]

stop celery and retry

http://celery.readthedocs.org/en/latest/faq.html#i-ve-purged-messages-but-there-are-still-messages-left-in-the-queue

[amaan-ahmad]

if you can find task IDs somehow for those scheduled tasks you can do this to mark it as revoked.
task will be removed from queue and will not be executed.

just tried with redis, it works.

celery -A {{your_celery_app}} control revoke {{task_id}} queue={{queue_name}}