Celery worker fails in Docker

[victor-cali]

I am trying to run a Flask application along with Celery in order to run some processes in the background. I want to do this in a containerized way. I have tried many different configurations however I cannot get the celery worker to run. Any hint, solution, idea, anything would be highly appreciated.

I am using mambaorg/micromamba:latest as base image for my application image, where I install the following (with environment.yml):

name: base
channels:
  - conda-forge
  - anaconda
dependencies:
  - python=3.10.6
  - mne=1.1.1
  - mne-features=0.2.1
  - ray-default
  - flask
  - celery
  - redis

My docker-compose file looks like this:

version: '2.4'

services:
  redis-service:
    image: redis:latest
    container_name: redis-service

  celery-service:
    image: myapp:1.0.0
    container_name: celery-service
    environment:
      - FLASK_APP=run
    volumes:
      - .:/app/
    depends_on:
      - redis-service
    command: celery -A run.celery worker --loglevel=info

  genfeatsbs:
    image: myapp:1.0.0
    container_name: myapp
    environment:
      - FLASK_APP=run
      - FLASK_DEBUG=1
    volumes:
      - .:/app/ # All my source code and my python initialization script 'run.py'
    ports:
      - '5000:5000'
    depends_on:
      - redis-service
    command: flask run -h 0.0.0.0

The __init__.py of my app looks like:

from celery import Celery, Task
from flask import Flask

from app.routes.execution_blueprint import execution_blueprint

def make_celery(app: Flask) -> Celery:
    class FlaskTask(Task):
        def __call__(self, *args: object, **kwargs: object) -> object:
            with app.app_context():
                return self.run(*args, **kwargs)
        
    celery = Celery(app.name)
    celery.conf.update(app.config["CELERY"])
    celery.Task = FlaskTask
    
    return celery

def create_app(config_file) -> Flask:
    app = Flask(__name__)
    app.config.from_object(config_file)
    
    celery = make_celery(app)
    celery.set_default()
    
    app.register_blueprint(execution_blueprint)
    
    return app, celery

And my celery configuration looks like:

    CELERY = {
        'broker_url': 'redis://redis-service',
        'result_backend': 'redis://redis-service'
    }

I have also tried adding the port (6379) and index (0).

My script to start my app and celery worker is:

from myapp import create_app

app, celery = create_app('app.config.Config')
app.app_context().push()

Not sure what is happening, but no matter what I do, I am not able to make the Celery worker run, the complete error is:

Traceback (most recent call last):
  File "/opt/conda/bin/celery", line 10, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/celery/__main__.py", line 15, in main
    sys.exit(_main())
  File "/opt/conda/lib/python3.10/site-packages/celery/bin/celery.py", line 235, in main
    return celery(auto_envvar_prefix="CELERY")
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/celery/bin/base.py", line 134, in caller
    return f(ctx, *args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/celery/bin/worker.py", line 348, in worker
    worker = app.Worker(
  File "/opt/conda/lib/python3.10/site-packages/celery/worker/worker.py", line 98, in __init__
    self.setup_instance(**self.prepare_args(**kwargs))
  File "/opt/conda/lib/python3.10/site-packages/celery/worker/worker.py", line 119, in setup_instance
    self._conninfo = self.app.connection_for_read()
  File "/opt/conda/lib/python3.10/site-packages/celery/app/base.py", line 818, in connection_for_read
    return self._connection(url or self.conf.broker_read_url, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/celery/app/base.py", line 877, in _connection
    return self.amqp.Connection(
  File "/opt/conda/lib/python3.10/site-packages/kombu/connection.py", line 201, in __init__
    if not get_transport_cls(transport).can_parse_url:
  File "/opt/conda/lib/python3.10/site-packages/kombu/transport/__init__.py", line 90, in get_transport_cls
    _transport_cache[transport] = resolve_transport(transport)
  File "/opt/conda/lib/python3.10/site-packages/kombu/transport/__init__.py", line 75, in resolve_transport
    return symbol_by_name(transport)
  File "/opt/conda/lib/python3.10/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/opt/conda/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/conda/lib/python3.10/site-packages/kombu/transport/redis.py", line 266, in <module>
    class PrefixedStrictRedis(GlobalKeyPrefixMixin, redis.Redis):
AttributeError: 'NoneType' object has no attribute 'Redis'

[Headmaster11]

Any solution on this?

[Nusnus]

Not sure what’s the problem with the original issue, but the official Dockerfile works.

A simpler Dockerfile from the new smoke tests is also fully functional.

They don’t have Flask installed, but adding it shouldn’t break anything.

LMK if it helps @Headmaster11 @victor-cali

[Nusnus]

You can also use the new pytest-celery plugin to test it.
See myworker example.