Vfrdtyky

I’m building a project using Flask, Celery and Docker.
The idea is to run time consuming processes from rest calls using celery and most of them involve calls to external apis.

First of all the problem I have is that when I start the container, tasks don't run at all and I don’t see anything in the logs except for:

INFO/MainProcess] Connected to redis://redis:6379/0

INFO/MainProcess] mingle: searching for neighbors

INFO/MainProcess] mingle: all alone

INFO/MainProcess] celery@34569b50965e ready.

I’m using a docker container for the flask application, another for the celery worker, and another for redis as a broker (which is used by celery and by flask-socketio.

• the flask app container has on it the celery definition and instance


• the celery container uses the flaskapp image but runs this command after activating the virtualenv: celery worker -A app.controller.celery -l info
  

Then I open the celery container’s log: docker logs server_celery_1 -f in order to monitor that the tasks are running.

Then I go open postman, make a request to the rest service in the flask app which delegates the task to celery,… but nothing happens.

Here is the code involved:

def make_celery(_app):

    celery = Celery(

        _app.import_name,

        backend=_app.config['redis://redis:6379/1'],

        broker=_app.config['redis://redis:6379/0']

    )

    celery.conf.update(_app.config)





    class ContextTask(celery.Task):

        def __call__(self, *args, **kwargs):

            with _app.app_context():

                return self.run(*args, **kwargs)



        celery.Task = ContextTask

        return celery





celery = make_celery(app)







@app.route('/module/run', methods=['POST'])

@jwt_required

def run_module():

    req = request.get_json()

    module = req.get('module')

    json_input = req.get('input')

    logger.info('Running module: ' + req.get('module'))

    res = do.delay(module, json_input)

    return JSONEncoder().encode({'status': 'Task: ' + str(res) + ' submitted.'})





@celery.task()

def do(module_name, json_input):

    logger.info('____ Running ____')

    modules.run(module_name, json_input)

**BUT ** IF I open the Celery events command line app to monitor the tasks (which looks like it’s not relevant when using redis… or is it?)

celery -A app.controller.engine.celery events

and run some tasks… (and nothing happens)… when I exit the celery events window by pressing CTRL+C a twice and suddenly I see logs in the logs feed for the celery container, the logs started to appear and the tasks start to run.

What am I missing?

Thanks a lot!

I kind of fixed this rare issue by starting the celery worker using eventlet:

celery worker -A app.controller.engine.celery -l info --concurrency=2 --pool eventlet

I don't fully understand why it doesn't work with the default one though.

I'd appreciate if anyone can bring some light on this.

Thanks anyways