Forums

Using multiprocessing.Queue in a Flask program?

For demonstration purposes I've tried to create a simple Flask program, which starts a background process to process long running tasks. I planned the communication between the Flask app and the background process can be done using multiprocessing.Queue. It works fine on my own PC, but at here the queue's get waits forever even though there are some messages in the queue. Interesting: the qsize() can see the waiting messages, but get can't.

By the way, if I put a message in that queue next to starting the background process, the background can get it. When the q.put() placed in the routed function, the consumer process can't read out that message.

from flask import Flask
from multiprocessing import Process, Queue
...
def consumer(pq):
    while True:
        msg = pq.get()
        print(f"Consumer: {msg}")


q = Queue()
p = Process(target=consumer, args=(q,))
p.run()
q.put("some message") # it is OK, consumer can get it
...
@app.route("/")
def index():
     q.put("other message")  # this increases the length of the queue, it visible 
                             # in the consumer process, but the get() waits indefinitely, 
                             # although pq.qsize() shows the correct length of the queue
     return "Hello"

You cannot start threads in web apps so your queue processor is not running. See https://help.pythonanywhere.com/pages/AsyncInWebApps/

Actually this isn't true, see: the first message has been delivered, so the Queue is working between the main and the consumer processes. But I forgot that the application here is running under a web server + wsgi software, while I ran it alone, if it were a standalone python application. When I run it from uwsgi, the Queue object works the same (wrong) way as here. (I don't understand, why can it see the correct length, but can't read the message, so it's not your problem ;) )

Thanks for letting us know!

O.K., it is working now! :) Actually the line "p = Process(...)" contained option daemon=True. I've changed it to daemon=False and ... it is working. :) Without daemon=True the parent process can leave orphaned children behind, but it runs in a container, I hope, this will not a problem.

That should not be a problem.