Forums

ISSUE: I have an output (from popen) that I need to stream/display line by line on a flask served page. PA is serving the full content all at once, that's not what I want. I'm aiming to have row by row displayed as what is happening on my local flask server testing. I've also tried this on an open file/log I've been reading too (not just the popen)

GOAL: I want to "emit" lines from either popen or log file read to the flask served webpage (as they are happening, some minor delay is ok)

MY ATTEMPT: I've been looking deep into PA Buffering article to solve my issue (since everything is on NGINX servers and there's load balancing happening) -- its not fixing my issue...Here's what I've tried this:

return Response(inner(), mimetype='text/html', headers={'X-Accel-Buffering': 'no'})

NOTES:

• All strings end with a \n on the yield statements

• The inner() function is kicking off the whole popen stuff too -- it works (I even did a sleep.time(1) for each yielded line) but it's not happy/working for me

ASK: Is there something else I should be doing here to get it to work properly? I'm totally lost here -- can someone please copy/paste a sample of code that would work?

Much appreciated!

-Doug Jr.

This code worked for me:

from flask import stream_with_context, Flask, Response
import time

app = Flask(__name__)

@app.route('/stream')
def stream():
    def generate():
        for ii in range(100):
            time.sleep(1)
            yield f'{ii}\n'
    response = Response(stream_with_context(generate()))
    response.headers['X-Accel-Buffering'] = 'no'
    return response

Bear in mind that this will tie up a worker for the entire duration of the sending and will cause your web app to be restarted after the connection has been held open for 5 min. So you'll need to include some code to ensure that that does not happen.

Thank you Glenn + PA staff -- this did help a bit to shed some light on my issue...I haven't solved it yet completely in my env but I think I have a clue/reason why...

I think there's something funky happening within python 2.7 here with flask and stream_with_context() inside of the PA flask site. I did the sample code above on a python 3.8 flask site (in PA) and it works great-- I know that it's possible.

I know, I know, I really need to get with the program and up my game to 3.X instead of holding on to the past -- I'm pretty sure this is due to some legacy crypto libraries that only existed in 2.7 and not yet in the 3.X world. I'm currently working through the port/changes to get into 3.8 (3.9 is a bit too green).

As you can maybe guess, I have 2 different PA accounts/sites where I'll do some testing and report back to this thread (with respect to python 2.7 and 3.8 and streaming content to the page). It might also be some strange "patching" I'm doing within 2.7....I haven't looked too deep into that matter but I'll consider it.

Thanks so much -- be back soon

Hmmm... the plot thickens a bit here....I may need some more sleuthing from PA staff or community.

UPDATE: I was able to have streaming content (lines) into a PA flask served page with both python2.7 and python3.8 with respect to the dropdown option inside the PA "Web" dashboard page.

CONTINUED ISSUE: Both of my PA sites are running python 2.7 -- one of my PA accounts/site isn't working correctly and I think it's possibly linked to the configuration/environment of python-- site A (the wrong site) is using a venv and site B (flushes are working correct) is the vanilla flask PA deployment (no venv). Further details regarding "not working"; it's where it flushes all the lines to the page instead of incremental flushes-- see URLs listed below.

NOTES: the version of flask module (for python 2.7) seem to be the same/close enough-- so that's probably not an issue or at odds here...I did a "python -m pip freeze >> current_modules.txt" to see if there's something going on (perhaps Werkzeug is an issue?)-- here's the result...

WORKING VERSION (SITE B):

-f /usr/share/pip-wheels
...(lots of modules/batteries included)...
Flask==1.1.1
Werkzeug==0.16.0

(if you want to see what I'm aiming to achieve) http://www.yasutoko.com/stream11

NOT WORKING VERSION (SITE A):

-f /usr/share/pip-wheels
...(other modules)...
Flask==1.1.2
Werkzeug==0.10.4

(if you want to see what the broken version looks like) https://douglasmccormickjr.pythonanywhere.com/stream11

ANOTHER CONFIG ISSUE: it appears that the older PA site (called site A) has the python version dropdowns of:

 2.7
 3.4
 3.5

but site B (newer, since I purchased it later within PA) has these options for python versions:

 2.7
 3.5
 3.6
 3.7
 3.8

Maybe that has something to do with it? I have no clue...It's strange to see different options in the menu/interface

POTENTIAL SOLUTION/ISSUE??: it just dawned on me that maybe http (=site B) vs https (=site A) is the problem? I'm grasping at straws here.

MY NEXT STEPS: Any suggestions of ways to help debug/solve would be greatly appreciated. I might just inevitably upgrade and alter some code chunks but I'd love to know why something was broken instead of brushing it under the carpet. Thanks!

I think you may have hit the nail on the head with your last post there!

It looks like there is a problem streaming HTTPS sites at the moment; HTTP works fine so long as you set the X-Accel-Buffering header, but HTTPS is doing some kind of buffering even with that header. We have a tentative fix, which we will hopefully be able to roll out tomorrow.

The rollout took a little longer than expected, but it's now live for the server where your website is running. Could you try now and see if the problem is fixed?

One thing to note -- the fix is only on a subset of our servers right now, so if you delete your website and re-create it, or if you create a new website, it might wind up on an unpatched server so the problem would still be there. If all goes well we will have patched all of the servers by the end of next week.

UPDATE: Apologies for a delay getting back to you and the team -- sidebar; you guys are incredible. I'm so impressed with what I can create/do within the tools/services you provide (ok, I'll stop gushing)

Yes, it works great on my site! The internal change on your servers (granting the allowance for HTTPS requests to stream the page via "yield" statements) definitely fixed things for me.

Thank you so much for everything you've done-- I'll get back to coding.

-Doug Jr.

That excellent to hear, many thanks for the kind words, and for confirming that it works now :-)