Forums

Flask app with tensorflow sessions not refreshing

I'm new to using flask, so be forgiving. I have a trained model from tensorflow that I want to load into my web app, so I need to start a one-time session to initialize my layer variables. I want to be able send post requests to the web app, so I need to start another session in the @app.route("/") function. However, if I have a session in the @app.route("/") function, the web app takes a long time to refresh, and eventually times out. This problem does not happen if I have the same two sessions in global scope.

To clarify, I think I've isolated the problem in my app.py file. The following code works:

from flask import Flask
import tensorflow as tf

app = Flask(__name__)

tf_config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=2)

# the first session
with tf.Session(config = tf_config) as sess:
    sess.run(tf.constant([1,2,3])

# a second session immediately following the first one in global scope
with tf.Session(config = tf_config) as sess:
    sess.run(tf.constant([4,5,6])

@app.route("/", methods = ["GET"])
def do_something():
    return "hello from tf"

But moving the location of the second tf session makes the refresh time out:

from flask import Flask
import tensorflow as tf

app = Flask(__name__)

tf_config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=2)

# the first session in the same location as before
with tf.Session(config = tf_config) as sess:
    sess.run(tf.constant([1,2,3])

@app.route("/", methods = ["GET"])
def do_something():
    # the second session in local scope; this times out the refresh
    with tf.Session(config = tf_config) as sess:
        sess.run(tf.constant([4,5,6])
    return "hello from tf"

Unfortunately, the latter is what I want in my web app. How would I fix this?

Edit: I almost forgot: I'm using Python3.6, and I already did

pip3.6 install --user --upgrade tensorflow
deleted-user-3609829 | 7 posts | March 20, 2018, 3:55 p.m. | permalink

Hmmm, I know Flask but not Tensorflow, so perhaps between us we can work out what's going on!

The first thing to mention is that threads won't work inside websites on PythonAnywhere. Might that be causing the problem? Is there any way to set up Tensorflow so that it doesn't spawn threads?

Staff giles | 11800 posts | PythonAnywhere staff | March 20, 2018, 6:24 p.m. | permalink

There is a way to configure tensorflow so that it doesn't spawn threads. I changed the tf_config variable to

tf_config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1, device_count={'CPU': 1})

which limits the number of threads to 1 and also the number of CPUs to 1, but it still doesn't refresh (no error messages in logs; it just times out).

deleted-user-3609829 | 7 posts | March 20, 2018, 6:47 p.m. | permalink

Could you try adding print statements after each line in your view to trace what's going on? They'll go to the server log, and hopefully that will at least tell us which line is blocking and causing the timeout.

Staff giles | 11800 posts | PythonAnywhere staff | March 20, 2018, 6:54 p.m. | permalink

I added the print statements as follows:

@app.route("/", methods = ["GET"])
def do_something():
    print("Point A")
    with tf.Session(config = tf_config) as sess:
        print("Point B")
        sess.run(tf.constant([1,2,3]))
        print("Point C")
    print("Point D")
    return "hello from tf"

The server log shows point A and B in that order. Points C and D were not reached.

deleted-user-3609829 | 7 posts | March 20, 2018, 7:04 p.m. | permalink

Sorry, I was unclear. Could you put a print in between 

with tf.Session(config = tf_config) as sess:

...and...

    sess.run(tf.constant([1,2,3]))

...?

Staff giles | 11800 posts | PythonAnywhere staff | March 20, 2018, 7:08 p.m. | permalink

Yes, I edited the previous post.

deleted-user-3609829 | 7 posts | March 20, 2018, 7:09 p.m. | permalink

Hmmm, so it's the tf.Session that's taking a long time. A bit of googling around on that suggests problems with CUDA, but I don't think that can be related, as we're not using GPUs here (and also the first tf.Session seems to be working without problems).

Maybe try a different tack -- I know this isn't the way you want to set it all up in the long run, but just for testing, what happens if you put all of the tensorflow stuff into the view? For example:

from flask import Flask
import tensorflow as tf

app = Flask(__name__)

@app.route("/", methods = ["GET"])
def do_something():
    tf_config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1, device_count={'CPU': 1})
    with tf.Session(config = tf_config) as sess:
        sess.run(tf.constant([4,5,6])
    return "hello from tf"
Staff giles | 11800 posts | PythonAnywhere staff | March 20, 2018, 7:15 p.m. | permalink

The same thing happens - the refresh times out. The same points are logged as before (A and B).

deleted-user-3609829 | 7 posts | March 20, 2018, 7:23 p.m. | permalink

It seems that if I do

@app.route("/", methods = ["GET"])
def do_something():
    do_something_else()
    do_yet_another_thing()
    return "hello from tf"

def do_something_else():
    tf_config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1, device_count={'CPU': 1})
    with tf.Session(config = tf_config) as sess:
        sess.run(tf.constant([1,2,3]))

def do_yet_another_thing():
    tf_config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1, device_count={'CPU': 1})
    with tf.Session(config = tf_config) as sess:
        sess.run(tf.constant([1,2,3]))

then it also works. However, the function do_something_else() is called whenever I make a post request (correct me if I'm wrong), whereas I just need it for initialization.

deleted-user-3609829 | 7 posts | March 20, 2018, 7:34 p.m. | permalink

quick thing to check- if you were to run the tf.Session and sess.run() code from a console, how long does it take to run?

Staff conrad | 4232 posts | PythonAnywhere staff | March 21, 2018, 4:49 p.m. | permalink

When I click the "Run" button on my app's main script, it takes around 2-3 seconds for the console to launch. When I call the do_something() function, it returns almost immediately.

deleted-user-3609829 | 7 posts | March 21, 2018, 4:51 p.m. | permalink

Re: the last code you posted -- yes, you're right, do_something_else will be called on every post. But the fact that it works sounds really odd. Just to confirm that I understand properly, if you do this:

@app.route("/", methods = ["GET"])
def do_something():
    print("Point A")
    tf_config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1, device_count={'CPU': 1})
    print("Point B")
    with tf.Session(config = tf_config) as sess:
        print("Point C")
        sess.run(tf.constant([4,5,6])
        print("Point D")
    return "hello from tf"

...then it blocks at point B, and doesn't return.

But if you do this (simplified version of your code):

@app.route("/", methods = ["GET"])
def do_something():
    do_something_else()
    return "hello from tf"

def do_something_else():
    print("Point A")
    tf_config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1, device_count={'CPU': 1})
    print("Point B")
    with tf.Session(config = tf_config) as sess:
        print("Point C")
        sess.run(tf.constant([1,2,3]))
        print("Point D")

...then it works. Is that right?

Staff giles | 11800 posts | PythonAnywhere staff | March 22, 2018, 12:48 p.m. | permalink