Forums

I'd like to have a long-running process that I can interact with from a django session. Why? the process takes time to start up. I don't want to impose that time on the user. need a way to communicate with this process. Is there a way to do that that does not involve a significant time delay (which is what I get using files as the communication device).

How much delay can you tolerate?

You mean you get delays writing to a standard file and polling it? You can use named pipes.

Note: This assumes that named pipes work reliably in the PA environment - someone more familiar with the architecture than I would need to jump in to answer that one.

This is best illustrated by example. Leave this Python script running in a bash window - this would be your long-running process:

import os
import sys

PIPE_NAME = os.path.expanduser("~/test_pipe")

def watch_pipe(name):
    while True:
        with open(name, "r") as fd:
            while True:
                line = fd.readline()
                if not line:
                    break
                print "Got: '%s'" % (line.strip(),)
        print "Writer closed pipe - reopening"

def main():
    if not os.path.exists(PIPE_NAME):
        os.mkfifo(PIPE_NAME)
    watch_pipe(PIPE_NAME)

if __name__ == "__main__":
    sys.exit(main())

Then open a new browser window with a new bash window and run this script - this would be whatever needs to communicate with the long-running process:

import os
import sys
import time

PIPE_NAME = os.path.expanduser("~/test_pipe")

def main():
    with open(PIPE_NAME, "w") as fd:
        for i in range(5):
            fd.write("hello, world\n")
            fd.flush()
            time.sleep(2)

if __name__ == "__main__":
    sys.exit(main())

There are some subtleties with named pipes, such as needing to reopen when the writer closes, but their usage is well documented all over the web - Google is your friend.

@jgmdavies has a good question about what an "acceptable" delay might be. there's always a danger of premature optimisation here, so it might be worth just trying it the naive way and seeing how bad it is. there might also be some client-side ways of managing the UX of the delay, just with ajax or similar.

@Cartroo's solution using named pipes is neat, but we tend to recommend that long-running processes are run from scheduled tasks, and they would be on a different server. but that's only a recommendation.

you would have to think about how you're going to manage this process on the web servers. Assuming you have a web app with three web workers, do you want one process per worker? Or just one on the server? Because if the latter, you'll need to figure out some way to only start it if it's not already running, and to restart it if it dies for whatever reason...

How much delay can I tolerate? No specific answer. Just exploring possibilities.

You mean you get delays writing to a standard file and polling it? Yes. Different shells can run on different servers. So there must be a shared (common) file store, and there is a delay of several seconds between saving a file and it being visible on a different server.

You can use named pipes.Same problem. Both programs must be on the same server!

Since this is a low priority project I will abandon for now the search for optimized response time. I have learned much by the inquiry.

I am using splinter to run a "headless" browser. There is a fairly long delay creating a Browser instance. I must do this, as the site I'm visiting requires a login, and that is done via a jQueryUI form. I must explicitly fill the user and password fields. then "click" login.