Personally I would really recommend staying away from threads in web apps, and especially staying well away from processes (i.e. os.fork()
). Webservers tend to use both threads and processes for their own purposes, and creating instances of your own for any other purpose can introduce subtle problems.
Also, HTTP in general isn't really designed for long-running requests - whilst they work, they tend not to be very robust. The chances of something going wrong in the middle of your request are much higher.
I suggest having a background work queue where your request submits items and then uses Javascript in the browser to periodically poll the progress. This has the boon that if something goes wrong with a request, the Javascript can simply retry again a few seconds later. Also, the user could in principle close their browser and come back later to see the progress. This approach is probably more effort, but if you care much about scalability and robustness then I really wouldn't suggest doing too much back-end work in the context of a single HTTP request.
This sort of thing is easier with beanstalkd
but until the PA devs have chance to get background daemons and the like working I wouldn't suggest that at present. You could do something with a background task and a scheduled job to restart it as necessary, but presently there isn't really a way to make it bullet-proof - the PA devs have spoken about support for background daemons, but they seem pretty busy right now so I wouldn't expect it for a little while.
Sounds to me like you're just doing IO in the thread (i.e. talking to remote email servers), so for now maybe you can just use non-blocking IO in a single thread instead of spawning multiple threads? In general non-blocking IO is a better approach than threading anyway - lighter on system resources and much more predictable. See the select
module for examples.