RuntimeError: can't start new thread : Forums : PythonAnywhere

RuntimeError: can't start new thread

I am trying to run a montecarlo simulation on the web app. I am using a package called Lithops to do this and offloading the calculation to IBM cloud functions, and then they send the results back. On my machine, all this works fine. There are times when it runs on here and other times when it doesn't. I get the following error:

   File "/home/jsolis/.virtualenvs/investart/lib/python3.9/site-packages/django_q/cluster.py", line 432, in worker
 res = f(*task["args"], **task["kwargs"])
 File "/home/jsolis/Underwriting_app/UW/montecarlo/montecarlo.py", line 257, in async_montecarlo
  result=IBM_cloud_functions(Property_data)
 File "/home/jsolis/Underwriting_app/UW/montecarlo/lithops_test.py", line 824, in IBM_cloud_functions
 fexec.map_reduce(montecarlo_rent_growth, iterdata, combine_forecasts)
 File "/home/jsolis/.virtualenvs/investart/lib/python3.9/site-packages/lithops/executors.py", line 363, in  
map_reduce
map_futures = self.invoker.run_job(map_job)
File "/home/jsolis/.virtualenvs/investart/lib/python3.9/site-packages/lithops/invokers.py", line 466, in run_job
futures = self._run_job(job)
File "/home/jsolis/.virtualenvs/investart/lib/python3.9/site-packages/lithops/invokers.py", line 205, in _run_job
 raise e
 File "/home/jsolis/.virtualenvs/investart/lib/python3.9/site-packages/lithops/invokers.py", line 202, in _run_job
 self._invoke_job(job)
 File "/home/jsolis/.virtualenvs/investart/lib/python3.9/site-packages/lithops/invokers.py", line 439, in 
_invoke_job
  future = self.executor.submit(self._invoke_task, job, call_ids_range)
 File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 170, in submit
self._adjust_thread_count()
 File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 193, in _adjust_thread_count
 t.start()
 File "/usr/local/lib/python3.9/threading.py", line 874, in start
  _start_new_thread(self._bootstrap, ())
  RuntimeError: can't start new thread

I read in one of the blogs that the system is limited to 128 concurrent processes. I set the invoke_pool_threads setting to 128. I am still getting that error. Not sure what else I can do. Thanks.

jsolis | 19 posts | Sept. 8, 2022, 6:39 p.m. | permalink

Existing processes on the machine also take up some of the threads. You will need to reduce the number of threads below 128 until it works.

glenn | 9498 posts | PythonAnywhere staff | Sept. 9, 2022, 9:11 a.m. | permalink

Is there a way for me to see how many threads are running at any one time?

jsolis | 19 posts | Sept. 9, 2022, 10:51 p.m. | permalink

If I set this up as either a scheduled task or always on task would this remove this thread limitation?

jsolis | 19 posts | Sept. 9, 2022, 11:10 p.m. | permalink

On a web server, there will be 4 threads per worker. On a task or console, there will be the threads that you start. The limit is 128 on all machines.

glenn | 9498 posts | PythonAnywhere staff | Sept. 10, 2022, 1:25 p.m. | permalink

I am using DjangoQ, aside from this thread issue how to leave the cluster activated. Right now, I open up a console and run python manage.py cluster, but how do I leave it on?

jsolis | 19 posts | Sept. 12, 2022, 5:08 p.m. | permalink

You need to use Always-on task. https://help.pythonanywhere.com/pages/AlwaysOnTasks/

fjl | 4352 posts | PythonAnywhere staff | Sept. 13, 2022, 8:53 a.m. | permalink