I am trying to run a montecarlo simulation on the web app. I am using a package called Lithops to do this and offloading the calculation to IBM cloud functions, and then they send the results back. On my machine, all this works fine. There are times when it runs on here and other times when it doesn't. I get the following error:
File "/home/jsolis/.virtualenvs/investart/lib/python3.9/site-packages/django_q/cluster.py", line 432, in worker
res = f(*task["args"], **task["kwargs"])
File "/home/jsolis/Underwriting_app/UW/montecarlo/montecarlo.py", line 257, in async_montecarlo
result=IBM_cloud_functions(Property_data)
File "/home/jsolis/Underwriting_app/UW/montecarlo/lithops_test.py", line 824, in IBM_cloud_functions
fexec.map_reduce(montecarlo_rent_growth, iterdata, combine_forecasts)
File "/home/jsolis/.virtualenvs/investart/lib/python3.9/site-packages/lithops/executors.py", line 363, in
map_reduce
map_futures = self.invoker.run_job(map_job)
File "/home/jsolis/.virtualenvs/investart/lib/python3.9/site-packages/lithops/invokers.py", line 466, in run_job
futures = self._run_job(job)
File "/home/jsolis/.virtualenvs/investart/lib/python3.9/site-packages/lithops/invokers.py", line 205, in _run_job
raise e
File "/home/jsolis/.virtualenvs/investart/lib/python3.9/site-packages/lithops/invokers.py", line 202, in _run_job
self._invoke_job(job)
File "/home/jsolis/.virtualenvs/investart/lib/python3.9/site-packages/lithops/invokers.py", line 439, in
_invoke_job
future = self.executor.submit(self._invoke_task, job, call_ids_range)
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 170, in submit
self._adjust_thread_count()
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 193, in _adjust_thread_count
t.start()
File "/usr/local/lib/python3.9/threading.py", line 874, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
I read in one of the blogs that the system is limited to 128 concurrent processes. I set the invoke_pool_threads setting to 128. I am still getting that error. Not sure what else I can do. Thanks.