Strange Behavior of scheduled tasks : Forums : PythonAnywhere

Strange Behavior of scheduled tasks

I have a script (a scraping script using the firefox driver) that works perfectly fine if I run it from a bash console with the correct virtualenv activated.

Up until friday the 21st, this was also working for several years as a scheduled task. I run it within the venv like so:

/home/tsgamingllc/venvtask/bin/python /home/tsgamingllc/tsgamingdjango/project-python-django-webapp-master/newscrape.py

Now suddenly starting on friday the scheduled task receives this error below. Once again, running this same exact script from the console (with venvtask active) works perfectly fine.

Traceback (most recent call last):
  File "/home/tsgamingllc/tsgamingdjango/project-python-django-webapp-master/newscrape.py", line 51, in fetch
    browser = webdriver.Firefox(profile)
  File "/home/tsgamingllc/venvtask/lib/python3.6/site-packages/selenium/webdriver/firefox/webdriver.py", line 80, in __init__
    self.binary, timeout)
  File "/home/tsgamingllc/venvtask/lib/python3.6/site-packages/selenium/webdriver/firefox/extension_connection.py", line 52, in __init__
    self.binary.launch_browser(self.profile, timeout=timeout)
  File "/home/tsgamingllc/venvtask/lib/python3.6/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 68, in launch_browser
    self._wait_until_connectable(timeout=timeout)
  File "/home/tsgamingllc/venvtask/lib/python3.6/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 108, in _wait_until_connectable
    % (self.profile.path))
selenium.common.exceptions.WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpvqatr4sw If you specified a log_file in the FirefoxBinary constructor, check it for details.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tsgamingllc/tsgamingdjango/project-python-django-webapp-master/newscrape.py", line 256, in <module>
    fetch()
  File "/home/tsgamingllc/tsgamingdjango/project-python-django-webapp-master/newscrape.py", line 56, in fetch
    browser = webdriver.Firefox(profile)
  File "/home/tsgamingllc/venvtask/lib/python3.6/site-packages/selenium/webdriver/firefox/webdriver.py", line 60, in __init__
    capabilities.update(self.options.to_capabilities())
  File "/home/tsgamingllc/venvtask/lib/python3.6/site-packages/selenium/webdriver/firefox/options.py", line 93, in to_capabilities
    desired["firefox_profile"] = self._profile.encoded
  File "/home/tsgamingllc/venvtask/lib/python3.6/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 175, in encoded
    zipped.write(filename, filename[path_root:])
  File "/usr/lib/python3.6/zipfile.py", line 1617, in write
    zinfo = ZipInfo.from_file(filename, arcname)
  File "/usr/lib/python3.6/zipfile.py", line 507, in from_file
    st = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpvqatr4sw/lock'

2022-10-23 13:06:27 -- Completed task, took 373.83 seconds, return code was 1.

tsgamingllc | 15 posts | Oct. 23, 2022, 5:08 p.m. | permalink

That looks like perhaps a crash has broken your FF profile in some way. Perhaps try clearing out the /tmp directory to see if that helps.

glenn | 9498 posts | PythonAnywhere staff | Oct. 24, 2022, 9:49 a.m. | permalink

I cleared out the temp directory with no change.

Any other ideas why this is happening only for scheduled tasks?

tsgamingllc | 15 posts | Oct. 24, 2022, 12:46 p.m. | permalink

Are there any noticeable difference between a console running a script in an activated venv and the scheduled task environment?

Was anything changed under the hood on friday that wouldve possibly changed something after several years of running correctly?

tsgamingllc | 15 posts | Oct. 24, 2022, 4:36 p.m. | permalink

Nothing's changed under the hood recently, so this is certainly a strange one. When you run it from a console, do you use "cd" to navigate to the directory containing the script? That would be one difference that might trigger an issue like this. If, in a console, you do something like:

 # activate venv
 cd /home/tsgamingllc/tsgamingdjango/project-python-django-webapp-master/
 python newscrape.py

...then that would differ in a scheduled task, because the working directory would not be the same. You could change the scheduled task to make it the same, like this:

cd /home/tsgamingllc/tsgamingdjango/project-python-django-webapp-master/ && /home/tsgamingllc/venvtask/bin/python newscrape.py

giles | 11788 posts | PythonAnywhere staff | Oct. 25, 2022, 5:45 p.m. | permalink

That's a good idea, but unfortunately didnt seem to make a difference. Still getting the same error. Any other ideas?

tsgamingllc | 15 posts | Oct. 25, 2022, 7:33 p.m. | permalink

So, how do you run it now? (just to be sure)

fjl | 4352 posts | PythonAnywhere staff | Oct. 26, 2022, 9:14 a.m. | permalink

In the console i do:

source venvtask/bin/activate
cd tsgamingdjango/project-python-django-webapp-master/
python newscrape.py

in the scheduled task i do:

/home/tsgamingllc/venvtask/bin/python /home/tsgamingllc/tsgamingdjango/project-python-django-webapp-master/newscrape.py

But following your instructions i also tried:

cd /home/tsgamingllc/tsgamingdjango/project-python-django-webapp-master/ && /home/tsgamingllc/venvtask/bin/python newscrape.py

tsgamingllc | 15 posts | Oct. 26, 2022, 3:14 p.m. | permalink

That last one should be fine. Does it give you the same error as you reported earlier?

glenn | 9498 posts | PythonAnywhere staff | Oct. 26, 2022, 6:33 p.m. | permalink

Yes. Still the error relating to Firefox profile and tmp directory.

Console method still works perfectly.

tsgamingllc | 15 posts | Oct. 26, 2022, 7:25 p.m. | permalink

Have you tried running it with Chrome instead of Firefox?

pafk | 2973 posts | PythonAnywhere staff | Oct. 27, 2022, 8:53 a.m. | permalink