Forums

Hello sir, madam,

<br> I wanna make an automated Selenium web scraping program and then push a notification to my Telegram bot weekly, but how can I link the path of "chromedriver.exe" that I've already uploaded on my Pythonanywhere account? And also I can't run it headerless because the website will reject the connection if without the header. Any solution, thanks in advance.

<br> Simon

You do not need to upload chromedriver as it is already there. See: https://help.pythonanywhere.com/pages/selenium how to use it.

It that possible not running these codes ?

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")

As I mentioned, the website required a header

I've tried again, I received this error this time.

Traceback (most recent call last):
  File "/home/simonnchong/main.py", line 13, in <module>
    driver = webdriver.Chrome()
  File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 76, in __init__
    RemoteWebDriver.__init__(
  File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: crashed.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

If the site rejects requests from headless chromium, then you cannot use PythonAnywhere to scrape it.

I'm getting the same error message about Chrome having crashed and "unknown error: DevToolsActivePort file doesn't exist".

Actual error message:

<pre><code>selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: crashed (unknown error: DevToolsActivePort file doesn't exist) (The process started from chrome location /usr/bin/chromium-browser is no longer running, so ChromeDriver is assuming that Chrome has crashed.) (Driver info: chromedriver=2.42.591071 (0b695ff80972cc1a65a5cd643186d2ae582cd4ac),platform=Linux 5.15.0-1017-aws x86_64)</code></pre>

My code has not been changed recently and it appears to have suddenly started failing with this error for the last few days. The last time I see a successful run was the morning of August 24, around 9:45 UTC.

The same code runs from my local machine with no issue. Did something change with chromium/chromedriver? I am on system image 'fishnchips'.

I see, thank you for your reply, have a nice day

@moviesurplus Upgrade your system image and make sure that you have "--no-sandbox" option like in the example https://help.pythonanywhere.com/pages/selenium

Thanks. The "--no-sandbox" flag did the trick.

Thanks for confirming that!