Can't use selenium in my web app : Forums : PythonAnywhere

Can't use selenium in my web app

I want to use selenium in my webapp. (please don't say it won't be recommended because of the performance issue..)

I've payed for my account to use the selenium and here is the issue : "403 Forbidden" error or "Connection Refused" error in logs

What's the wrong with my code?

Here is my code :

from bs4 import BeautifulSoup
from selenium import webdriver

@app.route("/devselenium")
def devselenium():
    driver = webdriver.Firefox()
    try :
        driver.get('http://www.google.com')
    finally:
        driver.quit()
    return redirect(url_for('index', mode='welcome'))

deleted-user-9500896 | 7 posts | Feb. 23, 2021, 6:19 a.m. | permalink

Try using headless Chrome instead of Firefox. That needs recent version of selenium and virtualization feature (that I'm enabling for you now). Then you can use it like so:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
browser = webdriver.Chrome(options=chrome_options)

try:
    browser.get("https://www.google.com")
    print("Page title was '{}'".format(browser.title))

finally:
    browser.quit()

pafk | 2973 posts | PythonAnywhere staff | Feb. 23, 2021, 10:11 a.m. | permalink

I was trying that code actually, but I've got the error like the below :

(myvenv) 13:33 ~/mysite $ cat test.py
from selenium import webdriver
from bs4 import BeautifulSoup

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
browser = webdriver.Chrome(options=chrome_options)

try:
    browser.get("https://www.google.com")
    print("Page title was '{}'".format(browser.title))

finally:
    browser.quit()

(myvenv) 13:33 ~/mysite $ python3.8 test.py
Traceback (most recent call last):
  File "test.py", line 8, in <module>
    browser = webdriver.Chrome(options=chrome_options)
TypeError: __init__() got an unexpected keyword argument 'options'

deleted-user-9500896 | 7 posts | Feb. 23, 2021, 1:34 p.m. | permalink

It looks like you need to upgrade Selenium in your virtualenv -- just "pip install --upgrade selenium" should fix the issue.

giles | 11800 posts | PythonAnywhere staff | Feb. 23, 2021, 2:28 p.m. | permalink

Sorry but still same error occurred :(

(myvenv) 16:15 ~/mysite $ pip install --upgrade selenium
Looking in links: /usr/share/pip-wheels
Requirement already up-to-date: selenium in /home/cyrooka/myvenv/lib/python3.8/site-packages (3.141.0)
Requirement already satisfied, skipping upgrade: urllib3 in /home/cyrooka/myvenv/lib/python3.8/site-packages (from selenium) (1.26.2)
(myvenv) 16:15 ~/mysite $ python3.8 test.py
Traceback (most recent call last):
  File "test.py", line 8, in <module>
    browser = webdriver.Chrome(options=chrome_options)
TypeError: __init__() got an unexpected keyword argument 'options'

deleted-user-9500896 | 7 posts | Feb. 23, 2021, 4:16 p.m. | permalink

Try running your code with

python test.py

glenn | 9498 posts | PythonAnywhere staff | Feb. 23, 2021, 5:34 p.m. | permalink

It works! Thanks for your support :D Btw, when I checked the python --version, it says the version : 3.8 then why the error occurred with the command : python3.8? What's the difference between them?? @.@

deleted-user-9500896 | 7 posts | Feb. 23, 2021, 10:06 p.m. | permalink

Oops.. sorry, my original problem is still opened :( I want to use my code in my webapp, not in py. My test code :

@app.route("/devselenium", methods=["GET", "POST"])
def devselenium():
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--disable-gpu")
    browser = webdriver.Chrome(options=chrome_options)

    try:
        browser.get("https://www.google.com")
    except Exception as e:
        print(e)
    finally:
        browser.quit()

And the error is comming :

There was an error loading your PythonAnywhere-hosted site. There may be a bug in your code.

Error code: Unhandled Exception

Debugging tips
The first place to look is at your web app page to ensure that there are no errors indicated there.
Next, check your site's server and error logs for any messages — you can view them here:
cyrooka.pythonanywhere.com.error.log
cyrooka.pythonanywhere.com.server.log
You can find helpful tips on the PythonAnywhere help site:
There's an ImportError in the logs
"403 Forbidden" error or "Connection Refused" error in logs
Database connection errors
There are many more helpful guides on our Help pages
If you get completely stuck, then drop us a line at support@pythonanywhere.com, in the forums, or using the "Send feedback" link on the site, with the relevant lines from your logs.

Can you please help me?

deleted-user-9500896 | 7 posts | Feb. 23, 2021, 10:16 p.m. | permalink

hmm- can I confirm that running it using "python" from within your virtualenv works?

if so, are you using the same virtualenv python for your webapp?

conrad | 4232 posts | PythonAnywhere staff | Feb. 24, 2021, 10:25 a.m. | permalink

The webapp is working fine without the selenium code. Yes, I'm using the same virtualenv as well. I'm facing the problem in very first step when I use the selenium in the webapp :(

deleted-user-9728347 | 4 posts | Feb. 24, 2021, 11:28 a.m. | permalink

*just to clarify, running the same selenium code from the console works?

conrad | 4232 posts | PythonAnywhere staff | Feb. 25, 2021, 12:05 a.m. | permalink

Oops, I found my mistake.. Now the webapp is working fine with selenium library. Thanks for your excellent support! :D

deleted-user-9500896 | 7 posts | Feb. 25, 2021, 5:10 a.m. | permalink