Forums

I have created a script to scrape some elements from a website.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service

url = "https://..."
ser = Service(r"C:\...")

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-gpu")
driver = webdriver.Chrome(service = ser, options=chrome_options)
driver.get(url)

unseen = driver.find_elements(By.XPATH,"//div[@class='vpRowHoriz flagUnseen vpRow DragElt']")
print(selen_ungelesen[0].text)
....

It works perfectly fine, as long as the --headless argument is not given. There are several suggestions on stackflow how to address this problem (change language by --lan, change window size, add a fakeuser, ...). None of them work for my script and it seemed to be a known problem. stackoverflow

Since the options "no-sandbox" ,"--headless", "--disable-gpu") are mandatory in pythonanywhere, I would like to try another webdriver. Firefox/Geckodriver seems to do the job regardless the --headless. Since on your selenium help page you only mention chromedriver I'd like to know if other webdrivers possible?

Firefox does not work correctly on PythonAnywhere.

Thanks for your reply! So there is simply no workaround for this issue? One of the main reasons for me to get the paid account was to use webscraping in combination with your task section.

What kind of errors are you getting when you use our default settings with Chrome?

My default settings before pythonanywhere were completely without this part:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-gpu")

As result from that I simply got the desired text from the element. (By the way, the "print(selen_ungelesen[0].text)" should of course be "print(ungelesen[0].text". Sorry for that.)

However, as soon as I add the -- headless-argument I get:

"list index out of range"

On stackflow I have read that --headless does not support all languages. Maybe thats the underlying problem but changing the language via Chrome_options did not help either.

Sorry, is ungelesen the same as unseen in your first post?

Yes, it is. I am really sorry for the confusion! :( :(

No problem. I assume that this part ser = Service(r"C:\...") is not relevant (it has a Windows-style path which wouldn't work on PA), right? It's hard to tell, since we can't reproduce that without knowing the url and all the actions you want to perform. Did you try the window-size option for the driver?

Yes, I tried the "window-size" option, it did not work. However, I have now combined it with "time.sleep()" and everything works fine. Thanks for the support!!

thanks

cant find elemnt with this optins so i wana more optins please to try , i tried time sleep and evry think not work

@mzr2yi We need more details to help you.