Forums

Hi,

I'm using Selenium Chromedriver to scrape data from 2 different websites. The function works fine for the first website but when I try to use the same function and driver session to scrape the 2nd website it doesn't work. I've taken a screenshot of the driver and the screen shows a message with "unrecoverable error" in it rather than the webpage I am trying to scrape. This is the screenshot: https://ibb.co/2Zq3P0V

This code runs without any problems on my local device on PyCharm.

This is the code to create the driver:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome(options=chrome_options)

Where are you running your code on PythonAnywhere? Could you have a look at appropriate logs or provide us with the traceback error message?

There are no errors from this. It just returns a null object from the website I'm trying to scrape. I think the site is most likely blocking requests from pythonanywhere. The site is https://flightaware.com/

Yes, that's certainly a possibility; many site whose administrators do not want them to be scraped will block incoming requests from IP addresses that they know belong to cloud computing environments.