Forums

I'm running code that I was running a few months ago, and the code is the same (I'm scraping the same website, which has been slightly updated), but it's orders of magnitude slower than it was previously. I think the functions that have gotten significantly slower include urllib.request.urlopen; they may also include driver.get() from selenium and requests.get().

The two options here are that something is different on the pythonanywhere side or that the website is doing something differently; unfortunately, I wasn't previously testing with any other websites, so I don't have a previous comparison here to test this out with. Has anyone run into anything like this?

No-one has reported anything like that to us, the support team -- though, of course, that doesn't mean that it's not happening for anyone else, just that no-one else has mentioned it :-)

It is worth mentioning that one thing that sites that don't want to be scraped sometimes do is create a blocklist of IP addresses -- or some other indicator to pick up scrapers -- and then serve pages slowly when they match that indicator. It's kind of a halfway house between blocking scrapers entirely, and letting them scrape without problems. So it's certainly possible that the site you're scraping is doing that.

Thank you. That's helpful.