I am using selenium to scrape some data as it is not bale to scrape with lxml. The problem is it run half way, for about 300 pages and hang there. there is no error or something to notice me. What actually happen?
Forums
selenium run half way
do you know if it's error-ing or just hanging? i would check two things:
- for the site you are scraping, does it have an access limit? Maybe they start blocking you if you hit them heavily
- are you opening/creating selenium browsers/virtual displays without closing them?
conrad
|
4232
posts
|
PythonAnywhere staff
|
just hanging...no errors 1. It did not seems so. Because if I restart, it can access again. 2. I did close the browser and virtual display every time i open the browser and close it before open the next one. I reduce the list to 250, it can work perfectly. Then I have to continue from 251 to 500. It is better it can run till the end that has 1000 pages
even I split into a few runs, it still hang. Every time in different pages
