Forums

Hi! i want to add selenium to scrapy spider, but i get an error saying that Chrome failed to start: exited abnormally. I tested it in local and in Colab. Not sure but i think it could be because of "xvfb-run".

My code in bash is: scrapy crawl congress_gen_d -o congress_gen_d.json

Thanks!

Are you able to tell scrapy to use headless Chrome instead of virtual display?

I'm using headless chrome and virtual display, how could i run without virtual display?

Here is how to use headless chrome directly: https://www.pythonanywhere.com/forums/topic/27202/#id_post_78144 To do that in scrapy, you should check the scrapy documentation to see if there's a way to configure how it runs chrome.

Thank you, i run succesfully the first time, but then all my scripts stops working until i detelet all my fetch processes. Do you know why is hapenning that?

It happens always, and restore when kill the fetches processes. Maybe which is the command for killing processes? It could work

How is your code handling that?

When i run like glenn suggest (without pyvirtualdisplay) first time it works, but second time chromdriver crash until i kill all fetchess processes, it doesn't happen when i use pyvirtualdisplay. In order to run scrapy with selenium i need to run without pyvirtualdisplay.

It sounds like your code isn't shutting something down when it exits; what are the processes that are running after your scrapy command completes?