Forums

I have a web scraping script that needs to use a proxy, so i installed Seleniumwire which adds additional features to to standard Selenium chromedriver to enable the use of proxies with user names and passwords.

My Script runs fine when i run one instance from bash or task, however I need to run multiple instances concurrently. When I run multiple tasks all fail to run with various errors occurring trying to open various pages.

I believe the issue is that all instances of the script are all using the same instance of Chromedriver.

Anyone successfully run multiple instances using selenium-wire?

After further testing it appears the issue is not specific to selenium-wire but with multiple instances of chromedriver.

I created a simple script using chromedriver that scapes a website every 15 seconds. Works fine on its own but if I try to run multiple instances at the same time it fails.

Yes. We have a process limit for each user on each machine and chrome starts a huge number of processes, so a number of chrome instances running together can easily breach the process limit.

Thanks Glen, that makes sense. Anyway to increase the process limit. Is that something I could pay an additional fee for?

We've doubled your processes number limit -- that may contribute to the faster CPU burn rate, so you may need to upgrade your CPU quota.

Hello. I got the same issue. May I know how many chrome instances I am able to open at the same time? Is it possible to increase my processes number limit? Thanks!

By default we allow 256 simultaneous processes per user -- I don't know exactly how many processes are being spawn by a chrome instance; I've doubled the limit for you, but it's advised to properly close browsers after they perform their job.