Forums

I have a web app that needs to do a bit of web-scraping. I am trying to use Selenium or Splinter, but I seem to keep running into problems.

With Splinter, my code looks like this:

from pyvirtualdisplay import Display
from splinter import Browser, driver

display = Display(visible=0, size=(800, 600))
display.start()

for retry in range(3):
        try:
            browser = Browser("firefox")
            break
        except Exception as e:
            print("Failed to start browser [%s]" % e)
            time.sleep(3)

The output looks like this:

Failed to start browser [Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.#012] Failed to start browser [Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.#012] Failed to start browser [Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.#012]

With Selenium, my code looks like this:

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()

for retry in range(3):
        try:
            driver = webdriver.Firefox()
            print("Successfully got webdriver.Firefox()")
            break
        except Exception as e:
            print("%s" % e)
            time.sleep(2)

And the output looks like this:

Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details. Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details. Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.

See our help page about using selenium on PythonAnyhwere: http://help.pythonanywhere.com/pages/selenium/

Hi Glenn, Thanks for the response. So, on the link you posted I see in bold letters: Don't use selenium from your web app

So I guess that answers that...

Now I guess I am wondering about Splinter--will that work with my web app or no?

Basically no. Also as a free user you can only access a whitelist- you wouldn't be able to access just any website.

OK bummer--thanks for the responses!