Forums

so it's been installed properly, yet i can't seem to even get the example in the dryscrape documentation to work properly in pythonanywhere... does anyone have any ideas?

once i get it to work with this simple example, i'll need to re-do the main code with dryscrape since selenium browser using Firefox 17 cannot render the java page properly anymore.

thanks!!

here's the error I get: webkit_server.EndOfStreamError: Unexpected end of file

here's the code:

import dryscrape
import sys

if 'linux' in sys.platform:
    # start xvfb in case no X is running. Make sure xvfb 
    # is installed, otherwise this won't work!
    dryscrape.start_xvfb()

search_term = 'dryscrape'

# set up a web scraping session
sess = dryscrape.Session(base_url = 'http://google.com')

# we don't need images
sess.set_attribute('auto_load_images', False)

# visit homepage and search for a term
sess.visit('/')
q = sess.at_xpath('//*[@name="q"]')
q.set(search_term)
q.form().submit()

# extract all links
for link in sess.xpath('//a[@href]'):
  print(link['href'])

# save a screenshot of the web page
sess.render('google.png')
print("Screenshot written to 'google.png'")

The reason that we haven't installed a newer browser is because they give strange errors and don't start correctly in our security sandbox. I think that it's unlikely that the webkit_server that dryscrape uses is working around any of the issues that we encountered.

We do want to be able to provide more up-to-date browsers, but we have not been able to fit in the time to implement that yet.

Hey there, any update on this matter? Is dryscrape already working? Thanks.

No, our virtualisation does not provide some of the stuff that webkit relies on.