Forums

Hi guys-

I have some code that I am running on my system here (different dataset) and I was hoping to pull a second set of data from PythonAnywhere, but I keep getting the following error :

Traceback (most recent call last):
  File "Jackson_Parcel_script.py", line 367, in <module>
    getLinks("https://ascendweb.jacksongov.org/ascend/%28biohwjq5iibvvkisd1kjmm45%29/result.aspx")
  File "Jackson_Parcel_script.py", line 33, in getLinks
    for link in bsObj.find(id="mResultscontrol_mGrid_RealDataGrid").findAll('a'):
AttributeError: 'NoneType' object has no attribute 'findAll'

My code looks like this:

for link in bsObj.find(id="mResultscontrol_mGrid_RealDataGrid").findAll('a'):
        parcel_link.append(link.text)
    #print(parcel_link)
    for test in parcel_link:
        clickable = driver.find_element_by_link_text(test)
        clickable.click()
        time.sleep(2)

I just dont understand because my code has been running for the last 3 hours on my system with no errors - any help would be great.

ES

so it looks like find(id="mResultscontrol_mGrid_RealDataGrid") is returning None on PythonAnywhere, whereas it returns an object when you run it on your own pc?

• are you definitely hitting the same urls?
• could the provider be presenting different responses to different users? our servers are in the USA, could that make a difference?

it might be worth adding a few debug prints into your code, to see what the actual pages you're getting contain...

Hi Harry and thanks for getting back to me.

I have checked the items you mentioned and "Yes" I am hitting the same urls and no there is not a different website being displayed to your servers.

After running a print PageSource I have noticed that the virtual browser never moves after the input and Enter of the search data. Code below works locally on FireFox but seems to hang up on the xvfb- run broser.

driver.get(link)
    time.sleep(3)

    inputElement = driver.find_element_by_id("mSearchControl_mParcelID")
    inputElement.send_keys(parcel_code+"*")
    inputElement.send_keys(Keys.ENTER)
    pageSource = driver.page_source
    bsObj = BeautifulSoup(pageSource)
    parcel_link.clear()
    pause = WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.ID, "mResultscontrol_mGrid_RealDataGrid")))

Any thoughts would be helpful.

RS

try using send_keys('\n\) instead of Keys.Enter? That sometimes makes a difference...

could it be that selenium is taking awhile to load the next page after you hit enter? What happens if say you put in a time.sleep and do something to make sure that the new page has been loaded? (eg: you could do a selenium.wait till a certain element in the new page is present.

Hi Harry -

I have tried changing over to a "/n" and that still doesn't execute the ENTER button - when I run the pagesource I am still on the enter parcel code screen.

https://ascendweb.jacksongov.org/ascend/%280yzb2gusuzb0kyvjwniv3255%29/search.aspx is the page I am trying to enter the data on. I am trying to enter 15-100* but I have no luck,

Any suggestions?

ES

Are you giving the browser enough time to load the new page?

Hi Glenn --

I tried adding waits but now I get the following message:

Traceback (most recent call last):
  File "Jackson_Parcel_script.py", line 372, in <module>
    driver = webdriver.Firefox()
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/firefox/webdriver.py", line 59, in __init__
    self.binary, timeout),
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
    self.binary.launch_browser(self.profile)
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 61, in launch_browser
    self._wait_until_connectable()
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 100, in _wait_until_connectable
    self._get_firefox_output())
selenium.common.exceptions.WebDriverException: Message: 'The browser appears to have exited before we could connect. The output was: b"\\n(process:9
442): GLib-CRITICAL **: g_slice_set_config: assertion \'sys_page_size == 0\' failed\\nError: no display specified\\n"'

Anythoughts? ES

[edited by admin: formatting]

I noticed that the line number is a bit further down now. Is the place that is raising this error your first attempt to create the driver? Or did you create it previously and you are somehow trying to create it again.

The driver is created near the end of the program ---- there are two functions:

Basically starting on line: 372: ----> Open web driver, Line 378 Call function which pass URL then back up to line 28 Function with the passed UrL

I also moved the driver - WebDriver(FireFox) up to line 19 and I still get the same error.

ES

"Error no display" usually means you've forgotten to use xvfb-run?

Hi Harry --

WE are almost there!!!!

At the end of the scrape of a detail page I need to go back one page and click on a new property and then scrape that page this happens x number of times in a loop. Unfortunately my browser back is not working. I have tried to different methods as you can see from the code below:

#driver.execute_script("window.history.go(-1)")
driver.back()

pause2 = WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.ID,  "mResultscontrol_mGrid_RealDataGrid")))

I have again printed the pagesource and have confirmed that I am still on the details page after executing the browser back command and giving a time.sleep(20) before the print(pageSource) to allow for page loading.

Any help on this final bug(I hope) would be great :)

how about just getting the url and going directly to the url instead of going back?

I wish I could but the url never changes it is all JavaScript and ajax.

Ok this is getting old --- so I finally figured out a work around, my code runs for 1.5 hours no problem. I shut it off and give it a new set of data to pull and all the sudden the code stops working, cann't find the attribute --- leave python anywhere come back and I get this error:

08:30 ~/Tax_Scrape $ xvfb-run python3.4 anywhere.py
xvfb-run: error: Xvfb failed to start

I would be pulling my hair out but I am already bald -- any ideas?

Thanks ES

I do not know - I kill the bash and start it over again and this time it runs --- I was hoping to run this as a scheduled script but it is starting to look like I need to watch this one for a while --- anyway call this thread closed