Forums

On Jan 30 one of my download scripts suddenly stopped working, generating the SSL error message "certificate verify failed" in from the file python3.3/ssl.py. I am quite confused about this because - it has previously been working flawlessly for more than six months - other scripts accessing other services in the same manner still work - the same script works fine when it is executed from own computer Anyone else who have experienced anything similar.

What site are you trying to access?

It is harvestapp.com. But I don't have a beginner's account, so it shouldn't be a whitelist issue.

I can get http://harvetsapp.com from my PythonAnywhere account just fine, so I think it may be an issue with the url library you're using (or your use of it). How are you getting the URL?

I am using httplib2 with python3.3. And it works fine with other sites from PythonAnywhere. And with harvestapp.com from my computer. And also with harvestapp.com from PythonAnywhere until Jan 29.

Could it some versioning issue? Was there perhaps some upgrade made at PythonAnywhere on Jan 29 or 30?

We did an upgrade on 3 February, so that's probably not the cause. Let me investigate a little further,

Right, it looks like I can repro the problem -- is this the error you're getting?

Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib2
>>> h = httplib2.Http(".cache")
>>> (resp_headers, content) = h.request("http://harvestapp.com", "GET")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1570, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1367, in _request
    (response, content) = self.request(location, redirect_method, body=body, headers = headers, redirections = redirections - 1)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1570, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1317, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1252, in _conn_request
    conn.connect()
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1044, in connect
    raise SSLHandshakeError(e)
httplib2.SSLHandshakeError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

It looks like you can work around the problem by disabling certificate validation, like this:

>>> h = httplib2.Http(".cache", disable_ssl_certificate_validation=True)
>>> (resp_headers, content) = h.request("http://harvestapp.com", "GET")
>>>

That feels a little unsatisfactory, though, so I'm investigating further.

Hmm. OK, my best guess is that on or around 30 January they changed their HTTPS certificate to one with a root certificate that we don't trust by default. We'll look into ways of solving that (perhaps we need to update our default set of trusted certificates) but in the meantime, I think adding the disable_ssl_certificate_validation=True to your code is probably the best solution.

Thanks for the help. It seems lika plausible reason.

Unfortunately I can not get it to work with the workaround you are suggesting. Maybe because I run python3.3 and not 2.7.2 like you do. This is the traceback I get when I do your test:

File "/usr/local/lib/python3.3/dist-packages/httplib2/init.py", line 1211, in request (response, new_content) = self.request(info['-x-permanent-redirect-url'], "GET", headers = headers, redirections = redire ctions - 1)

File "/usr/local/lib/python3.3/dist-packages/httplib2/init.py", line 1156, in request self.disable_ssl_certificate_validation)

File "/usr/local/lib/python3.3/dist-packages/httplib2/init.py", line 829, in init check_hostname=True)

File "/usr/lib/python3.3/http/client.py", line 1187, in init raise ValueError("check_hostname needs a SSL context with "

ValueError: check_hostname needs a SSL context with either CERT_OPTIONAL or CERT_REQUIRED

That's odd. Looking at the code for httplib2, there's definitely a difference between the Python 2.7 and Python 3.3 versions that would cause that problem; the Python 3.3 code sets a parameter called check_hostname to True. I think it's this bug, which was reported on httplib2 back in 2011 and still isn't fixed.

That makes me wonder if the best solution isn't to use a different Python HTTP library. For example, requests works fine:

>>> import requests
>>> requests.get("http://harvestapp.com")
<Response [200]>

If you really have to use httplib2, there is a patch proposed in the bug report which I've tested against my own PythonAnywhere account, and it looks like it works. Here's what to do if you want to apply it:

• Make your account's version of httplib2 editable by doing a local install: from a bash console, run pip3.3 install --user -I httplib2==0.8
• Edit the file /home/lennartohlsson/.local/lib/python3.3/site-packages/httplib2/__init__.py

• At around line 826, you'll see the following code:

  http.client.HTTPSConnection.__init__(
          self, host, port=port, key_file=key_file,
          cert_file=cert_file, timeout=timeout, context=context,
          check_hostname=True)
  

• Replace it with this:

  http.client.HTTPSConnection.__init__(
          self, host, port=port, key_file=key_file,
          cert_file=cert_file, timeout=timeout, context=context,
          check_hostname=not disable_ssl_certificate_validation)
  

That should fix things. But I'd definitely suggest switching to requests instead of httplib2, it looks like it's better-maintained.

Switching libs seems the right way to go. Thanks for the help.

Yup, it's probably safer than patching httplib2. If you'd had a load of code that depended specifically on it then it might have been worth the risk of patching it, but if you don't then switching is definitely a better choice.