Forums

I am trying to pull in ocean buoy data using a third party module, buoyant. I am able to get it to work locally but I am seeing this in the logs:

2018-07-22 22:32:53,312: [2018-07-22 22:32:53,309] ERROR in app: Exception on /send [POST]
...
2018-07-22 22:32:53,314:   File "/home/rafskov/deploy/app1.py", line 51, in send
2018-07-22 22:32:53,314:     waves = sf.waves
2018-07-22 22:32:53,314:   File "/home/rafskov/.local/lib/python3.6/site-packages/buoyant/buoy.py", line 188, 
in waves
...
2018-07-22 22:32:53,315:     matches = [k for k in dictionary.keys() if prop in k]
2018-07-22 22:32:53,315: TypeError: argument of type 'NoneType' is not iterable

If you are trying to pull in data from an external site, make sure that it is on our whitelist. Free users are limited to only access our whitelist over http/https, and we only add websites with api's that are clearly documented to the whitelist. If you believe that this is the case, please provide us with a link to the documentation.

The API docs are here: https://sdf.ndbc.noaa.gov/sos/ Overview of web data services here: https://www.ndbc.noaa.gov/docs/ndbc_web_data_guide.pdf

The module I am using parses CSV flat files which are made available to the public through the API. The exact subdomain is not on the whitelist but other subdomains for noaa.gov are whitelisted 3 times.

So would a whitelist issue actually be causing this NoneType error? Thanks for letting me know. Loving PA by the way and plan to upgrade and grow with it.

If a site's not on the whitelist, then when you do your request for the data you'll get back an HTTP status code of 403 and text saying "this site isn't on the whitelist" and stuff like that -- so it's likely that the CSV parser will just return an empty list, or None, or something else to signal the error. It might be a good idea to add some error checking to your code so that you can return an error if the status code from the server isn't 200 ("OK"), though I appreciate that might be tricky if the code you're using is from a third party...

Anyway, I've whitelisted sdf.ndbc.noaa.gov -- hopefully that will fix the problem. There is one potential problem -- in order to access sites on the whitelist, the code will need to go through a proxy. If it's using requests or some other modern HTTP library, this should all be automatic (we put the details of the proxy into standard environment variables which requests will pick up) -- but if it's using something older like urllib then it might need additional configuration. Probably best to give it a go and see if it works now :-)