Forums

Options for spatial support

Hello! I am new to pythonanywhere and somewhat new to python. First let me say to the people who created this site, I bow to your awesomeness. I am working on a little web service using bottle. The webservice will need to interact with some spatial data and I will need to perform 2 spatial functions. Buffer & Intersect. I have searched the forums and documentation. It seems that sqlite is supported here, so I am wondering if it will be possible to use the spatialite extension? I also see GDAL in the list, so that is another option. It looks like MySQL is supported. Does MySQL have spatial capability that I can utilize here? Are there any other options for spatial data that I have overlooked?

Thanks!

Hi mbruce,

This isn't something we have used. We installed Spatialite awhile ago because another user wanted to try it out. I'm not sure how their project actually turned out in the end. No news might be good news though!

One of the reasons people have been requesting that we add postgres is for the baked in spatial support. So perhaps this is another vote for that?

My first step in your situation would be to get a minimal working example with Spatialite. You really don't know until you try it out whether it will do what you want. If you do get it working we would love to hear about your experience back here in this thread.

Great! I will test out spatialite and let you know. Personally, I am grateful for whatever tools are at my disposal. If it has spatial capability - I can usually figure out how to work with it. I have not had the opportunity to test Postgre, but I know many people who love it- especially with ESRI supporting it and many people testing out cloud infrastructure.

Status: I tired import pyspatialite on all three consoles, does not look like is currently installed. I also tried uploading just spatialite.exe and running with popen, which I found out, I do not have permission to do. So next I tried to install pyspatialite via easy_install and pip. Both start out ok, but then error out with 'IndexError: list index out of range'. At this point I believe the problem is with the pyspatialite library (I tried it locally and got the same error) so I logged an issue on their site.

I was able to successfully import GDAL and create a buffer from/to shapefile format, so that will probably work for my initial testing. I will wrap back around to pyspatialite again as I don't want to be stuck with shapefile - it has some serious limitations. I also thought I could check to see if GDAL is willing/able to work with MySQL.

I will keep exploring to see if there are any other options..

Thanks for the update! spatialite.exe sounds like it might be a Windows binary, and PythonAnywhere's running on Linux -- is there a non-.exe binary in the package you got the exe from that might be the Linux one?

Oh! Yes I downloaded the windows version for testing locally and then uploaded it to this site. I will try again by downloading the linux one.. Thanks!

Also have been doing some testing with Shapely. Finding this to be a very nice high level python spatial library and covers most common spatial operations.

Great! Looking forward to hear how you get on.

ok, so I downloaded the Lunux version of the spatialite command line tool, uploaded it to my folder, and tried to run it from bash. It says: permission denied. Are we permitted to run programs from bash?

I also downloaded the spatialite library (an older one, v2.x), and tried to load it into sqlite. This had potential- but I got the error: Wrong elf class: elfclass32. I unpacked the newer one (version 4) but there were alot of files and there was no shared library- I think maybe it was not built) From what I have read about MySQL it has some spatial support, but for intersection it will only give you the bounding box. Did well with the buffer in Shapely, but for intersection with my line buffer - it does not support. GDAL will do the intersection - however- I will have to iterate through each point to see if it falls in the polygon. This will not do as I have many points and my end product is a webservice, which should not take all day to complete. I need a spatial index and I think I can only get that in a db.
+1 for postgre..

Wait.. dont hold me to that +1 yet. I am having a hard time finding info on Exactly what spatial support is included with Postgre out of the box and how to use it. (I am assuming you will not be installing PostGIS). Oracle also has a free version that can be installed on Linux, which comes with a scaled down version of Oracle Spatial called Locator. Locator is actually pretty good - it will do all of the expected spatial functions and you have PL/SQL and tons of documentation. Of course the downside is, if you decide to scale up to full GIS support - PostGIS is free whereas Oracle Spatial is .. eh .. not.

The permissions error is easily removed by adding execute permission to the file:

chmod u+x filename

However, this may reveal other issues, such as shared libraries missing or at different versions. Still, you'll quickly find out if you try!

Last time we investigated postgres postgis support was a trivial module install. So I don't think we would install one without the other.

@Cartroo - thanks for your help, this did remove the permission error, but now I get the error that spatialite is not a valid command. I am sure it is user error, as I do not have any experience with linux. In the past, I have downloaded the pre-built command line interface for windows, and ran it from cmd just by calling spatialite. No dependencies, I could move it around where ever I needed it, no troubles, it was very easy.

There is a newer version out now (4.0) and they did not pre-build anything for Linux. Since I could not get the older pre-built one to work, I have attempted to build the new one on my own here- but I get an error saying that it cannot find a valid C compiler. Any idea where that would be? (lol, I am pretty much in over my head but oh well, it is fun to try and figure it out).

@hansel - Very glad to hear that, I am putting my +1 back in the mix.

The PAW environment doesn't currently include compilers, I'm afraid. Sometimes the staff can compile things and make them available, but of course their time is limited so they might decide it's better spent getting Postgres going, for example.

Thank you. That is understandable.. I am having some success using OGR2OGR for the intersect, as it allows for an input clip file, so I should be able to work around spatialite. I will have to see how performance is with my dataset which is around 49,000 features- not overly large for GIS data - if I can ever get it loaded on here. I may have to break it up to upload it, then try to merge it back together..

Well, I got my test up and running and it was fairly quick to do, once I decided to go with OGR/OSGEO. Here are some reference materials on OGR that I bookmarked in case anyone finds it helpful:

Quick Tutorial: http://invisibleroads.com/tutorials/gdal-shapefile-points-load.html

Helpful Documentation: http://gdal.org/python/

Gotchas: http://trac.osgeo.org/gdal/wiki/PythonGotchas

And here is my little bottle webservice test:

import bottle
import json, os, osgeo, ogr
from bottle import route

@route('/')
def index():
    """ Display welcome & instruction messages """
    return "<p>Welcome to my simple spatial bottle.py powered web service !</p> \
    <p>How to invoke the web service? Just pass lat/long pairs to create a route, as below. \
    The webservice will buffer the line then intersect it with my point file, and you will \
    get a summary of total points per type in json format:\
<ul><li>http://mbruce.pythonanywhere.com/buffer?s=LINESTRING(-76.615 39.417,-76.642 39.478,-76.627 39.444)</li>\
</ul>"

@route('/buffer')
def buffer():

    # Store HTTP GET arguments
    googleroute = bottle.request.GET.get('s' , default=None)

    # Execute WebService specific task
    # create lineshape
    route = ogr.CreateGeometryFromWkt(googleroute)
    buff = route.Buffer(0.0025, 30)

    #open the source file, create a lyer for OGR to use
    input = '/home/mbruce/mysite/TestPointData_WGS84.shp'
    shapeData = osgeo.ogr.Open(input)
    layer = shapeData.GetLayer()

    #set filters.  This filter will only intersect the bounding box.
    #to get more granular you have to check intersection on each
    #feature

    layer.SetSpatialFilter(buff) # set buffer polygon as spatial filter


    #this loop creates an attribute filter for each distinct type, and
    #returns an array of the count of records for each distinct type
    count = 1
    list = []
    while (count < 6):
        layer.SetAttributeFilter("ACCIDENT_S = '0" + str(count) + "'")
        x = layer.GetFeatureCount()
        list.append(x)
        count = count + 1

# Sending back json
return json.dumps(
{
'result': list
},
    indent=4)

application = bottle.default_app()

Thanks, that's really useful!

I'm not sure I understood, I downloaded the source for spatialite 4.0 and tried to do a "make install" and it didn't work due to the lack of a c compiler.

The library is in my home folder, is there a way an admin can run:

./configure
make
make install

for this library? is there a way I can precompile it on my computer? (I actually got it from ubuntu's repository) sudo apt-get install spatialite-bin

same for geos, it's not installed

pysqlite I tried to install using: pip install --user pysqlite but that failed because of the missing c compiler as well

Yes, I think we will need the latest spatialite source compiled, but I am on windows. Do you have access to a Linux operating system? It couldn't hurt to try and compile it. There may still be issues with the shared library, but won't know until we try.

I could really use spatialite about now - my code is running very very slow and I am just getting a bounding box for my intersection. I still have to iterate through each point in the bounding box to see if it intersects my buffer.

I was under the impression that when used with OSGEO it would perform the geometry intersection with filter but that is not the case. It seems I can use OSGEO.OGR to test each point for an intersection but that will take way too long.

I've installed it on my computer with apt-get (yes I have ubuntu 12.04) I have no clue as to how to move the instalation onto PA, could you walk me through it?

https://www.pythonanywhere.com/forums/topic/429/#id_post_3058

Looks like it's getting installed :)

You can't in general move compiled code from one machine to another, especially on virtualised environments like PA. In specific cases you can get it to work if both machines share the same architecture and have exactly the same versions of all the libraries installed, but this is unlikely to be the case between PA and Ubuntu 12.04.

I'm afraid it's just a case of waiting until the PA boffins get around toe sorting it out (or adding a compiler to the system so we can compile on there directly), but at least it's on the todo list now.

Hi there,

just in case anyone else comes across this old thread, we have pyspatialite preinstalled for python2, and the spatialite C library is available for any python 3 libraries that want to use it (eg geodjango). More info here