Forums

Hi guys, something really weird is happening. I have a function to read a JSON to POST it to a database (Mongo DB). The JSON file has some keys like "$text", which starts with the "$" symbol. Now, this can't work on Mongo, so I did a recursive function that goes deep into the JSON, and remove every "$" symbol encountered.

Now, this works like a charm from Local Host. I'm able to save the JSON in Mongo with the "$" symbols removed. When I run the code in PythonAnywhere it doesn't work. This is crazy. It is exactly the same code!

I use PythonAnywhere and Mongo with no problems for weeks. Why am I failing here?

How is it failing?

The function is simply ignored and a 500 code Error arises. This is the same error code I get when I don't use the function in local host.

Normally if you get a 500 error, there will be a stack trace in the error log for your site -- the log is accessible from the "Web" page, and the most recent error will be at the bottom of the file. Is there anything useful there?

Nothing useful in the log. However, if I replace the recursive function with a For Loop which does exactly the same thing (on the same json) it works. Btw, in my experience when I work with Mongo, I get error 500 if something is wrong from the Mongo side (in our case it is probably due to the "$" symbol, which Mongo doesn't want). So in my opinion PythonAnywhere has problems with recursive functions. Is it possible?

PythonAnywhere just uses the normal Python interpreter -- we don't do anything special with respect to recursion.

Could you share the recursive code that you're using?

Sure, here it is:

def change_key_name(_dict, old_key, new_key):
 for k,v in _dict.items():
     if k==old_key:
         val = _dict[k]
         del _dict[k]
         _dict[new_key] = val
     if isinstance(v, list):
         for element in v:
             if isinstance(element, dict):
                 change_key_name(element, old_key, new_key)
     elif isinstance(v, dict):
         change_key_name(v, old_key, new_key)
 return _dict

Ok, so I really can't explain why it works locally and not on the webserver.

The problem is not recursiveness as such but the fact that you're changing the dict while iterating over it. Since Python 3.8 you'll get RuntimeError: dictionary keys changed during iteration.

I see you're using Python 3.8 in your web app, so the function will crash. Which Python are you using locally?

A quick fix, if you want to keep that funcion without further refactoring, might be iterating over a copy of the dictionary:

def change_key_name(_dict, old_key, new_key):
    for k,v in _dict.copy().items():
        if k==old_key:
            val = _dict[k]
            del _dict[k]
            _dict[new_key] = val
        if isinstance(v, list):
            for element in v:
                if isinstance(element, dict):
                    change_key_name(element, old_key, new_key)
        elif isinstance(v, dict):
            change_key_name(v, old_key, new_key)
    return _dict

Ok I'll try. Btw, I check now and I'm using python 3.6 locally (I was really convinced I was using python 3.8). Is this the reason why the function doesn't work? Just a different way of handling recursiveness in python 3.8 and 3.6?

Ok it works! Sooo, I will never doubt PythonAnywhere functionality again ahahah I really thank you!

I don't think it's how recursion is handled, rather how iteration is. Since your function implements recursion for side effects, the code is trying to change the data structure it's iterating over. I suppose that if you'd implemented that in a more "functional programming" approach, you wouldn't get the issue.

Anyway -- glad you got that working!