Forums

I have some implementation that is best served by pickling a pandas dataframe and storing it in a DB. This works fine if the database is sqlite but fails with a load error when it is MySQL I have found other people with similar issues on stackoverflow and google but it seems that everybodies solution is to use sql to store the dataframe. As a last resort I would go down that route but it would be a shame for this use case to do that. Anybody got a solution to get the same behaviour from mysql as sqlite here?

I simply dump the dataframe with

pickledframe = pickle.dumps(bodyframe)

and store pickledframe as a BinaryField

pickledframe = models.BinaryField(null=True)

I load it in with

unpickled = pickle.loads(pickledframe)

with sqlite it works, with mysql I get

Exception Type: UnpicklingError
Exception Value:      invalid load key, ','.

I don't know why MySQL and sqlite might treat binary fields differently. There are a number of ways where they differ in how they treat fields, though. Maybe it's just one of them.

You could try storing a base64 encoding of the pickle in a text field.

I am still scratching my head on this. Someone else on SO couldn't reproduce this with Django 2.0.5, mysqlclient 1.3.12 and pandas on python 3.6 so maybe is is not a simple sqlite vs MySQL thing. Think I will build a Centos VM to test it. I will update as I learn more.