Forums

Iam trying to send tar file client side.Tar is done using subprocess and while sending it client side iam not able to send with message :Out of memory

Below is the code snippet:

process = subprocess.run(['tar', '-cvf', '-', base_dir], stdout=subprocess.PIPE)
b = BytesIO(process.stdout)
w = FileWrapper(b)
response = Response(w, direct_passthrough=False)

Able to send files upto 3 GB, But failing with 4GB

Please suggest on how to proceed.

[edit by admin: formatting]

There is a 3GB per-process memory limit. You need to work on your data in chunks.

could you please give some example on how to send file in chunks using flask

You need to use a generator to read from the output of your subprocess.run and pass that in to the Response -- this blog post has some simple example code that you can adapt.

Hi

I have tried using the generator as suggested, Even then i was not able to send large files upto 4GB.

process = subprocess.run(['tar', '-cvf', '-', base_dir], stdout=subprocess.PIPE)
CHUNK_SIZE = 8192
def read_file_chunks(test):
       with BytesIO(process.stdout) as fd:
              while 1:
                        buf = fd.read(CHUNK_SIZE)
                       if buf:
                          yield buf
                       else:
                             break
  return Response(stream_with_context(read_file_chunks(process.stdout)), mimetype='application/octet-stream')

Please suggest on how to proceed.

If the file transfer takes more than 5 minutes, your web app will be restarted and the request will fail. You can tell if this is happening by looking for HARAKIRI in your server logs.