Chunked uploads with busyboy not working

[paulm17]

Hi there,

I got uploady working with multer quite easily, except it doesn't seem to support chunked uploading and well, it's based on busyboy. So why not go straight to the source.

I'm finding that I'm only getting the initial chunked saved.

I'm using nestjs and if I use the example: nestjs/nest#5979 (comment)

I get the console:

error: The "chunk" argument must be one of type string or Buffer. Received type object

Otherwise I'm trying to get it working on my own:

import { Controller, Post, Req, Res } from '@nestjs/common';
import path = require('path');
import os = require('os');
import fs = require('fs');

@Controller('dam')
export class DamController {
  //constructor() {}

  @Post('upload')
  async uploadFile(@Req() req, @Res() res) {
    const Busboy = require('busboy');
    const busboy = new Busboy({
      headers: req.headers,
    });

    const writtenFiles = [];
    let chunks;

    busboy.on('file', function (fieldname, file, filename, encoding, mimetype) {
      const saveTo = `${process.env.TMP_DIR}/files/${path.basename(filename)}`;

      const writeStream = fs.createWriteStream(saveTo);
      file.pipe(writeStream);

      const promise = new Promise((resolve, reject) => {
        file.on('end', () => {
          writeStream.end();
        });
        writeStream.on('finish', resolve);
        writeStream.on('error', reject);
      });

      writtenFiles.push(promise);
    });

    busboy.on('finish', function () {
      res.writeHead(200, { Connection: 'close' });
      res.end("That's all folks!");
    });
    req.pipe(busboy);
  }
  catch(err) {
    console.log(err.message);
  }
}

Doesn't work, like I say. I only get the first chunk.

Can someone please advise on both issues? I'd rather get the interceptor working, rather than trying to do this myself.

Thanks

[yoavniran]

Hey @paulm17, Im not familiar with busboy so I cant be sure why its not working.

If I had to guess, Id say it might have to do with the header(s) busboy expects to receive to tell it that the upload is chunked and therefore to expect additional requests that will contain the next parts.

For example, something along a header called: "X-Unique-Upload-Id"
This way it can group all requests with that same ID so when the last chunk arrives, the file can be combined out of all chunks.

This might help: https://stackoverflow.com/a/54551254/107976

[paulm17]

Thanks for the reply. I saw that earlier.

But another problem I have. Is that I am now able to send the chunked info. But only via the body. Not the header.

Can you please give an example using the tus-uploader how to do that?

Thanks

[yoavniran]

@paulm17 Im not sure what exactly the issue is and how Tus is related?
Do you have Tus implemented on the server?

Adding headers is very easy with Uploady. Here's an example from the chunked-uploady storybook:
(adds a unique header for the chunked upload)

import ChunkedUploady, { useRequestPreSend } from "@rpldy/chunked-uploady";
import UploadButton from "@rpldy/upload-button";

const UploadButtonWithUniqueIdHeader = () => {
    useRequestPreSend((data) => {
        return {
            options: {
                destination: {
                    headers: {
                        "X-Unique-Upload-Id": `rpldy-chunked-uploader-${Date.now()}`,
                    }
                }
            }
        };
    });

    return <UploadButton id="upload-button"/>
};

const App = () => {
    return <ChunkedUploady
        destination={{ url: "my-server.com/chunked" }}
      >
        <UploadButtonWithUniqueIdHeader/>
    </ChunkedUploady>;
};