Sending files collection as form-data

kt_

So I was tasked with creating an integration with external REST API. This external API has documentation, but no examples of requests.

Everything seems to work fine, but I can't seem to work out how to upload files. The docs say:

Send POST request as FormData.
Send files as:

files[0].file (type: file) - the file
files[0].name (type: string) - changes name of file

I'm not sure how to set this up in the request. Collections in post FormData are usually created by simply providing multiple parameters with the same name, e.g.:

languages: pl
languages: en
languages: de

But here these parameters should be key-value pairs.

I tried the following:

Name the parameters file.file and file.name. Didn't work.
Order parameter names: file0.file, file0.name. Didn't work.
Simply upload the file as "file" parameter. Didn't work.
Simply upload the file as file.file. Didn't work.

I'm using Postman for this. I'll appreciate any pointers.

dkf

@kt_ I think they're trying to say use normal file upload with multiple files. The way you usually encode that in an HTTP message is pretty horrific: it's a sequence of attachment sections in a multipart MIME message. With that level of nasty, you probably need to dump a real request by a browser to a file and work from there. Or that's what I did (and I really don't want to revisit that code! Though it did have the fun outcome of overloading a service at work when I used my automation to push a million files in inside the space of about 20 minutes… )

kt_

@dkf said in Sending files collection as form-data:

@kt_ I think they're trying to say use normal file upload with multiple files. The way you usually encode that in an HTTP message is pretty horrific: it's a sequence of attachment sections in a multipart MIME message. With that level of nasty, you probably need to dump a real request by a browser to a file and work from there. Or that's what I did (and I really don't want to revisit that code! Though it did have the fun outcome of overloading a service at work when I used my automation to push a million files in inside the space of about 20 minutes… )

Ho-oly shit! I hope you're not right. I asked them to provide a sample request, so should learn the truth, soon.

If this is the case, I guess it's a serious reason to choose a different vendor.

PleegWat

@kt_ There are three basic ways to send data on a HTTP request, recognizable by the mime type specified in the Content-Type header:

appliction/x-www-form-urlencoded. Typically used for traditional web forms without file uploads.
multipart/form-data, which is a MIME multipart message as @dkf is referring to. Typically used for traditional web forms with file uploads.
application/xml, application/json. Used for single-page apps. The data to be sent is encoded in an xml or json document by javascript, and sent via an AJAX request.

See also https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST

In this case, they are probably looking for the multipart/form-data. The names they want (value of the name= field in the Content-Disposition header of the part is almost certainly the exact string files[0].file and files[0].name. Sending an application/x-www-form-urlencoded message might work if they are using standard server software and you specify those exact same names, but if they're specifying form data then they probably only tested with multipart.

Lorne Kates

If you're looking into POSTing multiple files, read up on multipart/form-data boundary. Pay attention to the format, and the number of spaces. It'll fuck you up good otherwise.

Example of multipart/form-data

I am wondering if anyone can share with me an example of multipart/form-data that contains: Some form parameters Multiple files

anotherusername

@kt_ pretty sure you're literally supposed to have <input type="file" name="files[0]">. And if that doesn't work, then I'd try without the [0] part.

JBert

Back when I was experimenting with multipart/form-data I came across this Postman bug:

Multipart POST not working for me · Issue #576 · postmanlabs/postman-app-support

I'm having an issue posting a multipart form request with POSTMan where some of the items have binary data. I think POSTMan has an error with the boundary. I want to post a form that has three ...

Make sure you don't have a fixed Content-Type header in your request's Headers tab.

It also shows a few samples of using curl, though I don't recall if I tested those.

JBert

Also, a while ago I've built a client in .NET based on this StackOverflow answer which uses a bunch of string operations and Stream writes so that it doesn't need to construct the entire form-data request in memory:

Upload files with HTTPWebrequest (multipart/form-data)

Is there any class, library or some piece of code which will help me to upload files with HTTPWebrequest? Edit 2: I do not want to upload to a WebDAV folder or something like that. I want to sim...

However, if you can build it in .NET and in memory then using the MimeKit Nuget package will make it far easier (untested snippet though):

var formData = new Multipart("form-data");
var myFile = new MimePart ("text", "plain") {
    Content = new MimeContent (new MemoryStream(new byte[] { 79, 75}), ContentEncoding.Default),
    ContentDisposition = new ContentDisposition (ContentDisposition.FormData),
    ContentTransferEncoding = ContentEncoding.Default,
    FileName = "myfile.txt"
};
formData.Add(myFile);

var output = new MemoryStream();
string contentTypeHeader = formData.ContentType.ToString();
formData.WriteTo(FormatOptions.Default, output, true);

PleegWat

@JBert The problem there is the request body length. This has to be declared in the request headers using a Content-Length header, or chunked encoding has to be used. However, in the general case, you can't assume the receiving web server supports HTTP/1.1 requests, and because of that you can't use chunked encoding.

IETF - Internet Engineering Task Force

RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1

HTTP has been in use by the World-Wide Web global information initiative since 1990. This specification defines the protocol referred to as "HTTP/1.1", and is an update to RFC 2068. [STANDARDS-TRACK]

Technically, I think it is possible to determine where a multipart message ends based on the message data as well, but it's not in the RFC.

HTTP/2 does not have this problem.

JBert

@PleegWat Right, though my last code snippet shouldn't have too much trouble with that if you just call output.Length (the MemoryStream is after all a stream wrapper around an automatically sized byte array).

dkf

@PleegWat said in Sending files collection as form-data:

HTTP/2 does not have this problem.

It has a whole load of other problems, but that ain't one of them.