Uploading to S3 with Presigned Posts, HTMX and Alpine.js

Uploading files in a web application is always kind of annoying. It's such a common operation but it always seems to just bump up the complexity a little to do it well. The combination of files potentially being big, binary data and the various browser security restrictions around files just always seems to make things fiddly.

My goto these days is to use S3 for file storage, and use a presigned post to upload it. In essence the process is

1. Call an application endpoint with some details of the file to be uploaded
2. Application server uses it's private AWS credentials to "presign" an S3 upload request and return it to the client.
3. Client then uploads directly to S3 using the presigned request returned from the server

This way the application can enforce whatever permission checks are needed before granting the upload, but we're not having to worry about streaming a potentially large binary file through the applciation server.

Generating a presigned post

Generating the presigned post server side is pretty simple using the @aws-sdk/s3-presigned-post package.

This is slightly simplified, but basically the whole thing.

import { createPresignedPost } from '@aws-sdk/s3-presigned-post';
import { S3Client } from '@aws-sdk/client-s3';

const client = new S3Client({
  region: 'ap-southeast-2'
});
const { url, fields } = await createPresignedPost(client, {
  Bucket: 'my-s3-bucket',
  Key: `user-uploads/${request.body.filename}-${randomUUID()}`,
  Fields: {
    'Cache-Control': 'max-age=31536000, immutable',
    'Content-Type': request.body.content_type,
  },
  Expires: 300, // Only valid for 5 minutes
});

return reply.send({
  url,
  fields,
});

1. Accept a filename and a content type from the client
2. Add a server controlled prefix and UUID to the name so the client can't target a filename to overwrite another file
3. Generate a resigned post for the particular location in S3 where we want to put the file, including in the signed request the content type and cache control we want set on the resulting file in S3.

This basically returns everything needed for a S3 Browser Based Upload the url is where a multipart/form-data encoded form data should be posted to.

The fields is a set of fields to be included in the request to contain the key, signature and other information. Just add a file parameter containing the actual file and it is a complete presigned upload request.

Uploading from the client

This approach inherantly means that a file upload is two chained requests.

1. Get the url and fields for the pre-signed upload request from the application server
2. Do the actual upload with a POST to S3

In the past I've just done this purely in JavaScript on the client using the fetch api. But I wanted to try and do this with HTMX and Apline.js since that's what I'm using throughout my current application and will allow me to use standard HTMX functionality for loading states, errors etc.

So what I've ended up with is two chained forms. The user submit's one, which presigns the request, and then the second form is automatically submitted to do the actual upload.

<div x-data="{file: null, presigned: null}">
  <!-- Form #1 to actually upload the file-->
  <form hx-post="" hx-trigger="post-presigned from:next" enctype="multipart/form-data"
    @htmx:config-request.camel="
      $event.detail.path = presigned.url;
      Object.entries(presigned.fields).forEach(([key, value]) => {
        $event.detail.parameters[key] = value;
      });
      // File must be last so delete and re-add it
      const file = $event.detail.parameters.file;
      $event.detail.parameters.delete('file');
      $event.detail.parameters.file = file;
    ">
    <input type="file" name="file" @change="file = $event.target.files[0] || null"/>
  </form>

  <!-- Form #2 to request the presigned upload details -->
  <form hx-post="/presign-upload" @htmx:before-swap.camel.prevent="
    presigned = JSON.parse($event.detail.serverResponse);
    $dispatch('post-presigned');
  ">
    <input type="hidden" name="csrfToken" value="...."/>
    <input type="hidden" name="filename" x-bind:value="file && file.name"/>
    <input type="hidden" name="content_type" x-bind:value="file && file.type"/>
    <input type="submit" name="upload" value="Upload" />
  </form>
</div>

So, what's going on here:

1. The user selectes a file using the file input in Form 1. This is the form that will eventually be submitted to upload the file to S3.
2. We listen to the change event on the file input to store the file metadata into the Alpine.js data so that we can access the name and content type
3. The filename and content type are bound to hidden inputs in Form 2, the form we will submit to generate the presigned post
4. When the user submits the second form, it calls the application server and returns the presigned post
5. We use the htmxbeforeSwap event to intercept the resposne and shove it into the Alpine.js data instead of swappign it into the DOM.
6. We then fire the custom post-presigned event which triggers Form 1 to actually upload the file
7. We use the htmx:configRequest event to intercept the form submission and set the url to send to and add the fields from the presigned response to the file input that's actually in the form
8. There's a little bit of fiddly here because S3 requires the file to be the last field in the submission

Some of the weird back and forth here is because you need the file metadata (name and content type) to generate the presigned post in the first place. I can't see a way to do that bit without javascript unless we had a user enter them as normal form fields and hoped they got it right.

Removing the beforeSwap

I haven't actually tried this, but I think it would be possible to make this a bit more HTMX-ish. Instead of returning the presigned post details as JSON, we could return a set of input type="hidden" from the server.

Then we could do something like this

<div x-data="{file: null}">
  <!-- Form #1 to actually upload the file-->
  <form hx-post="" hx-trigger="htmx:afterSettle from:next" enctype="multipart/form-data"
    @htmx:config-request.camel="
      $event.detail.path = $event.detail.parameters.url;
      $event.detail.parameters.delete('url');
    ">
    <div id="presigned-post-fields">
      <!-- hidden inputs will be inserted here by htmx -->
    </div>
    <input type="file" name="file" @change="file = $event.target.files[0] || null"/>
  </form>

  <!-- Form #2 to request the presigned upload details -->
  <form hx-post="/presign-upload" hx-target="#presigned-post-fields">
    <input type="hidden" name="csrfToken" value="...."/>
    <input type="hidden" name="filename" x-bind:value="file && file.name"/>
    <input type="hidden" name="content_type" x-bind:value="file && file.type"/>
    <input type="submit" name="upload" value="Upload" />
  </form>
</div>

In theory at least this should work.Form 2 pushing a set of hidden inputs into the first form, and then that automatically triggering the upload form.

Could we remove the htmx:configRequest

We're still making use of htmx:configRequest here to set the url of the upload since that isn't being replaced. The reason we can't replace the entireity of the upload form is we need to keep the file input since the user will have already selected the file to get the metadata.

But using the HTMX Idiomorph extension we can morph the dom and keep the eixisting input field

So if we updated the /presign-upload endpoint to return an entire presigned form like this:

<form id="upload-form" hx-post="...url from presigning post..." hx-trigger="htmx:load" enctype="multipart/form-data">
    <input type="hidden" ... hidden inputs for fields from presigning post />
    <input type="hidden" ... hidden inputs for fields from presigning post />
    <input type="hidden" ... hidden inputs for fields from presigning post />
    <input id="file-upload" type="file" name="file" x-on:change="file = $event.target.files[0] || null"/>
  </form>

Then the initial page could look something like this

<div x-data="{file: null}">
  <!-- Form #1 to actually upload the file this will actu-->
  <form id="upload-form" enctype="multipart/form-data">
    <input id="file-upload" type="file" name="file" x:on-change="file = $event.target.files[0] || null"/>
  </form>

  <!-- Form #2 to request the presigned upload details -->
  <form hx-post="/presign-upload" hx-target="#upload-form" hx-swap="morph">
    <input type="hidden" name="csrfToken" value="...."/>
    <input type="hidden" name="filename" x-bind:value="file && file.name"/>
    <input type="hidden" name="content_type" x-bind:value="file && file.type"/>
    <input type="submit" name="upload" value="Upload" />
  </form>
</div>

The morph swap reatains the current file input so the file is still selected.

The htmx:load trigger submits the presigned form as soon as it's swapped into the page.

It works and it feels really clean and idomatic HTMX. I'm still using Alpine.js to read the file metadata for the presigned upload, still not sure if I can avoid that.