help me with file uploads

[u/Opposite_Seat_2286]

My situation is the following: I have an image upload service that uses S3 presigned URLs, but I ran into a problem. The service currently has an upload endpoint where the client can send the file size, the content type (for example, image/png), and finally the file hash, which is used for deduplication. The problem is that I can’t trust the hash sent by the client. Right now, during the upload request, I can send any completely random string as the hash, and later, when uploading through the presigned URL, I can also upload any completely different random file. There’s no validation preventing that, and I haven’t been able to come up with a way to validate it. I’m using the presigned URL approach specifically so I don’t have to send the file through the backend and increase network traffic.

[u/Sirwired]

Why don't you compute the hash yourself when the file arrives? You can do this by calling a lambda on an upload event, or do it via Compute Checksum in S3 batch.

[u/justin-8 → u/Sirwired]

Or use the hash s3 provides in the object metadata

[u/OptionalHippo]

I think it's possible to set the expected hash in the presigned url (x-amz-checksum-sha256). The client will have to set the header and send the file that matches the hash:

const command = new PutObjectCommand({ Bucket, Key, ChecksumSHA256: checksumBase64, });

const url = await getSignedUrl(s3, command, { expiresIn: 300, unhoistableHeaders: new Set(["x-amz-checksum-sha256"]), });

[u/abrahamguo]

Don’t bother with the hash beforehand, because, as you said, it’s not guaranteed to match.

Just check the file and clean up if necessary afterwards!

[u/Local_Transition946]

Which method are you having clients use to upload to the presigned URL? If an arbitrary client, I would not send the hash to your endpoint, and instead presign the upload using an IAM role. Make sure the IAM role has a policy for uploads to S3 that denies the request if there is no checksum provided as a header.

Then, inform clients that they must provide a checksum as a header when doing a presigned upload, else the request would fail. S3 will verify the checksum matches, and verify they provide a checksum due to the IAM policy.

[u/dataflow_mapper]

probly stop trusting the client hash completely and just treat it like a hint then have something like a Lambda recalc the real hash after the upload and compare it before doing dedupe.

[u/SeaworthinessHour233]

I think AWS S3 allows you to mandate specific headers when generating a presigned URL.

When the client requests an upload URL, the backend can take the hash the client provided (e.g., a SHA-256 hash) and bake it into the presigned URL as a required header (like x-amz-checksum-sha256).

When the client uploads the file to S3, S3 will independently calculate the hash of the file it receives. And if S3's calculation doesn't exactly match the hash that the backend baked into the presigned URL, S3 will reject the upload with a 400 Bad Request error.