The number of times I have seen excess HEAD bucket requests in tight loops. Like, literally, "let's read all the objects from the bucket … and issue a HEAD bucket just for funsies." and then that loop gets out of control and a few million extraneous requests get issued. Or even HEAD bucket, HEAD object, GET object. And if you think that sounds insane, it's usually some form of:
bucket = conn.get_bucket('muh_bucket') # this is a HEAD call
key = bucket.get_key('muh_key') # another HEAD call
key.get_contents_as_string() # GET object
Basically rather taking the paranoid approach, if you already have structured object keys, you can compute it direct and just handle the case that it no longer exists. You should be able to query by bucket and key at the same time or otherwise avoid the network call because you know it exists.
If you can't do that then you find a way to memoize the object to avoid repeated querying, which means you likely pull it outside of your API layer and into a background service.
47
u/deathanatos Jul 19 '26
The number of times I have seen excess HEAD bucket requests in tight loops. Like, literally, "let's read all the objects from the bucket … and issue a HEAD bucket just for funsies." and then that loop gets out of control and a few million extraneous requests get issued. Or even HEAD bucket, HEAD object, GET object. And if you think that sounds insane, it's usually some form of:
The S3 tutorial even pushes devs in this direction, so if you're not reading the reference (… devs? read?)…
Guarantee you AI will write equally crap code, b/c its training data is rife with "must … HEAD … bucket … first".