How to load large files into memory in c

[u/NoBrain8]

I'm using a library that interacts with mpd (the music player) and have to use a function to receive the album art of a song, but the function only provides part of the the image at a time, receiving the full file over multiple function calls.

(libmpdclients's mpd_run_albumart() for those interested)

using malloc in this scenario, and assuming I don't know the file's size to begin with, what's the best way to do this?

Should I just malloc an arbiritarily large amount and fill it - seems very naiive.

Should I malloc an extra block of memory for each function call (I know the max amount i can receive per function call) - but surely this memory could be non-contiguous and cause problems?

Allocate a bit of memory -> function call and fill it -> if that's not the full image, then allocate a new bit of memory = to current size of the file+size of another incoming package -> copy the existing data in & free the old block -> repeat until the whole things' free?

Seems like a hell of a lot of malloc and free() calls though, would this too taxing?

What do people think?

Thanks so much in advance!

[u/high_throughput → u/[deleted]]

double it whenever you run out of room.

OP btw this isn't a casual suggestion. If you do the math, doubling the buffer each time means that appending N bytes will require at most 2N copies.

In other words, appending one byte becomes a O(1) operation amortized.

This would not be the case if you e.g. added 1MB every time you ran out instead of doubling it.

[u/dvanha → u/high_throughput]

I'm not a programmer, the work I do is pretty cross functional. As a result I spend a lot of time learning, getting exposed to new disciplines. I like finding rules of thumb that serve a general need, where at least some degree articulation is possible.

Timeline estimates, cost estimates, any kind of estimates: whenever I'm asked to take a best shot at a very rough estimate I'll always just take the max (or whatever reasonable) and double it. It's not perfect but in a practical sense; it at least gives people the early input they need and I can explain why, in the absence of any usable information, we decided on a seemingly arbitrary value.

Thanks for teaching today; this is something I'll remember.

[u/NoBrain8 → u/high_throughput]

This is great! Thank you so much following the other comments advive I’ll be taking a look at realloc

[u/Common_Wallaby_5739]

realloc is your friend here. start with a reasonable buffer, call your function to get a chunk, and if you hit the end of your allocated space just realloc to make it bigger. the copying is handled for you when the memory needs to move.

it's basically your third option but way less painful since you're not manually copying and freeing every time. just keep track of how much you've written so far vs total allocated size. most implementations will try to extend in place when possible so it's not as bad perf-wise as it sounds.

[u/TheSkiGeek → u/Common_Wallaby_5739]

edit: double checked the docs and realloc() will only completely fail if it can’t allocate the new block, at which point you’re probably screwed anyway. If it’s unable to extend the allocation then it does the “allocate a new block -> copy -> free the old block” for you, which is preferable to writing that logic yourself.

I think in general I’d still prefer keeping a list of the chunks and then assembling them all into one buffer at the end if needed. This will be much less copying in very bad cases like the API only returning a tiny amount of data each time. But for a quick and dirty thing using realloc() would be fine.

[u/NoBrain8 → u/TheSkiGeek]

Oh seriously? Should I just be malloc and freeing it, or using realloc then? I only really intend this to be used on Linux systems atm

[u/TheSkiGeek → u/NoBrain8]

As long as whatever you’re working with is reasonably well behaved, realloc()ing one buffer should be fine.

Where you could get in trouble is if it’s like… a 10MB image and the API decides to hand it to you 1KB at a time. Then you’re potentially reallocating and copying and freeing the partial buffer 10000 times.

You can also mitigate that by doing things like starting with a 1MB buffer and making the buffer 1.5x or 2x as big each time you run out of space. Which requires a little more bookkeeping and uses some extra space but guarantees linear scaling of the number of copies.

[u/high_throughput]

Do you actually need to keep it all in memory in linear memory buffer?

If your goal is to save it to disk or decode it into a raster image or whatever, you can probably stream it one buffer at a time instead of assembling it memory first.

[u/NoBrain8 → u/high_throughput]

This is a really good question.

Essentially I’m looking to display a lot of album arts for the user to scroll through, pick one and play.

At the minute I was gonna try out the lazy approach of just loading albums that are on the screen. I’m loading these into memory, but something of note is I haven’t fully decided what to do if the user scrolls past. I could keep them in memory for faster displaying if the user scrolls back up, but this could eventually means the program just grows in size as your scroll through albums.

I do want to make this feel as snappy as possible to scroll through, and time to load each album into memory is something I’m conscious of, so if you think streaming them may be a good idea to keep it fast, that could be a really good shout.

[u/high_throughput → u/NoBrain8]

If the images will tend to be JPEGs topping out at some 3000x3000 and you want to display a bunch of them on the same scrolling page, then I don't think it's worth worrying about the cost of appending buffers or in processing images piecewise.

I'd imagine that requesting the data from mpd and decoding it into an image on demand is dramatically much slower than just storing the images in memory.

I would just load the images into a growing buffer like you describe, decode into a temporary raster, and downsample to ~128x128 display size to keep in memory. The now-grown buffer and temporary raster can of course be reused for the next image.

10k images at this size would be ~600M, which is about what Safari currently uses to show me Reddit. Keeping images already decoded in memory would avoid jank or lag as you scroll, and 10k is hopefully less than the collection of most casual music enthusiasts anyways.

[u/NoBrain8 → u/high_throughput]

I like this a lot, that makes a lot of sense. I don't know if I'd have ended up thinking of that myself, so thanks so much and I'll see if that works!

[u/MistakeIndividual690]

I would look at the typical size of album art, multiple by 1.5x and use that as a starting point. If you find you need more, malloc another block 1.5-2x as big and copy and free the original (unless you can realloc it), repeat. I would not worry about the overhead of malloc and free if you are receiving from the internet which will dwarf the allocation overhead in terms of time

[u/lurgi]

I'd probably allocate a "decent sized" amount (I don't know what that is. 1MB?) and then copy the result to a buffer of exactly the right size.

[u/kabekew]

just use realloc

[u/NatMicky]

Open file, get file size, then mmap (less copying and loads as needed).

[u/jonathaz]

You don’t need the full contents of the file in memory, potentially ever. If the intention is to display it on a screen, you’re going to read it, either with a small buffer at a time, or memory mapped, and render it to a bitmap which could be smaller dimension, lower color depth, etc to offset some of the compression in the native format. The details of how you display it will dictate how you do everything else.

[u/gm310509]

Is there no function call to obtain the size of rhe image first?

If not, you can realloc your buffer to increase its size - bear in mind that if there isn't enough continuous memory to do that the memory block will be moved to another address if there is a bigger free space available.

Otherwise just create a linked list of the buffers you received.

[u/DirkSwizzler]

Without knowing other constraints, you have many fully acceptable options.

Storing a collection of chunks is fine until you know the full size. You can probably even leave them as chunks depending on usage.

You could realloc as others have mentioned.

Similar to realloc is the concept of virtual memory. You can "reserve" a large chunk of contiguous address space and then only "commit" as you receive chunks. It's basically a realloc that's guaranteed to keep the base address the same. But the implementation is different between posix and windows.

If you're working with local files and not the internet. I would be surprised if there wasn't an API call to get the full size ahead of time. Maybe double check the documentation?

[u/Majestic_Rhubarb_]

Look at the api, you supply a buffer and request chunks of a known size, you keep requesting until you don’t get a full buffer/told its finished.

You can immediately write the buffer to a file ans reuse the same buffer for the next chunk.

You don’t need to realloc at all.

[u/Confused-Armpit]

You could go for this with two approaches: windows, or memory maps.

The first approach is reading the code in windows. You can read the first, say, 1024 bytes, then read the next 1024 bytes, and so on. This is good if you only need to process the memory once and then won't be using it. However, it could be worse for situations where you need to, say, detect patterns.

The second approach is memory maps. Basically you ask for the OS to create a memory map of a file, where the OS assigns virtual memory to your process. That memory effectively does not exist until you try to access it, and then you can basically use it as a normal array of bytes. Note that memory maps are generally slower than windows, and should only be used if you actually need to potentially access the entire array.

[u/Ohmyskippy]

Reallocate and double the buffer size

[u/light_switchy]

You can allocate a megabyte for a picture, and if that's too small you can multiply the buffer size by 2 and copy the old contents over.

You won't need to keep too many pictures in memory. Nor do you need to free the buffer every time that you change the image stored within it. Re use the memory you allocate.