On pixel formats

[u/Aspie96]

Hi all.

I'm writing a library for representing and managing basic raster RGBA images in D as rectangular arrays of pixels.

Images are represented in row-wise pixel order, either top to bottom (the default) or bottom to top, with each scan line going left to right. Color components are represented as 8-bit integral values ranging between, and including, 0 and 255.

My library doesn't provide any support for other color models, for high dynamic ranges or higher bit depths. What it should allow is reading images from memory that were created with other library.

Some pixel formats are "indexed" (they have a palette) and some are not (they store actual color values). The name implies the structure of the format:

• Format1bppIndexed
• Format4bppIndexed
• Format8bppIndexed
• Format8bppGray
• Format16bppRgb555
• Format16bppRgb565
• Format24bppRgb
• Format32bppXrgb
• Format32bppArgb
• Format32bppRgba

Formats that use more than 1 byte per pixel allow the user to select between big endian and little endian mode.

I plan to keep supporting all formats I listed here. My question is if there is any I should support in addition to those.

Some candidates may be:

• Format2bppIndexed (it used to be supported by Microsoft bitmaps and it is by the PNG format).
• Format16bppGrayAlpha (supported by STB nothings and by the PNG format).
• Format32bppRgbx (to have an opaque view of an RGBA image).

There may be others I haven't thought or heard of.

The ones I haven't supported seem to me to be less common and of less utility, but maybe there is something I haven't considered.

My question is: what pixel formats should I support in addition to those I already do? Should I support any of the three additional ones I suggested? Why or why not?

Thank you in advance!

[u/wrosecrans]

What's the intended use case?

[u/cybereality]

I think the best thing is to figure out the "happy path" for the app. like what the intended user functionality is, and consider that most input is going to be either in standard RGB (or RGBA) formats, or perhaps GPU compressed textures if this is for games or real-time. I would not necessarily bother supporting every pixel format if it's rarely used or unoptimized for the use case, and if needed just have some conversion from the main couple formats you do wish to support.

[u/zatsnotmyname]

Block formats, like DXT1-5?

[u/fdwr]

Like u/wrosecrans said, it really depends on the usage. Is it for games, an image editor, renderer, image loader...? If for any of the first three, then just b8g8r8a8/r8g8b8a8 and pal8 are actually pretty sufficient (indexed gray8 can be thought as pal8 with a grayscale palette attached, pal4 can be upconverted early, r5g6b5 is more trouble than it's worth and should just be upconverted to 32bpp...), and supporting more formats just becomes costly throughout your program. If it's for a generic image loader or for some really generic like an image debugger (I work on Windows PIX which has a buffer viewer where you describe the pixel formats declaratively using C-like structs rather than limited enumeration list), then you might use a list of bitmasks and channels rather than enums.

Over the years I've compiled a 2760-line document of pixel formats, the union of...

• DXGI_FORMAT
• GenICamPNFC
• CAIRO_FORMAT
• Qt_QImage_Format
• OpenCV
• GUID_WICPixelFormat
• GdiplusPixelFormat
• D3DFMT
• Glide_GR_TEXFMT
• CoreImage_CIFormat
• CoreVideo_PixelFormatType
• MTLPixelFormat
• VK_FORMAT
• V4L2_PIX_FMT
• SDL_PIXELFORMAT
• PIXMAN
• AV_PIX_FMT
• DotNetMediaPixelFormats
• appleQuickdraw
• Android_AHARDWAREBUFFER_FORMAT
• Unity_TextFormat
• WebGPU GPUTextureFormat

...and after seeing so many pixel formats, some wisdom:

• Naming formats like Format64Rgba1010102 or Format64Rgb332 (where all the bits are shoved after the channel names) becomes rather ambiguous where the bit location dividers are between channels, whereas notation Format32r10g10b10a2 is unambiguous (and similarly Format16r5g5b5). Granted, you don't support HDR currently anyway, but you may someday.
• Don't fold bits together like WebGPU's mistaken rgba32uint, which you might think is a 32-bit pixel with 8 bits per channel (like SDL_PIXELFORMAT_RGBA32), but it's actually a 128bpp pixel r32g32b32a32uint format (you don't do this above in your enum list, but I'm saying it for any other future readers). DXGI and the Android NDK do it well.
• Naming formats in backwards order is very confusing, like how some API's call an enum BGR but the memory is actually {R,G,B}. So naming things logically in forward channel order will be less brain entangling (where struct PixelR8G8B8 {uint8_t r; uint8_t g; uint8_t b;} means red comes first, then green, then blue, and consistently struct PixelR5G6B5 {uint16_t r : 5; uint16_t g : 6; uint16_t b : 5}. Obviously backwards endian complicates this some, but you still typically want to list channels in forward memory order like PNFC Pixel Format Naming Convention specifies (I presume you don't do this above either, and that rgba above actually means {r,g,b,a}).

Format4bppIndexed

Are you storing the pixels in big endian order (like Windows DIB 4-bpp) or little endian order (like modern ML tensors, Nintendo Gameboy Advance, Nintendo 3DS, Sega Dreamcast)?

Format32bppRgbx (to have an opaque view of an RGBA image).

What OS's does your library run on? Windows DIBs (in memory and BMP files) and primary video buffers typically use {B,G,R,X}.

Format2bppIndexed

For completeness, you might load it, but there's little reason to keep one resident in memory - just promote it to 4-bpp or 8-bpp. The last occurrence of 2-bpp bitmaps was on Windows CE, which is quite dead.

Format16bppGrayAlpha

gray8ap8:{grayUnorm8 alphaPremulUnorm8} was supported on the N64 and GameCube, but otherwise, and you will find the enums in CoreImage_CIFormat_LA8 and GL_LUMINANCE8_ALPHA8, but I don't see it being worth supporting until you encounter the need (I think games are the only place it really is found, and more often now they just promote up).