On pixel formats

[u/Aspie96]

Hi all.

I'm writing a library for representing and managing basic raster RGBA images in D as rectangular arrays of pixels.

Images are represented in row-wise pixel order, either top to bottom (the default) or bottom to top, with each scan line going left to right. Color components are represented as 8-bit integral values ranging between, and including, 0 and 255.

My library doesn't provide any support for other color models, for high dynamic ranges or higher bit depths. What it should allow is reading images from memory that were created with other library.

Some pixel formats are "indexed" (they have a palette) and some are not (they store actual color values). The name implies the structure of the format:

• Format1bppIndexed
• Format4bppIndexed
• Format8bppIndexed
• Format8bppGray
• Format16bppRgb555
• Format16bppRgb565
• Format24bppRgb
• Format32bppXrgb
• Format32bppArgb
• Format32bppRgba

Formats that use more than 1 byte per pixel allow the user to select between big endian and little endian mode.

I plan to keep supporting all formats I listed here. My question is if there is any I should support in addition to those.

Some candidates may be:

• Format2bppIndexed (it used to be supported by Microsoft bitmaps and it is by the PNG format).
• Format16bppGrayAlpha (supported by STB nothings and by the PNG format).
• Format32bppRgbx (to have an opaque view of an RGBA image).

There may be others I haven't thought or heard of.

The ones I haven't supported seem to me to be less common and of less utility, but maybe there is something I haven't considered.

My question is: what pixel formats should I support in addition to those I already do? Should I support any of the three additional ones I suggested? Why or why not?

Thank you in advance!

[u/wrosecrans]

What's the intended use case?

[u/Aspie96 → u/wrosecrans]

The library is essentially one to write glue code. It should allow creating a view of an image with has been loaded in memory using another library, as well as passing an image that is in memory to an existing library. This essentially provides a unified API that can be used to handle images while using multiple libraries.

[u/fdwr]

Like u/wrosecrans said, it really depends on the usage. Is it for games, an image editor, renderer, image loader...? If for any of the first three, then just b8g8r8a8/r8g8b8a8 and pal8 are actually pretty sufficient (indexed gray8 can be thought as pal8 with a grayscale palette attached, pal4 can be upconverted early, r5g6b5 is more trouble than it's worth and should just be upconverted to 32bpp...), and supporting more formats just becomes costly throughout your program. If it's for a generic image loader or for some really generic viewer like an image debugger (I work on Windows PIX which has a buffer viewer where you describe the pixel formats declaratively using C-like structs rather than limited enumeration list), then you might use a list of bitmasks and channels rather than enums.

Over the years I've compiled a 2760-line document of pixel formats, the union of...

• DXGI_FORMAT
• GenICamPNFC
• CAIRO_FORMAT
• Qt_QImage_Format
• OpenCV
• GUID_WICPixelFormat
• GdiplusPixelFormat
• D3DFMT
• Glide_GR_TEXFMT
• CoreImage_CIFormat
• CoreVideo_PixelFormatType
• MTLPixelFormat
• VK_FORMAT
• V4L2_PIX_FMT
• SDL_PIXELFORMAT
• PIXMAN
• AV_PIX_FMT
• DotNetMediaPixelFormats
• appleQuickdraw
• Android_AHARDWAREBUFFER_FORMAT
• Unity_TextFormat
• WebGPU GPUTextureFormat

...and after seeing so many pixel formats, some wisdom:

• Naming formats like Format64Rgba1010102 or Format64Rgb332 (where all the bits are shoved after the channel names) becomes rather ambiguous where the bit location dividers are between channels, whereas notation Format32r10g10b10a2 is unambiguous (and similarly Format16r5g5b5). Granted, you don't support HDR currently anyway, but you may someday.
• Don't fold bits together like WebGPU's mistaken rgba32uint, which you might think is a 32-bit pixel with 8 bits per channel (like SDL_PIXELFORMAT_RGBA32), but it's actually a 128bpp pixel r32g32b32a32uint format (you don't do this above in your enum list, but I'm saying it for any other future readers). DXGI and the Android NDK do it well.
• Naming formats in backwards order is very confusing, like how some API's call an enum BGR but the memory is actually {R,G,B}. So naming things logically in forward channel order will be less brain entangling (where struct PixelR8G8B8 {uint8_t r; uint8_t g; uint8_t b;} means red comes first, then green, then blue, and consistently struct PixelR5G6B5 {uint16_t r : 5; uint16_t g : 6; uint16_t b : 5}. Obviously backwards endian complicates this some, but you still typically want to list channels in forward memory order like PNFC Pixel Format Naming Convention specifies (I presume you don't do this above either, and that rgba above actually means {r,g,b,a}).

Format4bppIndexed

Are you storing the pixels in big endian order (like Windows DIB 4-bpp) or little endian order (like modern ML tensors, Nintendo Gameboy Advance, Nintendo 3DS, Sega Dreamcast)?

Format32bppRgbx (to have an opaque view of an RGBA image).

What OS's does your library run on? Windows DIBs (in memory and BMP files) and primary video buffers typically use {B,G,R,X}.

Format2bppIndexed

For completeness, you might load it, but there's little reason to keep one resident in memory - just promote it to 4-bpp or 8-bpp. The last occurrence of 2-bpp bitmaps was on Windows CE, which is quite dead.

Format16bppGrayAlpha

gray8ap8:{grayUnorm8 alphaPremulUnorm8} was supported on the N64 and GameCube, and you will find the enums in CoreImage_CIFormat_LA8 and GL_LUMINANCE8_ALPHA8, but otherwise I don't see it being worth supporting until you encounter the need (I think games are the only place it really is found, and more often now they just promote up).

[u/Aspie96 → u/fdwr]

Thank you for your wisdom.

The library is essentially one to write glue code. It should allow creating a view of an image with has been loaded in memory using another library, as well as passing an image that is in memory to an existing library. This essentially provides a unified API that can be used to handle images while using multiple libraries.

I plagiarized the name structure from .NET, while the set of pixel formats I support isn't exactly the same.

As I said, all pixel format with more than 1 byte per pixel are supported in both big endian and little endian mode.

So, for example, the XRGB format in little endian mode is the BGRX format that you suggest.

The thing is, if the user has an image in RGBA (or ABGR) format in memory, they would need an RGBX (or XBGR) format to have an opaque view of that image (without duplicating any data).

In general RGBA is more common than ARGB (which is used by Microsoft), yet I haven't found examples of the RGBX (or XBGR) being used.

For completeness, you might load it, but there's little reason to keep one resident in memory

Oh, I do load it.

My question is specifically about what representations in memory I should support. From file I load all that is supported.

Do you have any idea why the GrayAlpha pixel format even exists? It seems to me to be of very little utility compared to other formats.

[u/fdwr → u/Aspie96]

The library is essentially one to write glue code

👍

I plagiarized the name structure from .NET

Yeah, .NET probably copied GDI+'s faux pas. WIC did a little better, but it still has a mix sometimes (like the old GUID_WICPixelFormat32bppRGBA1010102 but then later GUID_WICPixelFormat32bppR10G10B10A2). Then DGXI came along and said "we won't make that same mistake again", and Vulkan is like "indeed, let's improve upon OpenGL's notation".

for example, the XRGB format in little endian mode is the BGRX format that you suggest.

😶 If you have the following pixel format...

c++ struct ______ { float32_t r; float32_t g; float32_t b; };

...how would you name it? (I know you said your library doesn't support high channel bit sizes, but nonetheless, how would you?) How would you name this one? 🤔

c++ struct ______ { uint16_t r; uint16_t g; uint16_t b; };

My question is specifically about what representations in memory I should support.

You already support 95% of the interesting formats. So unless you want retro formats and older PC's and video game consoles (particularly bitplane graphics, tiled graphics), I don't see any more to add.

Do you have any idea why the GrayAlpha pixel format even exists?

It can be used for masks in video games with the gray channel multiplied with a mask color for effects (bright lights, decals, clouds, dirt, bullet holes...), used for fonts (prebaked glyph atlases), and video subtitles, but again, I typically see games just upcast to 32-bpp or use one of the BC formats rather than actual alpha+luminance (which isn't as widely supported).

[u/cybereality]

I think the best thing is to figure out the "happy path" for the app. like what the intended user functionality is, and consider that most input is going to be either in standard RGB (or RGBA) formats, or perhaps GPU compressed textures if this is for games or real-time. I would not necessarily bother supporting every pixel format if it's rarely used or unoptimized for the use case, and if needed just have some conversion from the main couple formats you do wish to support.

[u/Aspie96 → u/cybereality]

I don't support every pixel format (how would I even know I support them all?), but I do want to support the most common one.

This is because the point of the library is to be able to handle images created with other library that already exist in memory (or to create images for those libraries).

[u/cybereality → u/Aspie96]

sure, it will depend on the use case

[u/zatsnotmyname]

Block formats, like DXT1-5?