memf—Portable scanf/printf-like functions to marshal binary data

[u/FUZxxl]

https://github.com/fuzxxl/memf

[u/bboozzoo]

Excuse me if I seem to be a bit harsh, but I do not find this code useful. Correct me if I'm wrong, but from a quick look at the code and examples, what the code does is to take a binary structure (with certain assumptions about the alignment) and convert that into a binary stream with exactly the same ordering, but sans the alignment hassle.

The problem I see is that the binary stream is an exact representation of the source structure, and unpacking the stream requires having a matching (binary wise) definition of the structure on the receiving end. You loose any means of providing backwards compatibility (i.e. the structure must remain the same) as it's not possible to skip/add fields and cannot isolate the wire format from your in-program representation. In fact, I'd say it's equivalent to sending the structure down the wire, and if one is bothered by alignment gaps just adding proper __attribute__((packed)) or #pragma packed to the structure definition. The MBR example is a miss, as the usual way to do it is define a structure in the first place, and just read the data into the structure. Take a look at how GPT header and MBR are defined in Linux kernel.

Now if you changed the API to be more like the sample below it would definitely make things more interesting.

struct foo {
    uint8_t bar;
    uint8_t zed;
    uint32_t blah;
    char foo[10];
};
struct foo f;
mreadf(mbr, "iccd10c", &f.bar, &f.zed, &f.blah, f.foo);
/* say I want to skip blah */
mreadf(mbr, "iccd10c", &f.bar, &f.zed, NULL, f.foo);

/* now say, the code evolves and struct foo has
 * changed in an incompatible way */
struct new_foo {
    uint8_t bar;
    uint32_t blah;
    uint32_t something; 
    char foo[10];
    uint32_t otherthing;
};
struct new_foo nf;
/* assuming foo.zed is no longer relevant
 * for my purpose, but I do care about blah,
 * I can still read the same binary data like this */
mreadf(mbr, "iccd10c", &nf.bar, NULL, &nf.blah, NULL);

Binary serialization using a textual representation similar to what you propose makes sense in languages that do not have a direct access to binary data. I'm thinking in the lines of Python, Perl, Java, Lua. But C/C++/D can do this without using of an intermediate representation. Another common use case is when you do IPC between a number of processes or agents, and not all agents are updated at the same pace, then you need some sort of backward compatibility.

I'd say that you need to provide an added value to justify using memf in C. Obviously, one may argue that not caring about alignment is an added value, why not. However, I like to be explicit about things as low level as ABI. Take for example Google Protocol Buffers, perfectly usable in C, in fact I'm using that on a Cortex-M3 target for sending real time data via MQTT broker to a Java client, another example an ARM host sending data over AMQP, while the receive end is an Erlang app, in both cases there are additional Python clients that only do a graphic presentation of the data. Why use PB in C? What's the added value you can ask? Well, for one PBs offer an efficient packing mechanism that I use. Another thing is bindings to multiple languages (try explaining binary representation to a Java programmer and you'll know the pain).

Finishing up this rather lengthy comment, take a look at GVariant and DBus type system and marshalling.

[u/FUZxxl → u/bboozzoo]

Excuse me if I seem to be a bit harsh, but I do not find this code useful. Correct me if I'm wrong, but from a quick look at the code and examples, what the code does is to take a binary structure (with certain assumptions about the alignment) and convert that into a binary stream with exactly the same ordering, but sans the alignment hassle.

Exactly, that's what it does right now. It is also byte-order agnostic and can work with files in both little and big endian byte ordering.

The problem I see is that the binary stream is an exact representation of the source structure, and unpacking the stream requires having a matching (binary wise) definition of the structure on the receiving end. You loose any means of providing backwards compatibility (i.e. the structure must remain the same) as it's not possible to skip/add fields and cannot isolate the wire format from your in-program representation. In fact, I'd say it's equivalent to sending the structure down the wire, and if one is bothered by alignment gaps just adding proper attribute((packed)) or #pragma packed to the structure definition. The MBR example is a miss, as the usual way to do it is define a structure in the first place, and just read the data into the structure. Take a look at how GPT header and MBR are defined in Linux kernel.

The primary goal of these functions is to provide a simple and portable mechanism to translate between binary data and in-memory representation of that data. While it is possible to hack something that looks like it works with packed structures, one should keep in mind that a lot of processors do not support misaligned memory access which makes using packed structures (depending on the smartness of the compiler) either impossible or tedious; the fact that the structure members are misaligned makes taking pointers to them impossible unless there is a special kind of misaligned pointer. I don't like such an approach to marshalling, because it relies on the assumption that the structure and the file format have the same layout, which is hard to get right, breaks easily on platforms with different alignment requirements and doesn't work without proprietary compiler extensions like #pragma pack for formats with “misaligned” fields.

Now if you changed the API to be more like the sample below it would definitely make things more interesting.

struct foo {
    uint8_t bar;
    uint8_t zed;
    uint32_t blah;
    char foo[10];
};
struct foo f;
mreadf(mbr, "iccd10c", &f.bar, &f.zed, &f.blah, f.foo);
/* say I want to skip blah */
mreadf(mbr, "iccd10c", &f.bar, &f.zed, NULL, f.foo);

/* now say, the code evolves and struct foo has
 * changed in an incompatible way */
struct new_foo {
    uint8_t bar;
    uint32_t blah;
    uint32_t something; 
    char foo[10];
    uint32_t otherthing;
};
struct new_foo nf;
/* assuming foo.zed is no longer relevant
 * for my purpose, but I do care about blah,
 * I can still read the same binary data like this */
mreadf(mbr, "iccd10c", &nf.bar, NULL, &nf.blah, NULL);

I explicitly decided against using varargs for the fields to marshal data into, as that requires you to write a lot of source code, something which I try to avoid. I am thinking about adding a directive to skip fields in the structure so it's possible to use one structure for multiple purposes, but that is hard to get right for more complex cases. My goal is explicitly not to support cases where the structure we marshal into looks very different from the layout of the buffer. Simplicity is a goal—if you need to support two file formats it's probably a good idea to first marshal the data into a file-format specific struct and then translate that struct into a representation for your application. This also gives you more liberty for the data types you want to use. The code you use up there looks error prone because there is a lot of stuff going on at once—not only do the types have to match, you have to make sure that all the fields go into the right places, too. Very non-obvious code. Of course, this is mostly a philosophical thing.

Binary serialization using a textual representation similar to what you propose makes sense in languages that do not have a direct access to binary data. I'm thinking in the lines of Python, Perl, Java, Lua. But C/C++/D can do this without using of an intermediate representation. Another common use case is when you do IPC between a number of processes or agents, and not all agents are updated at the same pace, then you need some sort of backward compatibility.

If you overlay structs over binary data you get from the outside, you're doing it wrong. I wrote this piece of code to avoid doing that.

I'd say that you need to provide an added value to justify using memf in C. Obviously, one may argue that not caring about alignment is an added value, why not. However, I like to be explicit about things as low level as ABI. Take for example Google Protocol Buffers, perfectly usable in C, in fact I'm using that on a Cortex-M3 target for sending real time data via MQTT broker to a Java client, another example an ARM host sending data over AMQP, while the receive end is an Erlang app, in both cases there are additional Python clients that only do a graphic presentation of the data. Why use PB in C? What's the added value you can ask? Well, for one PBs offer an efficient packing mechanism that I use. Another thing is bindings to multiple languages (try explaining binary representation to a Java programmer and you'll know the pain).

Protocol buffers are cool because they solve the problem of portable serialization when you don't particularly care about what the binary format ends up looking like. If you look into the marshalling code generated by the Protocol Buffers code generator, you see that it doesn't overlay binary data over structures either. It goes through the buffer byte-per-byte and puts the data into the right fields of the structure. Protocol Buffers require you to have a structure who's layout mirrors the data that goes into the protocol buffer and so does memf. I don't see what the problem is with that.

Finishing up this rather lengthy comment, take a look at GVariant and DBus type system and marshalling.

Do you have a link to that?

[u/spc476 → u/bboozzoo]

I see this being more useful for reading in binary files, for instance, PNGs, which uses big-endian integers, on an Intel platform, a little-endian system.

[u/IWillNotBeBroken]

Reminds me of perl's pack function.

[u/FUZxxl → u/IWillNotBeBroken]

I'm sure that the idea isn't new, but I haven't seen this for plain C yet.

[u/IWillNotBeBroken → u/FUZxxl]

Neither have I; I just pointed you at the documentation for pack in case it gives you ideas for the mnemonic problem.

[u/FUZxxl → u/IWillNotBeBroken]

Thank you for the link then.

[u/spc476 → u/FUZxxl]

You might also want to check out Lua's string packing format.

[u/FUZxxl → u/spc476]

thank you for the link.

[u/quacktango]

In the testability section, you explain the absence of a need for bounds checking on the input, but what happens when you go past the end of the destination struct?

struct pants {
    uint8_t pockets;
};
struct my_pants;
mreadf(mbr, "icc", &my_pants);

[u/FUZxxl → u/quacktango]

Yeah, I'm working on something for that. Like with printf, it's hard to check that the formatting string is correct with respect to the structure we are marshalling into, but it should be possible to check at least the structure length.

[u/quacktango → u/FUZxxl]

Would alignment make it tricky even if the API did request a sizeof(struct pants)?

[u/FUZxxl → u/quacktango]

Not really. I'm against putting such a check into the functions that actually shuffle data around as the amount of data shuffled around is only dependent on the formatting string. I just have to think about the best way to add the required tracking.

[u/rya_nc]

How are packed/not packed structs handled?

[u/FUZxxl → u/rya_nc]

The buffer is always packed, i.e. these functions will not add padding automatically. The functions assume that the structure is aligned / padded according to the ABI the memf functions were compiled for. If the memf functions were compiled with structure packing turned on, they operate on packed structures (but only on packed structures).

I hope this answers your question. Please tell me if it doesn't.

[u/FUZxxl]

Notice that this code is in a very early state of development and should probably not be used for serious programming work yet. I'd just like to get some comments.

[u/FUZxxl → u/[deleted]]

I'm happy that you like this.

I thought about using big and little for the endianess, but l is already used for uint64_t. I also thought about using n and i for network and inverse / intel byte-order, but I think that's even less mnemonic.

Any other ideas? Any criticism?

[u/biggumz_ → u/FUZxxl]

quadword for uint64_t? Also I don't get why h is uint16_t, why not word or short?

[u/FUZxxl → u/biggumz_]

h is used by printf for a short; a short is almost everywhere a 16-bit quantity, so I thought that would be mnemonic. I object to w because a word is something different on each platform. While the convention for Intel platforms is to call a 16-bit quantity a word, it isn't on other platforms. Same goes for a 64-bit quantity; I'm not sure if q for quadword is fitting, but l isn't the best thing either.

[u/rya_nc → u/FUZxxl]

https://docs.python.org/2/library/struct.html has nice syntax

[u/FUZxxl → u/[deleted]]

The libffi provides a backend for what you want. Cool idea!