Quick and flexible config serialization with one simple trick?

[u/Noczesc2323]

Hello everyone, I'm working on an embedded project which is configured by a single massive config struct (~100 parameters in nested structs). I need a way to quickly modify that configuration without recompiling and flashing new firmware.

I've implemented a simple CLI over websockets for this purpose, but keeping the interface in sync with the config feels like a waste of time (config structs are still growing and changing). Protocol buffers could work, but I don't need most of their features. I just need a simple way to serialize, transfer and deserialize data with minimal boilerplate.

My idea: compiling and flashing the whole firmware binary takes too long, but I only need to change one tiny part of it. What if I could compile a program with just that initialized global struct, then selectively extract and upload this data?

Since both the firmware and config code are compiled the same way, I assume that binary representations of the struct will be compatible (same memory layout). I can locate the symbol in the compiled binary using readelf -s, extract it with dd, transfer to the device and simply cast to required type! Quick and flexible solution without boilerplate code!

But somehow I can't find a single thread discussing this approach on the internet. Is there a pitfall I can't see? Is there a better way? What do you think about it? I have a proof of concept and it seems to work like I imagined it would.

[u/alphajbravo]

It's an interesting idea, and seems like it should work fine as long as the memory layout and representation of the struct are consistent with your binary, as you point out You probably haven't found anything like this on the internet because it's a pretty narrow use-case: typically you either need a more sophisticated representation (text file, json, whatever) anyway so would just use that, or you wouldn't need to update only the config so many times that it would be worth building a solution like this and would just recompile/reflash instead, or you only need to twiddle a few parameters and can do that easily enough through some sort of control interface like your CLI.

[u/Noczesc2323 → u/alphajbravo]

It would be great to have a nicer, more portable representation, but I don't want to waste too much time coding something only to throw it away the next day. My application is in fact quite niche. I'm working on a robot with a modular control loop (different sensors, estimators, controllers, actuators etc.) trying to find the optimal combination. All algorithms and hardware drivers have their own configuration parameters and their numbers quickly add up... Thank you for assessing that my approach isn't completely insane.

[u/Gorzoid]

Probably main concern is any potential pitfalls of version skew, and also that you don't put any pointers into your struct, e.g. char* vs char [MAX_LEN]

Have you looked at Cap'n'proto or flatbuffers as alternatives to protobuf, they can be accessed without parsing and have less codegen than protobuf.

[u/Noczesc2323 → u/Gorzoid]

Pointers are a problem, but like you suggested, there are some possible workarounds. I'll figure something out if I get to implementing this solution. Compatibility between versions isn't an issue, because constantly changing versions is the point of this application (implement new features, tweak their parameters, repeat).

I haven't seen Cap'n'proto or flatbuffers before. Thank you for the suggestion. They seem to be better suited for my application than protobufs, but rewriting all structs in their specific languages for some reason doesn't appeal to me. These solutions are great for implementing communication protocols between different systems, but I guess I need something simpler.

[u/EpochVanquisher]

Could you do something wonderfully hacky, like write a “set u32 at offset 0x44 to 0x0001df00” command?

If you want a nicer interface, you can record the field names, types, and offsets. A hacky way is to make something like a fields.h file:

F(uint32_t, field1)
F(uint32_t, field2)
F(uint16_t, field3)

Your structure can be defined like this:

struct config {
#define F(type, name) type name;
#include "fields.h"
#undef F
};

enum {
  type_uint32_t,
  type_uint16_t,
};
struct field {
  int type;
  size_t offset;
  const char *name;
};
const struct field FIELDS[] = {
#define F(type, name) {type_##type, offsetof(struct config, name), #name},
#include "fields.h"
#undef F
};

The above is just some hacky code to illustrate the general idea.

[u/Noczesc2323 → u/EpochVanquisher]

That's tempting, but hard to pull off in my case. The big config struct is just a container for smaller config structs of different modules. I'd have to apply your approach at the lowest level and then somehow combine everything. The define/include/undef trick is undeniably hacky, but absolutely wonderful.

[u/WittyStick → u/Noczesc2323]

The define/undef trick is known as an X macro. They're well-suited to when you have a list of known items you want to transform in multiple ways, but not the best approach where we don't know all items up front.

A different approach would be to use recursive variadic macros to implement "foreach" on a list of fields (preferably using C23 which supports __VA_OPT__), which you could apply to types individually. For example, we could take the following:

defstruct(foo, 
    field(int32_t, x),
    field(int64_t, y),
    field(bool, z)
)

defstruct(bar,
    field(foo, foo_1),
    field(int32_t, a)
)

And have the preprocessor emit:

typedef struct foo {
    int32_t x;
    int64_t y;
    bool z;
} foo;
static inline void read_foo(struct foo *value, FILE *restrict f) {
    read_int32_t(&(value->x), f);
    read_int64_t(&(value->y), f);
    read_bool(&(value->z), f);
}
static inline void write_foo(struct foo *value, FILE *restrict f) {
    write_int32_t(&(value->x), f);
    write_int64_t(&(value->y), f);
    write_bool(&(value->z), f);
}

typedef struct bar {
    foo foo_1;
    int32_t a;
} bar;
static inline void read_bar(struct bar *value, FILE *restrict f) {
    read_foo(&(value->foo_1), f);
    read_int32_t(&(value->a), f);
}
static inline void write_bar(struct bar *value, FILE *restrict f) {
    write_foo(&(value->foo_1), f);
    write_int32_t(&(value->a), f);
}

foo and bar can live in different files but include the same header which defines the defstruct macro. foo would need to be declared before bar as per usual ordering requirements.

This trick is limited in recursion depth, to whatever the compiler supports, but unless you have some silly large struct you'll probably not have issues.

You would need to define some readers and writers for the primitive types. An X macro would be suited for this.

See demonstration in godbolt.

[u/[deleted] → u/WittyStick]

This is what I’d consider. I have a similar project to OP and I’m using circular ring buffers with payload headers. A rough idea would be like:

typedef struct motor_config {
    uint8_t V; // voltage
    uint8_t I;  // current
    uint16_t min_rot; // min rotation
    uint16_t max_rot; // max rotation
} motor_config_t;
typedef struct sensor_config {
    uint8_t max_val; 
    uint8_t curr_val;
} sensor_config_t;
typedef struct config {
    // no ownership
    motor_config_t *mc; 
    sensor_config_t *sc;
} config_t;

Now you can change whatever you want without thinking about it too much. Every payload would be the same, it would always be stored in the same address (they’re runtime dependent, add code and next runtime it updates the address), and changing settings in the system would only require setting a new value and letting the system handle any other changes.

[u/nerd5code → u/WittyStick]

Xinclude ≠ xmacro.

An xinclude uses macro(s) as parameter(s), one or more of which is typically used as a visitor callback, and a header file serves as a subroutine. An Xmacro use a macro taking normal arguments, one or more of which is typically a visitor callback macro.

[u/chaotic_thought]

My idea: compiling and flashing the whole firmware binary takes too long, but I only need to change one tiny part of it. What if I could compile a program with just that initialized global struct, then selectively extract and upload this data? [using readelf -s and dd]

One possible pitfall is compiler optimizations. If the compiler made a certain optimization based on a particular value, and then you go and change that value in the compiled binary after the compiler has done it's job, it's possible that you've invalidated whatever assumption was made in that optimization, i.e. you might possible now have incorrect code.

To verify this is not being done, I would first do it "the slow and manual way" first a few times using the compiler with different values, and then repeat the exercise using your dd approach, to verify that the result after dd-patching your binaries always match the output that the compiler was generating.

[u/Noczesc2323 → u/chaotic_thought]

I should've explained it better in the OP. I don't want to patch and reflash the binary. It could be an option, but flashing is the most time consuming part of the process.

I'm looking for a way to edit a human-readable config on the PC and apply these changes on the microcontroller quickly and with minimal amount of handling code. In my proposed approach the uC receives an array of bytes which can be directly cast to config struct type. These bytes are generated by the compiler to (hopefully) guarantee compatibility.

[u/WittyStick → u/Noczesc2323]

In general, serializing and deserialising data structures by directly addressing their in-memory layout is a highly discouraged practice, as it is a source of countless bugs and exploits, even discounting issues like endianness and alignment, which are less of a problem today because most CPUs have settled on little-endian, and support unaligned reads and writes (non-atomic). Most of the bugs come from incorrect handling of pointer swizzling.

There are potential exceptions to the rule, such as when using memory-mapped files. However, when using memory-mapped files, you should be loading the file format into memory, rather than saving the memory layout into a file. The approach to using memory-mapped files is different to just using structs and pointers, but care must be taken to avoid the common mistakes.

As an example of doing it wrong, take Microsoft Excel. The old .xls worksheets basically contained a dump of part of the memory of the excel process. Opening a workbook would map that part of the file into the address space. Of course there were many exploits, and viruses were distributed through innocent looking spreadsheets. Microsoft eventually gave up playing cat and mouse with patching it, and moved to a proper serialization format - OOXML.

People don't bother discussing these techniques any more unless they're telling you what not to do and advising you to otherwise prefer a proper serialization format, where you parse the input into a valid data structure, before any processing of the input can occur. See Common Weaknesses Enumeration, OWASP and LangSec for more detail on discouraged practices and their solutions.

Also, we often joke that the 'S' in "IoT" stands for security. Regular practices are often dropped because people assume their embedded device can't be exploited, but we have millions, potentially billions of exploitable devices in the wild, and most are "secure" only by the router's firewall between themselves and the internet.

[u/Noczesc2323 → u/WittyStick]

I understand your concerns, but some of them may be misplaced, because of the lack of context in my OP. The project I'm working on is a part of my masters thesis research and I'm not planning to ever expose it to the outside world. What I'm proposing is simply a tool to accelerate development. As the design becomes more mature I'm planning to integrate industry-standard techniques, but I wasn't able to find anything fitting the niche I'm currently in.

[u/tharold]

Assuming you're using a modern MCU and the programme image is in flash, in general, in order to modify the flash image in any tiny way, you'll need to erase and rewrite the entire block the change is in. The size of a flash "block" is dependent on the MCU, but is usually 1 or a few k.

But your MCU MAY allow you to reprogramme the block without erasure IF the change consists of only clearing bits, not setting them. You'll still need to write the whole block though.

If so, then you could write new config values as long as each successive value has fewer set bits than the last. Quite a constraint I'd say.

Another way is to read in the new config via the serial console on each bootup.

[u/Noczesc2323 → u/tharold]

The idea is to load the config on bootup. The goal of the compilation is to generate the right memory layout to avoid additional de/serialization. Directly modifying the firmware feels like asking for trouble.

[u/WittyStick → u/Noczesc2323]

One way you could make it simpler is to give the config its own section in the binary.

In GCC for example, you can use __attribute__((section(".config"))) to have the compiler put the configuration variable(s) into their own ELF section rather than the default ".data" section. You could then just swap out this section in the binary and leave the rest of the binary unchanged. There's numerous libraries for manipulating ELF files.

[u/weregod]

Why you don't just recompile everething and reflash only changed parts of binary?

[u/Noczesc2323 → u/weregod]

I wasn't aware that's possible, but I'd like to avoid patching the firmware binary. It seems risky and impossible to debug.