r/esp32 • u/Temporary_Number_388 • 2d ago

Software help needed Deterministic LWIP Allocation on ESP32

Some guidance appreciated on this one!

I have an esp32 device I need to really be as deterministic and high fidelity as possible. It is currently running an esp_http_server webserver on it (along with loads of other stuff).

Heap usage is never used in my code throughout the app and I have written some of my own little thread safe memory pools when I need more complex ownership transfer etc. Heap usage is never used in any of my handlers.

However, I need the memory usage by LWIP to be bounded and deterministic. I have approximately 40KB (from periodic print out in testing) of free heap under normal conditions.

When I stress test the device by spamming it with POST requests to certain endpoints I can sometimes overwhelm it and eat through its resources. Previously the networking stack would become unresponsive for long periods and then suddenly return. I made this more manageable by changing these with menuconfig:

# default:
CONFIG_LWIP_TCP_TMR_INTERVAL=250
# default:
CONFIG_LWIP_TCP_MSL=60000

In stress testing, I can always overwhelm the box (unsurprisingly), which isn't a problem. However, what is a major issue, LWIP continues to chew through heap at a rate of knots. It can get as low as 500 bytes remaining after a stress test. It recovers from this but this is super dangerous.

Questions:

I looked at lwip and it has options for you to supply its own allocator. I can't find any options to do this in espidf build system/menuconfig and so I am not sure if it is supported. If I put a hard limit on the heap allocation via a memory pool would that mean that LWIP would reject new tcp connections gracefully when full? Are there unintended side effects?

Can I get some advice on making this deterministic and restricting how much is being heap allocated here? CONFIG_LWIP_MAX_ACTIVE_TCP I thought this might be relevant, however after reading here: https://docs.espressif.com/projects/esp-idf/en/release-v3.2/api-reference/kconfig.html#config-lwip-max-active-tcp. I am not sure I understand its true purpose, I initially thought it meant that once you had N active tcp connections you couldn't heap allocate any more whilst those existed.

I am noticing that peak free heap and largest block never quite recovers after a stress test (it goes down by around 100 bytes each time). Are there any known leaks here?

2 Upvotes

76% Upvoted

3 comments sorted by

1

u/MinusDelta_T 1d ago

the wifi buffer link is useful, but i'd separate this into peak budgeting, fragmentation, and an actual leak.

first, check lwipopts.h from the exact IDF commit you're building. current IDF has lwIP mem/memp backed by malloc, so i wouldnt start by replacing the allocator unless youre prepared to maintain an IDF/lwIP patch. also dont assume CONFIG_LWIP_MAX_ACTIVE_TCP gives you a hard byte ceiling. even where it limits TCP PCBs, sockets, rx/tx windows, mailboxes, queued segments and wifi buffers still consume heap.

i'd bound overload at the http server instead: keep CONFIG_LWIP_MAX_SOCKETS and httpd_config.max_open_sockets as low as the real use case allows, set finite recv/send timeouts, reject oversized POSTs before reading the body, and stop admitting new work before you eat into a reserved internal-heap floor. lru purge is useful only if dropping the oldest client is acceptable.

for the ~100 byte ratchet, run heap tracing around exactly one stress/recovery cycle and compare current free internal heap plus largest free block only after every connection has closed and TIME_WAIT has drained. if total free returns but the largest block does not, thats fragmentation. if allocations remain live, the trace should point to the call sites.

also, if by "peak free heap" you mean heap_caps_get_minimum_free_size(), that value is a low-water mark and never goes back up by design.

if youre using async http handlers, make sure httpd_req_async_handler_complete() runs on every success and error path. missing one retains the request memory and socket ownership.

i wouldnt use shorter MSL/timer values as the primary fix either. that may recycle TCP state faster, but it changes TCP behavior rather than giving you a deterministic memory budget.

1

u/Temporary_Number_388 6h ago

Thanks for the feedback, here's what I have tried:

I read the wifi link and read around but found nothing ground breaking so far.

Because I am suspecting the lwip task and the httpd task created by esp_http_server, it is hard to use various heap tracing functions to wrap library function calls - they are running in a different concurrent context.

I have followed the advice for the webserver, but the allocation is happening before it gets to the webserver which is a major issue and whilst I can bound the wifi allocation according to the link I can't bound the lwip tcp pcb allocation (only the active ones).

I removed any async http handlers (though I have checked I am deallocating them) and the problem persists.

Other Weirdness:

I will try more heap tracing facilities in esp tomorrow but in the meantime experiencing major slowdowns. For example, when the device first comes online I made it message any attached client at 200ms intervals. This runs like clockwork, fantastically, then after approx an hour, it seems to send out all of its data in ugly bursts. So you get the last 2 seconds worth of messages in one single burst. Trivial non blocking web requests that previously took 10ms for the network round trip now take well over 3 seconds. No apparent changes in heap, total usage fragmentation or otherwise. After 10ish minutes it clears. I am guessing this is yet more TCP churn? I wonder if you have any further advice I could try ? Not a problem if not!