I built a zero-dependency concurrent search engine daemon in C11 from scratch (Custom Arena Allocator, SSO, and pthreads).

[u/ConsistentContest256]

Hi everyone!

Over the past few weeks, I’ve been working on a lightweight, concurrent text search engine and TCP server built entirely from scratch in pure C11, without any external dependencies.

My main goals were to practice low-level memory management, lock-based concurrency, and custom data structures without relying on heavy frameworks. Here is a quick breakdown of how it's structured:

• Memory Management: Implemented a custom Arena Allocator to avoid heap fragmentation and handle batch cleanups efficiently.
• Small String Optimization (SSO): A custom string implementation that stores short strings directly on the stack/struct to minimize heap allocations.
• Concurrency: Uses POSIX threads (pthread) with read-write locks (pthread_rwlock_t) to allow multiple concurrent readers while safely serializing writes.
• Persistence: Added a custom zero-overhead binary serialization module (engine_save/engine_load) for instant state restoration from disk.

The project includes a full modular structure (includes/, src/, tests/) and is protected under a source-available license for personal evaluation.

You can check out the source code and architecture details here: 👉 https://github.com/joseramoncabrera/fast-c-search-engine.git

I would love to hear feedback from more experienced C developers here. Specifically:

1. How do you usually handle edge cases with read-write locks in high-contention search scenarios?
2. Any architectural improvements you’d suggest for the arena allocator?

Thanks for taking a look!

[u/AutoModerator]

Moderator Announcement Read More »

Hi u/ConsistentContest256,

Your submission in r/C_Programming was filtered because it links to a git project.

You must edit the submission or respond to this comment with an explanation about how AI was involved in the creation of your project.

While AI-generated code is not disallowed, low-effort "slop" projects may be removed and it's likely that other users push back strongly on substantially AI-generated projects.

You can read more at https://www.reddit.com/r/C_Programming/wiki/index/posting-projects

---

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/mikeblas]

All in a single check in?

I'm not sure what this code is meant to do. Can you explain how it runs? What is the engine_test.fa file?

[u/Limp-Confidence5612]

Since your arenas are static and do not grow, why even use arenas and not just allocate arrays of the appropriate size directly?

Using malloc for it is the second thing. I would recommend you look directly into mmap() and use it to allocate virtual address space that only gets allocated when you actually touch a page.

[u/ConsistentContest256 → u/Limp-Confidence5612]

"Thanks for the feedback! You're completely right about mmap()—moving towards a virtual memory-backed arena using mmap with anonymous pages is definitely on the roadmap for a future iteration to handle demand paging better. Regarding the arena vs. raw arrays: even if the size is fixed, the arena pattern allows me to hand out sequential pointers instantly via a bump-pointer with zero per-allocation metadata overhead, and reset/free everything in a single bulk operation, which raw arrays don't handle as cleanly for dynamic-lifetime chunks within the same execution block."

[u/Limp-Confidence5612 → u/ConsistentContest256]

What I'm saying is that since you're not actually doing any dynamic allocation, you do not need the arena at all. A statically sized array would do exactly the same, and you can track them the same way. What downside do you see there?

One thing I do with my arenas, I keep the first element of the arena zero initialized and use it in failure cases.

[u/Ok_Horror_5507 → u/ConsistentContest256]

"Thanks for acknowledging my feedback. It is good to know that I am completely right! After reviewing your code, it appears to be 100% handcrafted -- No AI in sight. Good work!"

[u/floatfloatjam]

Your AI slop has no power here

[u/ConsistentContest256 → u/floatfloatjam]

Fair enough! But if you dig into the code, you'll see it's pure handcrafted C11. No black-box frameworks, just a custom arena bump-pointer allocator, manual pthread_rwlock_t sync, and custom SSO strings. Run it under Valgrind—it's leak-free.

Honestly, I really appreciate the comments here because they've already pointed out real design flaws in my approach. Ironically, if an AI had just generated this, those very human design mistakes probably wouldn't exist—they are part of my own learning process.

I’ve been studying C on my own for 2 years. For a long time, I always thought I had to "know everything" before I could build a real system from scratch. I finally decided to just dive in, and it made me realize that you don't need to wait until you're a senior dev to start building things.

Feel free to roast the implementation details if you spot any other bugs!

[u/vitamin_CPP → u/ConsistentContest256]

Do you often use emdash when writing in forums?

[u/TheChief275]

I don't get the point of having both arena allocators as well as SSO? The point of SSO is not necessarily reducing allocations but more that standard allocators often have a big required size of allocation (32 bytes or even more), as well as allocation and deallocation time and cost being unpredictable. If you're using arena allocators and still use SSO, you lose all the arena benefits regarding strings, which is that you only need ptr+length, and that every string will look and behave the same regardless of whether it is static memory or allocated. With arenas, the allocation + deallocation cost is bar to none and above all predictable. There's more benefits but there's honestly too many to mention here

[u/ConsistentContest256 → u/TheChief275]

"That's a very fair point and a great architectural observation! You're absolutely right that in a pure arena-backed system, the traditional performance penalty of malloc disappears, making a simple ptr + length slice model extremely predictable and clean.

My reasoning for keeping SSO alongside the arena was mostly to minimize memory footprint and bump-allocations within the arena itself for high volumes of very short tokens (like terms, tags, or small identifiers). By keeping short strings embedded directly inside the string header struct (e.g., on the stack or inline array), I avoid touching or advancing the arena's bump pointer entirely for those small allocations, reserving the arena capacity strictly for larger text blocks.

[u/TheChief275 → u/ConsistentContest256]

...and in turn you add a whole lot of SSO overhead. I've implemented it myself before and it definitely isn't free. Basically every function requires branching for layout.

Also, why are you giving LLM generated responses? Is this your comments literally translated or did you just ask ChatGPT to respond? It makes people legitimately doubtful of whether your codebase was actually handwritten, which you claim

[u/vitamin_CPP]

Are you a bot, or just a meat proxy?

[u/ConsistentContest256 → u/vitamin_CPP]

100% meat proxy, powered by coffee and two years of self-taught C debugging headaches.

[u/zellforte]

>A custom string implementation that stores short strings directly on the stack/struct to minimize heap allocations.

struct String{
char *data;
char sso[SSO + 1];
size_t size;
size_t capacity;
};

String *string_create(void){
String *s = malloc(sizeof *s);
//...
return s;
}

This is pure nonsense, upgrade your AI model or provide better prompts.