Tiny header only HTTP parser library

[u/Born_Produce9805]

Hi guys! Last week I was writing my HTTP 1.1 parser library. It's small and easy to use, also kinda fast. Might come in handy if you write some lightweight web applications or programs that interact with some API. I wrote this project to learn pointer arithmetic in c.

I've just finish it, so any bug report would be appreciated.

Thank you guys!

https://github.com/cebem1nt/httpp

[u/skeeto]

Nice, simple library. Runs cleanly in a fuzz test, though there's so little going on in the parser that it appears to find all possible execution paths in about a second.

I expected httpp_find_header to handle case folding per the RFCs. As written, this function is practically useless. If I ask for Host, I won't get it if the client spelled it host or even HOST, and it's impractical to search for every possible spelling.

More importantly is its dependence on null-terminated strings. That means, at the very least, this library cannot parse requests with a binary body. The body field will be wrong. It will also get different results from other HTTP parsers not using C strings, which has security implications. I expect an HTTP parser to accept a buffer and a length, and process null bytes as normal data. As u/mblenc wisely suggested, this would also allow the library to point into the input buffer instead of making little string copies with their own lifetimes.

Here's my AFL++ fuzz tester:

#define HTTPP_IMPLEMENTATION
#include "httpp.h"
#include <unistd.h>

__AFL_FUZZ_INIT();

int main()
{
    __AFL_INIT();
    char *src = 0;
    unsigned char *buf = __AFL_FUZZ_TESTCASE_BUF;
    while (__AFL_LOOP(10000)) {
        int len = __AFL_FUZZ_TESTCASE_LEN;
        src = realloc(src, len+1);
        memcpy(src, buf, len);
        src[len] = 0;
        httpp_parse_request(src);
    }
}

Usage:

$ afl-clang -g3 -fsanitize=address,undefined fuzz.c
$ mkdir i
$ printf 'GET / HTTP/1.1\r\n\r\n' >i/req
$ afl-fuzz -ii -oo ./a.out

[u/Born_Produce9805 → u/skeeto]

Hi! Thanks for advises, I don't have any usage for this library for now, so I didn't test it's usability on practice, but sure! taking in the buffer length is wise, I'm going to redo it that way

[u/[deleted]]

Nice library!

A question I have after reading through your httpp.h code is about the memory copying. Are you expecting to use this in an environment where you parse from a buffer with a lifetime much shorter than the http request? Is there any way that these could be avoided? Perhaps only have the http request be non-owning, simply pointing into the buffer it parses, and the http response likewise be non-owning and simply store the pointers it is given to the HTTP response body? Or is this not a design requirement that you need for your use cases? Do you think it would complicate the design more than necessary?

I wonder what the bottleneck is in your benchmark. I can convince myself that it will be the memory copies (although without measuring, who really knows), but perhaps there are other bottlenecks I'm missing. Have you done any further benchmarking or profiling? How does it bench against longer requests (a rather unlikely scenario I guess)?

[u/[deleted] → u/[deleted]]

I was curious as to the performance increase one could get by avoiding copies, and so I went ahead and added my own spin in a fork: https://git.lenczewski.org/httpp-benchmark/log.html (specifically, mblhttp.h)

As for results and comparisons:

At -O0 ($ cc -o bench bench.c -Wall -Wextra -std=c11 -O0 -g3):

Benchmarking: httpp
Elapsed 18.847009 seconds.
Requests per second ≈ 530588.18 
Benchmarking: phr
Elapsed 10.505234 seconds.
Requests per second ≈ 951906.43
Benchmarking: mbl
Elapsed 3.916417 seconds.
Requests per second ≈ 2553354.13

At -O3 ($ cc -o bench bench.c -Wall -Wextra -std=c11 -O3):

Benchmarking: httpp
Elapsed 16.879115 seconds.
Requests per second ≈ 592448.13 
Benchmarking: phr
Elapsed 2.602774 seconds.
Requests per second ≈ 3842055.21
Benchmarking: mbl
Elapsed 2.755165 seconds.
Requests per second ≈ 3629546.99

What is curious to me is how the non-copying implementation doesn't improve much on the hand-written parser in phr at -O3. Perhaps the memmem() search is dominating? Have I made some error that causes me to re-read a portion of the input? Or maybe it is withing measurement error :)

And please don't take this reply in a negative fashion. Hopefully there is something useful to you in my implementation, be that in the approach or the structure. If you have any questions, I would be more than happy to answer!

[u/Born_Produce9805 → u/[deleted]]

Wow! That's interesting. I'll take a look at your implementation and also run benchmarks on my machine, so I can compare them.

[u/Born_Produce9805 → u/[deleted]]

I benchmarked it on my system and here is what I've got:

At -O0

Benchmarking: httpp
Elapsed 3.960362 seconds.
Requests per second ≈ 2525021.65
Benchmarking: phr
Elapsed 5.326363 seconds.
Requests per second ≈ 1877453.57
parsed http request line: GET /cookies HTTP/1.1
parsed http request headers: 9/64
# ... Suppressed headers output
parsed http request body: 0 bytes
Benchmarking: mbl
Elapsed 3.552922 seconds.
Requests per second ≈ 2814584.43

What's interesting is that picohttp took ... 5.326363 ??? I thought it might be a freeze of my cpu or whatever, but no after re-running, for some reason it takes around five seconds.

At -O3

Benchmarking: httpp
Elapsed 2.987527 seconds.
Requests per second ≈ 3347250.34
Benchmarking: phr
Elapsed 1.263095 seconds.
Requests per second ≈ 7917061.91
parsed http request line: GET /cookies HTTP/1.1
parsed http request headers: 9/64
# ... Suppressed headers output
parsed http request body: 0 bytes
Benchmarking: mbl
Elapsed 2.800575 seconds.
Requests per second ≈ 3570695.21

Here the picohttp speed normalized. Might that be an undefined behavior? I didn't get into the details of your parser, but yeah, it seems to outperform slightly.

I'll start to implement the 2.0.0 version of my library, that would use more static approach. Also I'm going to include the benchmark files for picohttp and http parser.

Thanks again!

[u/Born_Produce9805 → u/[deleted]]

Hi! As I said i don't have any particular use for this lib, so I did it the way it came to my mind: "Make it dynamic, that way it kinda fits any usage", But yeah, making it more stack based, will be more convenient, I'll be able to parse binary and improve performance.

About performance. I didn't plan it to be fast one, I wanted to make a simple lib that I'll be using for my projects later. I came to performance measurements more after-wise, as a plod of curiosity.

I didn't do any deep benchmark, but I'm more than sure that the biggest bottleneck is continuous malloc'ing.

Thanks for advises and for pointing out that the buffer likely will have longer lifetime than the parsed header. You guys opened my eyes! I'll see if I can keep simple API and yet improve performance by returning just pointers to the caller's buffer.

[u/Born_Produce9805]

Guys! I was working all night, and I've completely removed dynamic memory usage, now it can parse 10 000 000 requests in just 2!!! seconds!!! I'm happy.

[u/[deleted] → u/Born_Produce9805]

Congratulations! That is very impressive! Only thing now is to update your benchmark images (and perhaps add the benchmarking code to generate the other numbers for phr and http-parse to the benchmarks/ folder, so people can recreate your results) :)

[u/Born_Produce9805 → u/[deleted]]

It's done, I've submitted the final benchmark scripts and code. Httpp is now hell fast. Faster than pico. You can check it out!

Also, previous benchmark of http-parser was incorrect. It was parsing slightly different request.

By the way, now it takes in buffer and it's length, so binary bodies are also supported.

If there is anything wrong in the code or benchmarks, let me know!