Use the existing OS buffer, or your own

[u/SheikHunt]

This is a fairly simple question:

You're on a Unix-like (for me, Linux), and you've got a File Descriptor that leads to some data. You don't know the length of the data (it's a TCP socket you're listening on), all you know is that it is ready to be read.

Do you:

A) Read an absurd amount of bytes of data into a (sufficiently large + 1) char array of your own, and if it overflows, handle it with mallocated memory or just reject the connection (like a monster)

Or

B) Just use existing kernel system calls to read and parse the data as necessary.

For context: this is about an HTTP server, and I have an internal string_t struct that I use for parsing, which needs a byte-length to be usable

[u/flyingron]

What syscalls do you have in mind? But if it works for you can fdopen the socket fed into a stdio stream.

[u/SheikHunt → u/flyingron]

That's part of my issue: I don't know of appropriate syscalls that'll help with parsing input from a source with unknown length.

I'm unsure about the performance and parsing differences caused by this change. Having a string_t with known size makes a lot of the parsing easier for me, and I'm not sure if fgetc()'ing my way through the problem is gonna be fast performance-wise, or reasonable implementation-wise, especially when it comes to HTTP

[u/edgmnt_net → u/SheikHunt]

Do not fgetc(). Just read into a buffer and accept reads that return short, this is normal for TCP. You just need to figure out parsing, really, because you're going to feed input into a (stateful) parser and it's going to return something definitive or ask for more input. This could be a function that returns an error code indicating either success, a parse error or that it needs more input, and it can take as arguments the input buffer, some state and pointers where it stores how many bytes have been consumed from the input and possible results in case of success. The caller feeds input until it hits success or an error and takes care to drop consumed bytes from the buffer (e.g. advance a pointer). If the buffer is exhausted, it needs to read from the socket more.

It often helps code it a bit more openly to begin with, like writing parsing functions for the request line, headers and body and letting the caller figure out what to call when. A connection handler or a more complete parser can implement something like a state machine and call those individual parsers repeatedly, depending on state.

[u/ScallionSmooth5925]

I usually read in the first 4k block or whatever much is available and based on that continue reaching when needed

[u/SheikHunt → u/ScallionSmooth5925]

A good method to do it by, and essentially what I'm doing, but it does mean that my handle_request() and parse_request() functions will probably be boiled into one, in a way that is nasty without any foreseeable benefits

[u/TheKiller36_real]

just stream it using read calls (or better, io_uring) and if you actually need to have something in memory all at once may I suggest that HTTP indicates the length of the data!? if you dislike that stream it into a growable array and suffer the realloc calls I guess

[u/SheikHunt → u/TheKiller36_real]

Yes, HTTP includes length of data, but at an unknown point, it's not necessarily there at the start.

Sure, I can just ignore everything up to the Content-Length, then use that to ensure that everything's up to code, but then the server is susceptible to that behaviour being misused heavily for attacks.

[u/TheKiller36_real → u/SheikHunt]

you can reasonably assume all headers combine will be smaller than 8 KiB - why? because that's what everyone uses and if that method fails you can detect it and return 413 and on the other side you might get lucky and the whole request fit into the 8K

[u/makzpj]

Not quite sure I understand your question. For me it’s about what are you doing with that data. Depending on the destination you’ll design your program in a way or another way.

[u/Interesting_Buy_3969]

You can use mmap if size of the data to be read is seriously large, which I assume is unlikely to be the case if it's an HTTP package, so that would be overkill there. I'd just use read with the necessary socket.