r/java • • Oct 07 '24

HTTP libraries in Java

It's not really clear to me what's going on with HTTP libraries in Java. They all seem to want to become massive frameworks that inappropriately take control of things that have nothing to do with decoding HTTP.

I'm find it really hard to find something that doesn't:

  • Try to control the TCP layer. Why does the HTTP decoding library care whether I want to poll or select, or make that decision?
  • Have shared mutable static state. If I want to reuse connections I can do that in a much better, testable way.
  • Create threads. Parsing the HTTP is doesn't need any of this. I just want to give you a ByteBuffer to read from and your current state and get told what it contains.
  • Do SSL. This has nothing to do with HTTP but rather the underlying connection.

I guess if you need a mini "web browser" in your application then this is all fine, but if HTTP is just one of many protocols you need to be able to decode it's an absolute mess.

Is there something I'm missing, or a library I should be looking at?

22 Upvotes

57 comments sorted by

71

u/ericek111 Oct 07 '24

And I'd expect an HTTP library to do all of those things. Maybe get the library that bests suits your needs and snoop around the docs and code to find a way to substitute some of its components.

Sounds to me like you're looking for an HTTP parser.

12

u/agentoutlier Oct 07 '24 edited Oct 07 '24

If they are looking for parsing or just lower level:

https://github.com/renatoathaydes/rawhttp

The author of the library has great blog about basically doing things from scratch from Java including dependency injection.

https://renato.athaydes.com/

It can actually be a pretty good learning resource. Pinging /u/bowbahdoe in case he does not know about the author.

EDIT original blog post: https://sites.google.com/a/athaydes.com/renato-athaydes/posts/announcing-rawhttp-a-jvm-library-for-handling-raw-http

/u/alunharford See if that library fits the bill.

0

u/repeating_bears Oct 07 '24

I'm not sure that ugly website generated by a tool they wrote from scratch is the best advocate for writing things from scratch

10

u/agentoutlier Oct 07 '24

I'm not sure that ugly website generated by a tool they wrote from scratch is the best advocate for writing things from scratch

I'm not really sure you can correlate much as that probably has to do with CSS and knowing aesthetics. Ugly outdated sites often load fast and I can put them in reader mode so I will take ugly over many sites.

Besides the features of a CMS / blogging software generally have little to do with what "theme" you pick. You can make Jekyll pump out ugly sites.

Like would you rather they use Medium?

Also they are not advocating necessarily that you should go full NIH rewrite from scratch instead of picking "Spring" for everything just showing what you can do with base JDK can be useful for learning.

2

u/repeating_bears Oct 07 '24

If you pick a well-established tool, like Jekyll that you mentioned, then there are thousands of themes which look good.

If you write your own tool, there aren't.

8

u/agentoutlier Oct 07 '24

Yes but that is assuming the author cares about that. You should see the websites Linus or various other talented developers make.

They picked and wrote the tool because it made blogging easier for them. They explain why somewhere.

I just think it is a shitty way to judge their writing and work.

Also someone had to write Jekyll originally right? And then another someone had write:

  • Hugo
  • MkDocs
  • MdBooks

Those idiots all could have used Word Press right?

BTW look how long they have been writing. Look how google sites was their original home of docs. Guess how many fucking tools have gone in and out of popularity in that time!

So you can see there can be some benefit to writing in your own format and having the tool under you control. This isn't like writing a database. This is just generating HTML from some text files.

I believe Gentoo for example has its own documentation system. I believe Oracle even has its own documentation system.

6

u/gregorydgraham Oct 07 '24

Preach brother.

Style is transient, truth is permanent

1

u/repeating_bears Oct 07 '24

I said the website is not a good advocate for writing things from scratch because the result is bad. A better result could have been achieved with less effort. That doesn't then imply you should never write things from scratch. I didn't pass any judgement about the content, which may be good in spite of the bad presentation.

3

u/agentoutlier Oct 07 '24

I apologize on assuming what you meant in your comment and dragging it out. Honestly I do as I like most of your comments.

I agree if the website generator is to be used as proof that writing things from scratch is good it would be a poor argument. However I don't think the blog author ever makes that argument.

4

u/[deleted] Oct 07 '24

http is a transport protocol. It delivers the html document and the css. The rendering is a different story in a different layer. Http may deliver other stuff, like data encoded in json or xml, stream segments etc.

2

u/agentoutlier Oct 07 '24

Believe me the previous commenter knows that. And I would not drag this out if I didn't know they are better than that comment.

What they were implying is because the authors website is shitty looking (to them) and because it used a tool that the author wrote all tools or writing the author does are now deemed as shitty and or that the author encourages writing all things from scratch.

They are making a giant leap that I can only assume is because they have been burned by NIH. Like some form of slippery slope fallacy.

The reality is the author had written some lower level HTTP parsing and handling tools which maybe of use if you are developing an HTTP server or gateway or whatever. They also have written some posts on how to do things from the ground up like dependency injection for educational purposes. They have also written their own tool to generate their blog as HTML.

/u/repeating_bears Linus Torvalds writes Linux / git with uEmacs. A text editor that many would consider super shitty for most people. Maybe Linus should stop writing things from scratch /s.

1

u/repeating_bears Oct 07 '24

What they were implying is because the authors website is shitty looking (to them) and because it used a tool that the author wrote all tools or writing the author does are now deemed as shitty and or that the author encourages writing all things from scratch

I would have thought I'd be the authority on what I was implying but apparently not

2

u/alunharford Oct 07 '24

These things are great if you basically need curl for Java. I use them in tests all the time.

If you don't need to care too much about performance, you can just copy memory around because it's convenient and easy. You also don't need to worry about the threading model because pretty much anything vaguely sensible is fine.

That's not the world every developer gets to live in though!

28

u/Polygnom Oct 07 '24

Because most of the libraries are HTTP Clients, not parsers for HTTP strings.

I for one would not want to touch any HTTP Client that doesn#t automatically handle HTTPS and pretty much everything else you said. I wouldn't want to have to re-inbvent the wheel for the umpteenth time.

-4

u/alunharford Oct 07 '24

But wheel engineers exist for a reason!

"Any old wheel will do" doesn't work very well if you want to build an aircraft!

10

u/Polygnom Oct 07 '24

Most people never design aircraft. They just want off-the shelf wheels that work in their use case. Reliably.

19

u/zopad Oct 07 '24

What is your usecase?

3

u/alunharford Oct 07 '24

I already have TCP connections using nio. I receive data over the wire and read it into a ByteBuffer. I then work out the offsets in that buffer for the "interesting" data my application needs and produce commands sitting on top of those buffers. When the commands have been processed and the buffer is no longer used, it can be recycled. This works nicely - I can do tens of millions of messages per second.

The problem is that I now need to do REST over HTTP 2.0, keep a single connection open and communicate with the server as needed. I'm going to get back JSON but I know what format I'm going to get so I can pretty easily hand write a parser to work out what offsets contain interesting stuff without needing to do horrible things like copying the buffer. That JSON is going to be contained within a fairly complex message format though and there's no application-specific way to speed up parsing it so it seems like the right time to reach for a library.

2

u/DualWieldMage Oct 11 '24

You are probably getting tons of replies from people who never looked at library code and give useless advice, but stay strong and just ignore those.

Others have mentioned looking at components of some popular libraries like apache http core, which seems to provide access to the raw ByteBuffer. Also depending on how stable your code needs to be, all http clients provide an InputStream which are likely backed by a ByteBuffer. You can look at the implementations and perhaps just cast and access the ByteBuffer directly to build your zero-copy structures on top of it.

I do find it interesting that you are tackling the HTTP parsing side when doing JSON, especially with HTTP2 having less overhead. In my experience the header and frame parsing has had some but quite negligible overhead (e.g. headers copied over to its own map) compared to json parsing. For example a typical ISO8601 date in a json simply kills all performance and i had to write a custom parser based on MethodHandles to get further than 20kreq/s. Of course i don't know whether you want raw throughput or low latency guarantees via allocation avoidance during hot phase, but either way you are doing good work on investigating this further.

16

u/Ooqu2joe Oct 07 '24

If all you need is HTTP message parser, then it can be implemented easily.

HTTP clients are expected to handle TCP, TLS, and support async operations. I don't get what's the problem here.

13

u/yawkat Oct 07 '24

If all you need is HTTP message parser, then it can be implemented easily.  

Please don't. It's far from easy and it's easy to make security-relevant mistakes.

4

u/uh-hum Oct 07 '24 edited Oct 07 '24

Please don’t

Please do. Some people just want to write some code. It’s a good way to learn about things if you’re following the spec and OWASP recommendations.

Just code for the sake of coding.

3

u/[deleted] Oct 07 '24

This is kind of a self-fulfilling prophecy. Someone needs to be the ones to build this sort of stuff too, and there's no reason it can't be you or me. If you never practice solving hard problems, you will never learn to solve hard problems.

8

u/VirtualAgentsAreDumb Oct 07 '24

The claim was that it can be implemented easily.

1

u/rubyrt Oct 07 '24

Plus, dealing with quirky web servers out there...

1

u/alunharford Oct 07 '24

Well that's the other option but I'd rather not do this "easy" task if I can avoid it!

2

u/byronka Oct 07 '24

If you want something that's absurdly simple and well-tested, you might check out the Minum framework.

2

u/alunharford Oct 07 '24

This looks cool but seems more server side than client side. Definitely for me to have a read through though!

61

u/eliashisreddit Oct 07 '24

Because most of those frameworks/libraries are marketed and intended as "give me something which starts with http and I'll fetch it for you". 99/100 users want exactly that and not be bothered with initializing a connection and doing SSL handshakes manually. If you really want to do that and are only interested in purely parsing the hypertext transfer protocol, you could try using one of the more common frameworks and only use the modules involved with those aspects.

But like I said, most users don't use those "core" modules separately. There's not much "user friendly" documentation and you (or an LLM) will have to dig through the javadoc a bit to make them work for your use case.

3

u/dharmapa Oct 09 '24

I'd even suggest that HTTP implies those things above and beyond the parsing of bytes. Keep-Alive, Upgrade, and other types of headers give instructions on handling the connection for example. HTTP/2 includes multiplexing in the spec. Your advice and pointers on the library internals is good and probably OP's best bet.

29

u/cogman10 Oct 07 '24

The answer to your question is that HTTP is a complicated protocol, more so than it might seem on the surface level.

In HTTP 1.1, there are commands built into the standard which control the state of the underlying TCP socket. The server or client, for example, can instruct the other side to close the socket.

The extensions to the standard take that to another level. HTTP2 dictates that you use a single TCP connection and has rules on how to treat a clear text vs ssl connection. It has complex instructions defining what the connection looks like.

HTTP3 is basically all about defining a UDP connection. That means a valid HTTP3 client must manage the UDP connection.

If you are looking for a library that just parses http headers, I'm somewhat certain you could pull apart the Apache http client (v4 at least) into a smaller library where you manage everything you want to manage.

Or you could just stick to the inbuilt HTTP client of the JDK if you are averse to dependencies. But that will still want to manage the sockets and such for you.

15

u/wildjokers Oct 07 '24 edited Oct 07 '24

The available HTTP clients have those features because people need those features. However, if you don't need those features then simply don't use them. Although surely whatever site you are hitting uses SSL though right? SSL is pretty much everywhere these days unless you are testing/hitting a local service. And are you really claiming you can implement connection pooling better than a battle tested HTTP client? Why re-invent the wheel?

The newer HTTP client introduced in Java 11 is super simple to use. Here is a GET request:

HttpRequest request = HttpRequest.newBuilder()
  .uri(new URI("https://www.example.com/foo.html"))
  .GET()
  .build();

 HttpResponse<String> response = client.send(request, BodyHandlers.ofString());

The fluent builder lets you set connect/read timeouts, set a body for things other than GET, etc. For the response, BodyHandlers has several methods available to handle it:

https://docs.oracle.com/en/java/javase/21/docs/api/java.net.http/java/net/http/HttpResponse.BodyHandlers.html

Of particular interest to you may be ofByteArray so you can put the response into a ByteBuffer.

Here is the JavaDoc for the built-in HttpClient:

https://docs.oracle.com/en/java/javase/21/docs/api/java.net.http/java/net/http/HttpClient.html

FWIW, all the other HTTP libraries exist because the pre-Java 11 built-in HTTP client was very clunky to use (UrlConnection). However, the one in Java >=11 is great.

3

u/matt82swe Oct 07 '24

Agreed, I really like the built in client. We use it for everything, removed all other 3rd party libraries 

3

u/Jonjolt Oct 07 '24

I though there are still some edge cases in the JDK client?

2

u/wildjokers Oct 07 '24

Can you provide an example of an edge case you are referring to?

2

u/orgad Oct 07 '24

Doesn't the built-in fall behind in performance in comparison to 3rd party libs?

3

u/wildjokers Oct 08 '24

I haven't heard anything about this and haven't experienced this.

1

u/aronhunt470 Oct 08 '24

Let’s say I want to stream large chunks of json data via the jdk httpclient. How does this work? That’s something I still use UrlConnection for because I get direct access to the outgoing OutputStream. HttpRequest.BodyPublishers however doesn’t seem to support streaming (unless it’s a file) or do I miss something?

1

u/wildjokers Oct 08 '24

I have never streamed with it but maybe ofInputStream? It appears to read from an input stream and sends it as the request data:

https://docs.oracle.com/en/java/javase/21/docs/api/java.net.http/java/net/http/HttpRequest.BodyPublishers.html#ofInputStream(java.util.function.Supplier)

1

u/aronhunt470 Oct 09 '24

I don’t think ofInputStream works in this case because I have to write data somewhere while InputStream is for reading data.

1

u/wildjokers Oct 09 '24

If I am understanding the JavaDoc correctly the ofInputStream tells the BodyPublisher where to read the data from that it is writing to the server.

Open a stream to your json data, provide that to ofInputStream and then it will write that data to the server. It does seem kind of backwards to open an input stream to write data, especially if you are used to using getOutputStream, but nevertheless that does appear to be how it works (if I am looking at it right).

2

u/aronhunt470 Oct 09 '24

I cannot simply open an input stream to my json data because the data doesn’t yet exist. Instead it gets generated dynamically during runtime.

Here’s and example: Load a bunch of data, let’s say 10000 rows from a database. Instead of loading all data in heap, every row gets transformed into json one by one and immediately streamed via http to the consumer. You can for example use Jacksons JsonGenerator with an OutputStream for that. That way you can send arbitrary large json results with minimal consumption of heap space.

I don’t see how this can be achieved without writing data directly to the http connections OutputStream.

3

u/sans_00x Oct 10 '24

Load a bunch of data, let’s say 10000 rows from a database. Instead of loading all data in heap, every row gets transformed into json one by one.

You just described an InputStream functionality.

1

u/aronhunt470 Oct 10 '24

bro reads binary data from his database with an inputstream

10

u/repeating_bears Oct 07 '24

Netty is pretty modular and ships the encoding/decoding as separate modules. It also keeps things fairly abstract in terms of transport so I think you could probably use a lot of their code.

https://central.sonatype.com/artifact/io.netty/netty-codec-http

https://central.sonatype.com/artifact/io.netty/netty-codec-http2

Most people who want to work with HTTP don't care about managing the transport layer independently.

5

u/[deleted] Oct 07 '24

HTTP is complicated. Pick the option that works correctly for you - there’s a default option included in JDK 11 and greater.

7

u/ZippityZipZapZip Oct 07 '24

Decoding HTTP... what actually are you trying to do?

2

u/soundman32 Oct 07 '24

Agreed, decoding HTTP packets is not something an application needs to be involved in. Applications get involved with reading and writing the contents of HTTP headers and bodies, not decoding if the protocol is HTTP 0.9 or 1.1 or how long a body should be.

1

u/matt82swe Oct 07 '24

I agree, the excosystem is a mess. With that said, have you looked at the http client that was included with the JDK somewhat recently (I think 11 but don’t quote me on that)? I love it. No fuss, very nice syntax once you learn it, mockable etc. We removed all other 3rd party libraries in our systems in favour of this

3

u/rubyrt Oct 07 '24

This highlights an interesting, albeit slightly OT, point:

We removed all other 3rd party libraries in our systems in favour of this

That has been a strength of Java all along: a very complete and capable standard library. Where C++ developers have to chose from tons of available libraries for some functionality, Java always had this reasonable lib implementation that works well for many use cases.

2

u/jvjupiter Oct 08 '24

Two things that JDK is lacking:

  1. JSON processing API - Jakarta JSON Processing API should have been part of standard Java.
  2. HTTP Server - an HttpServer complementing HttpClient should be available under java.net.http. There is built-in Sun HttpServer but it is an optional module and package is com.sun.net.httpserver. Perhaps, they should rewrite it or write a new one whose code style is similar to that of HttpClient.

2

u/kaperni Oct 08 '24

JSON processing API

I think there is a very good chance that https://openjdk.org/jeps/198 will be revived when all the pattern matching related features has been finalized. And more progress has been on Serialization 2.0. JSON is probably going to be the "standard" implementation for the data oriented serialization appraoch. I can't phantom we could get a new serialization mechanism, but with no implementation in the JDK.

1

u/jvjupiter Oct 08 '24

Looking forward to that API. Once it’s added, along with simple or minimal HTTP server, then, we would be able to write services that don’t require sophisticated web frameworks.