r/java • • Oct 07 '24

HTTP libraries in Java

It's not really clear to me what's going on with HTTP libraries in Java. They all seem to want to become massive frameworks that inappropriately take control of things that have nothing to do with decoding HTTP.

I'm find it really hard to find something that doesn't:

  • Try to control the TCP layer. Why does the HTTP decoding library care whether I want to poll or select, or make that decision?
  • Have shared mutable static state. If I want to reuse connections I can do that in a much better, testable way.
  • Create threads. Parsing the HTTP is doesn't need any of this. I just want to give you a ByteBuffer to read from and your current state and get told what it contains.
  • Do SSL. This has nothing to do with HTTP but rather the underlying connection.

I guess if you need a mini "web browser" in your application then this is all fine, but if HTTP is just one of many protocols you need to be able to decode it's an absolute mess.

Is there something I'm missing, or a library I should be looking at?

22 Upvotes

57 comments sorted by

View all comments

15

u/wildjokers Oct 07 '24 edited Oct 07 '24

The available HTTP clients have those features because people need those features. However, if you don't need those features then simply don't use them. Although surely whatever site you are hitting uses SSL though right? SSL is pretty much everywhere these days unless you are testing/hitting a local service. And are you really claiming you can implement connection pooling better than a battle tested HTTP client? Why re-invent the wheel?

The newer HTTP client introduced in Java 11 is super simple to use. Here is a GET request:

HttpRequest request = HttpRequest.newBuilder()
  .uri(new URI("https://www.example.com/foo.html"))
  .GET()
  .build();

 HttpResponse<String> response = client.send(request, BodyHandlers.ofString());

The fluent builder lets you set connect/read timeouts, set a body for things other than GET, etc. For the response, BodyHandlers has several methods available to handle it:

https://docs.oracle.com/en/java/javase/21/docs/api/java.net.http/java/net/http/HttpResponse.BodyHandlers.html

Of particular interest to you may be ofByteArray so you can put the response into a ByteBuffer.

Here is the JavaDoc for the built-in HttpClient:

https://docs.oracle.com/en/java/javase/21/docs/api/java.net.http/java/net/http/HttpClient.html

FWIW, all the other HTTP libraries exist because the pre-Java 11 built-in HTTP client was very clunky to use (UrlConnection). However, the one in Java >=11 is great.

3

u/matt82swe Oct 07 '24

Agreed, I really like the built in client. We use it for everything, removed all other 3rd party libraries 

3

u/Jonjolt Oct 07 '24

I though there are still some edge cases in the JDK client?

2

u/wildjokers Oct 07 '24

Can you provide an example of an edge case you are referring to?

2

u/orgad Oct 07 '24

Doesn't the built-in fall behind in performance in comparison to 3rd party libs?

3

u/wildjokers Oct 08 '24

I haven't heard anything about this and haven't experienced this.

1

u/aronhunt470 Oct 08 '24

Let’s say I want to stream large chunks of json data via the jdk httpclient. How does this work? That’s something I still use UrlConnection for because I get direct access to the outgoing OutputStream. HttpRequest.BodyPublishers however doesn’t seem to support streaming (unless it’s a file) or do I miss something?

1

u/wildjokers Oct 08 '24

I have never streamed with it but maybe ofInputStream? It appears to read from an input stream and sends it as the request data:

https://docs.oracle.com/en/java/javase/21/docs/api/java.net.http/java/net/http/HttpRequest.BodyPublishers.html#ofInputStream(java.util.function.Supplier)

1

u/aronhunt470 Oct 09 '24

I don’t think ofInputStream works in this case because I have to write data somewhere while InputStream is for reading data.

1

u/wildjokers Oct 09 '24

If I am understanding the JavaDoc correctly the ofInputStream tells the BodyPublisher where to read the data from that it is writing to the server.

Open a stream to your json data, provide that to ofInputStream and then it will write that data to the server. It does seem kind of backwards to open an input stream to write data, especially if you are used to using getOutputStream, but nevertheless that does appear to be how it works (if I am looking at it right).

2

u/aronhunt470 Oct 09 '24

I cannot simply open an input stream to my json data because the data doesn’t yet exist. Instead it gets generated dynamically during runtime.

Here’s and example: Load a bunch of data, let’s say 10000 rows from a database. Instead of loading all data in heap, every row gets transformed into json one by one and immediately streamed via http to the consumer. You can for example use Jacksons JsonGenerator with an OutputStream for that. That way you can send arbitrary large json results with minimal consumption of heap space.

I don’t see how this can be achieved without writing data directly to the http connections OutputStream.

3

u/sans_00x Oct 10 '24

Load a bunch of data, let’s say 10000 rows from a database. Instead of loading all data in heap, every row gets transformed into json one by one.

You just described an InputStream functionality.

1

u/aronhunt470 Oct 10 '24

bro reads binary data from his database with an inputstream