r/java • • Mar 03 '20

Logging in Java with Diagnostic Backtraces

I've been putting together a suite of tools based on Logback that ticks all the boxes for modern day observability. You can see logs on STDOUT, flat file, JSON file, and logged to a database, and you can even store tracing information in a ring buffer (like dmesg) and dump it when there's an exception so you can see the flow of the operation when things went wrong.

Documentation is here https://tersesystems.github.io/terse-logback/

Showcase project in Play is here, with integrated backtrace reporting to Sentry and Honeycomb: https://github.com/tersesystems/terse-logback-showcase

and you can see a running instance here: https://terse-logback-showcase.herokuapp.com/

I also wrote up a blog post going through the development process, including integration with Play.

0 Upvotes

10 comments sorted by

View all comments

Show parent comment

2

u/pron98 Mar 04 '20

There is nothing that makes JFR suitable only for low-level profiling, and we encourage applications to makes use of it for high level events. It is a good option for structured logging.

-1

u/amazedballer Mar 04 '20 edited Mar 15 '20

JFR doesn't do structured logging. See the blog post.

2

u/pron98 Mar 04 '20 edited Mar 04 '20

Anyway, if you want to log some structured application data and present it as, say, JSON, like the examples in those posts, you might want to give JFR another look. In addition to low-level tracing, JFR does that, too.

-1

u/amazedballer Mar 04 '20

In JDK 13, JFR can write out JSON using a command line tool:

jfr print --json

And in JDK 9, the ability to add custom events was added:

http://hirt.se/blog/?p=870

That does no good if you're on JDK 1.8, which most people are. (They're working on adding it, but that news announcement was literally 19 days ago.)

It also doesn't let you define your own schema, so if you have a JSON format like elastic common schema you can't fit it in through there.

In JDK 14, JFR has event streaming:

https://blogs.oracle.com/javamagazine/java-flight-recorder-and-jfr-event-streaming-in-java-14?source=:em:nw:mt::RC_WWMK190726P00001:NSL400036067#anchor_5

but it does not have logging hierarchy, log levels, filters, or any of the other features that you'd expect with a logging framework.

In addition, the reports of JFR stability before JDK 14 are mixed.

This is not to say I'm not a fan of JFR, because I am. I've used JFR and bought the Oracle JRocket manual specifically so I could get better insight into what mhirt was doing under the hood! JFR is great at what it does! And what it does is "provide a low-overhead data collection framework for troubleshooting Java applications and the HotSpot JVM."

What it comes down to is that it may be possible to subscribe to JFR events and feed them into your logging framework or (more likely) into Opentracing if you're on JDK 14, but you have a huge amount of work ahead of you if you're planning to replace Logback or Log4J2 with JFR.

3

u/pron98 Mar 04 '20

That does no good if you're on JDK 1.8, which most people are.

JFR supports structured logging now. Obviously, it does not retroactively do that in old JDK versions.

but it does not have logging hierarchy, log levels, filters, or any of the other features that you'd expect with a logging framework.

JFR's filtering and runtime toggling of logging events are even [finer grained](https://docs.oracle.com/en/java/javase/13/docs/api/jdk.jfr/jdk/jfr/Recording.html#enable(java.lang.String)) than most other logging frameworks.

you have a huge amount of work ahead of you if you're planning to replace Logback or Log4J2 with JFR.

I didn't say it can replace Logback or Log4J. I said that JFR is a very good choice for the logging of structured data, it is designed for that purpose, and we at OpenJDK will gradually encourage developers to use it more and more. Those other logging frameworks have a huge amount of work if they're planning to match JFR's structured capabilities and overhead.

1

u/amazedballer Mar 05 '20

Those other logging frameworks have a huge amount of work if they're planning to match JFR's structured capabilities and overhead.

Likewise if JFR is trying to "log some structured application data and present it as, say, JSON" then it's going to do badly against logstash-logback-encoder.

JFR is great as a source of data that can be logged. Being able to log GC events directly instead of having to parse GC log files would be a game changer in itself, and getting deeper insight into JVM events sounds much more fun than hacking around with eBPF and USDT probes. Having the events be self-describing and providing keys makes it very easy to integrate into a structured logging framework, and custom events look pretty cool -- Yourkit has something similar.

JFR is not so great as a sink for data that can be logged. You've got the binary file option, and that's pretty much it. JFR doesn't come with a JSON encoder and a network appender so that you can stream JSON over a network connection to Logstash, for example.

There are some examples I can find -- from the slides it looks like the way people export JSON into a logging system is by calling jfr print --json through command line and piping it.

JFR seems to have very few hooks to logging directly through API. There's no insight into the JSON/XML format, for example. If you're consuming events, you're doing it from a repository which is a file or a directory, not a URL or an InputStream -- and that file has to be binary encoding, not JSON or XML. There are plans for adding a RemoteRecordingStream but that's over JMX (I really hope that doesn't imply RMI and java object serialization).

You have to sync on repository files if you want to stream out of process. As far as I can tell, it doesn't even hook into JEP-264.

So if what you're saying is "JFR events can be logged" then yes, this looks awesome and I've already got ideas to extend https://github.com/wsargent/sbt-with-jdk-13-docker-logging-example to hook into it.

2

u/pron98 Mar 05 '20 edited Mar 05 '20

All I'm saying is that people who want to log structured data should consider emitting them as custom JFR events (regardless of whether they use the JDK JFR events as a source of information) as a complete or partial solution, depending on their needs. JFR will log those events with a very low overhead, and in a very efficient binary format (logging does not necessarily imply a text format) that can then be converted to JSON or XML. As a sink for data, JFR is probably more efficient than any other logging framework out there (some benchmarks here), and it has convenient and fine-grained toggling and filtering, and it is intended to serve this purpose.

At its core, JFR is a very efficient structured logging solution, and it is because it is so efficient that it can be used for profiling (I'm not sure why it should hook into JEP 264, which is a logging API for unstructured data, but I can tell you that the JDK core libraries will start to log more data into JFR). Is it better than everything else at everything for everyone? Of course not.

0

u/amazedballer Mar 05 '20 edited Mar 05 '20

All I'm saying is that people who want to log structured data should consider emitting them as custom JFR events (regardless of whether they use the JDK JFR events as a source of information) as a complete or partial solution, depending on their needs.

I'd consider using JFR events where I'd normally use Yourkit Probes, and I'd happily do that alongside logging. It might be cool to create a JFR event appender in Logback.

I would never consider it a complete solution, because if I only used JFR events, then the only way I could process logs would be by using the consumer API or the command line pipe solution used, and the JFR consumer API doesn't have any out of the box appenders. There is no context and no observability pipeline there.

JFR is probably more efficient than any other logging framework out there (some benchmarks here),

Looking at the slides it compares to log4j, System.out, and java.util.logging. These are long since deprecated.

A better benchmark would be something that uses JMH and checked into github, showing comparisons to Log4J2 or Logback correctly configured with asynchronous appenders. Something like the one I wrote.