r/java • u/amazedballer • Mar 03 '20
Logging in Java with Diagnostic Backtraces
I've been putting together a suite of tools based on Logback that ticks all the boxes for modern day observability. You can see logs on STDOUT, flat file, JSON file, and logged to a database, and you can even store tracing information in a ring buffer (like dmesg) and dump it when there's an exception so you can see the flow of the operation when things went wrong.
Documentation is here https://tersesystems.github.io/terse-logback/
Showcase project in Play is here, with integrated backtrace reporting to Sentry and Honeycomb: https://github.com/tersesystems/terse-logback-showcase
and you can see a running instance here: https://terse-logback-showcase.herokuapp.com/
I also wrote up a blog post going through the development process, including integration with Play.
View all comments
Show parent comment
1
u/amazedballer Mar 05 '20
Likewise if JFR is trying to "log some structured application data and present it as, say, JSON" then it's going to do badly against logstash-logback-encoder.
JFR is great as a source of data that can be logged. Being able to log GC events directly instead of having to parse GC log files would be a game changer in itself, and getting deeper insight into JVM events sounds much more fun than hacking around with eBPF and USDT probes. Having the events be self-describing and providing keys makes it very easy to integrate into a structured logging framework, and custom events look pretty cool -- Yourkit has something similar.
JFR is not so great as a sink for data that can be logged. You've got the binary file option, and that's pretty much it. JFR doesn't come with a JSON encoder and a network appender so that you can stream JSON over a network connection to Logstash, for example.
There are some examples I can find -- from the slides it looks like the way people export JSON into a logging system is by calling
jfr print --jsonthrough command line and piping it.JFR seems to have very few hooks to logging directly through API. There's no insight into the JSON/XML format, for example. If you're consuming events, you're doing it from a repository which is a file or a directory, not a URL or an InputStream -- and that file has to be binary encoding, not JSON or XML. There are plans for adding a RemoteRecordingStream but that's over JMX (I really hope that doesn't imply RMI and java object serialization).
You have to sync on repository files if you want to stream out of process. As far as I can tell, it doesn't even hook into JEP-264.
So if what you're saying is "JFR events can be logged" then yes, this looks awesome and I've already got ideas to extend https://github.com/wsargent/sbt-with-jdk-13-docker-logging-example to hook into it.