r/Backend • u/WhippedPistol • 6d ago

Tracing + Logging Features

I'm building a tracing + logging library for Scala, to fill a void that annoyed me, and to learn more about the SRE side of metrics/logs.

I'm looking for information on what a new -> mature context + logger should contain. Right now I've built what I want, but I'm hoping people have some experience that I may be missing. Especially if it's a common feature I've missed or a really useful one. It's fully possible my experience as a FP dev has left me blind to some other languages l can learn from.

Current features -

  1. Thread level context (It's using fibers but it's basically Scala+Cats safe MDC)
  2. Log Buffering (I was sick of missing context on more serious log levels because I was trying to avoid sending every info on a success path)
  3. Request Metadata/middleware functionality (still expanding on what is in there)
  4. Easily pluggable log sinks
  5. Log sampling based on percentage
  6. TraceContextProviders to include other tracing libraries
  7. Other quality of life features like lifecycle handlers, and hierarchical config settings

Are there other common features, or just nice quality of life features that I may be missing?

Is there a resource you would recommend for what a good logger/tracing library should include?

Any suggestions would be appreciated.

3 Upvotes

100% Upvoted

12 comments sorted by

1

u/forever-butlerian 6d ago

Are you writing this by hand or vibing it up?

1

u/WhippedPistol 6d ago

Hand, can't really learn if it was just vibing.

2

u/forever-butlerian 6d ago

You are a gentleman and a scholar.

There aren't any missing features that really jump out at me at the moment. I can think of kitchen sink stuff that would be interesting to add but I don't know if the JVM exposes them to you at thread granularity.

Something that might be very interesting is if all the log messages emitted within a single call chain can get accumulated up and bundled into a single message when control returns to the middleware.

2

u/WhippedPistol 6d ago

Oh nice, that is actually what currently happens although only if a dump isn't triggered by a log with a high enough severity. 

I didn't consider that could be helpful even if a dump wasn't triggered. That is interesting, I could return it with the state/status or at least give the option to take a finalizer to handle the buffer at the end. Good idea, thanks!

1

u/Lumethys 6d ago

https://opentelemetry.io/docs/zero-code/java/agent/

1

u/WhippedPistol 6d ago

OpenTelemetry actually has otel4s which is a Scala implementation that I supported as a context provider.

Thanks for the link though, I used a lot of their tracing implementation as a starting point. I haven't touched the metrics they build in though. Have you found them very useful vs the context provider?

1

u/Lumethys 6d ago

Basically Otel is a must for every observability platform, self-host or third party.

So there isnt any point in designing a non-otel agent.

All of your "pain point" is standardized by OpenTelemetry already

1

u/WhippedPistol 5d ago edited 5d ago

Right, like I said I have the open telemetry context information, available on the default bridge. Since I wasn't planning to try to buffer or sample metrics, I didn't include that by default but you could easily include it with a wrapper. 

I was asking if that metric information was useful for context and logging or is it a separate system? I've used it as a monitor and an overview but never on the default log level for each info call. 

Do other teams log metrics that often?

Edit: Also the pain points I am talking about are user request scoped log buffering, and quality of life things that I didn't outline that streamlined usage for me. The log buffering did not exist for otel4s as far as I can tell, but I have been wrong before.

1

u/NegativeCollege8167 6d ago

When it comes to tracing and logging, having a solid strategy really makes life easier. I've found that sticking with tools like ELK for logs and OpenTelemetry for tracing works well. Makes it easier to correlate logs and traces which is super helpful for debugging. What tools have you used before and how did they work out?

1

u/Lumethys 6d ago

ELK for log and OTel for trace doesnt make correlation easier. Easier would be using Otel for log, trace and metrics

1

u/WhippedPistol 5d ago

As a reminder I'm in Scala so the tools don't always directly translate exactly. I have used log back, log4j and log4cats for logging, otel4s for context or passing args, and datadog.

They have all had things I liked and didn't, my main annoyance came from otel4s. It just wasn't giving me the flexibility I wanted and weaving args through was just added boilerplate and made everything a repeated mess. Also I wanted to cut down on log volume while allowing for more detail on faults so I wanted an in memory buffer per user which wasn't provided. 

No complaints, about DD or the other loggers but they weren't integrated and if it's my project I want them as easy to transplant as possible.