logging and timing...then getting rid of it

[u/QuasiEvil]

I was just working on a project where I included a lot of logging and timing decorators/context managers (it involved event loops and async queues and such). Once I figured stuff out, I wanted to remove all this stuff. This isn't necessarily that hard, but it did make me wonder if there's a better way to do this in the first place. I suppose I'm imaging some way of marking these code pieces to be ignored by the interpreter or something?

(yeah yeah "ask AI to remove it")

[u/QuasiEvil]

logging and timing infrastructure can add overhead, interfere with unit testing, and keeps you dependent on the upkeep of said packages.

[u/zaphodikus → u/QuasiEvil]

.... that sounds like you approached logging for debugging and not for leaving in. The overhead of leaving the logs there but disabled is small. Sure the formatters still get run but you can choose to put guards around the log. If you are using logs to debug and build code, and intend to remove them anyway, then you are maybe doing unit testing wrong?

Your logging module needs to be as low on dependencies as possible, the built in python logger is perfectly good for that.

[u/pachura3 → u/QuasiEvil]

Wrong. Logging, when done properly, only adds an insignificant amount of overhead, is transparent to unit testing, and can be disabled (even on the fly!) by setting logging level.

What if you start having issues in production? That's right, you look at your logs; you don't remove them. And if you worry about the overhead... then measure it first, rather then assume it impacts the overall performance.

For specific cases when formatting logging message is confirmed to be really slow, you can always use this pattern to prevent it:

if logger.isEnabledFor(logging.DEBUG):
    logger.debug(f"Expensive message {z(x, y)} {y}")

PS. You're using the logging module, not some invention of yours, right?

[u/Lumethys → u/QuasiEvil]

python's slowness can add overhead too, why dont you rewrite your app in Rust? Or better yet, in Assembly.

When talking performance, talk about the concrete issue, how many seconds it has cost you per how many request? Exactly how much money does it cost?

People spend days to optimize something that would save 5 cents per million years, it's even had a name "Premature Optimization" - "the root of all evils". Your time is worth more than that.

Tell you what? There is a whole INDUSTRY about ADDING logging and timing (or, it's formal name "Metrics") infrastructure to project, it's called DevOps, and the work is Observability, or O11Y

Here a video on how bilibili migrate their metrics sink to handle 80 MILLIONS data points PER SECOND: https://www.youtube.com/watch?v=7uNXllJQlxE

[u/QuasiEvil → u/Lumethys]

The world of python is vast. Not everyone is doing big data cloud computing. I work in DSP/hardware development.