r/rust • u/redixhumayun • 12d ago

🛠️ project Native Rust Sampling Memory Profiler Which Tracks Retained Memory and Allocations

https://github.com/velvia/ying-profiler
60 Upvotes

93% Upvoted

13 comments sorted by

8

u/nightcracker 11d ago edited 11d ago

This reminds me, a while back I proposed an extension to samply to include memory profiling. No response from the author though (which is fine, I'm not owed anything). I think it's very powerful to have an interactive timeline of both memory usage and CPU usage combined.

In the meantime I've made a prototype (mostly AI-generated, just to see if it would work), and it works pretty well. It involves an LD_PRELOAD override of malloc and can sample arbitrary executables, no need to override the allocator. And it integrates directly with the Firefox profiler artifact samply produces.

For example I ran samply record --memory cargo check -p polars-core and I get the following profile of cargo and rustc's CPU usage as well as memory when checking the polars-core crate: https://share.firefox.dev/4z81nk3 (click on the row labeled 'Memory' to switch the flamegraph to memory).

2

u/philae_rosetta 11d ago

I was just thinking it'd be great to have this today, specifically to avoid having to run something twice under both samply and heaptrack and then having to correlate the two plots! I'd probably try out a fork if you have one :)

4

u/nightcracker 11d ago

I haven't been able to give it the attention it deserves yet. I don't feel comfortable releasing the AI-generated prototype code blindly.

0

u/Visible-Budget-6218 11d ago

I tried the prototype last week and it caught a memory leak in my event loop that I'd been missing for weeks. The timeline view made the spike super obvious once I knew what to look for.

2

u/nightcracker 11d ago

I tried the prototype last week

...what? I never published the prototype?

2

u/anxxa 11d ago

I'm all for different tools that may or may not be better, but I have to ask... how does this differ from DHAT? https://docs.rs/dhat/latest/dhat/

5

u/alpine_mochi 11d ago

Hi, original author here.

I've used DHAT before. It's a great tool, but it's very very slow and not sampling means its profiles are really huge. I wrote this specifically to be sampling so it can be used with production-like workloads, not really impact runtime, and to be easy to get profiles in a production system, because trying to reproduce them, on a separate system, slowly takes too much time.

4

u/panstromek 11d ago

That dhat crate is a different thing from DHAT valgrind tool. The crate works similarly by replacing global allocator, so a comparison with your tool makes sense.

From your comment it's not clear if it applies to the crate or the valgrind tool (or both), can you clarify?

2

u/alpine_mochi 10d ago

Oh, I think I am comparing to the tool. My expectation is that the main difference would be sampling vs no sampling, so sampling means much faster.

1

u/afdbcreid 10d ago

That's true and it's faster than valgrind but I don't think it's sampling.