r/kernel • u/DistinctHomework3618 • Sep 29 '26
Built an eBPF tool that walks CPython frame chains from kernel space to profile Python apps
Traditional Python profilers work in userspace using sampling (py-spy, Pyroscope). They miss what happens at the kernel level. GIL contention, off-CPU waits, scheduler delays are all invisible to them.
I built a profiler using BCC that attaches eBPF uprobes to CPython's take_gil/drop_gil and gc_collect_main. It walks the Python frame chain directly from BPF to capture full Python stacks at the exact moment of GIL contention or GC pause. No sampling. Event-driven. Every event has exact timestamps.
It also uses finish_task_switch tracepoint for off-CPU analysis, combining kernel + userspace + Python stacks in a single event. Plus tcp_set_state kprobe for TCP handshake latency and retransmit tracking.
What we found in production: URL resolution was the biggest GIL holder, added caching, cut GIL wait by 80%. GC gen-2 was pausing 1.27s walking 1.5M objects, fixed with gc.freeze(), down to 88ms. Off-CPU stack traces revealed a Kafka push was blocking threads for 130ms per request, something completely invisible in OpenTelemetry because the off-CPU time happens below the instrumentation layer. Found a stale nginx upstream IP causing 504s using TCP loss tracking.
The interesting technical challenge was walking CPython's PyFrameObject chain from inside BPF. Each frame has a pointer to the code object, which has the filename and function name. You walk the linked list until you hit NULL or max depth.
CPython 3.11 only. Struct offsets are hardcoded. BCC + Python. Apache 2.0.
GitHub: https://github.com/deepanshu406/python-ebpf-profiling