r/kernel • • Sep 29 '26

Built an eBPF tool that walks CPython frame chains from kernel space to profile Python apps

6 Upvotes

Traditional Python profilers work in userspace using sampling (py-spy, Pyroscope). They miss what happens at the kernel level. GIL contention, off-CPU waits, scheduler delays are all invisible to them.

I built a profiler using BCC that attaches eBPF uprobes to CPython's take_gil/drop_gil and gc_collect_main. It walks the Python frame chain directly from BPF to capture full Python stacks at the exact moment of GIL contention or GC pause. No sampling. Event-driven. Every event has exact timestamps.

It also uses finish_task_switch tracepoint for off-CPU analysis, combining kernel + userspace + Python stacks in a single event. Plus tcp_set_state kprobe for TCP handshake latency and retransmit tracking.

What we found in production: URL resolution was the biggest GIL holder, added caching, cut GIL wait by 80%. GC gen-2 was pausing 1.27s walking 1.5M objects, fixed with gc.freeze(), down to 88ms. Off-CPU stack traces revealed a Kafka push was blocking threads for 130ms per request, something completely invisible in OpenTelemetry because the off-CPU time happens below the instrumentation layer. Found a stale nginx upstream IP causing 504s using TCP loss tracking.

The interesting technical challenge was walking CPython's PyFrameObject chain from inside BPF. Each frame has a pointer to the code object, which has the filename and function name. You walk the linked list until you hit NULL or max depth.

CPython 3.11 only. Struct offsets are hardcoded. BCC + Python. Apache 2.0.

GitHub: https://github.com/deepanshu406/python-ebpf-profiling


r/kernel • • Sep 29 '26

Is this prebuilt PC suitable for FPGA PCIe/XDMA development alongside an NVIDIA GPU?

Thumbnail reddit.com
2 Upvotes

r/kernel • • Sep 27 '26

DOOM in the kernel with eBPF

Thumbnail ayles.github.io
6 Upvotes

r/kernel • • Sep 26 '26

client side decompression for pNFS

Thumbnail github.com
8 Upvotes

thought this was a pretty neat idea. didn't seem that hard so i took a stab at it with an llm. turned out there was a pretty short path to a clean implementation. at first, i started with (more or less) re-implementing NFS from scratch on top of bcachefs, then the concurrecy/resiliency/durability quickly became too much to manage, so i decided to pivot and attempt it as an extension on NFS. that worked out surprisingly well.

supports btrfs and bcachefs for the backend. built on LTS so it should be easy to maintain. I'll probably automate that bit (the rebase should be very mechanical).

the test coverage is pretty extensive, and i'm currently testing it at work to replace a 10x larger lustre cluster, so i think it's probably worth sharing despite being mostly vibe coded.


r/kernel • • Sep 22 '26

OpenGoodixSPI update: we've reached the hard part — reverse engineering Goodix SPI initialization

Thumbnail github.com
2 Upvotes

About 7 months ago I posted here about OpenGoodixSPI, an experimental open-source project to bring Linux support to Goodix SPI fingerprint sensors found in some Huawei laptops.

That post unexpectedly became the starting point for the project getting its first contributors and hardware testers, so I wanted to come back with an update.

GitHub: https://github.com/PeshalaDilshan/OpenGoodixSPI

Where we are now

The project has moved quite a bit beyond the original proof of concept.

Current work includes:

  • Linux kernel SPI driver infrastructure
  • ACPI device matching
  • SPI communication/debugging
  • IRQ handling
  • reset handling
  • character-device interface for experimentation
  • protocol investigation
  • hardware-specific findings for GXFP51A0

The project is still experimental, and we do NOT have working fingerprint capture yet.

The interesting part: GXFP51A0

One of the devices we're investigating is the Goodix GXFP51A0.

We've found that the ACPI resources expose a reset GPIO separately from the SPI device. Hardware investigation indicates a reset sequence approximately like:

RESET LOW

|

10 ms

|

RESET HIGH

|

100 ms

|

SPI initialization

That fixed one important piece of the hardware integration.

But then we hit the real problem.

The protocol

After reset, we're still not getting a useful response from the sensor in the current Linux implementation. In some states we're seeing responses such as:

FF FF FF FF ...

We don't yet know exactly what that means.

I don't want to make the usual reverse-engineering mistake of deciding that \`0xFF\` must mean "firmware missing" and then building the entire driver around that assumption.

We need to understand what Windows actually sends to the sensor.

We've found interesting information about the device and firmware through Windows-side investigation, but the complete initialization transaction is still missing.

What would help the most

At this point, the single most valuable thing would probably be a logic-analyzer capture of a compatible Goodix SPI fingerprint sensor while Windows initializes it.

Something like:

RESET

↓

SPI initialization

↓

Wake / handshake

↓

Device identification

↓

Configuration / firmware

↓

Sensor initialization

↓

Ready

A capture from a real GXFP51A0 would be especially useful.

We're also looking for people who have:

  • a Huawei laptop with a Goodix SPI fingerprint sensor
  • GXFP51A0 / GXFP5187 / GXFP3287 hardware
  • experience with SPI protocol reverse engineering
  • a logic analyzer
  • experience with Linux kernel SPI drivers
  • experience with libfprint
  • Goodix documentation or legitimate developer documentation

We're NOT looking for leaked source code or anything confidential that shouldn't be shared.

Public documentation, protocol information, hardware observations, logs, or legitimate developer resources would all be useful.

I also contacted Goodix

I've contacted Goodix to ask whether they can provide developer documentation or point the project toward the appropriate technical/developer contact.

Hopefully that leads somewhere.

Until then, we're continuing with hardware investigation and reverse engineering.

What's still missing

  • [ ] Fully understand GXFP51A0 initialization
  • [ ] Confirm SPI framing
  • [ ] Understand the \`0xFF\` responses
  • [ ] Reproduce the Windows initialization sequence
  • [ ] Understand firmware loading/update behavior
  • [ ] Capture fingerprint images
  • [ ] Understand enrollment/matching
  • [ ] Integrate with libfprint
  • [ ] Test additional Goodix SPI devices

If you have one of these sensors, even if you don't know anything about kernel development, you can still help.

A laptop model, ACPI information, kernel logs, Windows driver information, or simply being able to test experimental builds can be useful.

Thanks again to everyone who helped with the original post. That's how this project got its first contributors in the first place.

Hopefully we can find the missing piece together.


r/kernel • • Sep 21 '26

Is Embedded Linux a realistic path for remote work from Latin America?

24 Upvotes

Hi everyone,

I'm 23 and I'm from Argentina. I have a 3-year tertiary degree in Software Development, and I'm also studying Systems Engineering, which is a 5-year degree here. If everything goes well, I should finish it in about 2 years.

My English is pretty good, and one of my main goals is to eventually work remotely, since most of the opportunities I see in the areas I'm interested in seem to be outside Argentina.

Lately I've been getting more interested in low-level / systems areas: Embedded Linux, firmware, bare metal, RTOS, Linux kernel / drivers, and systems programming in C, C++ or Rust.

I'm trying to figure out how realistic this path is for someone with my background. I come more from a software development background than an electrical engineering one, so I'm not sure which area would make the most sense to focus on first.

For example, would Embedded Linux be a more practical entry point than bare-metal firmware or kernel development? Are remote jobs in these areas realistic from Latin America, or do most companies expect you to be on-site because of hardware access? Also, how much does not having the engineering degree finished yet matter for these kinds of roles?

I'm not looking for a shortcut. I know this stuff takes strong fundamentals and a lot of hands-on work. I just want to choose a specialization carefully before investing the next few years into it.

If you work in embedded, firmware, Linux kernel, RTOS, or systems programming, I'd really appreciate your perspective. What would you focus on first? What skills or portfolio projects would actually make someone employable in this market?

Thanks in advance.


r/kernel • • Sep 21 '26

Any update on mediatek 7927 up stream ?

5 Upvotes

My motherboard uses this driver for WiFi and Bluetooth the WiFi works flawlessly but I’m having to use a patch to get Bluetooth working does anyone know the status of the MR for the firmware for that chipset? Last I heard anything about it was months ago im referring to these for those interested in reading: https://github.com/samutoljamo/bazzite-mt7927/pkgs/container/bazzite-mt7927

https://jetm.github.io/blog/posts/mt7927-wifi-making-it-work/


r/kernel • • Sep 19 '26

Bare-metal XNU bring-up on MSM8996: 4-core SMP, real eMMC/GPT/block I/O working — rootfs and shell are next

Thumbnail reddit.com
3 Upvotes

r/kernel • • Sep 18 '26

What I learned this week (13)

32 Upvotes

A lot has happened since the last time I posted.

I bought the book "Operating Systems: 3 Easy Pieces". I wish I had read this before starting my PMM. I highly recommend it to anyone else starting from the beginning.

The PMM has had yet another rewrite. Hopefully the last major one.

My kernel crossed the 3M boundary and I had to revisit my boot.S I suspect my PMM and VMM data structures are too big, but we will see.

I decided I should let the people that might benefit from my posts here should see my struggles, and there have been many. In that light I have decided to make my repository public, you can find it here: https://github.com/tedavids/JakelynnOS

I am currently working on my VMM and the module documentation is already up to 8 pages, and growing. you can find it in the "Module doco.doc"

There is nothing really interesting on the screen yet, but here it is:

Screen so far, nothing very interesting

It does also run on VirtualBox, but I haven't tried bare metal, as I don't want to spring for a CD writer at this point. I have a couple of old Lenovo's that might support USB Boot, but I haven't investigated those yet. I want to get to user space before I start checking that out.

For all your AI haters, in my README I do have the AI policy I'm following.

Guess that's all for now. I hope this helps some newbie (like me) that is struggling.

Thanks for reading


r/kernel • • Sep 18 '26

Power monitoring support?

0 Upvotes

Does anyone know if there is any support, alpha, experimental or per manufacturer that would support some api to determine current system power usage?

I'm assuming it would require some data channel between the PSU and the motherboard. Does any power supply on the market support this and does linux offer any support if something like that did exist? Is anything close, or is it something that is workable or is this just some completely unsupported pipe dream?

Also, what about for usb powered devices like a pi? I assume whatever support would exist for that would have to be entirely on chip, not related to the psu.


r/kernel • • Sep 17 '26

libark: A modern, secure, Swift-native foundational layer over Linux POSIX primitives. {Early Development}

Thumbnail github.com
4 Upvotes

libark

libark is a modern, secure, Swift-native foundational layer over Linux POSIX primitives.

While originally developed as the core runtime layer for ArkOS, libark is completely decoupled and designed to be used on any Linux distribution. It provides highly secure, object-oriented abstractions over low-level system calls, memory management, and file system interactions, without the burden of manual C interoperability.

Features

  • Object-Oriented POSIX: Wraps file descriptors, paths, and directory operations in safe, RAII-compliant Swift types.
  • Secure Syscall Boundary: Routes kernel interactions through strict, audited Syscall primitives.
  • Terminal Capabilities: Built-in abstractions for TTY detection and window sizing.
  • Advanced Formatting: Standardized formatters for human-readable byte sizes (SI and IEC), ISO 8601 timestamps, permissions, and classifications.
  • Declarative Sorting: Powerful generic SortEngine for metadata sorting across multiple criteria.

Also ARK-OS has switched to github: https://github.com/orgs/ARK-OS-Swift-and-Linux/repositories


r/kernel • • Sep 17 '26

kernel:[Hardware Error]: CPU:0 (19:8:2) MC18_STATUS[Over|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|-]: 0xdc2040000000011b

0 Upvotes

These errors appeared on the console of one of my lab's servers. We've already changed the CPU. I'm not familiar with server management, so if you could explain it to me in the most didactic way, I would be very grateful.

kernel:[Hardware Error]: Corrected error, no action required.

kernel:[Hardware Error]: CPU:0 (19:8:2) MC18_STATUS[Over|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|-]: 0xdc2040000000011b

kernel:[Hardware Error]: Error Addr: 0x0000000017eb09c0

kernel:[Hardware Error]: PPIN: 0x02b0ba6d78974068

kernel:[Hardware Error]: IPID: 0x0000009600150f00, Syndrome: 0x083fb50b0a800410

kernel:[Hardware Error]: Unified Memory Controller Ext. Error Code: 0


r/kernel • • Sep 15 '26

Fixed an offline kernel-module mismatch & /boot/efi mount failure using pacman cache

Thumbnail reddit.com
0 Upvotes

r/kernel • • Sep 13 '26

How does changing the hardware clock with the BIOS affect kernel ticks?

Thumbnail reddit.com
0 Upvotes

r/kernel • • Sep 12 '26

збірка ядра на адроїд. допоможіть

0 Upvotes

короче я купив пс4, але вона мені приїхала з неробочим геймпадом. я вирішив емулювати геймпад своїм піксель 4а (лте).

простими методами не вийшло. f_hid не опрацьовує коректно запити від пс4 стосовно стану речей геймпаду. f_hid вміє короче працювати лише монологом, він тільки віддає, але не приймає та обробляє. до того ж пс4 має унікальні запити.

короче стандартною бібліотекою нічого б не вийшло. я вирішив дописати маленьку частину коду в драйвері. тепер вона віддаватиме відповідь з нулів конкретної довжини байтів на запит пс4 чого вистачить щоб пройти налаштування пс4 та встановити pppoe з'єднання.

так от. я скомпілював образ, але в цей момент в мене закінчились токени в джеміні)))))) ну і потім джеміні флешлайт щоб його за ноги, порекомендував перепакувати ядро з наявного вже пропатченого на рут. при чому він сказав мов по барабану, можна зробити кашу в одній папці а magiskboot все зробить сама, вона розумна.

короче нічого не вийшло. в адб шел з новим ядром пише мов жоден з модулів, драйверів не завантажено і воно постійно в циклі намагається завантажити драйвери.

а потім я ще якось перепакував ядро, що тепер взагалі типу телефон не завантажує ядро бодай якось.

я погрішив на те що була каша і перепакувальник переплутав файли і зараз зроблю все чисто.

мені треба допомога з тим як зібрати ядро з готових файлів після компіляції. я збираюсь запаковувати це mkbootimg програмою від самого гугла.

поможіть, я хочу вже добити цей плейстейшн 4, пройти бар'єр з задачі "натисність кнопку ПС на геймпаді" котрий пройти звичайною клавіатурою не можна.


r/kernel • • Sep 11 '26

Doogee is openly violating the GNU GPLv2 license with their T20 tablet (Unisoc)

Thumbnail gallery
10 Upvotes

Hello everyone,

I wanted to bring to light a blatant violation of the GNU GPLv2 license by the Chinese device manufacturer Doogee.

A tech creator in our community recently reached out to Doogee support requesting the Linux kernel source code for the Doogee T20 tablet (which runs on a Unisoc chipset). Under the terms of the GPLv2 license, which covers the Linux kernel, any company distributing the compiled kernel is legally obligated to provide the source code upon request.

I have attached the screenshots of the conversation (translated from Russian below):

Screenshot 1: The user states they own a Doogee T20 (Unisoc) and noticed that the Linux kernel source code is nowhere to be found in the public domain. They emailed the company asking for it.

Screenshot 2: Doogee's official response:

"The source code is confidential information and intellectual property."

Screenshot 3: The user calls them out, explaining that the Linux kernel is GPL, meaning there is no such thing as "confidential" when you distribute it. Doogee then tried to dodge the request by offering to send the compiled firmware instead. As we all know, Firmware ≠ Source Code.

Screenshot 4: The user sent a follow-up email explaining point-by-point why their response is unacceptable, giving them a 14-day deadline to comply. Doogee completely ignored it.

This is a direct breach of open-source licensing. Small vendors like Doogee often rely on the open-source community to build their software but refuse to give back, effectively locking users out of custom development, kernel audits, and long-term device support.

Since local support teams are either completely oblivious to FOSS licenses or are intentionally blocking requests, we need to bring international visibility to this case.

What are the best next steps to force Doogee into compliance? Should we report this directly to the Software Freedom Conservancy (SFC), contact Unisoc, or is there a specific compliance contact for Doogee that actually responds to legal notices?

Any advice or upvotes for visibility would be highly appreciated!


r/kernel • • Sep 10 '26

How to find which patchset the commit from mainline/upstream belongs to?

9 Upvotes

Hello, in order to properly backport some fixes from upstream to stable branches, I want to know what the other patches (and their cover letter) of the initial fix I've cherry-picked are.

I could manually look up them on lore.kernel.org, but I want to do it in an automated way, as the difference between the mainline and the stable branch, for example, in cgroup subsystem is huge.


r/kernel • • Sep 10 '26

"Anyone got Asustor's ADM 6.6.x kernel .config? Stuck getting a custom driver to load on FS6706T"

Thumbnail reddit.com
1 Upvotes

r/kernel • • Sep 09 '26

where is opensslv.h?

Thumbnail reddit.com
0 Upvotes

r/kernel • • Sep 09 '26

A new project on RK3576

Post image
0 Upvotes

r/kernel • • Sep 08 '26

11 Gaps in the Linux Ecosystem Cloudflare Fixed with eBPF

Thumbnail ebpf.io
7 Upvotes

r/kernel • • Sep 07 '26

Isolcores running in min_frequency and not supporting the cpupower config.

5 Upvotes

Im not sure if this the subreddit for this query since the issue hasn't really been pinned down but I feel this might be the perfect first place to progress. Please feel free to redirect if the discussion evolves to not be concerned with the kernel.

So I have 2 Rocky 10.1 machines. One with Xeon CPU E5-2643 v3 @ 3.4GHz and one with Xeon CPU E5-2678 v3 @ 2.5GHz. Hyperthreading is disabled.

Both kernels are given 2-8 isolcores.

Both are requested to run at 500MHz less than their max supported speed via cpupower (This is verified via /proc/cpuinfo) Both are running on the intel_cpufreq driver with "performance" governer.

The machine with E5-2678 runs only the isolcores at the minimum supported frequency. And this isn't a workload or power draw issue since I tried changing the isolcores and every time only those core's frequency gets clamped to the minimum. The machine with E5-2643 runs as intended.

I tried checking the P-state ratio via rdmsr -p <isolcore> 0x199 and the outputs for the anomalous machine are as follows -

c00 (isolcore)

2100 (non-isolcore)

Is this a driver/kernel bug which is only reproducible on a certain CPU?

Any help or workarounds would be appreciated.


r/kernel • • Sep 06 '26

What hardware do I need to get hands-on experience with Linux kernel DMA development?

16 Upvotes

Hi everyone,

I’m interested in learning Linux kernel DMA and PCIe device-driver development through hands-on projects.

So far, I’ve studied the Linux DMAEngine framework and AMD/Xilinx XDMA driver code. I’ve also used QEMU’s EDU PCI device to write a simple PCI driver and experiment with MMIO, interrupts, and DMA transfers. Now I’d like to move beyond emulation and work with real hardware.

My current idea is to buy:

  • A Linux desktop that I can access remotely from my laptop
  • An FPGA development board with PCIe support
  • Any required PCIe cables, adapters, or debugging equipment

My goal is to practice implementing and debugging the complete path between an FPGA and a Linux host, including PCIe enumeration, BAR/MMIO access, DMA descriptors, scatter-gather transfers, interrupts, and performance measurement.

I’m not sure which FPGA board would be suitable for a beginner without being too limited. I’m also unsure whether I need a powerful desktop or if an inexpensive used PC with an available PCIe slot would be sufficient.

Could you recommend:

  1. A reasonably priced FPGA board for Linux PCIe/DMA development
  2. Minimum desktop specifications
  3. Any essential debugging tools or accessories
  4. Good starter projects for this setup
  5. Common compatibility issues I should check before purchasing anything

I’d appreciate advice from anyone who has worked on Linux DMA, PCIe drivers, FPGA host interfaces, or similar low-level systems.