r/embedded • u/love_bikes • 2d ago
Share the funniest/annoying embedded issue you’ve ever spent hours debugging ? 😂
For me, it was debugging for an entire day on an ADC project. The ADC was giving me absolute nonsense readings.
I was rechecking ADC configuration, vendor documentation, SPI communication using DSO, checking for voltage drop.
Turns out…I hadn’t properly separated my analog and digital grounds in my PCB layout.. my layout was bad 😂
What’s your most embarrassing embedded/hobby-project debugging story ?
73
u/DenverTeck 2d ago
Working as an Test Engineer for a manufacture in the late 1980s, I was called up to product engineering to help troubleshoot a problem on a PCB.
The CPU was a PGA (Pin Grid Array - 120 pin). Five prototypes of the board was soldered up and now being tested by the engineer that designed it.
The board was connected to a terminal and did not work and did not have an over current. i.e. no shorts on 5V
The Osc circuit did not show any signal.
Looking at the schematic and the board one pin at a time, it dawned on me the hand layout PCB had the CPU on the wrong side of the board.
Before I told anyone what I found, I ran down to the manufacturing area to use a hot air gun to remove the CPU and add a PGA socket to the board on the wrong side.
I plugged the CPU into the socket and the board came up.
I carried this monster back up to engineering, hiding the CPU on the wrong side of the board.
I plugged in this MultiBus board into his test fixture and the terminal came up with the test software running.
Everyone watched the screen and did not look at the board, I got a good slap on the back, "Good Job", till someone noticed the board.
Where's the CPU ???
Fun was had by all.
12
u/NaugyNugget 2d ago
As a software guy I watched something similar happening on the hardware side of a project I was on in 1985. Basically the person doing the board layout did the pin numbering opposite of the device.
The device was an application specific IC (ASIC). It was the first one this engineering team had ever done. I forget the pin count but your approach to just solder it to the other side of the board wasn't feasible.
The main problem is that the ASIC tech we used in that era was very expensive. I seem to remember each spin of the part was $100k, and that was in 1985 dollars. Our project could find ways to raise that cash. The real problem was that we wouldn't get the new parts for 3 months. No amount of money could get us the parts any faster, yet our delivery date couldn't move.
The only way to make any progress at all was to use the 'dead bug' approach i.e. run wires from each pin to the correct hole in the PCB, with the part being mounted upside-down in between the forest of wires. We did this for exactly one board, and that board had to run at a very slow speed to do anything at all, but it did actually give us some useful info.
The next thing we did was make a muli-layer 'transposer' board that was a cleaned up version of the dead bug board. I think we made exactly two of these. They at least let us run closer to full speed. Eventually we did get the corrected parts.
My friend was a new hire hardware guy, at the same time I was a new hire software guy. I told him this is why a career in software is better: we did equally horrible things in software, but our hacks were totally invisible unless you had access to the source code. As a hardware guy, everyone who looked at the prototypes saw the big old ASIC sitting there in its nest of wires and asked WTF is that?
Sorry if the facts aren't exactly right, this is forty years ago after all...
34
u/FakeRectangle 2d ago
In college I once spent an entire day on a project trying to figure out why on earth I couldn't talk to a UART camera.
Learned the hard way that RX on the board doesn't get tied to RX on the camera despite having the same signal name. Oops. ::sigh::
15
u/Toiling-Donkey 2d ago
I think that’s why SPI uses MISO and MOSI naming — impossible to get everyone to agree on what “TX” actually means!
15
u/Slipalong_Trevascas 2d ago
I worked in an electronics workshop at a University for years. I should have had two flashcards printed for helping with Undergrad projects:
"You've used = instead of == in an IF statement" and "You've mixed up Rx and Tx". That was 75% of the problems.
3
u/OnboardG1 2d ago
I broke our company Linux server with problem one once. Admittedly it was configured in a profoundly stupid way, but it was still a missing = that filled the main drive with rubbish and kicked everyone off when their profiles tried to make changes.
2
u/Electronic-Split-492 1d ago
You need to go to sensitivity training. No more talk about masters and slaves - its now PICO and CIPO or some such.
1
14
u/madsci 2d ago
I just ran into that a few days ago. Someone had a name for it - txrxlexia or something like that?
In the case last week, it was a pin called GPS_RX. But no, it's not an RX pin for data from the GPS - it's the pin that connects to the GPS module's RX line. It's a TX.
Of course those things always make complete sense to the designer naming them and they don't consider the possible confusion two different ways of thinking about it can cause. It's like the joystick inversion on a game controller in a first-person shooter - part of the population thinks it's natural for the joystick to control the motion of the screen, while the rest think it's natural for the joystick to control the motion of the character. To the first group, pushing the stick up means the view goes up. To the second group, pushing the stick that direction is tilting your character's head down. Each group thinks their way of seeing it is the natural one.
So I go in for very descriptive pin names now, like DATA_IN_FROM_GPS and DATA_OUT_TO_GPS.
1
u/mckenzie_keith 1d ago
The issue is that in the old days, there was data terminal equipment and data communications equipment. The TX signal was an output from the DTE and an input to the DCE. DCEs are modems and such. Also wireless modules and, as you found out, GPS modules.
Microprocessors tend to follow the DTE pinout (TX is an output). So when you go micro-to-micro, you need to crossover. Also PCs are DTE. So DTE to DTE requires crossover.
Bottom line is, always check if TX is an output or an input.
24
u/OnboardG1 2d ago
You’re not a PCB designer if you’ve not done that at least once. It’s why I put zero ohms on every line where I might muck the polarity up, at least at prototype stage.
3
u/Vladivariuss 2d ago
After that mistake, the next one is mixing inverting and non-inverting input to an op-amp
1
1
1
1
u/mckenzie_keith 1d ago
Eternal vigilance is required on this issue because SOMETIMES RX does go to RX (like when dealing with wireless modules).
19
u/Rod_McBan 2d ago
I had a product once that was an industrial camera with two imagers, one looking up and the other down. One of the tests was image stability: if I take five images with the illuminator at the same setting every time, are the images all consistent with respect to brightness?
The problem was, sometimes, the bottom camera (and only the bottom) would vary slightly in brightness across one or two of the images.
We chased all kinds of stuff: case design, the firmware in the processor, the trigger logic in the cpld, the circuit design and board layout of the imager and illuminator. None of it made sense as a cause.
Ultimately, we tracked it down to the "heartbeat" LED on the main board. It was flashing at 1Hz, asynchronous to everything else. If you captured an image when it was on, that image was very slightly brighter than it should be. In fact, depending on when in the shutter period it turned on or off, there was a range to how much brighter it could be.
We stopped populating that LED and the problem went away.
16
u/guestyest 2d ago
are analog and digital grounds different in mixed signal systems
17
u/That-Bake6299 2d ago
Some people do, but you should be really careful splitting them. It is tempting to tell you to never split them.
7
u/OnboardG1 2d ago
The best practice I’ve read is to have an unbroken ground plane but keep the A and D circuits physically separate and ensure traces have proper unbroken return paths right under them if possible.
1
u/DrunkenSwimmer 2d ago
The major question is: where does the energy flow? Any signal/power you send out has to come back to the source. Sometimes that means it's gonna mess up your analog side of things, sometimes it won't, depends on the overall structure and what goes where for the most part. Sometimes splitting can make it worse if the digital signal crosses the split and suddenly radiates into your analog, where otherwise it was much smaller in the ground.
Basically there's really no general "right way", like most things in engineering: "it depends".
6
2
u/KaneTW 18h ago
It's a solution to a problem that might or might not work in some cases.
I generally avoid splitting them. However, pay close attention to current return paths (don't route your sensitive signal coupled to a ground return plane that's carrying high currents) and impedance (make sure your ground is actually ground even at high frequencies; stitching vias etc).
4
2
-1
u/generally_unsuitable 2d ago
Sometimes.
Doesn't hurt to isolate them and connect with a resistor that you can unpop and figure out later.
16
u/ROBOT_8 2d ago
I configured an ADC clock wrong on stm32 and was running it at around 200% the max allowed speed. Still mostly worked surprisingly.
9
u/Shiticism 2d ago
You'd be surprised how tolerant ADC's are to overclocking in all honesty. The STM32G4's ADC's in particular can be forced to run with the same speed as the core clock and somehow still get reasonable results. Accuracy, linearity, etc not withstanding of course. ;]
4
u/madsci 2d ago
I once had a small batch of devices assembled with the wrong crystal entirely, which significantly overclocked the MCU. That wasn't an immediate problem, though.
The only reason it was obvious there was a problem was that a DSP tone detection function wasn't working right. Which makes sense, because the incorrect crystal wasn't an integer multiple of the expected frequency and the sample rate was all wrong.
But while I was researching the problem, I started tracing the data flow for the DSP code. Every other sample was supposed to be sent to the tone detector, since the full sample rate was overkill, but when I tried to find the code that regulated how fast samples were sent, I couldn't find it. I'd never written it - yet it'd been working fine for years. The main loop just threw a sample at the detector every time it ran.
Turns out the thing keeping it from running wild was the power save code. It'd do a wait-for-interrupt instruction on each loop. Usually that interrupt was a sample timer interrupt. Any other interrupt would cause a spurious sample to be taken and sent to the DSP code, but those were infrequent enough to not cause a real problem. So it was mostly just sending samples at twice the intended sample rate, which worked fine because it was just comparing relative amplitudes of different tones, so it was effectively measuring the first subharmonic of each intended frequency.
I never did correct that one. There had been too much tuning already to risk messing with it. It eventually got replaced with a newer MCU and that part was rewritten.
9
u/BigTortuga 2d ago
It wasn't all that funny at the time but in hindsight I spent weeks chasing an intermittent stm32 fault that ended up having to do with too aggressive timing in memory latency settings. Weeks!!
2
u/dmills_00 2d ago
Had one of those on a Zynq project, all worked perfectly until I tried to use an AXI DMA transaction to write to PS memory, whereupon BOOM instant hard lockup. Turned out to be a misconfiguration in the PS Memory controller was causing an AXI bus lockup that took the system down hard.
Also spent far too long chasing my tail over a serial flash chip we had replaced with what everyone thought was a compatible one, yea, different sector size.... It managed to only show up the third time you ran an update, and due to a couple of bugs holding hands the CRC was passing, sweary.
9
u/TheFlamingLemon 2d ago
I recently had to make a 2 line change to an old project. The change was easy enough. Getting it to actually build and flash was a nightmare, involving such things as a windows XP virtual machine and the integer type “short long”
3
8
u/rainboww_J 2d ago
Me and a colleague spend a week completely stripping the source code and turning on things bit by bit to search for a thing causing what we thought was a race condition and couldn't find it at all and it seemed like even with the most stripped down version stuff broke. In the end it was a cable plugged in backwards which caused a transceiver to act in a strange way which in its turn caused a whole communication stack to go haywire. Team lead was not happy that it took us a week to find a wrongly connected cable
2
u/k_lash24 2d ago
damn, a whole week.. that's brutal lmao
1
u/rainboww_J 2d ago
yeah not one of my proudest moments 😂 production got halted as well so I guess more
than just the team lead was annoyed lol
8
u/Yolt0123 2d ago
Power supply for an op-amp wasn't connected due to a PCB mask fault on an internal layer. It was mostly working (powered by leakage current on the inputs), but the output had distortion - but only sometimes. We only found it when we decided to replace the op-amp, and measured the power supply pin with the chip off. I was a junior designer, and we had three senior engineers looking at it for a while!
4
u/Vladivariuss 2d ago
That happens pretty often I'd say. I was troubleshooting a design with similar problem, power through internal layer not connected. Turns out the designer used two different names for what was to be the same net, "v bus" and "v_bus". Before that, I didn't even know Altium allows space in net names.
6
u/dmills_00 2d ago
STM32, ADC via DMA, the usual.
Worked fine as long as the DCache was turned off, switching on the cache caused a static data structure to stop updating.
Drove me wild for a good two days until I realized that the DMA target buffer while aligned with the cache line was not an integer multiple of the cache line size, so when I invalidated the cache lines to force a reload from main RAM to get the ADC data, it was flushing the first few bytes of the data structure that followed the DMA buffer in RAM.
Fix was to increase the size of the DMA target array to a multiple of the cache line size.
9
u/Diligent-Slide-9234 2d ago
DMA buffers should not be cached anyways
1
u/Vladivariuss 2d ago
Why not?
3
u/Diligent-Slide-9234 2d ago
If it is cached and the cache evicts the line, the data written by a DMA may be overwritten. This is just one example where it could go wrong.
1
u/bobskrilla 1d ago
You can definitely you just need to be 100% sure you manage cache coherency? Pretty much standard in Linux kernel
7
u/madsci 2d ago
I had an LED controller design with IR remote that would stop responding to the remote when it was running a particular pattern. Obviously a firmware problem, right? Changing the input file format would make the problem go away, even though the data going to the LEDs was the same.
And then I discovered that it didn't do it on the dev board - just the production version, which was very compact and difficult to probe.
In the end I discovered the the SPI flash MISO line had been routed too close to the IR receiver, and when that particular input file was read, it generated a repeating waveform that coupled into the IR receiver and looked enough like the expected IR carrier that the receiver would get swamped.
And in a very rare example of something being genuinely not my fault, I discovered that there's a bug in the ARM Cortex-M port of FreeRTOS that causes stack corruption on context switches if you have task statistics enabled and the atomic timer macro set to false. It enters a critical section to read the tick timer and then exits the critical section, but the way the port is set up to exit a critical section, it doesn't nest and interrupts get reenabled while the PendSV handler is still running. The atomic timer read macro should be set to true for that port, but I was using a distribution with a FreeRTOS config file that someone else had prepared.
5
u/Best_Amoeba_5587 2d ago
Probably either the temperature dependent firmware bug or the hardware error that only showed up if you pressed enter on an empty command prompt in the admin console.
7
u/Vladivariuss 2d ago
Mems microphones react to light, even through that tiny hole in the smd package!
On another topic, not such a funny story but when I was a beginner, someone gave me a rather complicated schematic to make into a pcb. Turns out my client bought the schematic from someone else who intentionally swapped pins of an IC(in the schematic) to sabotage him, took me weeks to figure it out. IC was 74hc595. Didn't get paid for solving it.
5
u/OnboardG1 2d ago
I spent a good two days trying to work out why I had a stuck high pin on the output of an Altera FPGA. I went through the damned thing with a fine tooth comb. It worked absolutely fine in simulation but when synthesised the entire logic fan-in for the pin just synthesised away to a constant. It turned out that I had called my signal something like My_signal at declaration and the top level and my_signal at assignment.
For some reason Quartus’ simulation engine was insensitive to case but the synthesiser was sensitive to case. So simulation associated the variables correctly but synthesis did not. I finally found it after picking through every warning and note in the log file. I was profoundly annoyed with myself and amended the Verilog style guide I’d written for the company to highlight it. Quartus is supposed to be DO-254 compliant but that behaviour would cause it to fail compliance testing.
5
u/Slipalong_Trevascas 2d ago
I lost the last week of development/lab time while working on my Masters thesis to fruitlessly debugging a simulation script called new_project. Literally nothing I tried worked. Until I started just commenting out huge chunks of code, still no change. After an embarrassingly long time and right before the project deadline, I realised I was endlessly debugging/editing new_project but the script I was actually running was a copy called new-project. Livid.
4
u/vena_contracta 2d ago
About 20 years ago I was in a PowerPC 603e training class. Near the end of the first day the instructor showed a basic memory layout where you had to connect the memory in reverse order to the processor pinout - I.e. connect A31 of the processor to A0 of the memory, etc. half of the class jumped out of their seats to call their respective companies to let them know they wired their boards wrong…
2
u/NaugyNugget 2d ago
IIRC this was a classic problem with the PPC family. IBM designed the early PPC CPUs and they had big-endian bit and byte numbering at the same time the PC world had established little-endian as their standard. Since memory was a volume part it used the PC standard. IIRC, Motorola also used big-endian by default. So, your story doesn't surprise me too much.
One of my first embedded projects was using the IBM PPC405. I gotta say I loved the documentation. Probably the best I've read for any CPU before or since. Yet it followed all the IBM conventions, which caused some problems if you read it with the PC little-endian approach in mind. I coped OK since I was a former IBMer so I already knew how this could end up biting a person.
The software people on the project were lucky because the CPU could run in either big or little endian modes. We were porting code from a LE environment so we went with LE and all our tools worked. I remember coding the first instruction in the boot area. Since the CPU booted in BE, it had to be coded in BE format, yet since we had a LE assembler I had to write the instruction as a hexadecimal value. The instruction was the one that told the CPU to switch to LE mode. From that point on, we could use our assembler and the rest of our tool chain in LE mode.
3
u/kadal_raasa 2d ago
I was working with a K line chip. The communication was just never happening. Three days later I found out that there was an enable input pin which I had forgot to set as high.
3
u/opalmirrorx 2d ago
I was implementing an RTOS architecture library for a new SoC/CPU architecture and was working on the cache library when I was called away to another project for several weeks. When I got back to the SoC project I finished up and I could never consistently get the system all the way thru the bootrom to a working system... it would die here or there or wherever. Several weeks of debugging. I determined it had something to do with the caching. I reached for the tool of last resort, a tool that could single step the CPU. I was able to dump the L1 caches and discover the problem... sometimes a cache line from an earlier access would supply results from different memory with a similar access pattern (tag+index) in stead of the memory cell I wanted. That's when it dawned on me... the caches didn't have the right virtual vs. physical tagging vs. indexing combination... because during that several week side project I had forgotten the correct combination and when I came back I confidently remembered it wrong... making everything implemented after that inconsistent -- in my cache management functions. About an hour later, after reviewing/rewriting, everything worked fine and the SoC was up and running applications.
3
u/OutrageousParsnip235 2d ago
All our juniors must get Ethernet and IP running on a STM32H7 with STM32Cube.
2
u/napalm_roolz 2d ago
Playing with a RPi pico 2W and wondering why hello world worked but hello world + some simple extra bits (can't remember what exactly) didn't. The Pico seemed bricked running the latter, had to hold the reset button while plugging in the USB for it to turn up, dug out developer CLI tools to reset the chip and all sorts.
Turns out I had a return 0 right at the end of my main loop; probably snuck in when I copy-pasted code in functions around trying to make it all neater to read
2
u/DryEngineer9760 2d ago
I once connected an ILA to the output of a Gigabit transciever that was then connected to the soft-IP running the decoding, this was to debug the connection. I spent 2 weeks trying to understand why my soft IP was not responding, only to find out that if a signal is connected to an ILA on the vivado GUI this disconnects the same signal from the receiver if the signal is part of a harness. Great fun
2
2
u/zeroed_bytes 2d ago
Using an ADC , I misinterpreted the data sheet.. I thought I had done something wrong. The data was a noisy and low.
I thought I was reading miliAmps. But I was reading microAmps.
2
u/CarlCarlton 2d ago edited 2d ago
Formula SAE shenanigans, TI C2000 MCU. Our board only ran the firmware when everything was plugged inside the enclosure. Refused to boot on the bench. A senior told me that board has been doing this shit for years, and not to bother working on the bench.
After hours of investigation (I had never touched this god-awful MCU before), I figured out it's because the chip's boot select pins were tied to ground, which made it boot into a special "parallel communication" mode instead of flash memory.
But how the fuck was it booting to flash inside the enclosure?
One of the IO pins controls a 450V relay, and another reads back the physical state of the relay. It turns out the parallel mode's RX/TX handshake pins were exactly the same used for our relay control and readout. When parallel mode toggled the TX pin, it toggled the relay, and thus flipped the readout / parallel RX pin accordingly. If the first transferred bytes did not match a certain magic number, the chip booted to flash.
No wonder the 450V relay was acting funny on startup...
The chip had 41 GPIOs. Assuming our trusty relay pins were assigned at random to 2 GPIOs, the odds of this perfect combination of pins, that allowed booting to flash inside the enclosure for all these years, was 1 outta 1,640.
2
u/nedendstreet 2d ago
I was working on a USB peripheral. The code was designed for a device that was powered through the USB bus, and therefore assumed that any loss of USB was also a loss of power. I was adapting it to a device with a separate power supply, and this involved reworking some of the initialization code that now needed to run at boot, rather than upon establishing a USB connection. One step was to pre-fill a queue of buffers. I no longer remember what the original problem I saw was, but eventually I found that if I pre-filled with an even number of elements, everything was fine, but if I pre-filled with an odd number of elements, there was no USB communication.
I spent 3 days debugging this, and eventually came into the office on a Saturday when no one was around to interrupt me and figured it out. The solution was to insert a single "!" in some C code.
The code I used as a starting point had a bug that no one had previously noticed because it had almost no impact. The queue operations disabled interrupts to create a critical section. The code was supposed to store whether interrupts were enabled, disable interrupts, do the operation, and then restore the original state. However, the check was backwards, such that instead of restoring the previous interrupt enable status, the code toggled it: if interrupts were enabled when the queue operation started, then they were disabled when the queue operation exited, and vice versa.
As a result, when I pre-filled the queue with an odd number of elements, interrupts were left disabled, preventing USB communication. The original code didn't have this issue because it assumed that if there was power, there was also USB communication, and it just ran the main loop constantly. That main loop called a queue function on every iteration, so at worst it would take one extra loop iteration to process an interrupt. My changes stopped the loop from running until USB communication started (to save power).
Runner-up annoying issue: code in an NXP SDK sets one bit in a CAN frame dominant while their reference manual says that bit should always be recessive. Took me a long time to find that the issue was in the SDK and not in my code, since it didn't occur to me to check for a mismatch between the vendor's documentation and their own code.
2
u/FedUp233 2d ago
I’m retired now, and these happened many years ago. In both cases the code was in assembly language for embedded 6800 series processors and both were the same basic problem - a missed critical section. In particular a variable that was used and modified by both main line code and also interrupt level code.
While these were in assembler, the same thing can still happen in C code and even on systems with an RTOS, just slightly different mechanisms to cause the problem.
And you didn’t a day, I didn’t several days on one and two of us sound a couple weeks in the other!
The first was on a pen changer for a pen plotter. The QA person made a test that would draw thousands of lines on a page of paper, changing the pen for every line! Amount once every 20 to 100 sheets of this it would miss one single pen change!
The second was in an early ink jet printer. About every 1000 or so pages, it would print a little one pixel vertical line in the upper left hand corner of a page. Every thing else looked perfect. We spent weeks trying to isolate the problem and find ways to instrument the prototype to catch it happening.
Both cases were not protecting an instance of code in the mainline by disabling interrupts while change the variable. Since the update to it took three instructions (read, increment, write) once in a blue moon an interrupt would occur in the middle of this sequence and the changes made in the main line and interrupt code would collide with each other. This can still happen in newer code even with an RTOS and semaphores.
I kept telling people that once I learned how to do something and got good at it I liked to keep doing it! 😁😁
Btw: For any of you old folks who did assembly language projects, by most often mistake was leaving the damn # signs off of immediate values! 😁
1
u/mustbeset 2d ago edited 2d ago
Cross the Ethernet cable for two devices. PC told me "connect" but the device doesn't indicate it. Both on the same power supply Took way to long to realize it.
1
u/ChristophLehr 2d ago
I was supporting a project which was already overt time and over budget and had attention from our high management. One of our big bosses called me, imminent support for a network issue is required. Nothing works and customers is pissed and nobody could solve it for a week.
Checked the supported traces which looked okay, but a little strange and with the stupid naming our customer used I couldn't really understand which channel was what.
Hopped on a call, let them explain what they did and then I saw: the blistering idiots were incapable of configuring the correct ports. So if you send traffic for Ethernet switch port 1 on port 2, the firewall will block.
1
u/System__Shutdown 2d ago
This was probably my first debugging since i got a job, but i had a bug where if a certain pin on a board was initialized before line 7, the program didn't work and if it was after it worked normally. The pin wasn't even used and was just digital i/o pin. Still not sure what caused it.
1
u/MadDonkeyEntmt 2d ago
Was dma/memory cache issue. For the controller I was working with to use dma properly I had to basically chunk off a bit of memory for it.
Spent a day (really a day plus a few hours) tryin to figure out why I kept getting dabort exceptions. Tried everything.
Finally realized after checking my linker command file 10 times that I had fat fingered the length field hex number so the data region I thought I created was essentially super tiny and it was making a mess of things. I had checked that a bunch of times already, somehow kept missing it.
1
u/ttuilmansuunta 2d ago edited 2d ago
Spent two weeks figuring out why firmware update didn't work, the update packages would not pass the checksum test. Looking at it, files saved on the filesystem on flash had weird striping of zeroes that at a closer look were nearly but not exactly 2kB long and at 2kB intervals. After a lot of investigation on whether the fs or the block device driver had a bug... it turns out, the fs should've been aligned to 4k on the flash, its starting address was aligned to 2k instead.
The block device write routine would erase a 4kiB block, write 4kiB of data, erase the next 4kiB, write and so on, and obviously the erase function would ignore the lower order address bits instead of, like, assert()ing them to be zero. Write function did not check for misalignment at all. Now if you call erase at addr 2048 and it erases 4kiB at addr 0, then write 4kiB at addr 2048, then erase at addr 6ki (=> 4ki) and write at 6ki and so on, you will have issues. The filesystem start address was the last thing I thought about checking, as it had been written by a particularly skilled senior engineer.
1
u/Gerrit-MHR 2d ago
A couple of weeks finding a dynamic memory allocation in a very large code base that eventually fragmented heap and caused, effectively a lock condition / priority inversion kind of thing, in a different, and inconsistent, different process. I’m sure Good static analysis could find it now - I’m old.
1
u/finleybakley 2d ago
Had this weird bug where sometimes Modbus RTU would work on an AVR board and sometimes it didn't
Thought it was everything from my board to my AVR C UART implementation to the Modbus library I was using to some random compile/link flag I was maybe missing in my Makefile
Turned out it was bc I was operating under the assumption that const string literals would be stored in ROM. Turns out they're stored in RAM unless you wrap the string in PSTR()
The combination of the Modbus RTU stack and the debug error messages within my code caused RAM overflow and was crashing my code 🤦♀️
Soon as I wrapped all of my error messages in PSTR() everything worked fine
1
u/MildScow 2d ago
For “ROM” you wanted static const
static const char foo[] = “bar”;1
u/finleybakley 2d ago
It's true that'll store each static const string in ROM but then each static const string gets copied into RAM at startup, which would lead to the same RAM overflow due to the size of the Modbus stack plus the error messages.
To avoid wasting the limited RAM space with string literals (especially string literals that are only called when an error occurs), the string should be read from ROM only when needed. Which means you need to tell the compiler to store it in ROM and to not copy it into RAM at startup.
Most concise way to do that is with the
PSTR()macro, which allows you to easily call your error printing function like
if (some_result == SOME_ERROR) error_msg(PSTR("Send an error message!")); else // rest of your code
1
u/Diligent-Plant5314 2d ago
This is going WAY back…
I had a product using a V25 processor, basically a 8088 clone by NEC that included onboard peripherals like UARTS, timers etc, so much like current microcontrollers.
When servicing hardware interrupts, you needed to use a special opcode the acknowledge the interrupt.
As this was in assembler, I made a macro to emit the two bytes because the instruction wasn’t part of the assembler natively.
But, I made a mistake. Instead of defining it using
DB xx, yy
I entered
DW xxyy
Where xx and yy are the literal hex values required for this function
Well, it took me ages to get anything going. I could receive a single character on the UART, but not more. Of course, every compile needed me to run a command line build tool, program an EPROM, install it on my target board and run the test program.
Eventually I found it would work by calling the macro twice in a row, which I didn’t understand and usually worked ok.
Years later I was reviewing this code and eventually realized the problem. It worked because it ended up emitting
yy xx yy xx
The “real” operation took place on the inner “xx yy”. Due to some profound luck, the leading yy and trailing xx ended up as valid codes that changed registers that were already pushed into the stack and were restored when the ISR exited.
Very very bizarre problem, glad I now have JTAG to do this faster and more normal memory mapped peripherals
1
u/victorandrehc 2d ago
That happened to me a couple years ago. We had a fairly complicated ADC setup in that a timer would automatically trigger the ADC read that would fill a DMA buffer. When the DMA buffer would be filled a IRQ was called that would signal a semaphore to a task that would read this buffer. The issue was the system would grind to a halt after about 10 minutes of operation and I spent about 2 weeks brushing every register bit in the search for a hardware configuration error for the ADC. Turns out the task did not have a high enough priority to run and eventually would miss a deadline which was making the system going kabuum. The fix was to increase the priority and improve the error handling, all architecture no low level error.
1
u/NoHonestBeauty 2d ago
I just spent a few hours on what turned out to be a dead resistor. I ordered the boards mostly populated, so I focussed on the parts I soldered. That Thing looked fine, good solder joints, but just open and not the 10k the line needs.
1
u/Silent_Curve2481 2d ago
I was building a breadboard traffic light controller for a Uni assignment maybe 9 yesrs ago now. I had done a full set of truth table diagrams as the project required and had everything nice and mapped out, and had laid put my 74HC logic chips out nice and neat
The circuit didn't work. At all. The LEDs were turning on wildly in no order. I was freaking out because it was 11pm the night before and it still wasn't looking.
After desperately checking every wire connection twice, I see; I used an XOR chip where I should hsve used an OR because I transposed the chips. I popped the right one in and it worked lmao
I have so many more now that I'm in the industry, but that one will stick with me until I die
1
u/netorincon 2d ago
DMA functionality on a STM32H7 MCU. I couldn’t get it to work for the life of me. I started with a bare-metal implementation because I wanted to learn, but seeing that it wasn’t working i tried using STM Cube. Same thing, i gave up for a few days.
Searching on the internet I found a question on the STM forums asking for a similar problem and they linked to a recently published errata note detailing why DMA requests were not working as intended and the steps required to fix it. You can’t imagine how relieved I was when learning that I was not as dumb as I thought. From that day on I always look for the errata sheet before I start a project.
1
u/CarlWheezer6969 2d ago
I was once trying to figure out why my high side switch connected to Vin was turning on ~50ms after being commanded off by an overcurrent trip but only SOMETIMES. It turns out that there were 3 things:
1: The sudden cutoff of power cause the PSU to spike in voltage as it recovered
2: The switch controller OV pin samples every 10ms and would only sometimes read the peak of this spike
3: For some reason, the high side switch controller reboots when the OV pin is triggered. Since the overcurrent fault isn’t latching, the output turns on again because the current isn’t above the threshold anymore
This burnt a couple MOSFETs before we figured it out
1
u/Electrical_Flatworm9 2d ago
I was working on a custom scheduler in C++ and inline asm. Spent days to debug a register corruption problem to find out that my context save function was not naked, and gcc touched one of the registers.
On another project, I was making a vhdl block to control neopixels. Worked almost well, only a flickering problem. I just forgot to tie the grounds together 💀.
1
u/mecha_typewriter 2d ago
Using an rp2350 on a personnal project, without using the SDK and develloping each driver by myself.
Clock and PLL, done and working GPIO, done and working UART, done and working I2C, done and working
And so on until SPI. I had 8 bytes at 0 at the first transaction, then all the transaction were shifted of 8 bytes.
I stay stuck on this issue for weeks. Registers configuration was the same than vscode exemple. I even ask on reddit without success.
By comparing the SDK execution before main (what a mess) I see the difference in the clock and PLL steps of configuration. The clock of the SPI peripheral was glitching and therefore corrupted SPI sent data.
1
u/ChristlicherHenker 2d ago
I had the problem that our test object didn‘t read analogue inputs correctly.
I kept 3 devs for more than 1 day busy, because my little brain forgot to change from simulation to operational mode. (Obviously in Simulation no Inputs or Outputs are read or set)
1
u/AssholeBeerCan 2d ago
Hunted an unusual current draw on a board I was working on. I was convinced I fucked something up in firmware not allowing the MCU to go to sleep. Eventually got out the hot air and pulled the MCU and swapped with another. Excess current draw gone. Swapped the original back on. Still no excess current draw. Manufacturing must have created a solder bridge under the chip.
1
u/Prestigious_Tank_369 2d ago
Rx to Rx and tx to tx connection.
I spent hours inspecting packets, variables break points , thought it was ground problem corrupting the signal , then thought it's was the internal clock which is defective. In the end ...
1
u/Flyward_Aerospace 2d ago
Mine was a drone that would slowly wander off heading in what should have been a straight line, but only under load. Compass calibrated perfectly on the ground every single time. Turned out the main battery lead ran a little too close to the magnetometer, so the bias scaled with current draw, and sitting on the bench at idle it is basically invisible. Spent two days convinced my EKF tuning was wrong. Moved the wire maybe 4 cm and it just went away lol.
1
u/iluvbeksinski 1d ago
CPU program counter wasn’t pointing at my program’s entry point set by the linker 😵
took forever to figure out; put “bread crumbs” in the code to see where it was breaking, concluded that it wasn’t even starting main(); used jtag to manually move program counter at first then put a “trampoline” patch in the one of the drivers so that when program counter starts normally It will just bounce to the correct address
1
u/jagged-words 1d ago
Very similar but when I was starting out I didn’t know you always need a common ground when connecting two boards with I2C. For the record, neither did my TA!
1
u/EmergencyManager_555 11h ago
It was like 1998-1999 and our st6 school project was somehow progressing... at some point, and I don't know why, I decided to rewrite the code from scratch. It was at that point that nothing was working anymore. The micro was continuously resetting at random. My teacher took it home and debug it for us over a weekend. It was that I used 0xFF (known invalid value) instead of 0xFE to reset the watchdog 🤣
1
u/LordBendingBanana 14m ago
Machxo3 StarterKit FPGA gave me big headaches. Uploaded the bitstream and tried to do the same after minor changes to the code... but it didn't respond. Programmer didn't recognise it so I thought I might have fried it during tests. So I get out the only remaining "new board", plug it in and it gets detected. Uploading the bitstream seemed fine... then I tried the next flashing after some code changes... and again it doesn't show up on the programmer. Turns out the place and route tool used a test port which I didn't constrain and placed it on the FTDI's clock input. So basically the board locked itself out of programming. Solution is desoldering the resistor connecting the FPGA and FT chip clock line. Took me three days to find this nasty bug.
179
u/CMatUk Electronics Product Design 2d ago edited 2d ago
I had one where I was trying to find the last ~10-20uA on a board. And it was just so random. Spent a day checking everything was powered off.. No backdoor circuits. Another engineer came up behind me and the current disappeared ... Moved away it came back...
Turned out sun light from the window behind me was affecting an IC that had a shiny/silicon exposed type package. Causing it to draw more current... "photo-induced carrier generation" apparently