We all hit bugs that make no sense. You debug for hours, the symptoms contradict each other, and eventually you run out of places to look. This was one of those.
Two identical boards, two different behaviours
I had two boards built from the same design. One I had soldered myself, and it worked flawlessly. The other came from the assembly house and behaved as if it was cursed.
- Random HardFaults that never happened in the same place twice.
- Logging the PC and LR registers at each fault pointed somewhere different every time, which looked like memory corruption.
- The only thing that helped was lowering the core clock by 10 MHz. That felt like duct tape over a structural crack.
Then SPI started acting up too. The Wi-Fi module would return corrupted data, and its interrupt line would trigger randomly, even at a reduced clock.
Chasing the wrong problem
With SPI misbehaving, signal integrity was the obvious suspect. I went through the stackup, conformal coating, series termination resistors and chip-select timing. Each change seemed to help a little, or simply moved the problem somewhere else. None of them actually solved it.
The real cause: power
The breakthrough came from looking at the MCU's own supply instead of its signals. The manufactured board had the VCAP capacitor populated with the wrong value: picofarads instead of microfarads, a factor of a million.
On MCUs with an internal core regulator, the VCAP pins connect the external capacitor that stabilises the regulator's output, the supply for the CPU core and its logic. The datasheet specifies microfarads of low-ESR ceramic capacitance there. With a few picofarads, the regulator can't hold the core voltage steady when the load changes, and the internal rail showed massive ripple.
That one part explained everything:
- Random HardFaults, because the core was running on a supply dipping out of spec.
- Interrupt jitter and timing problems.
- Flaky SPI, because the core and peripherals were sharing a noisy rail.
- Why a lower clock helped: less current, smaller load steps, less ripple.
After swapping in the correct capacitor, the ripple dropped to tens of millivolts. The board ran stably at its full 250 MHz, and the wireless link became rock solid, just like my hand-built unit.
What we check now
- Verify the regulator capacitors first on any board that shows random, non-reproducible faults. Measure the VCAP voltage with a short ground spring and look at the ripple, not just the DC level.
- Compare against a known-good board. When one unit works and another doesn't, a component-value check of the power section is often faster than any firmware debugging.
- Review the BOM and placement files for unit mistakes (pF, nF, µF) before releasing to production, and inspect first articles from the assembly house.
Even when a problem feels completely irrational, there's always a reason. Sometimes it just takes longer to find. Have you ever tracked down one tiny component that caused chaos across a whole system?