Jackpot Intermittently panic reboots mid job, cannot read the panic reason

Mostly Printed CNC - MPCNC Troubleshooting
1

I have been chasing intermittent panic reboots on a Jackpot for a while. I have ruled out a fair amount and I am stuck, so I would like a second opinion before I start changing hardware.

Setup
Machine MPCNC
Board Jackpot, TMC2209, board: Jackpot TMC2209
Firmware FluidNC 3.9.9, upgraded today from 3.8.0 by OTA
WebUI stock, served from the board
Config V1E reference MPCNC/Jackpot/JP1_MPCNC/config.yaml, three deliberate changes below
Stepping I2S_STATIC, pulse_us: 2, idle_ms: 255, planner_blocks: 32
Motors 5 off, run_amps 0.800, hold_amps 0.500, microsteps 8
Spindle Makita palm router, switched by hand, so NoSpindle
SD card 2 GB, FAT16, frequency_hz: 20000000, card_detect_pin: NO_PIN
Network STA mode to a Zyxel router, 2.4 GHz, channel 10 at 40 MHz
CAM Kiri:Moto 4.7.4
Sender normally the WebUI running files from SD. Also tried CNCjs over USB.
Host macOS

Differences from the V1E reference config, all intentional:

  • run_mode and homing_mode are SpreadCycle rather than StealthChop, for torque
  • axes.z.max_rate_mm_per_min: 900 (was 1200, now matches the reference)
  • homing.mpos_mm: 0 rather than 3

Symptom

The board panic reboots during a job. On reconnect the console shows:

[MSG:ERR: Showing startup log from previous panic]

Two other observations that may matter:

  1. During a job it sometimes pauses, as though waiting for the next instruction, then carries on. Sometimes it never gets the next instruction.
  2. After a panic the WebUI reports no SD card. A full power cycle brings it back. A soft reboot does not.
  3. It clusters. Some days are completely clean. Yesterday I ran the machine all day without a single failure. Today it failed repeatedly. Whatever drives it varies day to day rather than being constant, which is what makes it hard to pin down and hard to trust a good run as proof of a fix.

History

On 3.8.0: it panic rebooted on nearly every move. Moving from AP mode to STA mode made that much better, and I then ran 98 scripted jogs across two runs with zero crashes.

Today, still on 3.8.0: it crashed while jogging from CNCjs over USB, with the router off.

Upgraded to 3.9.9 by OTA. Config was preserved.

On 3.9.9: 168 scripted jogs over 20 minutes, zero crashes, and it returned to the start position exactly. Then the first real job panicked part way through.

What I have ruled out

The G-code. I scanned the file: no non-ASCII, no nulls, longest line 54 characters, only G0 G1 G2 G3 G21 G90 M30, no malformed lines. 2625 arcs, largest expanding to about 73 segments, about 24k segments in total. Worst arc radius mismatch is 0.074% of radius, well inside the usual rejection threshold. Nothing unusual at the point where the job was last seen running.

The router. It was switched off, on a separate mains circuit, with no dust extraction and its cable not in the drag chain. The jog crash happened with it off.

The SD card. On a Mac card reader it read 67 MB of files with zero I/O errors.

The config. It matches the V1E reference apart from the three deliberate changes above.

What I cannot do

I cannot read the panic reason. The panic handler writes to UART only, before the reboot, so telnet and the WebUI never see it. My Mac is USB-C only and my cable is USB-A to micro-B, which forces it through a dock that drops the CP2102N after a few seconds. A direct cable is on order.

I also checked whether $ESP420 reports a reset reason on 3.9.9. It does not, it returns only free memory and firmware version. I gather that was added in 4.x.

Questions

  1. Does the pause-then-continue behaviour point at SD read latency? My reasoning is that a slow read empties the planner and stops the machine, and a read that never returns blocks the task until the watchdog resets the board. That would also explain the missing SD card afterwards. Is that plausible, and is frequency_hz: 20000000 worth lowering even though it is the V1E reference value?

  2. Is the 3.9.9 watchdog freeze reported here the same thing I am seeing? I found a thread describing rst:0x8 (TG1WDT_SYS_RESET) on a Jackpot 3 running 3.9.9, fixed by moving to 4.0.5-pre3. I cannot confirm my reset reason without the cable.

  3. Is 4.x sensible on a Jackpot 1 right now, or should I stay on 3.9.9 until it settles? I have seen 4.0.4 described as having a job crash and a telnet bug, and advice to stay on 4.0.3.

  4. Anything obvious I have missed? I ran the same job in the air, router off and unplugged, nothing touching material. It panicked at about 48 minutes, 50.4% through. So cutting load and router EMI are both ruled out. I also polled the status twice a second for the whole run and saw no motion pauses, so the planner was never visibly starved during that run.

Happy to post the full config, the G-code analysis.

2

These types of issues are usually debugged by having FluidTerm connected while it’s running. That captures the panic.

Can you share the gcode? I’ve seen where large I/J values in G2/G3 moves can crash the controller.

3

I am interested. Have you tested this? We don’t typically move very fast for this to come into play but if it reasults in a tesed gain I am all ears. Also both can be tuned to more prefomance but I do not have any real way of testing subtle differnces.

This is where I would start for sure.

My gut answer to this is save your config, do a full erase and install 4.1.1 and mike’s 3.0.11 webui(or figui) and try again. I recently had random issues updating a very old build on a zen and all was fixed with a full wipe and a fresh install with the old config.
Second would be to make sure your board is clean and not caked in debris, I have seen a lot of dusty boards over the years and a little humidity would cause random issues.

4

Well, thanks…I am stuck in a rabbit hole seeing if we can gain any meaningful performance. :rofl: