The machine in question has been up and running for more than a year and the config.yaml hasn’t been changed in 10 months or so. Lately I have been experiencing boot issues where it boots to ALARM mode and complains about an Invalid config. Pressing the reset button on the ESP-32 board doesn’t fix the problem, but usually powering off for 20 seconds then back on tends to work. It’s been hard to troubleshoot because it is so random. Was just wondering if anyone else has experienced this behavior.
I did find one other post with a similar issue and his problem went away when he changed out the router that the ESP-32 was connecting to. Sounds unlikely but I can’t dismiss that as a solution. Just can’t figure out how a network issue temporarily corrupts the local file system on the ESP-32.
I just realized that I should have Claude look at the firmware code to see if that is possible.
That’s likely using AI for something it’s not really well suited for.
We need to get some details from you to start looking into your issue.
To start out with, when you next find the machine is in its’ funky state, send the $SS command from the terminal. Paste it’s output back here in the forum for us to look through.
Use the preformatted text icon on the forum editor toolbar… it looks like this: </>.
I don’t think it’s a WiFi problem if you can boot the board and access it that way. I had a similar issue where mine complained about a corrupt config for a while, even though nothing had changed as far as I knew. Maybe the UI overwrote something. To fix it, I just re-flashed it and started fresh. Just be sure to note any pull-offs or changes you’ve made. After that, it worked perfectly.
Thanks Jim. A few months ago, I would have agreed with you 100% on this, but I have found the Claude Opus (and Fable) are pretty good at tracking down issues as long as you give them enough info. Edit: The trick is to keep Claude in a sandbox where it can only see the code and not get distracted. I typically will open the source code as a project in VS Code and use the Claude plugin. That way you get less rabbit holes.
I was planning on updating the firmware but before I do, I will do some digging ($SS, etc.) and report back. Will probably be a few days.
The tools are getting better all the time.
With good understanding of the underlying system and good harnesses on your usage you can get amazing results. I’m not at the point where I’d get those kinds of results.
FluidNC continues to make big jumps forward, the last few releases worked through some issues. Maybe the 4.0.5 series will be a good one.
You didn’t mention which version you were running. There were some stability issues in some of the 4.0.x series builds.
Did you make changes to the 3.9.4 fork you’re running?
We are still recommending 3.9.9 as far as I know. I’m watching 4.0.5 to see if it is stable enough to recommend. There are significant feature improvements there, but the 4.0.x series has been changing so fast it’s had stability issues.
My money is on some type of power related issue. Maybe my power supply isn’t putting out the current that it used to or maybe a bad regulator or cap somewhere. The 152 error is fairly generic and is returned with every command when the machine is in the CONFIG ALARM state, and the “bad config” is a red herring and presumably means that it simply never loaded the config.yaml because it went into panic mode before it got to that point. It’s also a good bet that the WIFI is the trigger, but not the cause. I’ve been using ESP boards long enough to know that the majority of brownouts are caused by WIFI because it is the biggest power drain on the board. It is especially apparent on the cheap clones that have an under-rated 3.3v regulator on them. I am using a V1 board, so I know the 3.3v regulator is a good one. Hopefully the boot log might shed some light the next time it happens.
It was not a power issue, nor a bad connection on the card. I was able to duplicate the problem because it became less random and nearly permanent, so I pulled the ESP board and powered it up via USB and experienced the same issue.
The network that the ESP is connecting to is an ASUS AiMesh with a main router and one AiMesh node. The node was in a state which was causing a divide by zero error in the EspressIf WIFI module. Rebooting the main Asus router did not fix anything, but rebooting the node has temporarily fixed the issue. I realize this sounds unlikely, but my testing was exhaustive and I’m convinced it is correct. I believe the randomness of the issue was because whenever the ESP connected to the main router, it booted up fine, but when it connected to the node, it failed.
So, the bug is likely in the EspressIf WIFI code but because of where it was happening in the boot process, FluidNC was treating it as a config error and switching to panic mode, which loads a generic config, triggers an ALARM that cannot be reset, and insists the config error be addressed. Huge red herring. Specifically, the code can be found in the ../Machine/MachineConfig.cpp in the FluidNC source, but I noticed that file has changed since v3.9.4, so it is possible that they have addressed it. I haven’t had a chance to analyze the new code to verify that or not, just glanced at it. I should also note that despite the divide by zero error, the WIFI was working fine and I was able to get to the WebUI - just dead in the water due to the panic mode state. And $SS was useless as a troubleshooter, since technically there was no config problem at all.
Here is a link to a post that I mentioned which was experiencing a similar problem (just in case you think I’m crazy ):
Edit: I should also note that during the troubleshooting process I reformatted the ESP and re-flashed it. This solved nothing.
No. After I rebooted the node, the problem went away so I won’t be able to duplicate the behavior until the node starts doing whatever it was doing to cause the problem.
It is likely fixed in 4.0.5, not necessarily in the FluidNC code, but EspressIf has updated their libraries since 3.9.4 as well and I assume 4.0.5 is built with them.
Unfortunately, the ELF file associated with the firmware build I was troubleshooting with had been overwritten, so the exact location in the source code where the divide by zero error was occurring cannot be determined. If it happens again, I’ll flash the ESP with a build that will allow the memory locations of the compiled code to be translated to the source code. There were some other network issues, that I need to fix sooner than later, that I think were related to the same node, so I will likely do a firmware update on that router before it goes off the reservation again. Once I do that, the situation might not occur again.
):