Time for a fresh high-resolution picture of your as-tested configuration.
Did you pull everything except power (e.g. the endstops are disconnected, USB-C isn’t connected?)
Out of complete curiousity, what happens if you connect a return wire from a ground point on the jackpot over to a suitable ground point on the mac? Does it magically start booting all the time?
Software in this case is faulting when one of two drivers locations is populated, with any of a whole bunch of otherwise apparently good TMC2209s. This is the 2nd straight board to exhibit this issue, and the original board isn’t misbehaving for Ryan now that it’s been returned.
I can (and do!) fault software that can’t adequately respond in a non-lethal way to a missing hardware interface that shouldn’t be operationally critical. I’m definitely an odd duck that way, but stuff I’ve delivered is scattered about the solar system, so that thinking works in at least some scenarios.
As requested, this is the “good” configuration that boots.
I have looked at the Jackpot and ESP32 board with my microscope. I know Ryan tested it too. I don’t think there is anything physically wrong with the board or the drivers.
I’m embarrassed to say I don’t really know how I could do this, unless the body is grounded. One of the pins in the magnetic power connector is probably ground, but all I could really do is hold it in contact, and I really don’t want to damage my laptop.
I haven’t been using the laptop to boot all day. It just works as long as those drivers aren’t in those slots.
I could run a wire to the ground on my power supply, so they have a common ground? Does this help?
That might be worthwhile.
Do you have a pair of USB-C cables? You could plug one into the jackpot, one into the laptop, and then touch the outer shells of the USB-C cables together in the middle.
I’m trying to rule out some kind of power issue that would explain why some drivers don’t init, but somehow always do when the USB-C cable is present. I’m not sure exactly how that comes into play, unless maybe there’s something about your USB-C ESP32.
When you exchanged boards, did you keep the drivers AND the ESP32?
@vicious1 - Looking at your schematics, the pink IO Definitions aren’t present on U2 like they are on the others.
From the bottom of the board, The first (X) driver uses S0.0, s0.1, and s0.2. It then jumps over to s0.4, so.5, s0.7 for the Y driver, and s0.8 , s0.9 and s0.10 for Z.
What are s0.3, s0.6 used for? Do you remember discussion before in the design cycle about why Q3 and Q6 were floating and whether they should be?
I wonder if this is related…
Are all of these tests run with motors plugged in? Or no motors?
Does it behave differently with motors plugged in vs without?
Man this is crazy…but we are making some sort of progress.
I know how all of this is connected, and if anything I would expect issues coming from ABC, but XYZ are wired exactly as they are in the 6 pack, that is a proven system. ABC are the new multiplexed thing, XYZ are exactly as they are in the 6 pack from what I recall.
So I need to send out a micro USB version then, or do you already have some? I know the flaws of the USBC version, and the micro version and they are different. The whole issue with that logic is it works fine on my end. How is this an edge case, making it not work on your end. Either way I think we need to try a MicroUSB version to see if that prevents it. Heck maybe even a genuine esp32 just to verify.
What is it about your network that is messing with this and how it boots with those two driver sockets filled.
Just for kicks can you also verify socket “C” doesn’t mess with it when you get a minute please.
And just to be certain, if you only put a driver in “Y” it still doesn’t boot right?
X uses the lower IO pins on the 74AHCT595PW118. Then there’s a pair of floating outputs interspersed with the control lines for the Y and Z. Y and Z don’t work at times (but only at some times).
Then all the rest use the MUX UART scheme, and there’s only one floating GPIO in the set of muxed drivers.
I think that whatever is happening here, it impacts the IO going to the Y and Z stepper drivers in some way that pisses them off, preventing booting far enough that the WiFI stack gets loaded. Probably this is impacting UART communication between FluidNC and the stepper drivers in Y and Z.
Is it the case that X, Y, and Z are all using distinct UART intefaces on the ESP32? (I’ll go look at the schematic again to verify) I wonder what the ESP32 looks like inside as far as references. Maybe this is actually a case of a defective ESP32 that’s been jacking with us in a really weird way.
OK - will run back out and do those tests.
However, I did run a common ground to the PSU from the board, then from the ground pin on the TMC 2209 in the Y1 socket, from the ground on the ESP32, and finally from the USBC housing. 4 tests total, and none of them changed the behavior.
I honestly think it was just because before the multiplexing it made the layout easier. The UARTs were set up with more room. Someone else asked about the floating part and it doesn’t matter on those chips (from what I remember).
Previous unreleased board we made.
No XYZABC all use one channel.
The ST_UART pin they all share is connected to gpio-0 with a resistor…Mitch does not like that but I assume that is from the 6 pack as well. The resistor is there but it is a boot pin…but if that was a thing X would mess with it as well
I see that now. I see in the schematic that the TMCs also have floating pins. Too many short little unterminated antennas on these floating pins, perfect to pick up noise in a busy RFI environment. Noise to mess up serial communications, or noise to set them into oscillation.
I don’t like it either, but I suspect it’s because that is also the BOOT pin, the resistor is there in case the user pushes the BOOT button.
I think it’s time to swap the ESP32 as you suggested as I’m thinking there’s something about that coming into play here.
OK - this is interesting. I’ll show my work below. (now switched to XYZABC designations)
Summary - Only one of the X, Y, Z spots can be occupied. A, B, and C can all be occupied without changing the behavior.
Testing method - (X only, X+Y, X+Z, X+A, X+B, X+C, X+AB, X+ABC then Y only etc.)
If only X is occupied it boots. X also boots with A, B, C
If only Y is occupied it boots. Y also boots with A, B, C
If only Z is occupied it boots. Z also boots with A, B, C
X + Y , X + Z, Y + Z all fail to boot. (These are the only combos that failed, although I did not test a failed combo with ABC.)
I bought a : ESP32-WROOM-32E 4MB FLASH Genuine Espressif board from Digikey, but I can’t get it to work. I can flash it and see it when I boot from USB, but the one I have doesn’t appear to be compatible with Jackpot because it never boots, even on an empty board… I must have screwed up and ordered an incompatible version. I decided this is a distraction, and I set it aside for WLED or another use.
I don’t have another ESP32 here to test, my apologies. Because the V1E power supply didn’t appear to have any effect, I did switch back to my PSU for these iterations. (Actually, I switched back for the common ground experiment and didn’t change it.)
Would this work?
ESP32 4M 8M 16M Core Board WROOM-32E MicroPython (N8-8M Flash) https://a.co/d/6Zt1ymw
Is this the usb c version? Have you tried the other chip? The old version? Oops i see above you do not have anything else to test, sorry.
These are the ones linked in the Docs Amazon.com
Nothing about this is operationally critical, and we are definitely in a corner of a corner case here.
I agree it would be nice to have that log. But it also is not paid for by NASA. We get a lot of chances to fix it (unlike the stuff soaring through the solar system).
BTW, your stuff in the solar system: that’s cool.
Hmmm. Is the ST_UART the transmit from the drivers, or the receive?
When all the drivers are plugged in, and you power up from the PSU, the esp32 and all the drivers turn on at once. They send some little message at boot. Maybe if there are several, the boot pin gets lower than if there is only one. The boot pin goes low and the ESP goes into the boot loader mode. If you connect via fluidnc, it resets the esp, and this little driver handshake is already done. So the boot pin is left alone. If you power the esp via USB first, it gets through boot, and the TMC drivers don’t get configured, but when they get power, the esp is not put into the bootloader.
That kind of makes sense except:
Maybe it’s a slightly different ESP32 (perhaps different resistors or capacitors on it)?
Didn’t Ryan note something about this already (errata in the various clone ESP32s)
Or maybe they’re more likely to step on each other in a really noisy RF environment, or maybe there’s an interaction with the RF stack when the radio on the ESP is in some kinds of environments.
All you need to do is change either some element of timing to expose a race condition, or change the electrical noise environment to expose an EMI/EMC issue.
The interaction between those TMC 2209s only liking one peer in the first bank of 3 smells like either timing or EMC to me. I agree with you that when the USB-C cable and fluidterm are introduced into the equation it changes some aspect of this interaction and makes it work.
I’m at a research institution these days. My projects are no longer carrying the massive budgets like the US defense programs or big NASA missions. Have to do more with less. My last mission was an interplanetary mission that succeeded despite having “less than 1/3 the funding needed” (according to peer reviewers including some of those old NASA and commercial peers.) It also launched on time at the beginning of its launch window, and on budget.
You’d be surprised how often I encounter really good debug capabilities in simple inexpensive hardware and how often I find complete crap in hardware with lots of zeros in the price tag. Then again, with you working on autonomous vehicle software, I bet you wouldn’t find that surprising.
This weird problem we’re chasing here has to be an interaction of some kind with the ESP32 and the TMCs and it is certainly way out in some corner condition.
That doesn’t excuse not having better debug. Maybe the ESP32 just cant do it, but there are lots of tricks. For example, putting debug information into the least use parts of RAM, and on a reboot checking to see if there is debug to share from the last boot. If so, dump that to the UART or try and write it to nonvolatile storage. Maybe the ESP32 wipes the RAM on reset or something that makes that impossible.
I’m just noting that even on tiny embedded targets there are good design for test capabilities that can be built in. And an ESP32 is a remarkably powerful device for as inexpensive as they are.
This is 2 guys writing an open source firmware, for free, in their spare time.
It should not be a reach to understand that they are not handling every case, and certainly that they have not programmed for something they likely have never encountered.
I think it excuses it very much,
I’m sure they are open to accepting a PR with your solution and/or funding in order to make it happen.
