Welcome back, Anttix. Did you see the way I reorginzed things? I am not doing any git stuff in the workflows. I guess you are saying I could add a trigger to the ‘on’ for a new tag, and then add a step to release only if the build was triggered by a tag?
Darn. It looks like I’m still getting random build failures:
Maybe there is a way to have it trigger on a prerelease, and then edit the release to add the built artifacts.
Yep. The tag workflow would create a release, possibly using
Then each step in the matrix will upload its artifact to the release
I tried a few others such as GitHub - ncipollo/release-action: An action which manages a github release but the “official” one seems to be the best fit for matrix builds.
I kinda like the idea that we don’t need to use a git submodule. This thing is a pain to work with. The only concern I have is how does one “lock” the release to a specific Marlin version before pushing to a tag. Probably a script that captures the git hash of the Marlin repository in a file before pushing to a tag would do the trick.
I was expecting to release with Marlin’s versions, so we would only release with 2.0.6, for example. But I think we can also define a specific commit.
I did. It’s kinda neat. One thing I would do is that I would rename accessories to traits and perhaps make it so that if there is a file next to a “main” trait with a board or machine ID then it will be used to pull in overrides specific to that combination. E.g.
- traits/extruder - generic extruder
- traits/extruder.archim1 - Archim1 specific hacks
- traits/extruder.cnc - CNC specific extruder trait overrides
etc.
1 Like
That is interesting. It’s something we will have to play with. I am finding there are some things that are completely orthogonal, and others that do have interference. Like dual steppers, and the skr pro (because it has 6 drivers). I don’t need it to be completely generic, and explicit is better than implicit, but something like that could work well for that situation. I was thinking of fudging it, but in a clear way, and just letting the first script run, and then run another to undo the parts that are specific to that configuration.
Yeah, the challenge I saw was that build action has no idea which upstream version is being targeted when a tag is pushed. One can potentially encode Marlin branch/version in a tag name and have the action decode it but a file would be more reliable I think.
That was the reason I went with submodules because it is basically a file stored in git repository that contains the upstream hash. But all the logic around it turned out a bit of a pain to get right especially with automated checkouts where I had a ton of issues with shallow clones and other crazy monsters.
I’m explicitly defining which Marlin branch/tags to build right now. I am not using the forked/upstream logic to determine what is going on in Marlin. With each build, I am checking out 2.0.5.3 and then setting a configuration, zipping the result, and then trashing the whole thing.
I am also trying to follow bugfix-2.0.x. by building once/day. To add 2.0.6, I manually added 2.0.6 to the list of marlin’s to build.
It’s honestly a lot to build, and way more than we need. I really want to work on this a bit more, and get a release system out, but the hours it takes and the constant builds is driving me away.
Yeah, these matrices get large real quick and it only takes a few dimensions to do that. I had an idea to work around that by introducing version specific branches so a change generally only rebuilds all the machines for just a single version of Marlin or maybe for stable and bugfix-XXX but never for more than two.
Nightlies should only build against one upstream.
Don’t despair 
I’ve done a lot of release engineering and I always tell people that this takes calendar time to get right. One can often only poke it a few times a day. Setting expectations about progress accordingly will do wonders to stress levels and overall happiness.
1 Like
Thanks. I will be here. But I am seeing trouble coming with a new skr and a new version of Marlin out. I would really like to have at least a minimum viable product out soon so we can start doing things like providing LR firmware too. I am going to scale back the number of builds until we get a release strategy worked out.
I may also disable the nightlies until I get the release thing working. Just to clean up the workflows.
I don’t see any way to make the upload-release-asset grab an artifact instead of the zip fresh, do you?
Interesting. Looks like it’s throttling artifact upload and not actual builds.
EDIT: One thing I can think of to try is to combine Save Zip and Save Hex steps into one. The error message indicates the bottleneck is a database CPU usage (DBCPU) so maybe it can combine two or more files into a single database TX. This is a wild speculation but may be worth a try. It should be easy to move files into a temporary folder and then ask the plugin to pick all of these up at once.
EDIT2:
With the provided path, there will be 1541 files uploaded
This may also be an opportunity to optimize, maybe use a zip plugin to generate Marilin-source.zip or something to reduce the number of files Github needs to track.
No, it needs to run at the end of a “fresh” build task.
So very little of that means anything to me but just reading makes me feel like I am getting smarter. I am impressed as usual.
1 Like
Version that zips up Marlin sources and uploads only two files (HEX and Zip) is now running every 2h from GitHub - anttix/MarlinBuilder: MarlinBuilder
Let’s see if this improves stability. The only annoyance here is that the result is a zip in a zip but that extra step if someone wants to rebuild the sources should not be a big deal for nightlies.
1 Like
BTW Github is working on a feature that should reduce copy paste between workflow files https://github.com/actions/runner/issues/438
1 Like
The zip in a zip is a good idea. At least, having it in one file is much easier to share.
That composite workflow is just what I need. I can’t tell if it’s been merged to the point where I can use it or not. It looks like the PR they referenced was merged, but I have no idea what the process is from their development to the point where it gets used to parse my workflows.
Me neither, but given there has been no announcement in their blog, it’s probably still WIP and the PRs we see are related but all of the pieces are not in place yet.
When I get a minute, I’ll try it. I am wondering if maybe it is active, but not yet documented or something. I have no idea what their workflow is, but when I was poking around with my phone it looked like there was a project using it already (something that wasn’t in the github or actions account).