I don’t find any examples how to extract the upload_url that is needed as an input to upload-release-asset plugin, but I believe the event payload should be available in some context variable (just didn’t find the syntax with first try) and upload_url is certainly part of it.
If this works, then it may be possible to shave a step off of the release process. Github has a fairly nice web UI for creating a release that allows you to select a branch and automatically creates a tag from the top commit of that branch. The workflow would then react to the release::created event, build off of the freshly created tag and upload to release. Once all is done, one can hit publish.
I did that, without the uploading files to the release, on sandify:
I stage all the code I want in master, then at some point I will make a release, filling out the details in markdown, and that will trigger an update to the githubio page. I do a quick check a few minutes later to make sure it looks right and then share it on the forums and twitter.
Thinking through the workflow for this though, I think there are probably more steps:
Scheduled build on bugfix, to give us a heads up on problems coming down the pike. This is probably the lowest priority though, honestly. We would almost never download these, unless someone had a really neat feature just merged, or there was a problem building it, and we needed to build an issue around it.
Updating our configurations for the latest Marlin. Like what happened with 2.0.6.
a. We would edit the workflow files to build up some .zips with that new version as a base, look out for configuration problems, build errors.
b. We would want to download the build products and vet them on a machine. It would be great to crowd source that (so I’m already thinking this might be tough).
c. We would want a release at that point to get it out the door. The release would only include the latest version of Marlin. Ideally, we would have already released 2.0.5.3, and the release this week would have been 2.0.6.
Developing for a new board or a new configuration.
a. We would add the workflow, change the settings, etc. We would work on a topic branch and use github actions to check for errors and build candidate builds.
b. The person developing the changes would be downloading the zips and inspecting them and trying them on the new boards.
c. This would be approved when it looked good, and we would merge it into main.
d. At that point, do we edit the previous release? Probably not. Do we make a new version and a new release? Probably. We could also wait until the next release to include it.
Bugfixes or configuration changes on the existing firmware. This is probably similar to 3.
a. Develop on a topic branch.
b. Developer downloads and checks .zip files from a few configurations.
c. Merged into main when it looks like it is solved.
d. At that point, we’ve changed the versions, so we would make a new release.
In cases 2,3,4, we are ending up at the release page on github. I can’t figure out a good way to do anything better. I would really like to present the users with something a bit cleaner, and with some directions on which version to pick, but I don’t have a good way to do that without really leaving room for a ton of errors. In all those scenarios, I think we are working on the build products created in the topic branches, and then trusting that they will be identical when we do the release.
There are also manual triggers. We could have a release trigger that only happens through the web interface. Other than the explicit nature of that, I don’t see a benefit over using the release trigger to do that.
So I’m at the point now where my two main goals are going to be:
Get the builds robust. I think your zip approach is promising. I am also ready to drop a ton of configurations until we have it stable.
Get the zips into the release page based on a specific manual trigger.
Best way to crowd source testing is with RC-s or release candidates. There is a concept of a pre-release in github that can be used to model this. Continue publishing RC-s until comfortable with the feedback and then turn the final RC to an actual release.
I am not sure if we want a lot of topic branches in the main repository. Every PR is essentially a topic branch but the difference is that these are easy to rebase and thus keep history clean. Branches, especially the ones that are long-lived are evil. Merging is messy, rebasing is going to break people who have it checked it out etc. It’s better to avoid this. In practice we’ve found that it’s often better to merge changes early and just turn them off if they are not ready to be used yet rather than having long lived branches and dealing with the maintenance hassles of what is effectively forked code.
I’m not advocating for along lived branches. I know git well enough to avoid that. I doubt there would ever be two at the same time. If there was, it would be rare, and I know git well enough to fix most issues.
The RC releases is a good idea though. I don’t like the idea of trying to point a forum user to the artifacts to try to test something out. Apparently you need to be logged into github, the links aren’t very intuitive to read, and they could be deleted later.
I’m assuming that you could trigger the same build on a PR, or push, and release, but look at that release.upload_url to determine if you ran the release upload step? Something like:
(Apologies for R-rated language in the commit message)
Next step is to test with a tag push. Then cleanup.
Turns out GitHub UI will not actually “create” a release nor a tag if you “save a draft”. Thus there seems to be no way to trigger a workflow for an unpublished release from the web UI.
Thus you either publish, which makes the release visible right away (probably not a big deal but may feel weird), or you have to push to a tag form a command line so a draft release can be created from a workflow.
Making a public (but maybe pre-release) release is fine to trigger a job. I see you have two release triggers. Is it smart enough to only do the work once?
It looks like it is making it’s way through their release process. This release from today includes it:
I still don’t know when they actually install this on the runners I get when I run an action, but my guess is they are rolling it out now, since they just did this release.
BTW, their release workflow is pretty neat:
They have 5 build products they need in their matrix. They do a check, then build (with the matrix) and save the artifacts. Then they download the artifacts to the workspace, and then add each item to the release manually. I suspect it is because they are doing some other work in those release-uploads.
I think this branch is set up as a simple test. I am basing this off of the draft documentation, or maybe it is more like a requirements doc? Anyway, it doesn’t seem to work yet, and I’m not surprised.
I don’t know I just three letter’d it to get something to trigger so I can see if it actually works.
On the other side of things, it looks like I have accumulated 20+ full builds with no artifact upload failure yet. How often did you observe these throttling errors? I don’t know if I would call the fewer uploaded files idea success yet, but it is indeed promising.
It’s too late, I’ve already cherry-picked the one zip commits. They are in main already. I think it has improved in quality. But I thought that after the max-parallel: 1, and that broke the second day.
That looks like it works. Two improvments I think:
We should probably either replace “created” with “published” or add them both because if one hits “save draft” and then later publish, created event is not triggered. This was an issue I spent so much time scratching my head about until I found the link I posted above. I’ll do some testing to determine exactly how triggering works.
release:
types:
- created
We probably do not want to release different Marlin versions. I am thinking that maybe I’ll write a quick local module which can read marlin version from a file in the repository and store it in an “output variable”. Alternative would be to store the version in the workflow, luckily the version of the workflow that runs is tied to the git hash so in theory one does not need a separate file. However this way nightlies may need their own set of workflow files. I think a module would be better.
EDIT: Plus a few more minor nits like suppressing builds specifically on tag pushes instead of limiting builds to “main” branch, using tag name as V1_VERSION if release build is triggered etc.
Yep, published seems to be the best option. Created does not trigger if you save a draft and listing both created and published will trigger twice if you hit publish right away (w/o saving a draft).
I tested the release strategy, which is a little klunky, but it is good that it is a little manual:
a. Created a branch
b. Changed the version in src/core/version
c. Manually forced the version of all the builds to be the version of Marlin I wanted
d. Created a release, or a prerelease tagged with that branch.
Results: https://github.com/jeffeb3/MarlinBuilder/releases
Bugfix on rambo actually did break yesterday. I can see which commit it was, and I have provided a small point upstream to that error. I don’t want to end up being the arduino Marlin police, so I may need to work on getting an arduino build upstream as well. I also reported the 2.0.6 Archim not working in arduino issue, but so far all I’ve heard is, “32bit boards should use platformio”.
I just three letter’d it to get something to trigger so I can see if it actually works.