(sorry, this got really long while I wasn’t watching)
Testing can be thorough and consistent without necessarily being arduous or expensive. I’d suggest building very explicit, step-by-step test procedures that can be executed by a very basic skill-level user. Remember the goal of testing is to identify defects (or ideally ensuring there aren’t any). Testing does not have to include defect troubleshooting/investigation/remediation -that’s where the expensive expertise comes in. The easier testing is to complete, the more likely you are to get complete test results.
There are probably other more expert folks available, but when I’ve documented test plans, I’ve used spreadsheets or tables that serve both as user/tester instructions and result collection tools. In a spreadsheet, I’d have a header worksheet to capture tester name and hardware variations, then a separate worksheet for each isolated test activity to be completed. The worksheets might be things like:
- Compile and Upload Firmware
- Exercise LCD UI and verify version
- Browse SD contents, load and run test gcode file
Each worksheet would have columns like Step #, Description of user action to be taken (e.g. “Rotate knob 5 clicks clockwise”), Expected result (“LCD scrolls down to display “Settings” menu”), Pass/Fail, and Actual Result (a free entry field where tester describes what the failure was, e.g. “LCD display scrolled up”). If the tester can workaround a single-line failure, they continue with the subsequent tests, if not they note that testing could not continue.
This makes it is easy to do consistent testing even with different users executing the tests, and to collect consistent results, including some “hints” if things don’t go as expected.
Whoever collects the test results does the final analysis, not the tester. Obviously, if all the individual steps pass, the test passes, but someone else needs to make the call as to whether a minor unexpected result requires fixing before the release.
It seems daunting to put this together, but you really only have to do it once, with minor updates as features change through the releases.