Yeah, it seems to be that even a “real” dust collector isn’t going to help much because of that small port. If you’re building it into a bench, you can box most of it in; lots of examples on YouTube. Then a dust collector might be more effective, or a box fan like you suggest.
Personally, I’d start with a box fan or two because even if they don’t work out in this application,they’ll tell you whether a dust collector might be better, and box fans are never useless to have around. Worst case, you could build them into a shop air cleaner with your filters (also plenty of examples on YouTube).