Yeah, that’s definitely a consideration. I prefer the approach of using a pattern bushing rather than a following bit with a bearing. That also helps by making it a lot harder to nuke your pattern, in my experience.
The bushing approach has the advantage of being a fixed height relative to the base rather than the bit so you can use it for shallow trenches/dados or making a deep cut in a bunch of steps. The main downside is that the bushing is always bigger than the bit so you need to have a pattern that reflects that.
If you’re doing stuff in CAD then the change in dimension is a lot easier to manage. It’s relatively easy to dimension structures as the size you want + the bushing clearance or to draw the structure you want and then use an offset tool to expand it.
It can also be a little bit of work to get the bushing lined up exactly with the bit but for most stuff I’ve done it doesn’t matter if there’s a small ~0.5mm offset. I just make sure to either not rotate the router as I’m moving it if I want the pocket size/shape to be accurate or keeping the router aligned with the path I’m following, depending on the goal. There are methods for getting it aligned but I found them a bit too finnicky with the bases on palm routers. In the end I made myself a large acrylic base and cut a step in it for a pattern bushing that I could accurately align and leave in place. I use the spare base from the router in my MPCNC to do all my general purpose stuff and keep that base as purely a pattern follower base.