I agree with Tony and I will add this.
I think that is relative. You are suing the largest diameter bit I have ever seen one of these routers swing. In terms of Material Removal Rate (MRR) I bet you are doing pretty good, actually. Now my next tip is probably not going to save you a ton of time, but…
A smaller endmill, I suggest 1/8, 1/4"-(maybe even 1/2") is going to remove the same amount of material (by moving a faster) but the Tram issue will be much less prevalent. That size endmill is also more useful for other things.
To actually tram it in, you will find the tape trick does work and doesn’t require much. Typically, I get things real nice with 4 layers of masking tape or less. It only goes on the inner part of the clamp, and only 1/4 - 1/2 of the surface (as in 12 to 3 o’clock, or 12 to 1:30). I have a tramming tool on printables and thingiverse.