If you push sideways and it moves 3-5 mm, that is quite a lot, even for a tall machine.
The Z distance from the cutter tip to the gantry rails is rather long and that lever arm makes the deflection worse. Raise the spindle within the mount as much as possible. If you’re not cutting any deeper than the length of your bit then you could raise the router so that the collet is level with the ends of the Z tubes. Then prop up the workpiece as high as possible so when the Z axis is all the way up, the tip of the bit is not much above the workpiece.
Even so I think there is something wrong with your setup that is allowing the Z axis to pitch away from vertical. I know you said you tightened all the bolts but it’s hard to know how much, and the instructions say not to tighten too much. Check if all bearings in the center assembly are in contact with their respective rails. That could indicate that you can go tighter in those specific areas.
Are you using a single flute bit? That can make a difference.