What toolpath are you using for the 4mm holes? I’m guessing that you used a contour toolpath for the holes and the exterior of the plate. I just checked, and the Bore toolpath will mill a 4mm hole with an 1/8" bit.
The Bore toolpath is made for cutting holes. Do the outside and the large hole using a Contour tool path. Generate a second toolpath to bore the holes. Then select both toolpaths when post processing. A single g-code file will be generated.
I’ve always used 2D Contour toolpaths for outline cutting. I’ve used Ramp toolpaths a couple of times for steep but not perpendicular edges. I don’t know what your hold down strategy is, but 2D Contour has tabs, and Ramp toolpath does not.
Feeds and speeds depends on a number of factors including type of wood, size and type of the endmill, and whether you have a variable or fixed speed router. One thing I did not understand when I started out is that it is possible to go too slow as well as too fast. Go to slow and your bit burns and grinds, but where that too slow is will depend on the size of the endmill, router speed, and number of flutes on the endmill. I recommend doing some test cuts of square depressions in a piece of plywood at different settings.