I am not saying it can not be done just anything under 1mm is extremely difficult to point a finger at. we have dimensional errors, and squaring issues. Square is simple to adjust, dimension is a hard road. Some people have just had bad collets out of the box, but that was a dimensional thing. Square test parts would really help me, to help you, and you need to be able to measure them to the accuracy you require. A circle will work but you are throwing away the extremes but if you accurately mark the axis we can make full use of your calipers at least.
If you cut a square(or circle), or draw one, you can measure each axis and each diagonal. You can cut or draw a ~380mm square on your machine. That gives you ~537 diagonals, and with that you are measuring the difference between them. Again, or a accurately marked circle.
So a standard caliper is accurate to about 0.01mm or 0.0005". So in a perfect world you could get ±0.01 over 537mm, at best but most people only have a 150-200mm caliper.
Step one is cut the largest square/circle you can, measure it to the best of your ability. We need each axis, at least, and both diagonals. Then do it again and see what numbers you get a second time…and more if you really want to get into repeatability.
From there we will see if you are even close dimensionally, and then we can discuss how square you are. Square is extremely easy to adjust with M666, much more so than screws, but we need more than one number to reference. I know the holes seemed to work better for you but we can not help you with that, we have to have numbers.
The larger problem usually by far is always dimensional accuracy. You have belts, pulleys, bearings, routers, collets, part flex, rail flex, and endmills that all add inaccuracies.
I am absolutely in for seeing how far you can dial in a giant 15"x15" bed, and I will help in anyway I can.
Lets see some numbers. I am not really sure how to start to help without lots of nerdy numbers for me to use.
This only tells me how far you are off, not where and by how much. I have no frame of reference to tell you what is wrong or where, or even if that is + or -.