Well, good old ASCII art…
So take Y Max as up and X Max as right on your screen.
If the / measurement is longer, you need more pull-off on the left hand side, closer to Y Min.
If the \ measurement is longer, add it to the right hand side.
Naturally, subtracting from the other side is also possible.
The diagonals measurement enhances the out-of-square amount, but the actual amount of change gets diminished because the belts are outside of the cut area. Using the amount of the difference as an adjustment is a fudge, and isn’t guaranteed to get you a correct result, so you keep repeating the process. Some time ago, I did a CAD drawing that you input your machine parameters (Namely distance between Y belts), the Y length of your rectangle, and entered your diagonal measurements into and it would tell you how much to adjust by. It’s in the forums here somewhere…
How to: Square your Lowrider by adjusting the end stops with M666
Right. There it is.
Edit: M666 is the Marlin method of adjusting pull-off. We do it more individually with GRBL/FluidNC.