Should I skip roughing pass cutting XPS foam?

This is following up on a different thread that ended up not being related. So i moved it back to this thread.

@robertbu - following up on the z axis issue from the other post.

So it makes sense to start from the spoil board, move it up and re-zero at the correct height.
what i can’t figure out is how to set the X,Y.
I’m trying to shave off some of the material to get down to the the required thickness. if set my z axis to the desired height then I would not be able to set my X and Y for the facing mill to be centered at the corner of the stock.
It would intersect with the material I want to remove.


red = stock to remove and is not precise
Green = desired stock height.
How would I account for the necessary offset?