Think the extension designers have pretty much answered this question.
Take any short, medium or long extension, slip the adjustment collar up to the highest
setting, and measure the amount of extension from the seat on the collar to the top of the extension.
My guess would be about 5.0" (12.7 cm) based on a NP 26 cm max (medium size) extension I just found under my chair.
As far as how much is too much, measure the front of the lowest batten in your sail from the foot, then see how far up inside the mast your extension goes.
If it extends beyond the height of the lowest batten (the foot batten) then you need a little shorter extension.
This is not critical on RDM masts and probably not on small wave and freeride sails, but the higher end sails, with lots of shape in the foot of the sail, can have some real problems if the mast base extends beyond the front of the lowest batten and "stiffens"
the mast in an area where the designer intends it to start bending to allow the foot batten to rotate freely.
Yes, you can often "fix" the poorly rotating foot batten by backing off the tension, but then you get wrinkles and less than the design amount of draft in the draftiest and most power making area of your sail.
Hope this helps,