It's actually less mystical and much more practical. The fastest sails are very sensitive to batten tension, and most sailmakers apply battens the starboard side of the sail. While much effort has been made to equalize the effect of batten tension between the port and starboard tacks, such as the modern batten tensioners seen on Gaastra and Maui Sails or alternating the side on which each subsequent batten sits.
Given the above and the need for very high skin tension, sails tend to represent the intended shaping when on starboard tack. This makes then more stable and probably faster on starboard than port, at least when all other things are equal.