I'm about your same size and also sailed wide boards with big sails. I've not raced formula, but I did own various formula boards. For me, 9.6 was the biggest sail I ever felt comfortable with. (Loft 02). I had a Sailworkds Retro 9.5 and a Sailworks Race 9.8 (forget the model). I could sail them both, but they were just too big to be fun. (The Rertro 9.5 was pretty close, however.) I did sail a friend's 11.0 Retro with his Bic Techno Formula once. It was sailable, in very light wind (took nothing to plane) but I felt it would be way too big if the wind got any stronger.
I'm sure there is a lot of technique to sailing big sails, and I suppose with work and effort I could have used the bigger ones. But the main problem was almost as soon as there was enough wind to plane the big sails got "too big".
For me, a cam sail automatically feels at least a meter bigger (and I'm biased - I don't like cam sails for other reasons). But this may just be a matter of "feel" and nothing else.