RE: 70 cm fin or after market boom head?
On your first question:
"Can a better fin ,Deb, C3, Vector, hurricane, and others custom brands
inprove the lift of the board and the upwind angle?"
Yes. if you did alot of testing, you might find one of these custom fins that
gives you a very minor improvement, but onc again, you DO NOT have a full on formula board, so you will never (even with the very best fully tested fin for the FF-158) match their upwind performance or early planing performance.
Your board is somewhat "detuned" to make it more fun and easier to sail, but in the detuning process it lost a little tail width which affects the early planing, and the rockerline is changed slightly which diminished the upwind capabilities.
No, I do not have any experience with the Maui Sails boom head.
I know that a stiffer boom head helps a little with control, and may help to mantain sail shape when going upwind as high as possible, but I don't think you are going to see a measureable difference unless you thoroughly test what you have, record all the GPS tracks and windspeeds/directions and then change the boom head and go out in nearly identical conditions and record the same data; then do a data comparison.
The board and fin would have to be the same, same sail, rigged as closely as possible to the same way.
I don't like being a "naysayer" here, but I don't think you are going to get the full Formula board performance you seem to want unless you get a full on Formula board.
But, if you get the formula board, you will be giving up some reaching speed, some of the joy of "magic carpet rides" on the FF-FT. to gain a very small amount of upwind speed and angle and perhaps 1 knot earlier planing.
And remember, in order to get the best early planing in the light an fluky Raleigh area winds, you will be tuning your 11.7 NX pretty much the opposite from the way you would tune if for really good upwind performance in stronger winds.
It's a very nice sail with lots of performance designed in, but you can't have tons of draft to get going early and lots of twist and a really fine entry to go upwind in stronger winds at the same time.
Hope this helps,