I looked up the FreeX 130 (nice board) and see it comes with a 39cm fin. I am guessing that the OFO is around 40cm to 46cm based on the supplied 39cm fin. Most here seem to be pointing the finger at your fin, saying it is to big and sending you upwind. I disagree with them (sorry everybody). My reason for this is: Without seeing your fin, it would be impossible to say for sure if you need a smaller fin. If you take 5 fin brands of the same size, they will "all" feel very different. Example: you would need a C3 Venon 45cm to get the same power/lift as a Select SL7 39cm, 6cm difference. Size alone is not a good guide line as there are to many elements for fins.
I do not think that buying another fin will solve your problem. As Ken say's, a bigger fin will drive you upwind more efficiently c/w a smaller fin, but a big fin will not automatically go upwind (unless you are seriously over-powered).
Without actually seeing your gear set up, we are all just speculating, but I am pretty sure that the problem is with your sail set-up. The HTS is only a guide line, it is marked at the average beam reaching position. Moved back 3cm to 5cm will allow easier downwind runs. On my Norths I have my lines about 5 to 6cm behind the HTS, I always found the recommended setting was very wrong for me.
Also, if you down haul to the max VTS, the sail will breath easier and will allow excessive power to exhaust, this will stop extra pull on your back hand and in turn will stop you pushing against the fin with your back leg, sending you upwind.
Your outhaul ideally should be slightly negative to help downwind, but if your worried about getting back upwind, set it at neutral. Do not set it with positive outhaul as this also send you upwind.
Short harness lines will also send you upwind. If you have a pair of longer lines, try them out, or maybe you can butcher your existing lines to make the longer. Longer lines really help for downwind.
Deck plate and footstrap position will make a big difference as well. Deck plate forward, easier upwind, pull it back and easier downwind, same for the footstraps.
All in all, if your settings are upwind biased, yes your board will try to creep upwind and make it uncomfortable downwind.
Is there an experienced sailor/guru at your local beach that look at your kit for you?
This is just my thoughts and feeling on what might be wrong, and hope you find out soon what the problem is. Good winds