The other Olympic classes can get around the course with the sails they have on board, without reefing or changing. They don't have to go back to the beach to change sails as the wind changes. I think most only allow one set of sails to be used each regatta, just like the RSX does.
You'll never find a 470 or Tornado, for example, racing under mainsail alone, and I've never seen them go downwind without a spinnaker up unless something has gone very wrong. A 49er is actually extremely difficult to get downwind in a breeze without a spinnaker up so they don't have much of a choice. Not a single Olympic class has changed to smaller sails in strong winds since about 1965.
Changing sails IS easier said than done, and it's not politics. Courses are normally set well offshore, away from the land effects to allow fairer racing. Race committees are under pressure to get races completed. Event sites are often crowded.
To stop the racing while the windsurfers decide to go in and change sails can mean holding up the racing for the vast majority of the sailors (those on boats) and then a huge muddle on shore. I've hit the beach on an Olympic class regatta on a board at the same time as 70 or so Lasers wanted to get up the ramp; getting up onto the beach, finding space to change sails and then going against the flow would have been very hard.
And what does the RC do when conditions change? If sailor X has a physique and skills that allow him or her to use a big sail across all conditions, why should they have to sit and wait because person Y, say a lightweight light-air sailor who cannot handle a big sail in big winds, goes in to change gear?
If the wind has gone from 13 knots to a perfect 20 knots, does the RC wait until the last lightweights struggle across the finish line in a heat, then allow those sailors to struggle back to the beach, then wait while they re-rig, then wait for the last one to get back out to the start? Or do sailors just gamble that they will be able to go back in and change rigs before the next start? Won't the sailors with a RIB and pit crew have a huge advantage?
Olympic classes have been pretty popular at times - I can remember in Division 2 days going from the worlds of the most popular class of the time (basically F42 Raceboard/slalom) to a Div 2 worlds and being amazed at how big and professional the Olympic class worlds were. Getting Olympic status can really hurt the numbers of a class, but the Olympic class worlds in boards are normally bigger than the FW worlds in terms of entrants and nations.
I think if you check the actual numbers around the world, you'll find that those who want to race in light winds are NOT limited in numbers, compared to FW. In many major sailing countries, the "light wind" boards are at least as popular. This isn't attacking FW, just pointing out that it doesn't seem to be the vast majority of racers as sometimes implied.