What's your weight again?
Whenever you are sailing a "sinker" (for your weight) sail size (and sheeting angle) become much more critical.
If you don't have quite enough sail size, you won't get going easliy, even with "flutter pumping".
Also smaller sails are much more sensitive to proper sheeting angle and it's very easy to oversheet/undersheet them.
This is why you find that under 6.0 m2 there are sail sizes by 0.2 m2.
If you were riding on larger waves (head high and more) and really "wavesailing" rig size becomes even more critical because you need to be able to accelerate quickly, jibe quickly, and depower the sail on the wave face.
So, smaller wave/B&J/high wind sails are designed with less overall draft and this is why it's easy to undersheet/oversheet.
On a 6.5 m2 rig, you will find that you can move the boom end 2-3 inches between no power (undersheeted too much) and stalled (oversheeted too much).
On your 4.2 m2 rig the 2-3 inches will shrink to maybe 3/4"-1.5" (maybe even less.
So you have to become alot more sensitive to the "feel" of the little sail when it's making
Also, if you are accustomed to heading well off the wind on much larger sails (7.5 m2 +) to get going in marginal conditions, this just doesn't work so well on smaller sails as they do not have the draft depth to make their best power at much below 120 deg.
This is why the high wind speed sailors look for areas where the speed course can be set up 110-120 deg. off the prevailing wind direction.
It's the fastest and easiest control direction at full speed but unless you have 30 knots+
windspeed you can't get going on the little narrow "speed boards" they are sailing.
Yes, you might have had better luck getting going (from a beach start or waterstart) with your 4.7 m2 rig, but you might also find that the 4.7 m2 would be a little hard to control.
But, when sailing sinker boards, it's always better to have a little more power than you really need because you can undersheet (or ease your sheeting angle a little) and dump a bit of power.
If you don't have enough power to get going it's really tough.
A quick story, from my own experience, to illustrate thse points.
My first ride on a "sinker board" was in Cape Hatteras, out to the "Reef" (3 miles) in the middle of the Pamlico Sound.
My friends girlfriend (she was a really good high wind sailor) wanted to go out there so she picked her little board and a 4.2 m2 wavesail.
I didn't have any boards smaller than around 100 liters so she said I could use the little "Gorge board" that they had stashed under their house.
I think it's about 75 liters and a early 90's "no nose" design.
I rigged a 4.8 m2 wavesail and off we went......well, off she went.
I tried to beachstart and the little board just sank to the bottom as soon as I put weight on it.
She sailed back in and told me to move further out to the windline.
I tried just inside the windline a couple more times, and the little board started forward then just sank to the bottom and stopped.
When I got just past the windline it was amazing.
I did exactly the same as I had in less wind inside the windline, but the little board just took off.
At some point they gave me that little board and I still have it as my "high wind" board.
Since it's a "no nose" design (maybe the little board you were on was a no nse as well)
it's always a struggle when I first get on it because I have to keep a foot more forward to force the nose off the wind.
Once it starts to light up, then back into the straps and that little board is a rocket ship.
It's fast, jumps really easy, but it's very sensitive to sail size and sheeting angle, and I always kinda hate it right at first.
As soon as I adjust to the no nose and sinker design, then I love it.
Hope this helps,