Personally, we watch eastern martial films, there is going to be one spinning from prone, legs flailing as if to strike at another; kicking their knees, hamstring the back of knees, so the other will collapse, causing the assailant to withdraw or be hit.
And same thing with a sword or axe; that swinging it around one's head from prone is not as difficult as one thinks if aiming for a leg and a warrior should be train for any possibility, not him standing in place sword-fencing? Still he vulnerable, hence the -4 AC and some mention as I searched through PHB of no shield being allowed, but uncertain if that was for some grappling attack? Still it would make sense if someone was pushed to either protect self with shield or attack with sword, but not both. The -2 you suggested Iron Ranger makes sense, it not easy, but not impossible.
But it should not be common, no warrior wants to be knocked prone. But the negatives can be constant, for example: did he fall on a wet glass covered floor? a banana? or was he bowled over by a raging bull or an ogre?
How would a Magician respond or an agile Thief? Will some person with a Con of 4 rise as quickly or as equal as one with Con 18?
Make adventurer roll a Test of Strength to attack or a Test of Dexterity if using a rapier of dagger to complete manoeuvre? Depending how he got prone in first place?
Or a test of Constitution for hitting floor and not have wind taken out of them? Of course hitting mud is better than hitting jagged rocks, so bonuses and negatives apply.
1/2 move makes sense also... if not a 1/4 if he fighting and rising and spinning and rolling and avoiding a giant's foot. Also heavy armour might slow some. As state an agile thief should be fast with a knife not a warrior on full plate spinning to his feet with a greatsword. Go for the groin Boo!!!
The list is endless, it is making the rule in the moment that in the end makes sense?