This may be something that other referees rule differently, and I know folks could say, "But you wrote the game!"; notwithstanding, I do feel it's a matter of preference.
For me, a charge attack is not the same as standard melee. A charge attack is always a single attack -- mutliple attacks do not apply. For example:
[*] Fighter makes charge attack with broad sword. If he hits his enemy, he does +2 damage; in either case (hit or miss) he loses his dex bonus to AC or is at least penalized -1 to AC.
[*] After charge attack he goes into his normal melee routine. Say he is 3/2 with the sword. He will get one attack on the round that follows the charge, and two attacks next.
In a nutsell, I do not allow multiple attacks on a charge and in fact treat it as a separate type of combat action. It has so many benefits already: double the distance and bonus damage.
HYPERBOREA- A Role-Playing Game of Swords, Sorcery, and Weird Science-Fantasy