I would apply the same penalty to encumbered, and the same impossibility for heavily encumbered. It would apply to all the physical abilities, not the intelligence- and wisdom-based abilities.
HYPERBOREA- A Role-Playing Game of Swords, Sorcery, and Weird Science-Fantasy