I always pictured Voormis as more straight up hairy Neandrethals and not necessarily ape-man like, which would be more akin to gorillas. (in my mind).
Here is some good description of Voormis from the Seven Geas by CAS
Few of these caverns could be reached by men without the use of ropes; and the Voormis, who were possessed of quasi-hunan cunning, would hurl blocks and rubble upon the heads of the assailants. Most of the caves were narrow and darksome, thus putting at a grave disadvantage the hunters who entered them; and the Voormis would fight redoubtably in defense of their young and their females, who dwelt in the inner recesses; and the females were fiercer and more pernicious, if possible, than the males.
As for the Voormis themselves, they were indeed a misbegotten species; but it was hardly necessary, in explaining their generation, to go beyond the familiar laws of nature. They were merely the remnant of a low and degraded tribe of aborigines, who, sinking further into brutehood, had sought refuge in those volcanic fastnesses after the coming of the true Hyperboreans.
They had spent the night beneath its lowermost crags, keeping a ceaseless watch, piling dead branches on their fires, and hearing on the grisly heights above them the wild and dog-like ululations of those subhuman savages, the Voormis for which the mountain was named. Also, they heard the bellowing of an alpine catoblepas pursued by the Voormis, and the mad snarling of a saber-tooth tiger assailed and dragged down; and Ralibar Vooz had deemed that these noises boded well for the morrow's hunting.He and his men rose betimes; and having breakfasted on their provisions of dried bear-meat and a dark sour wine that was noted far its invigorative qualities, they began immediately the ascent of the mountain, whose upper precipices were hollow with caves occupied by the Voormis. Ralibar Vooz had hunted these creatures before; and a certain room of his house in Commoriom was arrased with their thick and shaggy pelts.
Also, much was said regarding the genesis of the Voormis, who were popularly believed to be the offspring of women and certain atrocious creatures that had come forth in primal days from a tenebrous cavern-world in the bowels of Voormithadreth.
Ralibar Vooz inferred that the buttress was inhabited by some person nearer to civilized humanity than the Voormis, who were quite ignorant of the use of fire.
Raphtontis rose toward the lowest cave, and hovered at its entrance while Ralibar Vooz climbed precariously behind him amid a heavy barrage of bones and glassedged flints and other oddments of less mentionable nature hurled by the Voormis. These low, brutal savages, fringing the dark mouths of the dens with their repulsive faces and members, greeted the hunter's progress with ferocious howlings and an inexhaustible supply of garbage.
They stood only half erect, and their shaggy heads were about his thighs and hips, snarling and snapping like dogs; and they clawed him with hook-shaped nails that caught and held in the links of his armor.