I haven't played it or had copies of the books so I'm not too sure how they work, but 2e Ravenloft had rules for those specifically, you could probably borrow them though again, I'm not sure how much adaptation would be required. If you really want, the d20 SRD has sanity rules that originally came from Call of Cthulhu d20, but it'd be a bit more work to get that integrated with the Hyperborea rules. Quick and dirty, one could probably use the standard saving throw system to do it too, probably the best option if you don't want to add new rules.

A few of the more unnatural monsters (e.g., elder things) can cause insanity on a failed saving throw, but there is no generalized system for this. Frankly, I don't think having relatively hardened fantasy characters lose sanity from seeing skeletons or fish-men would be a valuable addition to the game. These are not Arkham academics.

Nice 1 Iron, I agree...

In Mongoose Conan D20 the Terror of the Unknown roll is each time you meet a monster. It 3.0 rules so, DC equals 10 plus half the HD of creature.

To convert to Hyperborea  should not be to difficult by comparing HD, a 2nd-level fighter vs a 3 HD creature/monster could gain a -1 negative to a saving throw (ST) (I would normally choose the prime ST for class). Where as a 4th-level fighter vs the same 3 HD creature/monster would gain a +1 bonus to the ST?

I think the players will generate their own fear and play it through their character because the events will dictate how they react without any rolls of dice?

lol, I tend to play the character as is, so big scary thing, is a big scary thing, but your right some may shrug their shoulders and play on, hence Fear rolls...

For Horror/Fear, I've used the higher of Wisdom or Intelligence as a line, as both can be used, Wisdom to apply past knowledge to the situation, and Intelligence to understand the situation at hand and apply understanding to the situation. EX, A Wise, but an unintelligent person could apply past knowledge and lore to a situation to overcome the fear/horror of it. An Intelligent, but Unwise, could understand the situation and overcome their fear/horror using their brain to understand the Why/How in the current situation.