Obviously, to a party of adventurers, it's basically a dinner gong.
Yeah, I think that's the key, right? They are already fighting minotaurs, snake-men, giant lizards, skeletons, zombies, ghouls, mummies, evil necromancers and so on. They've heard about gorgons (and stuff way worse).
Plus, quite frankly, it just seems lame/disappointing to grow up watching it happen in Clash of the Titans only to find out that on the rare occasion it's relevant in a fantasy RPG, your referee has imposed penalties/risks so onerous as to effectively neutralize the whole concept. YMMV and all of that.