Over at Zen of Design, Damion comments on aggro circles vs "realism":
"It would be trivially easy to make it so that every monster within eyesight attacked you when they saw you walk by or attack their friend."Attack on sight? Realism? Firstly, wild animals in the wild don't attack you on sight .. they ignore you, they prowl, they sneak, they stalk, they move away, they bathe lazily in the sun .. they don't behave like mobs in WoW: blatently ignore you until some binary switch flips, and then immediately and mindlessly charge ferociously to attack.
It's this binary nature of mobs which is unrealistic and unsatisfying. Sure, there are exceptions scattered about - there's a bird critter in Nagrand that "senses danger and flees", and there are some mobs inside instances that immediately take flight and bang some gong or some other non-attacking action. But these are the very rare exceptions, and would seem to be specially programmed for just those specific mobs.
Currently, in WoW, apart from some rare exceptions programmed into instances, the usual pattern of responses for mobs is the following: first, completely ignore you until you get too close, then immediately charge to attack, and keep fighting until either dead, or (for some creatures but not others) when severely wounded attempt to flee .. but then turn around and come back. Every time, every monster, again and again, and again.
Once you've twigged onto this pattern, the only other factor to work out is just how big the aggro circle is. That's it. No further thinking required. Commence grinding now.
How come more creatures don't try to run away at the first sign of danger? You take a whack at a zebra in the real world and it will flee, perhaps getting a kick in as it goes .. but you try the same on a zhevra on the barrens and it will be right there with you, attacking you back.
Not every creature needs to have a custom AI behaviour, that's a lot of work, you'd just need maybe a half dozen or so basic response patterns. Every creature of a given type would respond in the same manner, so it's deterministic and solvable. You've just got to work out how each time for that general species, and probably make safe assumptions according to whether it's a herd animal or loner, whether it's a predator or not.
How much more interesting would it be if the zhevras would immediately flee, and keep fleeing, only turning to fight to the death if you somehow get them below 50% health? If there was a quest to gather zhevras tails you'd need to solve this puzzle .. probably involving a few team mates spread out so it has nowhere to run, or maybe have most of your group waiting behind a log while one goes out and scares it towards you. Don't have a group? Try maneuvering such that the direction away from you is into some dead end gully. Maybe you could scare it off in the direction of some natural predator, and let it finish it off. Puzzle solved.
So, a half dozen response patterns .. it would help if the mobs had more than three choices available (ignore, fight, or flee). Can you imagine the mayhem and the fun if certain predatory cats would generally ignore you, would move away if you breach one radius, but attack if you got really close ... and would also sneak up close if they see you are distracted, perhaps attacking if they see you are wounded ... or running off again back to it's usual spot if they see you more than capably dispatching your current distraction. Deterministic and predictable behaviour, but typical for a predator creature, whether it be a lion in the Barrens, or a panther in Stranglethorn Vale.
So, design the mobs AI such that it has multiple possible responses, and design these responses such that they don't all commit the player to combat. Provide the possibility for the player to gently probe the encounter, to gather intelligence, to learn something about the mob before committing to combat. Doing so sets up the game play of making predictions about how the mob will react, and then testing it, all without every experiment being potentially lethal.
Some types of mobs might respond to being stalked by closing ranks. If you slowly approach them, they slowly group up, such that if you then attack you're within the aggro radius of many more. Puzzle. Solution: wait until they are scattered, then rush in to attack just one while it's away from it's friends; or try to outflank it from it's herd and then close in on it.
Some mobs might put on a show of ferociousness once you breach some outer radius, howling and charging right at you .. but then pulling up short, snarling at you, daring you to further intrude on its territory. It has a large reaction radius, one where it will defend it's territory, but it's real aggro radius is actually quite smaller.
Imagine some creature that has a very small aggro radius, but a large reaction radius. One where if it sees any of it's kind being attacked inside that reaction radius it will then also aggro. These would be a real bugger to go hunt, and be a nasty surprise the first time you sneak up on one. Maybe the key to this puzzle is to notice that such a mob has an innate curiosity to any intruders of it's reaction radius, meaning it slowly wanders over in your direction. So, let it wander over, draw it in, then once it has wandered far enough away from it's fellows that is when you spring your trap. Lower level spawns of this mob response would curiously sniff groups of intruders, but have higher level spawns be a bit smarter and only approach intruders who have no colleagues within 40 yards. For those, you'd need to have one of your party be the bait, with the others a bit further away. Puzzles with solutions.
Some randomness can be added to further spice the mix. The usual way the aggro radius mechanic works is that as soon as you breach that circle, and I mean immediately, they will respond. So here's a simple change to the mechanic .. make it a % chance of response, evaluated every tick. Stumble a bit too close and maybe it will attack, maybe it won't, but stay too close and it damn well will eventually. If you're lucky, or foolish, you might even be able to walk right up to the mob before it responds. For added realism, you could have the mob emote in some way even when it doesn't decide to attack, just as a warning or signal to the player that they actually are now within the aggro radius.
The mobs don't need to have real AI programmed in, just some variety of responses, and these responses don't need to be custom programmed for each mob but instead just a broad variety of general response patterns with some tweaking of the variables for each mob. Decide whether it's first instinct is to fight or to flee if you breach the aggro radius, add a reaction radius outside of the aggro radius, is it aggressively protective of it's territory if you breach the reaction radius, or is it cautiously curious and will stalk you, or is it skittish and will move away from you, is it a herd/social creature and will move towards others of it's own kind?