I've largely dealt with this by using random rolls to determine which target, from those closest to it, that a monster will go after. That and morale has been enough, surprisingly, to have some really fun combats, partially, because I have stuck to mostly creatures of animal intelligence. But I've wanted something more sophisticated for a while.
Traditionally, I think the solution is supposed to be to look up the monster in the monster manual and you find out about its behavior and ecology there. But that means you have potentially infinite monster behaviors to memorize as DM. That's not a good option. I think traditionally the encounter dice are supposed to give the DM valuable info too, but because it just generates a range of numbers it isn't going to solve that memorization problem: high or low numbers will presumably lead to different corresponding actions by different creatures.
In this post by Alexis of the Tao of D&D looks at creatures by intelligence and one of the factors is how they react during encounters. That has been bubbling in the back of my head for years now as well as an article from the Dragon that talked about intelligent monsters targeting high threat party members like magic users.
I think I have finally come up with a solution that simplifies the idea while keeping much of the flavor. I can assign all creatures one of a small number of behaviors. That will give me some consistent behaviors without requiring too much to remember. I would put them into these six categories:
- Pack hunters
- Lone hunter
- High Threat
Mindless creatures, like zombies, would function the way I've been handling monsters, they attack whatever's closest with a bit of randomness for equally close targets.
Vermin would only physically attack if they far outnumber the target or are cornered, but they will shadow the party and try and steal small trinkets and food.
Scavengers would behave similar to vermin with a lower outnumbering ratio needed. They would also become aggressive over any carcasses the party produces fighting other creatures.
Pack hunters would try to encircle the party and, also dependent on the numbers of each group, might start probing attacks by stronger pack members to see if the whole pack should swarm in.
Lone hunters might shadow the party for a while before darting in quickly in an attempt to carry off the smallest party member.
High threats are like mother bears with cubs or elephant bulls in must, unless you show sign of submission and back away quickly, they will attack ferociously. (hmm, that seems a little too similar to mindless).
What I like about this is that players could learn these behaviors and gain a sense of expertise about the dangers in the wilderness as their characters become veterans. And all I have to do is decide which behavior type traditional monsters fall under. Are stirge vermin or pack hunters? And that gives the cool possibility of having some stirge that act as vermin, the bluish-black ones, and some as pack hunters, the reddish tinged ones. Parties will learn to pay attention to that small difference.
Are there any other broad behavior types that are left out? Maybe lie-in-wait type hunters, but those are more like traps that would then attack mindlessly.
(I know this still doesn't deal with sentient foes, but I have an idea that six additional categories might work for them, including guerrilla tactics and such.)