I've adopted the habit of periodically firing up Task Manager and checking the memory usage of my running programs. Most of the time, everything is as it should be, but occasionally, I'll notice bizarre situations where an application has decided to misbehave. If you're particularly protective of system resources, Process Piglet can give you some warning before things get out of control.
As the developer explains, Process Piglet is designed to alert you of when a program decides to quickly exceed its average memory usage, giving you "advance[d] notice" before it gets too greedy. It does this by monitoring a newly-started application's memory consumption over a 30-second period and using this as a baseline.
If the program then allocates memory beyond this baseline, you'll be notified via a dialog just about the notification area of the task bar.
A similar tool for CPU usage, called Process Tamer, is available from the same developer. Essentially, it lowers an application's priority if it begins to hog too much CPU time. As you'd expect, it's possible to have it ignore certain programs.