There is no option to filter by user group, no. If you have some structure to your users names, and in particular if the department is encoded in the username, you can filter that way. If not, I think there's no way.
Turning it on for all users shouldn't be a huge overhead, if that's what you are concerned about. In fact in a production system I'd almost say it should be turned on for all users. It has been on my systems from day 1.
Steve.