You can add an "evaluator" to any action there are a few examples in the forums and in the XML documents in the basic system and there are documents on in various wiki's. You will need then to write a custome evaluator that takes a list of groups that a person must be in before the action will become available.
You will also have to interfere with the bulk action drop down on the document library. This is also configurable with evaluators
