I want to customize the users search ( particularly when assigning a user in a workflow ) to let the user serach by groups. First, he selects the group, after that, he chooses a user which belongs to the group selected.
This sounds like a reasonable thing to do and is entirely possible as a customization. I'd probably start by specializing the existing person finder component. Go look at how the group picker is built, then look at how the person finder is built, then mash up the two.