I need to make some custom group selector for bpm:groupAssignees. The main goal is to limit selected groups by subgroups of some group. I know that it is possible, see <a>http://www.appnovation.com/blog/form-control-members-group</a> but I have no idea how to do it
You can create your own custom webscript which fulfill your requirement with the same response as OOTB webscript used in object finder component.
Now, 1. In authority.ftl provide you custom webscript URL in below parameter. finderAPI: Alfresco.constants.PROXY_URI + "{Custom Web Script URL}/${field.control.params.itemFamily!"node"}", OR 2. You can pass from share-config-custom.xml file also.