Hi,
You would have to create a custom form control to be used instead of the standard one that would provide the additional action you want to happen, and you may also have to deal with launching the resulting dialog/modal popup manually as well as you might be acting outside the standard "actions" framework here.
Basically, Alfresco uses its form service to render forms for the View/Edit Task screen, so the form you see depends on the specific workflow task you're looking at - each task in a workflow can have different forms if required. The form configuration for each task controls what fields to show for the current task instance and also decides what control to use to show them.
Out of the box, for the Items field (the list of documents associated with a workflow instance or task instance) is usually rendered using the \share\WEB-INF\classes\alfresco\site-webscripts\org\alfresco\components\form\controls\workflow\packageitems.ftl control.
You would have to either override this with a custom one, or configure the forms you want to use a different, custom control when rendering this field.
Controls are Freemarker templates that are merged together with the main form webscript response and usually create the main boilerplate html for the control and optionally instantiate a client side Javascript object to do any heavy lifting (creating buttons, registering event listeners etc).
This particular control uses one of the more complex client side components, the Alfresco.ObjectFinder so there could be some not insignificant effort involved in getting your head around where to make changes here.
Regards
Steven