I solved it. I just added a permission and removed the disableAction so that a Consumer who does not have say Delete permission will not have that action.
<!– Download document –>
<action id="document-download" type="link" label="actions.document.download">
<param name="href">{downloadUrl}</param>
<permissions>
<permission allow="true">Delete</permission>
</permissions>
</action>