You could possibly to do this with some deep customisation but nothing out of the box I don't believe anyway
You could extend the existing Start Workflow action, or create a new action that in addition to starting a Workflow on a document, also checks if there is already an existing workflow on a document before hand?
When a document becomes part of a workflow, there will be a small Indicator icon explaing that the Document is part of X workflows. If you view that document via the node browser, it will have two parents "cm:folder" (the folder the doc lives in) and "bpmackage". You could do a check for "bpmackage", and if that parent does exist, then skip running the workflow.
Well there is a workaround. Here is what we had to do: once the workflow gets started on particular file we move that file to the processing folder. User that started the workflow cannot open that folder from the alfresco explorer (unless he knows the direct url which is not that easy to know).
In the task dialog, we have removed the folder path and link to the folder and we also excluded those files from the search results.
So now when you start the workflow, user cannot access the folder where that file is and cannot start another workflow nor edit or delete that file.
We do have a report where user can track the state of that workflow and from that report user can always open (download) the file for later reference.
Sounds crazy, but client asked us to do what ever we can to minimize possibility for errors made by users. And they are pretty happy right now