Removing it from the site activities won't block the user from accessing the document.
On the contrary, if you choose to put your document in a place the normal users can't access (another site, a folder with restrictive persissions), your document will be safe AND it won't appear in the activities list.
So I'd suggest you go with the second option. Which is also the only option for which you don't have to code anything!