Auditing was not working on direct downloads but all other internal reads were being audited. For example if you apply an auditing template on a file, show_audit.ftl was getting audited for getReader.
Auditing direct downloads worked fine when PostgreSQL was replaced with MySql with all other configurations remaining the same.
Cache shouldn't affect auditing and download as actual contents are not stored in memory as far as I know.