For any Java application that is active, there are multiple memory areas that the application manages. One of those is the heap which you can configure/limit via -Xmx. Others are e.g. code store (previously often referenced as PermGen), stack, constant pools… Look at <a href="https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-2.html">the structure of Java virtual machine</a> to learn more. Long story short: -Xmx does not limit the total amount of memory a Java process can take.
No, there is no "easy" way to limit RAM usage, because any "easy" way would gloss over application requirements and end up in a horrible performance scenario. You can limit the RAM usage technically, but you need to understand what which memory area does, how it is used by Alfresco over time and how it affects performance.
