Folder rules can invoke Java classes that are instances of ActionExecuterAbstractBase. That means you can write your own Java class and make a call to any other class on the classpath.
Folder rules can also execute JavaScript. That's because there is a Java class that extends ActionExecuterAbstractBase that knows how to run JavaScript.
So, your first option is to write a Java class, extend ActionExecuterBase, and call your class from there.
But you said you want to call JavaScript and your JavaScript is going to call a class. That's kind of "the long way around" but it's perfectly fine. In that case, you'll need to write a Java class that is a scope processor extension. Basically it just exposes your Java class to JavaScript. That class can call the other Java class that is on the classpath, and because it is a scope processor extension, it will be exposed as a root variable in Alfresco's server-side JavaScript.
All that's left to do at that point is set up your folder rule, use the "Execute Script" action, and point to your server-side JavaScript that makes the call to your new root variable.
