This is unfortunately not possible with default / low-level permissions only as there is no distinction about which type of node is about to be created. Using a custom permission model, a customisation of Repository-tier web scripts, and some customisation of the Share document library "create content" menu, it would be possible to achieve something like this for Alfresco Share, but it would still be possible for members of that group to create folders via e.g. CMIS / WebDAV (mapped network drive). A solution to that would be to implement Java-based behaviours which throw exceptions as a kind of "veto" when members of the group create folders, but that would potentially generate errors in CMIS / WebDAV clients which had checked permissions before to determine they could create folders to later then be thwarted with an unexpected error.
So you see: It is possible with some trickery and limitations, but it would always take quite some hacks for a rather incomplete overall solution.
