Hello,
I suspect you have already adapted the SOLR configuration to make use of the larger RAM, e.g. re-configured the various caches in solrcore.properties.
One thing I definitely would try to do is adjust the default template for the "keywords" field, e.g. what the "test" phrase will be processed with to determine the final query. Removing some of the default fields - specifically TAG - might speed up the query significantly, as fewer field caches may need to be initialized / loaded from disk into memory or fewer sub-queries may need to be fired.
Have you compared the final query from the global search with the search in site? Unless there is a major difference in the query (other than the PATH restricting to site) I would not expect a huge difference in performance unless the result set is significantly smaller, e.g. the Repository has a lot less to do in post-query permission checking (which, by the way, did not apply to SOLR performance in Alfresco 4.0, but was kind of re-introduced somewhere in the 4.1.x Enterprise line as well as 4.2).
Regards
Axel