topic safely crawl all documents via webscript in Alfresco Archive https://connect.hyland.com/t5/alfresco-archive/safely-crawl-all-documents-via-webscript/m-p/280985#M234115 <HTML><HEAD></HEAD><BODY><SPAN>what i am trying to do is:</SPAN><BR /><BR /><SPAN>find all nodes in the repo and get their file size. also get all versions of the node and calculate the overall filesize of the node and its versions. </SPAN><BR /><BR /><SPAN>how can i safely crawl every document in the repository?</SPAN><BR /><BR /><SPAN>searchservice is going to hit the result-limit easily, even if i increase the limit, the searches wont return results.</SPAN><BR /><BR /><SPAN>by traversing recursively through the repository i also seem to fill up the solr caches</SPAN><BR /><BR /><BR /><PRE class="language-none line-numbers"><CODE><BR />private static void traverse(List&lt;FileInfo&gt; context) {<BR />&nbsp;&nbsp;&nbsp; for (FileInfo node : context) {<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (node.isFolder()) {<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; traverse(fileFolderService.list(node.getNodeRef()));<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; else {<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // is file = do stuff<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }<BR />&nbsp;&nbsp;&nbsp; }<BR />}<BR /><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN></CODE></PRE><BR /><BR /><PRE class="language-none line-numbers"><CODE><BR />… :44,186&nbsp; INFO&nbsp; [solr.component.AsyncBuildSuggestComponent] [Suggestor-alfresco-1] Loaded suggester shingleBasedSuggestions, took 267411 ms<BR />… :53,005&nbsp; WARN&nbsp; [cache.node.nodesTransactionalCache] [http-apr-8080-exec-1] Transactional update cache 'org.alfresco.cache.node.nodesTransactionalCache' is full (125000).<BR />… :17,075&nbsp; WARN&nbsp; [cache.node.aspectsTransactionalCache] [http-apr-8080-exec-1] Transactional update cache 'org.alfresco.cache.node.aspectsTransactionalCache' is full (65000).<BR />… :17,081&nbsp; WARN&nbsp; [cache.node.propertiesTransactionalCache] [http-apr-8080-exec-1] Transactional update cache 'org.alfresco.cache.node.propertiesTransactionalCache' is full (65000).<BR />… :19,938&nbsp; WARN&nbsp; [alfresco.cache.contentUrlTransactionalCache] [http-apr-8080-exec-1] Transactional update cache 'org.alfresco.cache.contentUrlTransactionalCache' is full (65000).<BR />… :19,991&nbsp; WARN&nbsp; [alfresco.cache.contentDataTransactionalCache] [http-apr-8080-exec-1] Transactional update cache 'org.alfresco.cache.contentDataTransactionalCache' is full (65000).<BR />… :49,599&nbsp; WARN&nbsp; [org.alfresco.nodeOwnerTransactionalCache] [http-apr-8080-exec-1] Transactional update cache 'org.alfresco.nodeOwnerTransactionalCache' is full (40000).<BR />… :27,516&nbsp; WARN&nbsp; [cache.node.childByNameTransactionalCache] [http-apr-8080-exec-1] Transactional update cache 'org.alfresco.cache.node.childByNameTransactionalCache' is full (65000).<BR /><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN></CODE></PRE><BR /><BR /><BR /><SPAN>i understand it is an antipattern to grab everything at once, but i don't know of any service/api that allows me to page the results into batches/pages, </SPAN><BR /><SPAN>please enlighten me <img id="smileysad" class="emoticon emoticon-smileysad" src="https://connect.hyland.com/i/smilies/16x16_smiley-sad.png" alt="Smiley Sad" title="Smiley Sad" /></SPAN><BR /><BR /><SPAN>version: 5.0.c</SPAN></BODY></HTML> Wed, 20 Jan 2016 10:44:40 GMT jaeni 2016-01-20T10:44:40Z safely crawl all documents via webscript https://connect.hyland.com/t5/alfresco-archive/safely-crawl-all-documents-via-webscript/m-p/280985#M234115 what i am trying to do is:find all nodes in the repo and get their file size. also get all versions of the node and calculate the overall filesize of the node and its versions. how can i safely crawl every document in the repository?searchservice is going to hit the result-limit easily, even if i in Wed, 20 Jan 2016 10:44:40 GMT https://connect.hyland.com/t5/alfresco-archive/safely-crawl-all-documents-via-webscript/m-p/280985#M234115 jaeni 2016-01-20T10:44:40Z Re: safely crawl all documents via webscript https://connect.hyland.com/t5/alfresco-archive/safely-crawl-all-documents-via-webscript/m-p/280986#M234116 <HTML><HEAD></HEAD><BODY><SPAN>It's not the solr cache that is filling up, its the transaction cache.&nbsp;&nbsp;&nbsp; </SPAN><BR /><BR /><SPAN>The node service does give you all results, so apart from being slow that is O.K.&nbsp;&nbsp;&nbsp; </SPAN><BR /><BR /><SPAN>The problem is that your code is executing in a single transaction and at some point there will likely be a limit on the number of database rows that can be updated</SPAN><BR /><BR /><SPAN>What the alfresco code does itself in those situations is to use the batch processor to break up your huge transaction into smaller chunks.&nbsp;&nbsp;&nbsp; That's probably what you want to do here.</SPAN><BR /></BODY></HTML> Wed, 20 Jan 2016 11:38:48 GMT https://connect.hyland.com/t5/alfresco-archive/safely-crawl-all-documents-via-webscript/m-p/280986#M234116 mrogers 2016-01-20T11:38:48Z Re: safely crawl all documents via webscript https://connect.hyland.com/t5/alfresco-archive/safely-crawl-all-documents-via-webscript/m-p/280987#M234117 <HTML><HEAD></HEAD><BODY><SPAN>Is there an easy example that i could re-use? if found user-rename tool but i am even more riddled now</SPAN></BODY></HTML> Wed, 20 Jan 2016 13:33:12 GMT https://connect.hyland.com/t5/alfresco-archive/safely-crawl-all-documents-via-webscript/m-p/280987#M234117 jaeni 2016-01-20T13:33:12Z