why concurrent user upload files by cmr service is very slow
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2013 07:13 AM
Hi,
I use multi-thread(10 thrads) to invoke alfresco cmr nodeservice to upload file into alfresco.
But it is very slow,only 3 files(1M per file) per second.
everybody can help me,give me some suggestion to optimization it?
thanks.
cj.tin
I use multi-thread(10 thrads) to invoke alfresco cmr nodeservice to upload file into alfresco.
But it is very slow,only 3 files(1M per file) per second.
everybody can help me,give me some suggestion to optimization it?
thanks.
cj.tin
Labels:
- Labels:
-
Archive
5 REPLIES 5
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2013 08:07 AM
I don't think an answer is possible given the lack of detail in your question. You will need to do some investigation yourself to narrow down on the problem first.
At minimum, please post your code and details of your system including the configuration.
At minimum, please post your code and details of your system including the configuration.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-05-2013 04:28 AM
Hey,thanks your reply.
My code as bellow,
And i remark these invoke function in DbNodeServiceImpl.java,and the speed can be 150 files per second.
// checkPendingDelete(parentRef);
// ParameterCheck.mandatory("parentRef", parentRef);
// ParameterCheck.mandatory("assocTypeQName", assocTypeQName);
// ParameterCheck.mandatory("assocQName", assocQName);
// ParameterCheck.mandatory("nodeTypeQName", nodeTypeQName);
// invokeBeforeCreateNode(parentRef, assocTypeQName, assocQName, nodeTypeQName);
// nodeIndexer.indexCreateNode(childAssocRef);
Why ?
My code as bellow,
private void start(String[] args) { try { preFolderName="D"; sumFiles=0; sumFileSize=0; int start=1; int end=100; folderSize=2; if (args.length>0){ start=Integer.parseInt(args[0]); } if (args.length>1){ end = Integer.parseInt(args[1]); } if (args.length>2){ folderSize = Integer.parseInt(args[2]); } if (args.length >3){ fileNo=Integer.parseInt(args[3]); } writeFileContent=false; if (args.length >4){ writeFileContent=true; } tmpFolder=new File("c:/tmp"); NodeService nodeService= EcmService.me.getNodeService(); NodeRef root = nodeService.getRootNode(StoreRef.STORE_REF_WORKSPACE_SPACESSTORE); ChildAssociationRef child = nodeService.getChildAssocs(root, ContentModel.ASSOC_CHILDREN, QName.createQName(NamespaceService.APP_MODEL_1_0_URI, "company_home")).get(0); NodeRef companyHome=child.getChildRef(); File[] tmpFiles = tmpFolder.listFiles(); files=new FileContent[tmpFiles.length]; int k=0; for (File fi : tmpFiles){ FileContent fc=new FileContent(); fc.readFrom(fi); files[k++]=fc; } List<Thread> threads=new ArrayList<Thread>(end-start+1); sumFileSize=0; long beginTime=System.currentTimeMillis(); for (int i=start;i<=end;i++){ final int threadNo = i; final int finalEnd = end; final Folder folder=new Folder(companyHome,preFolderName + i); NodeRef folderNode= GetFolderCommand.getOrCreateFolder(folder, true); folder.setNodeRef(folderNode); Thread thread = new Thread(new Runnable() { @Override public void run() { System.out.println(String.format("upload thread %d/%d start…", threadNo, finalEnd)); try { AuthenticationUtil.runAsSystem(new AuthenticationUtil.RunAsWork<Object>() { @Override public Object doWork() throws Exception { uploadFilesToFolder(threadNo, folder); return null; } }); } catch (Exception e) { e.printStackTrace(); } } }); threads.add(thread); thread.start(); } for (Thread thread: threads){ thread.join(); } long endTime= System.currentTimeMillis(); double span = (endTime-beginTime)/1000.0; double sumSizeMB=sumFileSize / 1024 /1024.0; StringBuilder msg=new StringBuilder(); msg.append("upload totalFiles=").append(sumFiles).append(",\t totalSize="); msg.append(sumSizeMB).append("MB"); msg.append(",\tspent seconds=").append(span).append(","); System.out.println(msg); msg=new StringBuilder(); msg.append(String.format("%1$.4f MB/s , \t", sumSizeMB / span)); msg.append(String.format("%1$.2f Files/s , \t",sumFiles/span)); System.out.println(msg); } catch (Exception e) { e.printStackTrace(); } }
And i remark these invoke function in DbNodeServiceImpl.java,and the speed can be 150 files per second.
// checkPendingDelete(parentRef);
// ParameterCheck.mandatory("parentRef", parentRef);
// ParameterCheck.mandatory("assocTypeQName", assocTypeQName);
// ParameterCheck.mandatory("assocQName", assocQName);
// ParameterCheck.mandatory("nodeTypeQName", nodeTypeQName);
// invokeBeforeCreateNode(parentRef, assocTypeQName, assocQName, nodeTypeQName);
// nodeIndexer.indexCreateNode(childAssocRef);
Why ?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-05-2013 05:41 AM
Can't see anything wrong with that code. I was interested in your how your transactions overlapped but you havn't listed the code of uploadFilesToFolder so I can't tell.
Looking at your last method listed If you are using lucene (again guessing due to lack of detail) then the indexing and metadata extraction is in-transaction so 3 per second may be reasonable. For comparison my old laptop used to do 2 document ingestions per second with lucene.
It may also be worth you looking at the bulk file import tool since it does this sort of stuff already.
Looking at your last method listed If you are using lucene (again guessing due to lack of detail) then the indexing and metadata extraction is in-transaction so 3 per second may be reasonable. For comparison my old laptop used to do 2 document ingestions per second with lucene.
It may also be worth you looking at the bulk file import tool since it does this sort of stuff already.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-05-2013 06:01 AM
thks your reply.
i remark these code:
// checkPendingDelete(parentRef);
// ParameterCheck.mandatory("parentRef", parentRef);
// ParameterCheck.mandatory("assocTypeQName", assocTypeQName);
// ParameterCheck.mandatory("assocQName", assocQName);
// ParameterCheck.mandatory("nodeTypeQName", nodeTypeQName);
// invokeBeforeCreateNode(parentRef, assocTypeQName, assocQName, nodeTypeQName);
// nodeIndexer.indexCreateNode(childAssocRef);
and it be fast and i can access alfresco by web-client and share application.
and also,i have another performance problem:
i query files by lucene and sort by cm:created field,it fast.then i query same files and sort by my custom field,it's also a datatime field,same config as cm:created,it be slow.
i invoke by searchService in cmr package.
i remark these code:
// checkPendingDelete(parentRef);
// ParameterCheck.mandatory("parentRef", parentRef);
// ParameterCheck.mandatory("assocTypeQName", assocTypeQName);
// ParameterCheck.mandatory("assocQName", assocQName);
// ParameterCheck.mandatory("nodeTypeQName", nodeTypeQName);
// invokeBeforeCreateNode(parentRef, assocTypeQName, assocQName, nodeTypeQName);
// nodeIndexer.indexCreateNode(childAssocRef);
and it be fast and i can access alfresco by web-client and share application.
and also,i have another performance problem:
i query files by lucene and sort by cm:created field,it fast.then i query same files and sort by my custom field,it's also a datatime field,same config as cm:created,it be slow.
i invoke by searchService in cmr package.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-10-2013 08:44 PM
up,and also,i have another performance problem:
i query files by lucene and sort by cm:created field,it fast.then i query same files and sort by my custom field,it's also a datatime field,same config as cm:created,it be slow.
i invoke by searchService in cmr package.
i query files by lucene and sort by cm:created field,it fast.then i query same files and sort by my custom field,it's also a datatime field,same config as cm:created,it be slow.
i invoke by searchService in cmr package.