topic Re: why concurrent user upload files by cmr service is very slow in Alfresco Archive

why concurrent user upload files by cmr service is very slow

nocturn9 — Tue, 03 Dec 2013 12:13:25 GMT

Hi,I use multi-thread(10 thrads) to invoke alfresco cmr nodeservice to upload file into alfresco.But it is very slow,only 3 files(1M per file) per second.everybody can help me,give me some suggestion to optimization it?thanks.cj.tin

Re: why concurrent user upload files by cmr service is very slow

mrogers — Tue, 03 Dec 2013 13:07:50 GMT

I don't think an answer is possible given the lack of detail in your question. You will need to do some investigation yourself to narrow down on the problem first.

At minimum, please post your code and details of your system including the configuration.

Re: why concurrent user upload files by cmr service is very slow

nocturn9 — Thu, 05 Dec 2013 09:28:31 GMT

Hey,thanks your reply.

My code as bellow,


   private  void start(String[] args) {
        try {
            preFolderName="D";
            sumFiles=0;
            sumFileSize=0;

            int start=1;
            int end=100;
            folderSize=2;

            if (args.length>0){
                start=Integer.parseInt(args[0]);
            }
            if (args.length>1){
                end = Integer.parseInt(args[1]);
            }

            if (args.length>2){
                folderSize = Integer.parseInt(args[2]);
            }

            if (args.length >3){
                fileNo=Integer.parseInt(args[3]);
            }

            writeFileContent=false;
            if (args.length >4){
                writeFileContent=true;
            }

            tmpFolder=new File("c:/tmp");

            NodeService nodeService= EcmService.me.getNodeService();


            NodeRef root = nodeService.getRootNode(StoreRef.STORE_REF_WORKSPACE_SPACESSTORE);
            ChildAssociationRef child = nodeService.getChildAssocs(root, ContentModel.ASSOC_CHILDREN,
                    QName.createQName(NamespaceService.APP_MODEL_1_0_URI, "company_home")).get(0);

            NodeRef companyHome=child.getChildRef();

            File[] tmpFiles = tmpFolder.listFiles();
            files=new FileContent[tmpFiles.length];
            int k=0;
            for (File fi : tmpFiles){
                FileContent fc=new FileContent();
                fc.readFrom(fi);
                files[k++]=fc;
            }

            List<Thread> threads=new ArrayList<Thread>(end-start+1);
            sumFileSize=0;
            long  beginTime=System.currentTimeMillis();

            for  (int i=start;i<=end;i++){
                final int threadNo = i;
                final int finalEnd = end;
                final Folder folder=new Folder(companyHome,preFolderName + i);

                NodeRef folderNode=  GetFolderCommand.getOrCreateFolder(folder, true);
                folder.setNodeRef(folderNode);

                Thread thread  = new Thread(new Runnable() {
                        @Override
                        public void run() {
                            System.out.println(String.format("upload thread %d/%d start…", threadNo, finalEnd));
                            try {
                                AuthenticationUtil.runAsSystem(new AuthenticationUtil.RunAsWork<Object>() {
                                    @Override
                                    public Object doWork() throws Exception {
                                        uploadFilesToFolder(threadNo, folder);
                                        return null;
                                    }
                                });

                            } catch (Exception e) {
                                e.printStackTrace();
                            }
                        }
                    });


                threads.add(thread);
                thread.start();
            }

            for (Thread thread: threads){
                thread.join();
            }

            long endTime= System.currentTimeMillis();
            double  span = (endTime-beginTime)/1000.0;
            double sumSizeMB=sumFileSize / 1024 /1024.0;

            StringBuilder msg=new StringBuilder();
            msg.append("upload totalFiles=").append(sumFiles).append(",\t totalSize=");
            msg.append(sumSizeMB).append("MB");
            msg.append(",\tspent seconds=").append(span).append(",");
            System.out.println(msg);
            msg=new StringBuilder();

            msg.append(String.format("%1$.4f  MB/s , \t", sumSizeMB / span));
            msg.append(String.format("%1$.2f  Files/s , \t",sumFiles/span));

            System.out.println(msg);


        } catch (Exception e) {
            e.printStackTrace();
        }
    }
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

And i remark these invoke function in DbNodeServiceImpl.java,and the speed can be 150 files per second.

     // checkPendingDelete(parentRef);
     // ParameterCheck.mandatory("parentRef", parentRef);
     // ParameterCheck.mandatory("assocTypeQName", assocTypeQName);
     // ParameterCheck.mandatory("assocQName", assocQName);
     // ParameterCheck.mandatory("nodeTypeQName", nodeTypeQName);
     // invokeBeforeCreateNode(parentRef, assocTypeQName, assocQName, nodeTypeQName);
     // nodeIndexer.indexCreateNode(childAssocRef);

Why ?

Re: why concurrent user upload files by cmr service is very slow

mrogers — Thu, 05 Dec 2013 10:41:32 GMT

Can't see anything wrong with that code. I was interested in your how your transactions overlapped but you havn't listed the code of uploadFilesToFolder so I can't tell.

Looking at your last method listed If you are using lucene (again guessing due to lack of detail) then the indexing and metadata extraction is in-transaction so 3 per second may be reasonable. For comparison my old laptop used to do 2 document ingestions per second with lucene.

It may also be worth you looking at the bulk file import tool since it does this sort of stuff already.

Re: why concurrent user upload files by cmr service is very slow

nocturn9 — Thu, 05 Dec 2013 11:01:03 GMT

thks your reply.
i remark these code:

// checkPendingDelete(parentRef);
// ParameterCheck.mandatory("parentRef", parentRef);
// ParameterCheck.mandatory("assocTypeQName", assocTypeQName);
// ParameterCheck.mandatory("assocQName", assocQName);
// ParameterCheck.mandatory("nodeTypeQName", nodeTypeQName);
// invokeBeforeCreateNode(parentRef, assocTypeQName, assocQName, nodeTypeQName);
// nodeIndexer.indexCreateNode(childAssocRef);

and it be fast and i can access alfresco by web-client and share application.

and also,i have another performance problem:

i query files by lucene and sort by cm:created field,it fast.then i query same files and sort by my custom field,it's also a datatime field,same config as cm:created,it be slow.

i invoke by searchService in cmr package.

Re: why concurrent user upload files by cmr service is very slow

nocturn9 — Wed, 11 Dec 2013 01:44:38 GMT

up,and also,i have another performance problem:

i query files by lucene and sort by cm:created field,it fast.then i query same files and sort by my custom field,it's also a datatime field,same config as cm:created,it be slow.

i invoke by searchService in cmr package.