why concurrent user upload files by cmr service is very slow

[nocturn9]

Hi,

I use multi-thread(10 thrads) to invoke alfresco cmr nodeservice to upload file into alfresco.
But it is very slow,only 3 files(1M per file) per second.
everybody can help me,give me some suggestion to optimization it?

thanks.
cj.tin

[mrogers]

I don't think an answer is possible given the lack of detail in your question.   You will need to do some investigation yourself to narrow down on the problem first.

At minimum, please post your code and details of your system including the configuration.

[nocturn9]

Hey,thanks your reply.

My code as bellow,

   private  void start(String[] args) {
        try {
            preFolderName="D";
            sumFiles=0;
            sumFileSize=0;

            int start=1;
            int end=100;
            folderSize=2;

            if (args.length>0){
                start=Integer.parseInt(args[0]);
            }
            if (args.length>1){
                end = Integer.parseInt(args[1]);
            }

            if (args.length>2){
                folderSize = Integer.parseInt(args[2]);
            }

            if (args.length >3){
                fileNo=Integer.parseInt(args[3]);
            }

            writeFileContent=false;
            if (args.length >4){
                writeFileContent=true;
            }

            tmpFolder=new File("c:/tmp");

            NodeService nodeService= EcmService.me.getNodeService();


            NodeRef root = nodeService.getRootNode(StoreRef.STORE_REF_WORKSPACE_SPACESSTORE);
            ChildAssociationRef child = nodeService.getChildAssocs(root, ContentModel.ASSOC_CHILDREN,
                    QName.createQName(NamespaceService.APP_MODEL_1_0_URI, "company_home")).get(0);

            NodeRef companyHome=child.getChildRef();

            File[] tmpFiles = tmpFolder.listFiles();
            files=new FileContent[tmpFiles.length];
            int k=0;
            for (File fi : tmpFiles){
                FileContent fc=new FileContent();
                fc.readFrom(fi);
                files[k++]=fc;
            }

            List<Thread> threads=new ArrayList<Thread>(end-start+1);
            sumFileSize=0;
            long  beginTime=System.currentTimeMillis();

            for  (int i=start;i<=end;i++){
                final int threadNo = i;
                final int finalEnd = end;
                final Folder folder=new Folder(companyHome,preFolderName + i);

                NodeRef folderNode=  GetFolderCommand.getOrCreateFolder(folder, true);
                folder.setNodeRef(folderNode);

                Thread thread  = new Thread(new Runnable() {
                        @Override
                        public void run() {
                            System.out.println(String.format("upload thread %d/%d start…", threadNo, finalEnd));
                            try {
                                AuthenticationUtil.runAsSystem(new AuthenticationUtil.RunAsWork<Object>() {
                                    @Override
                                    public Object doWork() throws Exception {
                                        uploadFilesToFolder(threadNo, folder);
                                        return null;
                                    }
                                });

                            } catch (Exception e) {
                                e.printStackTrace();
                            }
                        }
                    });


                threads.add(thread);
                thread.start();
            }

            for (Thread thread: threads){
                thread.join();
            }

            long endTime= System.currentTimeMillis();
            double  span = (endTime-beginTime)/1000.0;
            double sumSizeMB=sumFileSize / 1024 /1024.0;

            StringBuilder msg=new StringBuilder();
            msg.append("upload totalFiles=").append(sumFiles).append(",\t totalSize=");
            msg.append(sumSizeMB).append("MB");
            msg.append(",\tspent seconds=").append(span).append(",");
            System.out.println(msg);
            msg=new StringBuilder();

            msg.append(String.format("%1$.4f  MB/s , \t", sumSizeMB / span));
            msg.append(String.format("%1$.2f  Files/s , \t",sumFiles/span));

            System.out.println(msg);


        } catch (Exception e) {
            e.printStackTrace();
        }
    }
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

And i remark these invoke function in DbNodeServiceImpl.java,and the speed can be 150 files per second.

     // checkPendingDelete(parentRef);
     // ParameterCheck.mandatory("parentRef", parentRef);
     // ParameterCheck.mandatory("assocTypeQName", assocTypeQName);
     // ParameterCheck.mandatory("assocQName", assocQName);
     // ParameterCheck.mandatory("nodeTypeQName", nodeTypeQName);
     // invokeBeforeCreateNode(parentRef, assocTypeQName, assocQName, nodeTypeQName);
     // nodeIndexer.indexCreateNode(childAssocRef);

Why ?

[mrogers]

Can't see anything wrong with that code.   I was interested in your how your transactions overlapped but you havn't listed the code of uploadFilesToFolder so I can't tell.

Looking at your last method listed If you are using lucene (again guessing due to lack of detail) then the indexing and metadata extraction is in-transaction so 3 per second may be reasonable.   For comparison my old laptop used to do 2 document ingestions per second with lucene.

It may also be worth you looking at the bulk file import tool since it does this sort of stuff already.

[nocturn9]

thks your reply.
i remark these code:

// checkPendingDelete(parentRef);
// ParameterCheck.mandatory("parentRef", parentRef);
// ParameterCheck.mandatory("assocTypeQName", assocTypeQName);
// ParameterCheck.mandatory("assocQName", assocQName);
// ParameterCheck.mandatory("nodeTypeQName", nodeTypeQName);
// invokeBeforeCreateNode(parentRef, assocTypeQName, assocQName, nodeTypeQName);
// nodeIndexer.indexCreateNode(childAssocRef);

and it be fast and i can access alfresco by web-client and share application.

and also,i have another performance problem:

i query files by lucene and sort by cm:created field,it fast.then i query same files and sort by my custom field,it's also a datatime field,same config as cm:created,it be slow.

i invoke by searchService in cmr package.

[nocturn9]

up,and also,i have another performance problem:

i query files by lucene and sort by cm:created field,it fast.then i query same files and sort by my custom field,it's also a datatime field,same config as cm:created,it be slow.

i invoke by searchService in cmr package.