topic Re: Speed up import in Nuxeo Forum

Speed up import

abc_xyz — Fri, 16 Mar 2012 14:47:34 GMT

Based on the nuxeo-platform-importer package we have developed our own importer tool. Up to one million documents the average import speed is about 250 docs/sec. Importing further documents takes more and more time and the average speed goes down to 50 docs/sec and less.
/>
/>We followed all performance-relevant instructions for postgres DB described here. My question is if anyone knows further measures to speed up import since we had to import several millions documents.

Re: Speed up import

ben_ — Fri, 16 Mar 2012 15:22:30 GMT

Yes importing few millions of documents is a question of days.

A much faster way is to generate the ad'hoc SQL dump and to populate the database with the PostgreSQL copy instruction. This is possible if the data layout to import is simple.

ben

Re: Speed up import

bruce_Grant — Fri, 16 Mar 2012 22:14:31 GMT

The big thing I found to help with mass import tuning was batch size - that is number of documents created before a commit (save) is performed. Too small and the overhead is large (per transaction). Too big and Postgres complains (not to mention a hickup runs the risk of losing all the documents in the commit).