topic nuxeo storage for big amount of data, on multiple disk ? in Nuxeo Forum

nuxeo storage for big amount of data, on multiple disk ?

Quentin_ — Fri, 06 Jan 2012 17:58:15 GMT

Hi,

I'd like to store million of documents, so they can't fit all on a single disk.

I read in the documentation that there is a "cluster" part in nuxeo, but it looks like only the DB is clusterised ?

Can nuxeo shard my documents accross my clusters, automatically (if a disk is full, it will go to the other cluster member etc.) ?

Thanks

Re: nuxeo storage for big amount of data, on multiple disk ?

Florent_Guillau — Mon, 09 Jan 2012 11:04:39 GMT

Millions of documents can easily fit on a single disk, provided the disk is big enough. Which means that the disk is not an actual single hard drive but a storage solution like a NAS or a SAN.

You shouldn't as Nuxeo to deal with disk management, with is a pure storage issue.

Nevertheless if you really want to do it you could go have a look inside the nxserver/data/binaries/data/ directory and make the 256 first-level directories (most of which you'll have to create if your database isn't yet much filled) into symbolic links to directories mounted on other disks.

Re: nuxeo storage for big amount of data, on multiple disk ?

Quentin_ — Mon, 09 Jan 2012 11:14:44 GMT

Ok, I should have say Po of documents, but nevermind I've got my answer, it's not up to Nuxeo to solve this storage problem.

Re: nuxeo storage for big amount of data, on multiple disk ?

Florent_Guillau — Mon, 09 Jan 2012 11:58:43 GMT

Another answer is that the file-storage layer in Nuxeo is pluggable, so if you want you can implement a new one that does what you need. For instance we have a backend that stores in S3 (Amazon S3 Online Storage). You just have to implement BinaryManager and a few related classes.