Storing hundreds of millions of files in HAMMER (1 or 2)

[Mehmet Erol Sanliturk]

On Wed, Jul 15, 2015 at 6:30 PM, Sepherosa Ziehau <sepherosa at gmail.com>
wrote:

> Maybe just use large file, and sub-index the chunks of a large file
> and add open/read/write/lseek/close like APIs for users.  You will
> have more control than using database.
>
> On Wed, Jul 15, 2015 at 11:58 PM, Michael Neumann <mneumann at ntecs.de>
> wrote:
> > Hi,
> >
> > Lets say I want to store 100 million small files (each one about 1k in
> size)
> > in a HAMMER file system.
> > Files are only written once, then kept unmodified and accessed randomly
> > (older files will be access less often).
> > It is basically a simple file based key/value store, but accessible by
> > multiple processes.
> >
> > a) What is the overhead in size for HAMMER1? For HAMMER2 I expect each
> file
> > to take exactly 1k when the file
> > is below 512 bytes.
> >
> > b) Can I store all files in one huge directory? Or is it better to fan
> out
> > the files into several sub-directories?
> >
> > c) What other issues I should expect to run into? For sure I should
> enable
> > swapcache :)
> >
> > I probably should use a "real" database like LMDB, but I like the
> > versatility of files.
> >
> > Regards,
> >
> >   Michael
>
>
>
> --
> Tomorrow Will Never Die
>


In reality , an operating system is one of the "best" data base management
system .

Question by Michael Neumann is very important with respect to this fact .

I was thinking to use DragonFly BSD for such a task , but it seems that it
is not useful on that issue , because assumption about that files should be
large is not so much suitable for an operating system . Then XFS seems to
be a good alternative in Fedora Server edition .


Thank you very much .


Mehmet Erol Sanliturk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dragonflybsd.org/pipermail/users/attachments/20150715/0dc924a2/attachment-0003.htm>