Storing hundreds of millions of files in HAMMER (1 or 2)
sepherosa at gmail.com
Wed Jul 15 18:30:07 PDT 2015
Maybe just use large file, and sub-index the chunks of a large file
and add open/read/write/lseek/close like APIs for users. You will
have more control than using database.
On Wed, Jul 15, 2015 at 11:58 PM, Michael Neumann <mneumann at ntecs.de> wrote:
> Lets say I want to store 100 million small files (each one about 1k in size)
> in a HAMMER file system.
> Files are only written once, then kept unmodified and accessed randomly
> (older files will be access less often).
> It is basically a simple file based key/value store, but accessible by
> multiple processes.
> a) What is the overhead in size for HAMMER1? For HAMMER2 I expect each file
> to take exactly 1k when the file
> is below 512 bytes.
> b) Can I store all files in one huge directory? Or is it better to fan out
> the files into several sub-directories?
> c) What other issues I should expect to run into? For sure I should enable
> swapcache :)
> I probably should use a "real" database like LMDB, but I like the
> versatility of files.
Tomorrow Will Never Die
More information about the Users