[GSoC] HAMMER2 block compression feature

Thu Apr 25 14:28:50 PDT 2013

Thank you for your answers.

About git: I don’t have experience using it, but I have used SVN while
working on some team projects. I’m not sure how similar they are, but in
any case I’ll try to learn how to use it as soon as possible.

About HAMMER2: I was looking at the code located in github [1]. I can’t say
that I understand how it works in-depth, but I think I understand the
basics and it’s only a question of time for me to get fully familiarized
with it.

I think that the code related to compression and encryption functions would
be located in separate files, like hammer2_compression.c and
hammer2_encryption.c, so there won’t be many changes in other code. I
assume that one change would be that before writing something it would
check attributes of a directory and if compression/encryption is enabled,
it would pass the data to compression/encryption functions. In case of
compression function, it would probably return the resulting size of new
block along with the already compressed data. So, the main code would write
it then just like it does now, perhaps taking into account the new size of
block.

Hopefully, this is a plausible approach… For now, I don’t have any specific
questions, but probably I’ll have them in the future if I’ll get to more
specific details of implementation.

One thing I would like to ask is if it’s OK to send several proposals
related to HAMMER2? The reason I’m asking this is that, generally, I’m
equally interested in both compression feature and encryption feature, even
though compression feature would be my first choice. But I would gladly do
any of those.

I am confident that I can succeed with encryption feature too because
overall it is a similar proposal and I have completed a course on
cryptography, so it’s not unfamiliar to me. I would choose an encryption
algorithm and would implement it using a prototype approach first and then
gradually integrating it in HAMMER2 code. In fact, I assume that
AES128-encryption would be OK and I’m relatively familiar with that
algorithm.

I plan to send an official proposal regarding compression feature through
Google Summer of Code page by this weekend and, if it’s OK, another
proposal regarding encryption feature by the same time or a bit later.

Thank you.

[1] https://github.com/DragonFlyBSD/DragonFlyBSD/tree/master/sys/vfs/hammer2

Daniel

On Thu, Apr 25, 2013 at 4:34 AM, Justin Sherrill
<justin at shiningsilence.com>wrote:

> Daniel - One additional item that may be useful is learning something
> about git, the revision control tool DragonFly (and many other
> organizations) use.  You may already have encountered it or something
> similar, but I don't see specific mention in your email.  It will be
> the preferred way to store your code and to make it accessible for
> others.
>
> I can't tell you if there's a strong candidate for block compression -
> the application period just started, so there's nothing yet.  Rather
> than second-guess what's available, apply with a project proposal that
> has what you most want to do.  That will do you the most good.
>
> On Tue, Apr 23, 2013 at 3:14 PM, Daniel Flores <daniel5555 at gmail.com>
> wrote:
> > Hello.
> >
> > My name is Daniel Flores and I’m a student from Spain. I study in
> Barcelona
> > School of Informatics, which is a faculty of Polytechnic University of
> > Catalonia, and it’s my last year before I graduate.
> >
> > I sent a mail with this proposal to Matthew 4 days ago, but I didn't get
> a
> > reply yet, so I decided to send this to the mailing list, as it is
> suggested
> > on Dragonfly's GSoC page. Before submitting any official application, I
> > would like to ask for your opinion and criticism regarding my proposal
> and
> > whether I have the required skill and adequate planning to successfully
> > finish this project.
> >
> > I’m interested in implementing the block compression feature in HAMMER2,
> > which is described on Dragonfly’s GSoC page [1]. The feature consists in
> > that any written block should be compressed, if possible, to 50% of its
> size
> > or less (in powers of 2 – 32 KB, 16 KB and so on, the initial size of
> block
> > being 64 KB). It should be possible to set that compression mode on any
> > directory, so that new contents written in it and its subdirectories
> would
> > be compressed.
> >
> > I believe that I can succeed because I have programming experience in
> C/C++,
> > since we use those languages for most practices in our university. I also
> > completed courses centered on algorithms and their implementation in C or
> > C++. I also have an interest in operating systems, their concepts and
> their
> > development. I completed a course on Operating Systems development where
> I
> > and a partner implemented a simple linux-based kernel called “ZeOS”
> (using C
> > and, at times, x86 assembler). It’s a very simple kernel that has a task
> > scheduler, system calls, exceptions, interruptions, memory management
> and a
> > file system. We were able to successfully run it on bochs emulator and it
> > passed all tests needed to successfully finish the course.
> >
> > If you wish, you may take a look at it [2].
> >
> > Sadly, it’s documented in Spanish, but I just want to provide it as an
> > example of code that I can produce.
> >
> > In spite of it being a kernel implementation, I think that my skills and
> > knowledge aren’t yet the best suitable for kernel development projects,
> so
> > instead I prefer to do something moderately difficult, yet related to
> > operating systems, like HAMMER2 block compression feature.
> >
> > My initial outline for project planning is the following:
> >
> > 1. (Starting May 27 or before – June 17) During community bonding period
> I
> > would get myself familiarized with the community, Dragonfly BSD
> system/code
> > and “hammer2” branch code. I would also refresh my knowledge on
> compression
> > algorithms and would complete a design for the future implementation,
> like
> > choosing the appropriate algorithm and having defined everything needed
> to
> > be done. I have read Matthew's design document for HAMMER2 [3] and, as I
> > understand, the algorithm #2 is not yet determined, so I believe this
> should
> > be decided before the actual coding period will start. Also, more
> detailed
> > planning for coding period would be done, with clear milestones and
> > deliverables.
> >
> > The deliverables for this phase would be in the form of documents that
> would
> > describe the design, the chosen algorithm(s) and the calendar with
> > milestones.
> >
> > 2. (June 17 – July 29) I would start by coding a prototype application
> that
> > would implement the defined design, consisting of algorithms #0, #1 and
> #2.
> > I would need it to insure that the algorithm is implemented correctly
> and to
> > do all necessary debugging and refining before integrating it in
> HAMMER2. I
> > would create a series of tests to prove that the algorithm works well. If
> > time allows, I would also try to study its behavior statistically and to
> > improve it, if possible.
> >
> > The deliverables would be code of the prototype as well as any supporting
> > applications (to generate tests, for example) and tests themselves.
> >
> > 3. (July 29 – September 16) After confirming that the prototype works
> well,
> > I would start actually integrating it into HAMMER2 filesystem. At this
> > point, the task would be gradually adding prototype’s code without any
> > conceptual modification into HAMMER2 and testing if it works correctly in
> > real-life. A series of testing applications with appropriate system calls
> > would be created and the results checked for correctness. Also, the
> > appropriate hammer2 utility command to set the compression feature to
> > directories would be created and tested.
> >
> > The deliverables would be the code of implemented feature and application
> > tests for it.
> >
> > 4. (September 16 – End of GSoC 2013) Hopefully, all known bugs would be
> > corrected at this point, so I would take the time to do final
> refinements to
> > documentation, to write a detailed description of what was implemented
> for
> > future developers and to finish everything needed to start submitting my
> > code to Google.
> >
> > Generally, my philosophy is to start “as soon as possible”, so if some
> phase
> > is done before the deadline, I would immediately go to the next phase. If
> > project goes smoothly and there is some time left, I would be happy to do
> > small refinements or bugs fixes on general system or HAMMER2.
> >
> > During the coding period (June 17 – September 16) I don’t have any
> > commitments (like courses or trips) at all, so I should be able to easily
> > dedicate 40 hours per week or more to this project. Prior to June 17 I
> have
> > my studies and some exams, but generally it should not prevent me from
> > getting familiarized with code and getting the design/algorithm decisions
> > done, because my workload is not very big.
> >
> > I don’t have any past experience with Open Source or generally big
> projects,
> > but I believe that I can accomplish this task since it is generally
> related
> > to C programming and algorithms, and I believe that I have sufficient
> > knowledge of those fields from my university courses.
> >
> > Currently I am getting myself familiarized with Dragonfly 3.2 (for i386)
> > running it on Oracle VM Virtual Box 4.2.12, but I’m able to install it on
> > physical machine, if necessary. I’m also trying to learn as much as
> possible
> > about BSD in general and particularly Dragonfly.
> >
> > If you think that my set of skills is more appropriate to some other
> project
> > related to HAMMER2 or Dragonfly, I would be grateful if you pointed at
> them.
> > I also would consider working on HAMMER2 block encryption feature, since
> it
> > is similar in requirements to my current proposal, if there is already a
> > strong candidate to implement the block compression feature.
> >
> > Thank you.
> >
> > [1] http://www.dragonflybsd.org/docs/developer/gsocprojectspage/
> > [2] https://dl.dropboxusercontent.com/u/56646280/zeos.zip
> > [3] http://leaf.dragonflybsd.org/mailarchive/users/2012-02/msg00020.html
> >
> >
> > Daniel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dragonflybsd.org/pipermail/kernel/attachments/20130425/d26298d5/attachment-0003.htm>