DragonFlyBSD Project Update - colo upgrade, future trends

[Matthew Dillon]

For the last week I've been testing out a replacement for Monster, our
48-core opteron server.  The project will be removing Monster from the colo
in a week or two and replacing it with three machines which together will
use half the power that Monster did alone.

The goal is to clear out a little power budget in the colo and to really
beef-up our package-building capabilities to reduce the turn-around time
needed to test ports syncs and updates to the binary package system.
Currently we use two blades to do most of the building, plus monster
sometimes.  The blades take almost a week (120 hours+) to do a full synth
run and monster takes around 27.5 hours.  But we need to do three bulk
builds more or less at the same time... one for the release branch, one for
the development branch, and one for staging updates.  It just takes too
long and its been gnawing at me for a little while.

Well, Zen 2 to the rescue!  These new CPUs can take ECC, there's actually
an IPMI mobo available, and they are fast as hell and cheap for what we get.

The new machines will be two 3900X based servers, plus a dual-xeon system
that I already had at home.   The 3900X's can each do a full synth run in
24.5 hours and the Xeon can do it in around 31 hours.  Monster will be
retired.  And the crazy thing about this?  Monster burns 1000W going full
bore.  Each of the 3900X servers burns 160W and the Xeon burns 200W.  In
otherwords, we are replacing 1000W with only 520W and getting roughly 6x
the performance efficiency in the upgrade.  This tell you just how much
more power-efficient machines have become in the last 9 years or so.

This upgrade will allow us to do full builds for both release and dev in
roughly one day instead of seven days, and do it without interfering with
staging work that might be happening at the same time.

--

Future trends - DragonFlyBSD has reached a bit of a cross-roads.  With most
of the SMP work now essentially complete across the entire system the main
project focus is now on supplying reliable binary ports for release and
developer branches, DRM  (GPU) support and other UI elements to keep
DragonFlyBSD relevant on workstations, and continuing Filesystem work on
HAMMER2 to get multi-device and clustering going.

John Marino (marino) pioneered the 'synth' system for dports and continues
to help out, but his focus is on RavenPorts for now.  Rimvydas Jasinskas
(zrj) and Antonio Huete Jimenez (tuxillo) have done a huge amount of work
bringing dports back into operational form and getting the FreeBSD ports
sync stuff working better.    DragonFlyBSD is still using synth, and will
probably remain on synth for the foreseeable future, though there is always
some discussion about how best to move dports forwards.  It's an excellent
build platform for us.

Francois Tigeot (ftigeot) has done a ton of work taking DRM up to
Linux-4.7.10 and this has worked very well for Intel iGPUs.  We are now
finally starting to dive into Linux's 'amdgpu' subsystem which is much
older, in order to modernize our AMD support (which is still deficient).
 Numerous other people have spent a considerable amount of time helping
test GPU support and tracking down bugs.  The work is ongoing.

I apologize for only writing everyone's names in plain ascii :-)

Right now HAMMER2 makes for an excellent single-device filesystem
(extremely well given that it supports writable snapshots and compression
out of the box), but it remains deficient when it comes to expandable
storage, multiple devices, and clustering.  This will be active work for me
but honestly the amdgpu support has to come first so it's still going to be
a long-haul for HAMMER2.

The mailing lists are not seeing much if any activity any more.  This is
more a generational issue... people kinda prefer web-based forums these
days and younger generations do not use mailing lists at all for group
stuff (not really).  Even the devs almost universally use IRC and not
mailing lists for discussions now (its kinda too bad that we don't have a
permanent irc log stored on DFly servers for posterity).  So we are looking
into potentially shifting user interaction to a web-based forum, perhaps
this year, and retiring the mailing lists, leaving just an archive for the
mailing list.  Possibly sometime this year, so look for action on that
upcoming.

Our other main developers: Sascha Wildner (swildner), who keeps the tree in
tip-top shape and catches numerous issues, Justin Sherrill (JustinS) who
handles our distribution rolls and the DragonFly and general BSD blog on
our home page, Sepherosa Ziehau (sephe) who focuses on the network
subsystem, Tomohiro Kusumi who helps with HAMMER and HAMMER2, Peter Avalos,
and more.  I will list more down below.

--

It's hard to say how the future will develop.  There are only three
open-source operating systems in the entire world that really pull it
together on having a complete, modern, SMP kernel:  Linux, DragonFlyBSD,
and FreeBSD.  And that's it.  We also have NetBSD and OpenBSD and I'd kinda
like to know what their plans are, because the future is clearly going not
only multi-core, but many-core.  For everything.  But as I like to say, for
SMP there are only three at the moment.  One can't dispute that Linux has
nearly all the eyeballs, and DragonFly has very few.  But OpenSource tends
to live on forever and algorithms never die... I think there is a place for
all of these projects and there really aren't any alternatives if you want
a sparkling clean system that doesn't have too many layers of abstraction.
At the current Juncture DragonFlyBSD is doing well and there are no plans
to slow down or stop.

There are many other developers who help out with DragonFlyBSD on a regular
basis, or drop in from time to time, as well as past developers who did an
awful lot of work.   For this I am going to run the names out of the git
log in alphabetical order, so I don't miss anyone (hopefully).  And to
'User' and 'Charlie Root'... we will never know who you were, but the party
is still going!

-Matt

Aaron LI
Adam Hoka
Adam Sakareassen
Adrian Chadd
Aggelos Economopoulos
Alex Hornung
Alexander Kuleshov
Alexander Polakov
Alexandre Perrin
Antonio Huete
Antonio Huete Jimenez
Antonio Nikishaev
Aycan iRiCAN
Ben Woolley
Bill Yuan
Brad Hoffman
Brills Peng
Charlie Root
Chris Pressey
Chris Turner
Chris Turner
Chris Wilson
Christian Groessler
Constantine A. Murenin
Daniel Bilik
Dave Hayes
David P. Reese
David Rhodus
David Shao
David Xu
Diederik de Groot
Dimitris Papastamos
Dylan Reinhold
Ed Schouten
Eirik Nygaard
Eitan Adler
Francis GUDIN
Franco Fichtner
Fran\xc3\xa7ois Tigeot
Gregory Neil Shapiro
Gwenio
Hasso Tepper
Hidetoshi Shimokawa
Hiroki Sato
Hiten Pandya
Ilya Dryomov
Imre Vadasz
Imre Vad\xc3\xa1sz
Imre Vad\xc3\xa1sz
Jan Lentfer
Jan Sucan
Javier Alc\xc3\xa1zar
Jean-S\xc3\xa9bastien P\xc3\xa9dron
Jeffrey Hsu
Jeremy C. Reed
Jeroen Ruigrok/asmodai
Joe Talbott
Joerg Sonnenberger
Johannes Hofmann
John Marino
Jordan Gordeev
Joris Giovannangeli
Justin C. Sherrill
Levente Kurusa
Liam J. Foy
Lubos Boucek
Magliano Andrea
Markus Pfeiffer
Matt Dillon
Matteo Cypriani
Matthew Dillon
Matthias Rampke
Matthias Schmidt
Maurizio Lombardi
Max Herrgard
Max Herrg\xc3\xa5rd
Max Okumoto
Maxim Ag
Michael Neumann
Michael Neumann
Michael Neumann
Mihai Carabas
Nicolas Thery
Nicolas Thery
Nolan Lum
Noritoshi Demizu
Nuno Antunes
Nuno Antunes
Peeter
Peeter Must
Peter Avalos
Pierre-Alain TORET
Robert Garrett
Robin Hahling
Rui Paulo
Rumko
Samuel J. Greear
Sascha Wildner
Scott Ullrich
Sepherosa Ziehau
Simon 'corecode' Schubert
Simon Arlott
Simon Schubert
Simon Schubert
Stathis Kamperis
Sylvestre Gallon
Thomas E. Spanjaard
Thomas Nikolajsen
Tim
Tim Bisson
Tobias Heilig
Tomasz Konojacki
Tomohiro Kusumi
Ulrich Sp\xc3\xb6rlein
User
Venkatesh Srinivas
Victor Balada Diaz
Vishesh Yadav
Walter Sheets
YONETANI Tomokazu
Yellow Rabbit
Yonghong Yan
Zach Crownover
b86
dumbbell
glebius
hrs
jkim
minux
rnoland
sinetek
zrj
\xc3\x81kos Kov\xc3\xa1cs

-Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dragonflybsd.org/pipermail/users/attachments/20190722/780565e5/attachment-0002.htm>