|HF:|| How many people were there on the release engineering team, and
what were their duties?
|TV:|| The 1.5 branch release engineering team had nine members on it,
and still prospectively retains eight of them for the 1.5.1
release, depending on who volunteers the time to do the job.
That is, for the people who volunteer selflessly their available
Personally, I wasn't even expecting to be on the release engineering team. I was recruited for the task by Wasabi Systems (http://www.wasabisystems.com/), who devoted several weeks of my work time to the task of pushing NetBSD 1.5 out the door. That significantly helped wrapping up the release, and made me take notice that there is real industry interest, not just volunteer interest, in a product such as NetBSD.
|HF:|| What were the major jobs during the release engineering cycle?
|TV:|| The most time-consuming job was certainly pullups. CVS branch
maintenance is not a pretty sight - particularly when the
maintainer must also provide audit trails for large changes. We
have scripts that partially automate this, but they certainly
aren't perfect, and need input data from the pullup requests in
some very specific formats. This is a place where better tools
will definitely find a place in NetBSD developer use in the near
I also made sure that, while I was executing pullup requests, I personally read diffs of the merged code (and compiled it, where feasible to my available host systems) before committing to the branch. I don't necessarily understand all the code innards intimately, but extra eyeballs and compilers help to catch the occasional mechanical merge error.
Besides pullups, there was the coordination of the build team for all the architectures, but I'll get to that in a moment.
Other tasks included arranging the files on the FTP server using a new layout that helps us reduce distribution space for architectures that share the same base CPU and executable format; preparing the release announcement (which Charles Hannum revised for final release); and preparing the first officially supported, downloadable NetBSD CD-ROM images, which I'll also mention later.
|HF:|| The release was delayed a bit, what were the reasons for this?
|TV:|| The concept of "delayed" in proportion to the 1.5 release is
probably a bit skewed, and I'm partly to blame for saying it was
delayed for a "long time". NetBSD 1.5 has so many new features
that even I can't recount more than one-sixth of them without
resorting to the release announcement and change summary, and
it's still reasonably close to its target release (end of
September) compared to some prior NetBSD releases.
NetBSD's biggest release impediment is also its most important feature: we released 1.5 on 10 base CPU types comprising 20 groups of hardware architectures ("ports"). Taking in to account I/O buses, MMUs, and system controllers that another OS would consider to be different platforms, the actual number of specific hardware platforms is somewhere closer to 40. That's no small feat, and next to CP/M (which used the same CPU type throughout), NetBSD runs on more hardware platforms than any other full-featured OS in history.
Part of the slowdown was that most of the people working on release engineering were doing the work on personal time, making the available time for the task unpredictable. This will probably always be a small problem, but we have some amazingly devoted volunteers, and now some background business support as well.
Toward the end of the release cycle, we decided to delay a few days here and there to pull in and test some very important security fixes to the IPv6 network tunnel interfaces, BIND (which was updated before release to 8.2.2-P7), and tools throughout the tree in a compiler-aided printf() formatting string sweep. All though the release cycle, we did bring in many minor yet important functionality improvements for dozens of tools from the development sources into the 1.5 release.
As an unexpected benefit from the longer release cycle, though, NetBSD 1.5 has had a longer testing cycle than prior major releases.
|HF:|| So how did you know when it was the "right" time to do the
release at the time it happened, and not delayed for another
|TV:|| The NetBSD developer base is quite diverse, and opinions about
the "right" time to release have been just as different from one
developer to another. I had my own personal crisis when I was
recruited to help out - I'm a bit of a perfectionist when
developing my own software, and I was placed in the contrasting
position of keeping the NetBSD 1.5 release cycle as close to
schedule as feasible.
We went through several "beta release" cycles, each one shorter than the prior one, and released after a beta cycle that involved nearly no machine independent pullups. We could have gone through more beta cycles ad infinitum, but the time came to say, officially, that the system was up to the task of production use, and any further changes would wait until 1.5.1. We're still finding and fixing more bugs all the time, of course.
This is a learning exercise, and we're working on improving the process. I have a feeling the release cycle will keep more to a schedule in 1.6, and future patch releases such as 1.5.1 will take quite a bit less time to accomplish.
|HF:|| With the many different machines supported by NetBSD, how was the
binary distribution available on the FTP server created? Did you
have all machines available and do all the builds yourself?
|TV:|| Though NetBSD can cross-build on a number of platforms now, we
don't have a solid procedure yet to use for building releases
with a cross compiler. So, even on VAXen, we currently build
natively on almost every platform we support. (The VAX example
is a bit tongue-in-cheek; our VAX builder, Matt Thomas, built the
whole system and uploaded it within a day. Most m680x0 platforms
took nearly three times as long.)
16 people did the actual building, with two handling more than one port's build. Though some port maintainers were too occupied to do the builds personally, I found more than enough volunteers to take up those jobs. The actual build time was only a few days from tag to completion, and all platforms were building simultaneously.
NetBSD 1.5 also increased in binary size a bit from 1.4 and prior releases. Feeping creaturism, or something like that. The typical base binary release - gzip compressed - is right around 70MB including X11 binaries, and that's not counting the third-party packages from NetBSD "pkgsrc".
|HF:|| Some ISO images were put up recently on the FTP site, after the
release was available for some time. You said you have some
special ideas to create them on the fly?
|TV:|| Before I became directly involved in NetBSD 1.5 release
engineering, Wasabi Systems also wanted to put together a CD-ROM
distribution. Also, the demand for CD-ROM images from the
FTP-downloading public has been extremely high lately, thanks to
ever-increasing available bandwidth for client users.
In hopes that I could address both these issues, I put together a flexible CD-ROM image building tool that could take any combination of architectures and put them together with a minimum of duplicated code and a maximum of compatibility with host OS's such as Windows, MacOS, and AmigaOS. It also makes the CDs bootable on up to 7 architectures (thus far). NetBSD source users can find it in NetBSD-current under src/distrib/cdrom, and this tool will be more fully documented on the NetBSD website later this week.
There's more to the issue of downloadable ISOs, though. CD-ROM images, if downloadable, contain a huge amount of redundancy. Over 98% of NetBSD CD-ROM data is simply duplication of the gzipped distribution files available elsewhere on the FTP site - a huge waste of diskspace, especially if mirror sites want to pick up the images, and if we were to provide images for multiple releases or multiple combinations of platforms.
I have been working on a side project to find a way to reduce this redundancy. The result is a new tool, to be released at the end of this week at http://www.duh.org/expack/, which strips the redundant file data from the file content, reducing image size to no more than a few MB (and as little as 400KB on non-bootable architectures). At download time, ftpd runs a special filter program that interpolates the redundant data into a fully populated image, with checksum verification. This filter has little more system overhead than "cat". With this tool, FTP servers can provide CD-ROM images for download, while using up only 40MB of extra diskspace for all architectures.
|HF:|| The ISO images on ftp.netbsd.org contain one architecture per CD.
Will there be a CD (or CD-set) with all architectures?
|TV:|| Multiple-architecture CD-ROM images may be available unofficially
on the ftp.netbsd.org website as a trial run of the expack tool I
mentioned before. However, the easiest method for obtaining
NetBSD on CD-ROM, particularly for multiple architectures, is to
buy a prepressed distribution.
A list of all known CD vendors is available at http://www.netbsd.org/Sites/cdroms.html. One thing to note about NetBSD distributions is that - unlike Linux - all NetBSD distributions use the same source code tree. Some CD vendors build the sources themselves, but most use the same binaries that the NetBSD Foundation releases on ftp.netbsd.org.
|HF:|| If you choose to do release engineering for the next release
again, what will you do differently?
|TV:|| Well, I am staying on the 1.5 release engineering team to help
put out a 1.5.1 patch release, hopefully sometime in February.
The patch releases aren't aiming for the kind of functionality
differences between NetBSD 1.4 and 1.5; they mainly just fix
problems found after the release or fixed in the interim.
I noted before that NetBSD 1.5 runs on 10 CPUs on 20 ports. The NetBSD-current development sources are fast approaching 14 base CPU types and 40 ports. We're expecting about 30 of those ports to be part of the next major NetBSD release (tentatively numbered 1.6). That release won't be a simple one.
With that said, I may stick around to volunteer for 1.6 release engineering. Tedium aside, this has been a rather fun thing to do.
|HF:|| Do you have an idea what the NetBSD users can look forward to for
|TV:|| Probably the most noticeable feature will be some support for
symmetrical multiprocessing on i386 (and possibly other)
platforms. NetBSD has been a little behind in SMP development
from the other BSD's, but I have to mention again that because of
the number of platforms NetBSD supports, we try Very Hard to come
up with abstractions that work on more than just one platform.
SMP may not be entirely complete in 1.6, but there's quite a bit
in the NetBSD-current development tree as I write this; it looks
There's also active work on making the NetBSD base system more granular, using our existing packaging infrastructure used for "pkgsrc". It's likely that NetBSD 1.6 will be installable using a set of packages much like commercial Unix-like OS's, allowing admins to install as much or as little of the base system as desired.
Some platforms will be making the switch from the historic a.out binary format to the more well supported ELF format. And another project ("UBC") has been integrated that unifies the concepts of page cache and buffer cache, both increasing general system speed and correctly fixing the historic problem of consistency between mmap() and regular file access.
Of course, there will be more platforms as I mentioned earlier. They include, among others, Hitachi Super-H system-evaluation boards and the Sega Dreamcast, SGI workstations, more UltraSPARC systems, and a few older systems including the OMRON LUNA m68k's, original MIPS Corporation workstations, and the Acorn Archimedes. One developer also plans to bring back the BeBox port to an official release.
|HF:|| Todd, thanks a lot for your time. Is there any final word that
you'd like to tell to the DaemonNews readers and/or the NetBSD
community out there?
|TV:|| Sure. If you haven't tried NetBSD yet, you don't know what
you're missing. http://www.netbsd.org/