Of course it runs ... pkgsrc - NASA's Pleiades Supercomputer
flew by on twitter, thanks to imil:
NASA uses pkgsrc on their Pleiades supercomputer.
the NASA page,
``Pleiades, one of the world's most powerful supercomputers, represents NASA's state-of-the-art technology for meeting the agency's supercomputing requirements, enabling NASA scientists and engineers to conduct modeling and simulation for NASA missions. This distributed-memory SGI ICE cluster is connected with InfiniBand in a dual-plane hypercube technology.
The system contains the following types of Intel Xeon processors: E5-2680v3 (Haswell), E5-2680v2 (Ivy Bridge), E5-2670 (Sandy Bridge), and X5670 (Westmere). Pleiades is named after the astronomical open star cluster of the same name. ''
The system architecture consists of 163 racks of machines
from SGI (still alive!) with Suse Linux as their operating system. Check
for all the juicy details!
New binary releases for NetBSD on Raspberry Pi, including 7.0 RC1
NetBSD runs on many machnes, and the Raspberry Pi is one of them.
Getting the stock distribution is not that easy, and to help
in getting things going, Jun Ebihara is providing
ready-made images for quite some time.
Ansible & EC2 - Playbooks for orchestrating NetBSD into the cloud
As follower of my blog you have
getting NetBSD instances started in Amazon's EC2 cloud
with a simple web application deployed on one EC2 instance
and the database on another one.
These blog articles were very detailed on purpose, to have full
logfiles available just in case needed. I have used these logs to
talk about Ansible and Amazon's EC2, so things can be looked at
without actually running anything. As it turns out this was good,
because the 32bit NetBSD instances that I've used during my
pkgsrcCon demonstration actually decided to do a kernel panic, and the presentation
was a bit more on the theoretical side than I originally planned.
Managing NetBSD with Ansible, First Steps (Updated) Background: Plan, Build, Run
I've mused a lot about the transition from
system planning over system setup to system operation recently.
Different kinds of databases may be involved, that serve
different purposes: On the planning end, topics like
version management, license management, life cycle management
and enterprise architecture management come into play.
When this transits into operation, other aspects like
system and network configuration get more important,
while others fade away.
At the end, someone or something has
to make adjustments to systems in order to get them
into service, and keep them there.
This is either done with a lot of manual labor
to keep documentation (CMDB, or whatever serves)
and the systems in sync, or - more likely - documentation
will grow stale quickly and rot away.
With a growing set of hosts, the latter is not an option,
and transparency becomes increasingly important.
This is where system "orchestration" tools like cfengine,
puppet, chef and ansible come into play.
You write a system definition there, and the machine is
configured according to this definition automatically:
Configuration files can be adjusted, accounts created,
software packages installed, services enabled, etc.
System Orchestration Alternatives
Looking at the state of affairs, many people seem to be
fond of Puppet these days. This comes with a pretty long list
of dependencies (and thus complexity to maintain), and
configuration files have to be written in Ruby (which I don't know).
So I chose Ansible as alternative to investigate - it comes with very little overhead
(no separate daemon on neither the central machine nor any
of the configured systems, just using SSH) and its own
"Playbook" language seems easy enough to start with, yet
complete enough to be used in large environments (as the
list of customers listed on the homepage shows).
Packaging Ansible, First Try
My first try to get going with Ansible on NetBSD was to use
the version 0.9 archive available, instead of using a GIT checkout.
The included (GNU) Makefile
tries to determine the version from the git checkout.
Which the archive is not. Using information not available in the archive either.
Checking if the source code was checked out from GIT by looking
if there's a git binary available is not the best idea here,
and so I dropped the ball, cursing the fact that not all
the world is Linux. Sort of.
Looking briefly at a fresh git checkout that this worked there,
but as NetBSD's pkgsrc doesn't support git checkouts (yet, as far
as I know?), I moved back to the 0.9 archive.
Packaging Ansible, Second Try
Leaving out the git-based version games, the next fun was
to get the Python setup script to install the files in the right
place. Having missed much pkgsrc development recently, and not being into
Python either way, this proved unneccessary - Joerg Sonnenberg
pointed me at existing pkgsrc infrastructure for Python
programs, and with them the program was packaged successfully
in no time.
For now, the package is available
on my website,
I'll look into review and import into pkgsrc next.
Ansible on NetBSD - What Works, And What Needs More Work
Modules tested successfully on NetBSD 6:
user add/delete (without system=true)
Modules tested unsuccessfully on NetBSD 6:
user add/delete (with system=true)
service (needs work)
The git version (and the upcoming 1.0 release) also include
a "pkgin" module that can be used to manage packages.
Unfortunately the git-version of the module cannot easily be used
with the 0.9 version, so this has to wait.
FOSDEM talks/slides: pkgsrc on MirBSD, pkgin (Updated)
Last weekend the Free and Open Software Developer Meeting (FOSDEM)
happened. As in previous years, there was a booth manned by
people from various BSD-projects, and there were also
two(?) talks related to NetBSD and its related project pkgsrc:
NetBSD and MirBSD developer Benny Siegert gave a talk titled
"pkgsrc on MirBSD" -
see his slides! pkgsrc is a framework for packaging and building 3rd party applications from source. Besides MirBSD, it runs on many
other platforms like Linux and Mac OS X.
While building from source is fine, it costs a lot of time.
pkgsrc can also create binary packages, and to manage those,
there is "pkgin", a binary package manager.
Its developer, Emile 'iMil' Heitor
introduced it in a talk -
see the slides (PDF)!
Thomas 'wiz' Klausner pointed out that there was
actually a whole "BSD Devroom", and there were
many BSD-talks there, including:
MINIX3 and BSD, by Arun Thomas
The Lua Scripting Language in the NetBSD Kernel, by Marc Balmer
Touch your NetBSD - towards tablet integration, by Pierre Pronchery
Introduction to pkgsrc, and to package creation in NetBSD, by Noud de Brouwer
pkgin, a binary package manager for pkgsrc, also by Emile Heitor
Automated package building, by Nicolas Thauvin
Besides other talks that were less focused on NetBSD/pkgsrc,
this event shows that there's a pretty active group of
BSD advocates in Europe that manage to advocate "BSD"
in its entirety pretty well. Well done, guys!
NetBSD ketchup - news from my mailbox
Here's another bunch of NetBSD-related news that has
been lingering in my inbox for far too long:
restore CD is available based on NetBSD versions
for information on what it is and how to use it.
A negative symbol lookup cache was added
to NetBSD's loader
for shared libraries and shared objects, ld.so_elf, by
``I've been researching why Evolution from GNOME takes over 5 minutes to load on my quad core amd64 beast. It boils down to dlsym looking for a symbol that does not exist directly and as such examining every needed library. However, the current implementation does not remember what libraries it as already checked. Normally this isn't a problem, but with the way Evolution is built the search chain is massive.
With this patch, Evolution (without the patches to and a glib I added to pkgsrc a few days ago) loads in under 2 seconds (5 seconds with initial disk thrashing). ''
The NetBSD Logo
is available in many variants, but a new variant was submitted
via www@ these days by "Tim" - which is actually plain HTML,
SafeNet's ProtectDrive is
``a full disk encryption solution that encrypts the entire hard drive of laptops, workstations and servers, as well as USB flash drives, to protect data in the case of the theft or loss of a hardware device.''
How do you implement such preboot authentication and
harddisk encryption software,
esp. if you want to provide thinks like LDAP integration for
the user/key handling and two-factor authentication?
Little is known, but rumors say the 32bit version of the software
is based on NetBSD, as is backed by
this worker bio info:
``Duties: Working on pre-boot restricted environment with loads before operation system and implemented on NetBSD.
Ported and optimized the KDrive X server to NetBSD.
Developed and implemented user secure authentication interface with smart card support.
Environment and tools : NetBSD (3.0), C/C++, FLTK''
A german-language introduction of pkgsrc on OpenSolaris
was given by Michael 'kvedulv' Moll at the Munich
OpenSolaris User Group back in march.
``I have now fairly good (i.e., it works for me) support for the
MINI2440 on NetBSD with support for the following:
- S3C2440 UART
- DM9000 (MAC+PHY)
- S3C2440 SD Controller
- S3C2440 DMA Controller
- S3C2440 IIS Controller
- FriendlyArm 3,5" LCD Display
- S3C2440 USB Host Controller (OHCI)
- S3C2440 Touch Screen
- UDA1341TS audio codec
Currently, support for three things on the S3C2440 are missing:
- S3C2440 NAND Controller
- S3C2440 USB Device Controller
- S3C2440 RTC
I've also created a stage2 bootloader for use with u-boot, which
ensures that the value of bootargs is passed to the NetBSD kernel.
At this point I have only tested the code with the 64Mb version of the
More news from NetBSD land: inside and outside the kernel
My inbox is still overflowing with NetBSD related news,
so here is the next chunk for you:
People owning a Marvell PCMCIA WiFi card can how have
a look at
the malo(4) driver.
If you use this on a gumstix ARM board, have a look
as some fiddling with drvctl(8) is needed to get the
card recognized properly.
KIYOHARA Takashi has declared the
porting effort as finished.
Support for the baex, connex, verdex and verdex-pro
modules is available, as is support for the support modules.
Staying in the "embedded" corner, KIYOHARA Takashi has
that Plathome's OpenBlockS600 (AMCC 405EX) can now
boot NetBSD via NFS. See the posting for dmesg output.
comes with a AMCC 405EX PowerPC CPU,
two GigE ethernet ports and a bunch of other
goodies for a price of about $600US.
Getting back to the NetBSD code, a number of interesting changes
were made in the previous weeks. The first to mention is that
David Young has continued is work on
the new shutdown order
for device drivers:
``cgd, dk, dm, md, raid, and vnd gracefully detach from the device
tree during shutdown. I believe that ccd is the only virtual disk that
does not detach.''
This allows having arbitrary stack of file systems, and still
have them unconfigured properly in the right order on system
Moving from userland inside the kernel, David Holland has
proceeded with work to
unhook LFS from UFS.
Historically, the Log structured File System was written after
the Berkeley Fast File System. With the idea of sharing
the core "Unix File System" code for both file systems,
this resulted in a strong relationship between LFS and FFS,
which was/is not always the best for the advantage and
stability of either one:
``sharing ufs between both ffs and lfs has
made all three entities (but particularly lfs) gross. ffs and lfs are
not similar enough structurally for this sharing to really be a good
Another major addition to the NetBSD kernel was made recently
by Darren Hunt:
``Courtesy of CoyotePoint Systems, I've been working on a port of
DTrace [...] to NetBSD for i386.''
``DTrace is a comprehensive dynamic tracing framework created by Sun Microsystems for troubleshooting kernel and application problems on production systems in real time. Originally developed for Solaris, it has since been released under the free Common Development and Distribution License (CDDL) and has been ported to several other Unix-like systems.
DTrace can be used to get a global overview of a running system, such as the amount of memory, CPU time, filesystem and network resources used by the active processes. It can also provide much more fine-grained information, such as a log of the arguments with which a specific function is being called, or a list of the processes accessing a specific file. ''
So much about about the NetBSD code for now. Of course having
all those fine features added screams for an immediate (*cough*)
release, which brings me to the fact that
NetBSD 5.0.2 has been released:
`` NetBSD 5.0.2 is the second critical/security update of the NetBSD 5.0 release branch. It represents a selected subset of fixes deemed critical for security or stability reasons.
Please note that all fixes in critical/security updates (i.e., NetBSD 5.0.1, 5.0.2, etc.) are cumulative, so the latest update contains all such fixes since the corresponding minor release. These fixes will also appear in future minor releases (i.e., NetBSD 5.1, 5.2, etc.), together with other less-critical fixes and feature enhancements. ''
I'm closing for today by pointing to three NetBSD-related events:
going this weekend (Feb 20/21 2010)
Volunteers are wanted to setup & man a NetBSD booth at
FrOSCamp 2010 Zurich, Switzerland, on Sep 17/18 2010
pkgsrcCon 2010 will be held in Basel, Switzerland, from
May 28ths to 30ths 2010.
pkgsrc via git
Did you always feel like
exposing your soul to the dark side
but couldn't find a good reason for?
Maybe here's one for you, found via
The DragonFly BSD Digest:
Matthew Dillon hass etup a GIT pkgsrc repository
to give DragonFly users and developers a more reliable(?) way
to track the pkgsrc tree.
Right now this is just a copy from cvs every 15 minutes, so it won't
allow changes back to pkgsrc, but apparently it's much faster to download
via git than it is via cvs.
For a quick start, run "git clone git://avalon.dragonflybsd.org/pkgsrc.git".
Matt Dillon's mail
for a bit more information.
Catching up - various items (and not source-changes, this time)
Many things have happened in NetBSD-land in the past few weeks,
and as I've been slacking^Wbusy again, here's just a digest
of things that I haven't seen mentioned
elsewhere so far, in
BSD-related radio-show "bsdtalk" has published an interview
with NetBSD's Andrew Doran in its
March 2009 issue. Besides covering Andrews work,
the upcoming NetBSD 5.0 release is also discussed.
Cross-compiling pkgsrc packages is a long-standing dream, and it's
yet waiting for someone to do it. For the time being, Jared McNeill
has come up with an
HowTo on how to build 32bit packages on amd64
(and probably other 64bit systems).
Jared McNeill's been hacking on more stuff recently, and one thing
includes changes to the framebuffer console support on x86 (i.e.
both i386 and amd64). In short,
the recent changes
are just a stop on the way to move the splashscreen code and
esp. image data from the kernel to userland. I.e. that you can put
into your /boot.conf in the future. But we'll see a separate
announcement when that part is done. Let's stay tuned! :)
Martti Kumparinen has tackled generating a UFS file system on
a "large" (~5.5TB) disk. As the process is not straight forward,
posted a howto that may help in the future.
Any takers for adding comments and integrating this into
The NetBSD Guide? :)
Manpages are a major component of every Unix system. If you have ever
tried to write such a manpage, you 'll have learned that they are
in a funny text-based format similar to LaTeX and HTML, with its
own processor - *roff. There are several *roff implementations, and
the one used in NetBSD currently is the GNU implementation. To provide
an alternative here is good for both removing GPL'd code from the
NetBSD codebase, and also because groff is written in C++, which
is slow to compile, and - well - requires a C++ compiler.
A change for that situation may arise eventually, as Kristaps
Dzonsons has been working on a groff replacement to format
Unix manpages recently. See
his homepage for further information.
I've talked about Xen support for PCI passthrough
and Manuel Bouyer has finished his work to get full support
for passing in access to specific PCI devices from the Xen
Dom0 to DomUs. See
his posting to port-xen
for more details!
The NetBSD operating system supports many different hardware
and CPU platforms. For a specific platform, binaries are
compiled with a specific compiler, and there is a set of
binaries for each platform. This results in a rather big number
of different sets of binaries - currently about 50.
A different approach
with historic precedence
is to have one binary work on may hardware platforms,
so-called "fat" binaries.
Qt is a user-interface library found in widespread use in the
Unix/Linux world. It's not exactly small, and its prerequirement
of the X Window System doesn't it make a #1 choice for embedded
systems at the first look. A Qt variant - Qt/Embedded - can be
ran without X, though, and which thus avoids all the configuration
and hardware support trouble of X in one go.
On NetBSD, Qt/Embedded could talk to the wscons driver directly,
and Valeriy 'uwe' Ushakov has posted about his work on
patches to adopt Qt/Embedded to wscons.
Who's first to post some screenshots?