hubertf's NetBSD blog

Google Summer of Code 2010 NetBSD swcryptX Project Suggestion (Updated)

Mon, 22 Feb 2010 22:57:00 GMT

I've been thinking of a neat-o project for this year's Google Summer of Code:

Abstract:

Overview of operation

The opencrypto(9) framework exists to coordinate hardware acceleration in NetBSD. Applications of the framework can be inside the kernel like the FAST_IPSEC IPsec implementation, or in userland like OpenSSL with the "cryptodev" engine. Crypto drivers can be realized in software or in hardware. Hardware drivers can be used to instruct e.g. the AMD Geode LX's AES block or a HIFN chip to perform cryptographic operations. Upon system startup, the crypto drivers at the opencrypto(9) framework, telling what operations they can perform. When an operation is required later, the framework will look which crypto device is currently not busy, and offload the operation to that device. Upon completion, the result is fed back to the application.

The following image illustrates the components and their interaction.

Limitations

Offloading the cryptographic requests involves some overhead. Data needs to be transferred to the hardware and back. On systems with a slow CPU, this overhead is relatively small compared to the operation speed of the CPU. On faster CPUs, the overhead becomes more of a burden, making the benefit of the crypto hardware negligible.

As examples, while a hifn(4) chip can provide worthwhile speedups on 500MHz and 1GHz CPUs, no performance win is experienced on a 2.4GHz CPU.

Proposal

The communication overhead involves data transfers over a PCI bus, which is of relatively low speed compared to today's modern CPUs. Preventing the data transfer is a worthwhile goal. In coordination with today's modern multi-core CPUs, using one or more CPUs solely for the purpose of crypto acceleration, a measurable improvement of crypto performance is expected. At the same time, no special hardware requirements beyond the CPU exist. This allows turning standard contemporary systems into fast crypto systems easily.

The following image illustrates the idea of interoperation between a CPU core that runs the kernel and application codes and three cores that are dedicated to crypto code.

Implementation Roadmap

This is where it gets fishy. ;) ~~The existing opencrypto(4) framework probably needs to be make MP-aware at the same time, employing proper use of NetBSD's locking framework.~~ (Already done) The existing swcrypto(4) needs to be adjusted for operation on multiple CPUs at the same time. A way to decide how many CPUs are dedicated to run swcrypto(4) instances. CPUs that run swcrypto(4) need to be taken out from the usual NetBSD CPU scheduling so that they are available exclusively for crypto.

Requirements

In no particular order:

Know how to build and install a kernel
Understanding of fine grained SMP and locking
How to use NetBSD's kernel threads, code-wise
How to interact with NetBSD's scheduler, code-wise
Tell the scheduler to pin a specific kernel thread to a specific CPU
Interaction between applications (IPsec, OpenSSL) with opencrypto(9), code-wise
Interaction of crypto providers with opencrypto(9), code-wise
Hardware! You won't be able to do this without at least two CPU cores in your machine. The more the better.
Benchmarking & a test setup for it

Project Applications

Please follow the NetBSD Project Application/Proposal HowTo if you're serious to work on this project.

If you have any questions let me know, public discussion should be led on the tech-crypto@ list.

More news from NetBSD land: inside and outside the kernel

Sun, 21 Feb 2010 14:20:00 GMT

My inbox is still overflowing with NetBSD related news, so here is the next chunk for you:

People owning a Marvell PCMCIA WiFi card can how have a look at the malo(4) driver. If you use this on a gumstix ARM board, have a look as some fiddling with drvctl(8) is needed to get the card recognized properly.
Speaking about Gumstix, KIYOHARA Takashi has declared the porting effort as finished. Support for the baex, connex, verdex and verdex-pro modules is available, as is support for the support modules.
Staying in the "embedded" corner, KIYOHARA Takashi has announced that Plathome's OpenBlockS600 (AMCC 405EX) can now boot NetBSD via NFS. See the posting for dmesg output. The OpenBlockS600 comes with a AMCC 405EX PowerPC CPU, two GigE ethernet ports and a bunch of other goodies for a price of about $600US.
Coming from hardware to software, pkgsrc-2009Q4 was released some time ago, and of course binary packages are available for a number of platforms: 5.0/macppc, 4.0.1/sparc and 5.0.1/sparc, 4.0.1/i386, 5.0.1/i386 and the same for 4.0.1/amd64 5.0.1/amd64. Also, binaries of pkgsrc-2009Q3 are available for 5.0/shark.
Google's Summer of Code was a big success for NetBSD and all of the Open Source community, and it seems there will be one again this summer.
Preparations are in an early stage, but there are already a FAQ and a timeline as well as the Program Terms of Service. From the NetBSD side, we're always happy for project suggestions (please use our mailing lists for discussions), and in NetBSD, we are currently working on out projects page. If you plan to submit a proposal for a project with NetBSD, please see our project application/proposal form If you plan to submit a proposal for a project with NetBSD, please see our Project Application/Proposal HowTo.
Getting back to the NetBSD code, a number of interesting changes were made in the previous weeks. The first to mention is that David Young has continued is work on the new shutdown order for device drivers: ``cgd, dk, dm, md, raid, and vnd gracefully detach from the device tree during shutdown. I believe that ccd is the only virtual disk that does not detach.'' This allows having arbitrary stack of file systems, and still have them unconfigured properly in the right order on system shutdown.
Another major change that went into NetBSD recently is that terminfo was imported into NetBSD-current. Terminfo replaces termcap, but provides a backward compatible termcap interface. This move follows discussion from last summer, and docs by The Open Group (the people who make things like the POSIX standard and the Single Unix Specification), which indicates that the termcap specification will be withdrawn in the future.
Moving from userland inside the kernel, David Holland has proceeded with work to unhook LFS from UFS. Historically, the Log structured File System was written after the Berkeley Fast File System. With the idea of sharing the core "Unix File System" code for both file systems, this resulted in a strong relationship between LFS and FFS, which was/is not always the best for the advantage and stability of either one: ``sharing ufs between both ffs and lfs has made all three entities (but particularly lfs) gross. ffs and lfs are not similar enough structurally for this sharing to really be a good design.''
Another major addition to the NetBSD kernel was made recently by Darren Hunt: ``Courtesy of CoyotePoint Systems, I've been working on a port of DTrace [...] to NetBSD for i386.'' Citing Wikipedia, ``DTrace is a comprehensive dynamic tracing framework created by Sun Microsystems for troubleshooting kernel and application problems on production systems in real time. Originally developed for Solaris, it has since been released under the free Common Development and Distribution License (CDDL) and has been ported to several other Unix-like systems.
DTrace can be used to get a global overview of a running system, such as the amount of memory, CPU time, filesystem and network resources used by the active processes. It can also provide much more fine-grained information, such as a log of the arguments with which a specific function is being called, or a list of the processes accessing a specific file. ''
The code is available in NetBSD-current. I haven't looked into this yet, but I'm looking forward of reports and blog postings if the wikipedia command line examples work.
(I think like with ZFS, Dtrace could use a hand with documenting the NetBSD side of things. Any takers?)
The last kernel change to mention is related to security: mapping the address 0 from userland was disabled. This issue went through the press late last year, and it this is now addressed in NetBSD, too. Those that still beed to map address 0 can do so via the USER_VA0_DISABLED_DEFAULT kernel option or the vm.user_va0_disable sysctl.
So much about about the NetBSD code for now. Of course having all those fine features added screams for an immediate (*cough*) release, which brings me to the fact that NetBSD 5.0.2 has been released: `` NetBSD 5.0.2 is the second critical/security update of the NetBSD 5.0 release branch. It represents a selected subset of fixes deemed critical for security or stability reasons.
Please note that all fixes in critical/security updates (i.e., NetBSD 5.0.1, 5.0.2, etc.) are cumulative, so the latest update contains all such fixes since the corresponding minor release. These fixes will also appear in future minor releases (i.e., NetBSD 5.1, 5.2, etc.), together with other less-critical fixes and feature enhancements. ''

I'm closing for today by pointing to three NetBSD-related events:

There's a hackathon going this weekend (Feb 20/21 2010)
Volunteers are wanted to setup & man a NetBSD booth at FrOSCamp 2010 Zurich, Switzerland, on Sep 17/18 2010
pkgsrcCon 2010 will be held in Basel, Switzerland, from May 28ths to 30ths 2010.

Have fun meeting the gang!

Setting up a PPPoE server with NetBSD

Mon, 15 Feb 2010 21:56:00 GMT

I had an urge to configure PPPoE today, but didn't have a DSL line handy to test it. So, with some help from martin@, I got to prod NetBSD into becoming a PPPoE server.

I've put my notes here.

Open issues:

Whenever I terminate a client session ("ifconfig pppoe0 down"), the server's pppoe0 interface goes down as well. Before new connections are served, I have to do a manual "ifconfig pppoe0 up".
When I add "query-dns=3" to the client side, the connection is terminated immediately after it's established. I don't know how to offer DNS servers via PPPoE/LCP, but at least just leaving the connection open instead of terminating it would be nice.

I understand this method is only for one/few PPPoE connections. If more sessions are needed, this may not be the best option. It still served my purpose well. YMMV.

Musing about git's object store efficiency

Fri, 12 Feb 2010 17:26:00 GMT

I'm currently looking at git to see what it can and cannot do, and one thing I've looked today is how effective the backing store mechanism is. To recall: CVS stores a list of patches between versions in a single file, and git stores each new revision in full in a separate file in the so-called object store. Is that an issue for NetBSD? Let's see;

One of the more frequently updated files is the i386 port's GENERIC kernel config file, which is at revision 1.963 right now. This means that since it's import into CVS, 963 different revisions have been made. In CVS, all those files are kept in a single GENERIC,v file. In git, this puts 963 files on the file system. A bit of a difference.

Looking at the space requirements for storing the repository data itself, the GENERIC,v file is 883,233 bytes[1]. Extracting all 963 versions from revision 1.1 to revision 963 results in disk space usage of 32,805,828 bytes[2,3]. And that's not counting the overhead of 962 inodes and the related directory bookkeeping.

In other words, the git model requires about 37 times the space that CVS does.

Sure the example file is not exactly one with an average number of revisions, and I know that git offers some more efficient storage methods via "pack" files, but investigating those is left as an exercise to the reader. :-)

[1] Obtained via rsync from cvs.netbsd.org:

% ls -la GENERIC,v 
-r--r--r--  1 feyrer  wheel  883233 Feb 12 16:57 GENERIC,v

[2]

% mkdir extracted
% chdir extracted
% sh -c 'for i in `jot 964`; do echo $i ; co -p -r1.$i ../GENERIC >GENERIC-`printf %04d $i` ; done'

[3]

% cat extracted/* | wc -c
 32805828

Hackathon, February 19-22 2010

Fri, 05 Feb 2010 23:00:00 GMT

Aleksej Saushev writes on a bunch of lists:

We're running 13th Hackathon February 19-22 2010,
come and join us on IRC channel #netbsd-code at FreeNode (irc.freenode.net).
You may choose other ways to participate, if you find it more convenient.
See Hackathon page for updates: http://wiki.netbsd.se/Hackathon13


Goal

Fix as many bugs as possible, close as many PRs as possible.


Details

In previous years we have accumulated a lot of problem reports.
Some of them relate to no longer supported branches (e.g. 2.0)
and were occasionally fixed during regular code work.
Some of them relate to hardware developers don't have access to.
Some of them may be too easy to fix, but noone looks at that part
of code (e.g. documentation).
Some of them relate to packages already removed or heavily reworked.
Some of them relate to packages in exotic environment (Solaris, Interix)
and developers cannot test if the bug is present or not.

You can find more in PR database at http://netbsd.org/support/query-pr.html


More specifically, David Holland maintains categorized lists of PRs at

  http://www.netbsd.org/~dholland/buglists/

of which "Wanted for 5.1" 

(http://www.netbsd.org/~dholland/buglists/51-WANTED.html)
and "Stuck" (http://www.netbsd.org/~dholland/buglists/STUCK.html)
are of particular interest.

Another hot area is pkgsrc 
(http://www.netbsd.org/~dholland/buglists/pkgsrc.html).


IMPORTANT NOTE: You don't have to be programmer to be able to help us,
you can help us with generic testing support.

Come, bring your favourite booze, and join in!

Turn your ALIX into KITT (Updated)

Tue, 02 Feb 2010 10:18:00 GMT

Drop the following script into /etc/rc.d/alixkitt, put "alixkitt=yes" into /etc/rc.conf and enjoy:

#!/bin/sh
# Turn ALIX2c front LEDs into running LED
# http://www.gifninja.com/Workspace/59f3f22e-5f40-4937-936c-1dc0d6fbe690/output.gif
#
# Copyright (c) 2010 Hubert Feyrer <hubertf@NetBSD.org>

# PROVIDE: alixkitt

$_rc_subr_loaded . /etc/rc.subr

name="alixkitt"
start_cmd="alixkitt_start"
stop_cmd="alixkitt_stop"
pidfile="/var/run/${name}.pid"

alixkitt_sleeptime=1.0          # 0.5

alixkitt_setled()
{
        gpioctl -q -d /dev/gpio 6 $1
        gpioctl -q -d /dev/gpio 25 $2
        gpioctl -q -d /dev/gpio 27 $3

        sleep $alixkitt_sleeptime
}

alixkitt_start()
{
	touch $pidfile
        (
                alixkitt_setled 0 0 0

                while [ -f $pidfile ] ; do
                        alixkitt_setled 0 1 1
                        alixkitt_setled 1 0 1
                        alixkitt_setled 1 1 0
                        alixkitt_setled 1 0 1
                done
        ) &;
        echo $! >$pidfile
        echo Started pid $!
}

alixkitt_stop()
{
        if [ -f /var/run/alixkitt.pid ]; then
                read pid <$pidfile
                echo Stopped pid $pid
                kill $pid
                rm $pidfile
                alixkitt_setled 1 1 1
        fi
        exit 0
}

load_rc_config $name
run_rc_command "$1"

P.S.: There's still a bug left in that stopping immediately re-starts the process. Got a patch? Update: Axel Scheepers suggested doing the while-loop only as long as the PID-file exists. Changed above.

Toggling Alix2c LEDs

Sun, 31 Jan 2010 23:50:00 GMT

I've found that the three LEDs at the front of a PC Engines Alix2C board can be toggled on and off via their attachment to the GPIO bus. Here are the commands for this:

left:	gpioctl -d /dev/gpio 6 2
middle:	gpioctl -d /dev/gpio 25 2
right:	gpioctl -d /dev/gpio 27 2

Unfilling my inbox: NetBSD news from the past few weeks - ACPI, NUMA, Xen, and more

Sun, 31 Jan 2010 16:54:00 GMT

Herre are some more things that I've caught in my inbox for too long, and I'm finally finding some time to sum them up here:

NetBSD's "let's move kernel parts to the userland" RUMP project is still under heavy development, and in order to make testing of compatibility after kernel changes easier, a new command "rumptest" was added to build.sh: ``Basically you say:
./build.sh ${yourargs}�tools ; ./build.sh ${yourargs} rumptest
Where yourargs are what have you, e.g. '-U -u -o -O /objs'.
The latter builds only the rump kernel libs and uses some ld+awk magic to figure out if things go right or not. This is to avoid having to install headers and build libs (which is too slow since a full build is too slow). The magic is not a substitute for a full build, but it is n+1 times faster and works probably 99.9% of the time.
The scheme uses a number of predefined component sets (e.g. tmpfs+vfs+rumpkern) to test linkage. They are currently listed in build.sh. This area probably needs some work in the future. It would be nice to autogenerate the combinations somehow.
If things go well, you get something like this:

===> Rump build&link tests successful ===> build.sh ended: Wed Nov 18 20:10:59 EET 2009
'' See Antti's Antti's mail to tech-kern: on how to tell if things didn't go so well, and what to do in that case.
According to Wikipedia, ``Non-Uniform Memory Access or Non-Uniform Memory Architecture (NUMA) is a computer memory design used in multiprocessors, where the memory access time depends on the memory location relative to a processor. Under NUMA, a processor can access its own local memory faster than non-local memory, that is, memory local to another processor or memory shared between processors.''
Supporting NUMA in a contemporary (i.e.: Intel centric) SMP-enabled operating system requires following a bunch of standards, two of which are parsing of two tables, the System Resource Affinity Table (SRAT) and the System Locality Information Table (SLIT). Both tables are accessible via the Advanced Configuration and Power Interface (ACPI), and according to the German-language Wikipedia, the SRAT is used to assign local memory to local threads to boost their performance, and the SLIT defines the "distance" of the nodes among themselves, which is used to determine the "nearest" memory if local memory is not enough.
Now, Christop Egger has posted patches to add an ACPI SLIT parser and an ACPI SRAT parser. See the two postings for dmesg pr0n from his tests on an 8-node system.
Staying with ACPI and Christoph Egger, he found that even though the ACPI spec defines an ACPI device for fans, BIOS vendors and OEMs do their own thing. To accommodate things like the fan sensor found in the ACPI Thermal Zone in his HP Pavillion DV9700 laptop he has proposed a driver to extend the acpitz(4) driver with fan information. That way, envstat(8) can be used to display the ran's RPMs:
```
[acpitz0]
  Processor Thermal Zone:     56.000   95.000                       degC
                     fan:       2840                                 RPM 
```
Staying with driver games, iMil writes me that there's documentation on getting DRI, AIGLX, Composite and Compiz going with NetBSD 5.0 available in the O(ther)NetBSD Wiki now.
The documentation covers how to enable the Direct Rendering Manager (DRI), setting up and configuring Modular X.org, assuring that everything's in place, and how to get Compitz going. Mmm, wobbly windows at last! :-)
While we're talking funky desktop stuff: Marc Balmer has submitted a patch to get touchpanel support for ums(4). ums(4) is for USB mice, and in contrast to mice, touch panels need to deal with absolute numbers, not relative numbers.
Back to the guts of the kernel, another patch suggested by Christop Egger was for adding x2apic. What is x2apic? X2APIC is ``an Intel-only feature but can also be found in virtual environments with support for CPU apic id's > 0xff.
I.e. Xen 4.0 (not yet released) supports 128 CPUs in HVM guests with the CPUs enumerated with even apic id's. That means you need x2apic for the 128th CPU :) ''
While speaking of Xen: Xen 4.0 is coming soon, and there's a call to help testing it on NetBSD!
Install Mercurial, check out latest Xen sources, apply a bunch of patches, build and install. Examples of commands are given, in addition to changes required for /boot.cfg etc.
Report your findings to port-xen!
Last one for today: Michal Gladecki, Editor-in-Chief of BSD Magazine writes: ``We are happy to announce that BSD Magazine is transforming into a free monthly online publication. The online version of BSD Magazine will stay in the same quality and form. It will look like the BSD magazine one is familiar and comfortable with. Please sign up to our newsletter at www.bsdmag.org and get every issue straight to your inbox. Also, you can now download any of the previous issues from our website. The first online issue -- 2/2010 -- is coming out in February. Please spread the word about BSD Magazine. '' Click!

So much for today. I still have a bunch of news items in my inbox for next time, but let's call it good for today.

Unrelated, I've been playing with git a bit over the past few days, and wile I have a number of questions building up (which will be subject to tech-repository or so), what I can say today is that the speed of "git pull" with NetBSD's git repository and my 1MBit DSL line reminds me a lot of the times when I used SUP with my 56k modem - it took forever, too. :-(

A colorful collection of NetBSD news from the past few weeks

Tue, 19 Jan 2010 15:17:00 GMT

AKA "I've been slacking again, and there's a whole pile of stuff here now that I'm putthing into one blog posting". Here we go:

Initial support for the FriendlyArm Mini2440 board has been announced by Paul Fleischer. In a later update, most of the hardware is reported working, and the patch is available for review & comments. Furthermore, the touch screen is usable, and Qt/Embedded was built on top of wscons.
Force10 Networks Receives Common Criteria Security Certification for Its High-Performance Ethernet Switch/Router Products. According to the article, ``Common Criteria evaluations entail formal rigorous analysis and testing to examine security aspects of a product or system. Extensive testing activities involve a comprehensive and formally repeatable process, confirming that the security product functions as claimed by the manufacturer. Security weaknesses and potential vulnerabilities are specifically examined during wide-ranging evaluation and testing.
FTOS is the operating system software that runs on Force10 switch/router product lines, including the E-Series, C-Series and S-Series platforms. Based on NetBSD, FTOS leverages a distributed, multiprocessor architecture that delivers highly scalable protocols and reliability. By delivering the same OS across its entire switch/router line, Force10 ensures that customers benefit from stable code, a consistent configuration environment and simpler software management. ''
While there: Force10 Networks Delivers Ethernet-Optimized Platform for MPLS Core Networks: ``Force10 Networks, Inc. [...] announced the immediate availability of MPLS (multi-protocol label switching) functionality for its ExaScale E-Series core switch/routers. [...] The ExaScale platform combines high-density, non-blocking, line-rate 10 GbE switching and routing with robust MPLS LSR support at 1/5th of the cost of a traditional core router, enabling carriers to fully capitalize on the economic advantages of Ethernet.''
Create Bootable Live Linux USB Drives with UNetbootin: ``UNetbootin allows you to create bootable Live USB drives for a variety of Linux distributions from Windows or Linux, without requiring you to burn a CD. You can either let it download one of the many distributions supported out-of-the-box for you, or supply your own Linux .iso file if you've already downloaded one or your preferred distribution isn't on the list.''
And of course we all know that NetBSD is Linux, right? See the list of supported distributions:

The homepage mentions that NetBSD 4.0 is supported, maybe someone wants to give them an update on what's up with NetBSD 5.0? Would be nice to see that on the list!
Ever wondered what happened with the BSD Certification recently? There's a video from the talk BSD Certification Group: A Case Study in Open Source Certification available that talks about the goal of the project, the two exams (BSD Associate, BSD Professional), and what's going on behind the scenes.
With the move from XFree to X.org, the X server for the DNARD Shark's NetBSD/shark lost support for accelerated X. Thanks to Michael 'macallan' Lorenz, hardware-accelerated X for NetBSD/shark is back now: ``I finally got around to start working on an Xorg driver for the IGS CyberPro 20x0 family found in rev. 5 Sharks, Netwinder etc. - currently the driver is built only on shark and supports only the VL variant found there. Adding support for PCI chips is trivial though, just needs extra probing. The driver supports autoconfiguration ( X -configure should yield something almost usable, only DefaultDepth needs to be adjusted).''
Staying with cool platforms, here's a quick procedure to run NetBSD/sun2 5.0.1 on The Machine Emulator (TME) (see pkgsrc/emulators/tme), compiled by Izumi Tsutsui. Who's first do get a pkgsrc bulk build done? :-)
Jed Davis has committed the RAIDframe parity Summer-of-Code project. See his posting for the details. The project ``drastically reduces the time RAIDframe spends rewriting parity after an unclean shutdown by keeping better track of outstanding writes (thus, "parity map"). The tech-kern archives have more details [...]
This feature is enabled by default on all sets (other than RAID 0). It can be administratively disabled with the new "raidctl -M" flag, which is described in the changes to the raidctl(8) man page; however, the I/O overhead for updating the parity map is expected to be unnoticeable in practice.''

So much for now. There is more in the pipe, but that will have to wait for now. Good night!

NetBSD and ISO 8859-2

Sun, 17 Jan 2010 22:47:00 GMT

Flying by one one of our mailing lists, I found this gem: ``This short article aims to help people that want to use ISO8859-2 character encoding in NetBSD/i386's wscons(4) console. It is very brief, so if you understand Slovak language, I recommend you to read NetBSD Slovak & Czech HOWTO instead. ''

The article describes how to teach wscons to use a ISO 8859-2 font, and also how to adjust the wscons keymap to either a czech or slovak keyboard mapping.

I think that gives support for ISO 8859-1, -2 and -15, so now where are your patches for the rest?

Hardware crypto with Geode LX based ALIX board

Wed, 13 Jan 2010 16:55:00 GMT

I'm using an Alix 2d13 machine as home router for quite some time now - uptime was 158 days today. The board has a 500MHz AMD Geode LX800 CPU, which can do AES crypto in hardware. Today, I've found time to play with the glxsb(4) driver, and get some numbers:

The 'numbers' are in 1000s of bytes per second processed.
crypto   type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
sw       aes-128-cbc       3583.29k     3931.51k     4037.32k     8113.75k     8205.61k
hw       aes-128-cbc       1200.70k     4470.18k    11729.65k    17328.05k    34006.33k

The command "openssl speed -evp aes-128-cbc -elapsed" was ran each time, in the first case with glxsb(4) disabled (boot -c, disable glxsb), in the second case with the driver enabled.

Still, I find those numbers interesting in comparison to those of a Soekris 5501, esp. as the machines have the same CPU & clock rate.

I couldn't find a way to switch use of hardware crypto off in software, anyone got a hint? openssl(1)'s "-engine cryptodev" seems to always use hardware crypto when it's there, and leaving out that switch uses the same engine, so no difference. I've found a few sysctls under kern.*, but apparently none seems relevant to my goal (not to speak of the lack of documentation...)

sysupdate - an addition for etcupdate(8) (Update #2)

Tue, 12 Jan 2010 12:54:00 GMT

I wanted to update a few machines with latest netbsd-5 and -current today, and found that there's a certain lack of tools to aid in the process: we have etcupdate(8) and postinstall(8), but nothing to help extract the userland sets (let alone do the same thing for the kernel, esp. now that modules must be put into place, too).

To save me from re-typing the same commands, I've hacked a script "sysupdate" that's intended to run with a new kernel booted, and which will then 1) extract all the sets where mtree files are in /etc/mtree, and 2) run etcupdate for etc (and xetc, if in /etc/mtree).

Here's an example run:

    % sh ~/tmp/sysupdate -s /usr/cvs/src-current/obj.i386/releasedir/i386/binary/sets -n
    ---------------------------------------------------------------------------
    Updating your userland now!
    ---------------------------------------------------------------------------

    New userland: NetBSD 5.99.23 (GENERIC) #0: Sun Jan 3 02:13:00 CET 2010
    Current kernel: NetBSD vmnetbsd6 5.99.23 NetBSD 5.99.23 (GENERIC) #0:
     Sun Jan 3 02:13:00 CET 2010 feyrer%mini.local@localhost
     :/usr/cvs/src-current/obj.i386/sys/arch/i386/compile/GENERIC i386

    Really update? (y/N) y
    Updating sets...
    Extracting base.tgz
    tar -plzx -C / -f base.tgz
    Extracting comp.tgz
    tar -plzx -C / -f comp.tgz
    Extracting games.tgz
    tar -plzx -C / -f games.tgz
    Extracting man.tgz
    tar -plzx -C / -f man.tgz
    Extracting misc.tgz
    tar -plzx -C / -f misc.tgz
    Extracting modules.tgz
    tar -plzx -C / -f modules.tgz
    Extracting tests.tgz
    tar -plzx -C / -f tests.tgz
    Extracting text.tgz
    tar -plzx -C / -f text.tgz
    Extracting xbase.tgz
    tar -plzx -C / -f xbase.tgz
    Extracting xcomp.tgz
    tar -plzx -C / -f xcomp.tgz
    Extracting xfont.tgz
    tar -plzx -C / -f xfont.tgz
    Extracting xserver.tgz
    tar -plzx -C / -f xserver.tgz
    Updating etc files...
    etcupdate -s /usr/cvs/src-current/obj.i386/releasedir/i386/binary/sets/etc.tgz \
     -s /usr/cvs/src-current/obj.i386/releasedir/i386/binary/sets/xetc.tgz
    Done.
    %

The script is available in the current-users@ mailing list archive.

Update: Chavdar Ivanov tells me that there's a package pkgsrc/sysutils/etcmanage that comes with a script INSTALL-NetBSD which does the same as mine (plus more). Seems that's not established enough that it's the communicated way for updating (yet?). Given that the script uses perl, maybe that's some time off. :-P

Another option that uses toosl that come with NetBSD out of the box (ksh!) is available here - thanks to Dalibor Gudzic for the hint and the script!

Update #2: Asmodehn sent me mail that he also has made a script for performing system updates. How about someone collect all these into one, and import something into NetBSD that at least provides minimal functionality?

Mounting fixed and removable disks with KDE

Mon, 28 Dec 2009 12:50:00 GMT

If you're among the people running KDE, and if you want to be able to have KDE handle removable and fixed disks properly during the HAL subsystem, there's a suggestion by David Brownlee to achieve this.

To get removable devices automatically mounted and a file browser pop up, add the following to /usr/pkg/etc/PolicyKit/PolicyKit.conf:

        <match action="org.freedesktop.hal.storage.mount-removable">
                <return result="yes" />
        </match>

David writes that ``[w]ith the above plugging in a USB flash drive into my amd64 laptop pops up a notification window, clicking on it opens dolphin, and clicking on the device in dolphin correctly mounted it.''

A similar thing can be achieved for fixed disks, by adding the following data:

        <match action="org.freedesktop.hal.storage.mount-fixed">
                <return result="yes" />
        </match>

There are still discussions if/what part of this should be enabled by default, but I think it's nevertheless useful to document this here. Enjoy!

Catching up on source changes

Thu, 10 Dec 2009 00:30:00 GMT

I've dug through my source-changes archive once more, and there are entries from February(?!?) to mid-October 2009 in there that I think may be of interest to casual followers of NetBSD.

Before I go into the details, let me add a hint on the "source-changes-full" mailing list, which is like source-changes, but the mails there also contain diffs of the changes (as long as they are below 1MB in size). The -full list is not archived on mail-index.netbsd.org, and I must have missed the announcement. It's definitely useful!

So, what interesting changes have there been to NetBSD-current recently? Here we go:

New / updated drivers:
- sdtemp(4) reads the on-DIMM temperature sensors following the JEDEC Standard No. 21-C Section 4-7.
- gpio(4) got support for Intel ICH southbridges as found on Intel SS4220-E (ICH7) and Acorp 6A815EPD (ICH2) motherboards
- lom(4) supports the LOMlite "Lights Out Management" boards found on Sun hardware like the LOMlite found on the Sun Netra t1 and the LOMlite2 found on Sun Netra T1/X1 and Sun Fire V100/V120.
- An initial version of a SD/MMC driver for the Winbond W83L518D (and probably W83L519D) Integrated Media Reader with PNP bus attachment glue was added.
- acpiwmi(4) was added to added, to further support ACPI development. See the specs for more details.
- piixide(4) now supports Intel 3400 SATA
New kernel interfaces and library functions include:
- getdate(3) and getdate_err(3) were added according to POSIX.1 and the Open Group
- RAIDframe was sped up massively when checking parity after a system crash with the help of a parity map. This work was done during this year's Google Summer-of-Code.
- A "netbsd.inet.icmp.bmcastecho" sysctl was added to disable replies to the broadcast address.
- Floating point functions f{min,max,dim}{,l,f}(3) were added
- udl(4) adds support for DisplayLink DL-1x0/1x5 based USB LCDs and USB-VGA adaptors
Random other additions and changes include:
- hdaudioctl(8) can be used to manipulate hdaudio(4) devices
- The NetBSD/playstation2 port was removed due to a lack of developer support. It's still available in CVS.
- The NetBSD/i386 bootloader now knows a "dev" command to list available boot devices and their size.
Newly imported and updated 3rd party software includes:
- pfsync from OpenBSD 4.2, adopted in this year's Google Summer of Code.
- PF from OpenBSD 4.2 was imported.
- Multicast DNS ("Bonjour") support was added, based on Apple's mDNSResponder. Enable with "dns: mdns" in /etc/nsswitch.conf.
- GNU binutils were updated to version 2.19
- Many of the bazillion of X.org components were updated - ways too many to note in detail.
- BIND was updated to 9.7.0b1
- Flex 2.5.35 was imported
- A new m4(1) was imported

Logical Volume Management (LVM)

Wed, 09 Dec 2009 23:37:00 GMT

Adam Hamsik has been working on Logical Volume Management (LVM) support in NetBSD during the Google Summer of Code and beyond. The code is a from-scratch implementation of the same interfaces found in the Linux kernel, and as a result, the same userland tools can be used. The code is now enabled in NetBSD by default, i.e. it will be part of NetBSD 6.0 (quite some time down the road, but time to test and shake out the bugs is *now*).

Besides the code, there's also documentation available in the NetBSD Guide.

Happy Volume Managing!

TeX Live 2009 - including NetBSD binaries

Mon, 09 Nov 2009 23:32:00 GMT

Today, TeX Live 2009 was released. After teTeX is no longer maintained, TeX Live is *the* Unix (and other) TeX distribution. And with the binary release come binaries for NetBSD: ``Executables for the cygwin and i386-netbsd platforms are now included, while the other BSD distributions have been dropped; we were advised that OpenBSD and FreeBSD users get TEX through their package systems, plus there were difficulties in making binaries that have a chance of working on more than one version. ''.

The NetBSD binaries were provided by Jukka Salmi. Mmm, community involvement. Thanks Jukka!

NetBSD vs. Microkernels - From Kernel to Userspace and Abroad

Sat, 07 Nov 2009 00:59:00 GMT

A few weeks ago I made a comment about NetBSD not being a "microkernel" despite it's move to kernel modules. Antti Kantee wrote back to me (Hi Antti!) reminding me of his work on RUMP, PUFFS and the like, which I think does definitely deserve mentioning in that context:

With RUMP, PUFFS and the like, Antti is set on a mission to define interfaces between kernel components, and move those kernel components from inside the kernel to the userland. As a result, the (once) kernel-code is now running as "simple" userland processes that have all the benefits like memory protection and not taking down the whole system in case of an error. Examples that Antti has been bashing on started with moving file systems to the userland, which resulted in PUFFS and ReFUSE, an implementation of the Linux inferface of File systems in USErland (FUSE). Not stopping at file systems (which are traditionally used for accessing data on storage media, like e.g. for ntfs-3g), Antti went further and moved into getting network services like SSHFS going - in userland, based on FUSE.

Not stopping there, more recent works include:

USB device driver support, to run NetBSD's USB stack in userland - and with it a bunch of USB device drivers as well, e.g. storage (umass) and wireless adapters.
Running NFS in userspace, and with it the underlying TCP/IP stack? Well, we've had nfsd(8) for quite some time, but this time it's the same code running in userland as in kernel. The thread also answers questions on why we want to virtualize nfsd+networking with a process and why do we want to do that by reusing kernel code.

Remember when NetBSD got (re)FUSE? All of a sudden we got a bazillion of filesystems back then. So why not turn this development around, and make RUMP available as an interface for all sort of drivers to other operating systems. That way, they can get our file systems, usb stack and drivers, etc., and run them in userspace as well. Developing a driver on one operating system, and using it on many - a wet dream would come true!

Sounds impossible? Ye fear not, it's been done! Arnaud 'stacktic' Ysmal already has ported RUMP to FreeBSD and Linux, the work is available via pkgsrc/misc/rump, and there is also Arnaud's page on Rump on non-NetBSD Operating Systems for more information.

Whew... lots of research and development going on in this area, and - getting back to the initial topic - we may well see an operating system in the future that moves from the monolithic to a microkernel approach, and it may or may not be called NetBSD. Fact is, that a lot of research is going on in that area, on NetBSD, here and now. Hats off, Antti!

On the difference between "data" and "information"

Sat, 07 Nov 2009 00:11:00 GMT

Thanks xkcd, from the information scientist inside me!

Harddisk image cloning for Unix - g4u 2.4 released

Sun, 25 Oct 2009 23:05:00 GMT

g4u ("ghosting for unix") version 2.4 has been released. g4u is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common set up on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported.

Three years of time have passed since the last full release of g4u. Here's a list of what's new / changes in g4u 2.4:

Major new supported device types include bluetooth keyboards and SD/MMC cards - feedback highly appreciated!
Lots of new drivers. Too many to list, please see the g4u section of my blog at http://www.feyrer.de/NetBSD/blog.html?-tags=g4u for details.
Based on the NetBSD development version from Sep 2009
Source builds native and without root privileges on NetBSD 5.0 and crossbuilds also without root privileges from Mac OS X (tested) and probably others (untested; expected: Solaris, Linux).

The g4u 2.4 release is available on the g4u homeage at

http://www.feyrer.de/g4u/

Fun around daylight saving time (DST) and Unix timezone handling in general

Sat, 24 Oct 2009 14:42:00 GMT

It's that time of the year again when I and my friends never know when daylight saving time is switched from summer- to winter time. Do you know? And if you've heared in the news today... do you know when the next change is?

There's an easy way to find out, using the zdump(8) utility that comes with about every Unix machine (at least with NetBSD and Mac OS X - is there anything else relevant? :), and the timezone information stored in the /usr/share/zoneinfo directory.

To find out when DST is switched on/off, run the following command:

% zdump -v /etc/localtime
...
/etc/localtime  Sun Mar 29 00:59:59 2009 UTC = Sun Mar 29 01:59:59 2009 CET isdst=0
/etc/localtime  Sun Mar 29 01:00:00 2009 UTC = Sun Mar 29 03:00:00 2009 CEST isdst=1
/etc/localtime  Sun Oct 25 00:59:59 2009 UTC = Sun Oct 25 02:59:59 2009 CEST isdst=1
/etc/localtime  Sun Oct 25 01:00:00 2009 UTC = Sun Oct 25 02:00:00 2009 CET isdst=0
/etc/localtime  Sun Mar 28 00:59:59 2010 UTC = Sun Mar 28 01:59:59 2010 CET isdst=0
/etc/localtime  Sun Mar 28 01:00:00 2010 UTC = Sun Mar 28 03:00:00 2010 CEST isdst=1
/etc/localtime  Sun Oct 31 00:59:59 2010 UTC = Sun Oct 31 02:59:59 2010 CEST isdst=1
/etc/localtime  Sun Oct 31 01:00:00 2010 UTC = Sun Oct 31 02:00:00 2010 CET isdst=0
/etc/localtime  Sun Mar 27 00:59:59 2011 UTC = Sun Mar 27 01:59:59 2011 CET isdst=0
/etc/localtime  Sun Mar 27 01:00:00 2011 UTC = Sun Mar 27 03:00:00 2011 CEST isdst=1
/etc/localtime  Sun Oct 30 00:59:59 2011 UTC = Sun Oct 30 02:59:59 2011 CEST isdst=1
/etc/localtime  Sun Oct 30 01:00:00 2011 UTC = Sun Oct 30 02:00:00 2011 CET isdst=0
...

The data is given relative to Universal Time Coordinated, which you can determine on a NetBSD system by running "date -u":

% date
Sat 24 Oct 14:30:24 CEST 2009
% date -u
Sat 24 Oct 12:30:26 UTC 2009

And if you wonder what timezone you're actually in, you probably knokw that /etc/localtime is a symbolic link to a file that fits to your exact timezone, with all related information:

% ls -la /etc/localtime 
lrwxr-xr-x  1 root  wheel  33 May 26  2007 /etc/localtime -> /usr/share/zoneinfo/Europe/Berlin

If you always wanted to know what timezones there are, have a look at /usr/share/zoneinfo:

% ls /usr/share/zoneinfo/
Africa       Chile        GB-Eire      Israel       NZ-CHAT      UCT
America      Cuba         GMT          Jamaica      Navajo       US
Antarctica   EET          GMT+0        Japan        PRC          UTC
Arctic       EST          GMT-0        Kwajalein    PST8PDT      Universal
Asia         EST5EDT      GMT0         Libya        Pacific      W-SU
Atlantic     Egypt        Greenwich    MET          Poland       WET
Australia    Eire         HST          MST          Portugal     Zulu
Brazil       Etc          Hongkong     MST7MDT      ROC          iso3166.tab
CET          Europe       Iceland      Mexico       ROK          posixrules
CST6CDT      Factory      Indian       Mideast      Singapore    zone.tab
Canada       GB           Iran         NZ           Turkey
% ls /usr/share/zoneinfo/Europe/
Amsterdam    Chisinau     Kiev         Moscow       Sarajevo     Vatican
Andorra      Copenhagen   Lisbon       Nicosia      Simferopol   Vienna
Athens       Dublin       Ljubljana    Oslo         Skopje       Vilnius
Belfast      Gibraltar    London       Paris        Sofia        Volgograd
Belgrade     Guernsey     Luxembourg   Podgorica    Stockholm    Warsaw
Berlin       Helsinki     Madrid       Prague       Tallinn      Zagreb
Bratislava   Isle_of_Man  Malta        Riga         Tirane       Zaporozhye
Brussels     Istanbul     Mariehamn    Rome         Tiraspol     Zurich
Bucharest    Jersey       Minsk        Samara       Uzhgorod
Budapest     Kaliningrad  Monaco       San_Marino   Vaduz

So now that we all that, here's a last cutie: suppose you want to log into a machine on the other end of the world, and still run an application that should use your local timezone, not that that the machine is in. Changing /etc/localtime is not an option, but you can do it on a per-process base by setting the "TZ" environment variable:

% date
Sat Oct 24 14:35:06 CEST 2009
% ssh remote.example.org date
Sat Oct 24 12:35:45 UTC 2009
% ssh remote.example.org env TZ=Europe/Berlin date
Sat Oct 24 14:35:58 CEST 2009

Enjoy!