hubertf's NetBSD Blog
Send interesting links to hubert at feyrer dot de!
 
[20091107] NetBSD vs. Microkernels - From Kernel to Userspace and Abroad
A few weeks ago I made a comment about NetBSD not being a "microkernel" despite it's move to kernel modules. Antti Kantee wrote back to me (Hi Antti!) reminding me of his work on RUMP, PUFFS and the like, which I think does definitely deserve mentioning in that context:

With RUMP, PUFFS and the like, Antti is set on a mission to define interfaces between kernel components, and move those kernel components from inside the kernel to the userland. As a result, the (once) kernel-code is now running as "simple" userland processes that have all the benefits like memory protection and not taking down the whole system in case of an error. Examples that Antti has been bashing on started with moving file systems to the userland, which resulted in PUFFS and ReFUSE, an implementation of the Linux inferface of File systems in USErland (FUSE). Not stopping at file systems (which are traditionally used for accessing data on storage media, like e.g. for ntfs-3g), Antti went further and moved into getting network services like SSHFS going - in userland, based on FUSE.

Not stopping there, more recent works include:

Remember when NetBSD got (re)FUSE? All of a sudden we got a bazillion of filesystems back then. So why not turn this development around, and make RUMP available as an interface for all sort of drivers to other operating systems. That way, they can get our file systems, usb stack and drivers, etc., and run them in userspace as well. Developing a driver on one operating system, and using it on many - a wet dream would come true!

Sounds impossible? Ye fear not, it's been done! Arnaud 'stacktic' Ysmal already has ported RUMP to FreeBSD and Linux, the work is available via pkgsrc/misc/rump, and there is also Arnaud's page on Rump on non-NetBSD Operating Systems for more information.

Whew... lots of research and development going on in this area, and - getting back to the initial topic - we may well see an operating system in the future that moves from the monolithic to a microkernel approach, and it may or may not be called NetBSD. Fact is, that a lot of research is going on in that area, on NetBSD, here and now. Hats off, Antti!

[Tags: , , , , , , , , , , ]



[20081114] Catching up - what happened in NetBSD-land between mid-August and mid-November
OK, I've been slacking^Wbusy for the past weeks, but I hope things will get a bit better now. For a start, here's a catch-up of the things that accumulated in my inbox in the past ~two months:
  • Google Summer of Code is over for some time, but apparantly no final report has emerged so far (shame!). Still, a number of individual status reports came by on the official lists:

    I know of at least one other project (uvc) that has completed but that I didn't see a report here - maybe I've missed it. Anyways, GSoC was another big success this year. Thanks, Google!

  • Speaking of Adam Hamsik and Logical Volume Management (LVM), Adam has continued his work in that are, and he has written a device mapping library that interacts with his kernel driver. This allows to interact with his GSoC project without using any GPL code! See Adam's posting for more details.

  • Force 10 Networks, producer of 10gbit switches that use an operating system based on NetBSD, have added a new feature as part of their FTOS operating system: VirtualView, which provides virtualization of Force 10 based equipment. From the xchange article: ``Force10 Networks Inc. this week introduced VirtualView software for benchmaking, troubleshooting and managing virtualized environments based on Force10 gear.''

    More information is available from the Force 10 Networks homepage, plus in articles by fibresystems.org, light reading, Zycko, and SmartBrief.

  • Following the latest hype in portable computers, NetBSD has created a netbook page that intends to list models and the extent to which they are supported. Your contributions are most welcome here! (Contact me for sending updates and hardware :-)

  • Zafer Aydogan has made RSS feeds available for CVS commits to single files - see his mail to netbsd-users for more details.

  • New security advisory were released that I've missed in my last update:

  • A project that's been ongoing for quite some time is the move from "old-school" loadable kernel modules (LKMs) to new-style kernel modules. Important changes include the fact that modules can be either linked into the kernel at build time, or loaded into the kernel at runtime from the same file. Also, the bootloader was modified to load modules after the kernel, e.g. for a RAM-disk like the one that is used by the INSTALL kernel.

    In the same line, some parts are starting to be moved out of the GENERIC kernel, and installed as modules that can be loaded by the new framework then. The start is made by POSIX semaphores as a first step and proof-of concept, even if some details are still under hot debate, e.g. what the file system layout for modules is, and if the belong to the kernel and its build process, or to the userland.

  • While talking about splitting the kernel into modules, Antti Kantee has continued his work to move parts of the kernel into userspace, in particular running file system code as userland in his RUMP, and puffs and (Re)FUSE works. The idea is to provide the interfaces that file systems need in the userland, and the result is that you can run code that used to run inside the kernel in userland now.

    Another subsystem running in the kernel that could be moved to userland by providing appropriate interfaces with the rest of the kernel is the network stack, and Antti has moved just that to the userland. See Antti's mail to tech-net@ for more information on this impressive work.

  • NetBSD has shipped XFree in previous releases, and people who wanted to use X.org had to install it from pkgsrc. That's all fine, but to get a modern X, one had to compile things, as no precompiled binary packages are made available for many platforms. This is changing now, and NetBSD is getting X.org integrated via a reachover infrastructure which is also enabled for crosscompiling.

    The "user interface" for this is still in flux, but after some detour ("build.sh -V MKXORG=yes", without -x), "build.sh -x" now builds whatever X is considered the default for the platform. Some platforms already default to use X.org as X, and more will come, as changes that were made to NetBSD's copy of XFree are adopted to X.org.

    Platforms that use X.org by default now are macppc (see here and here), sparc sparc64, shark, amd64 and i386.

    As X.org is at Revision 7 now, it's installed in /usr/X11R7, which will lead to a lot of interesting effects. pkgsrc is already prepared for the new layout, but there are still many minor details that will need adjusting to the new directory. If you find one, post your patches to tech-x11.

  • Besides the GNU C compiler, there's the BSD-licensed Portable C Compiler around for some time now. It doesn't offer the same support as its GNU cousin yet, but this may change now: The BSD Fund is currently doing a fund drive to get money to enhance PCC. The goal is to raise $12,000US to improve support for core compiler functionality as well as support for C99, gcc compatibility and the amd64 architecture. See the project page for further details.

  • The NetBSD 5.0 release cycle has started! There's a netbsd-5 branch in CVS, daily binaries are available for testing, and some of the highlights of the upcoming release include file system journalling for FFS via WAPBL, and X.org.

    To help testing of NetBSD on Cobalt machines, Izumi Tsutsui has made a NetBSD 5.0_BETA based version of the Cobalt restore CD available. Enjoy!

  • As the final point today, a word on NUMA support from Christoph Egger. Non-Uniform Memory Access is needed in massive parallel systems where some nodes have RAM more tightly associated than others, where the RAM is further away, resulting in different access times for different regions of memory. In order to support this, Christop Egger has made first steps.

    His example implementation uses information from ACPI, and shows some heavy dmesg-pr0n from a 16-core machine with four sockets. Yumm!

So much for today. With the NetBSD 5.0 release cycle started, I'd like to encourage everyone to test the release branch, report errors, send patches as well as beer and choccolate to make this the best release that we've ever had.

[Tags: , , , , , , , , , , , , , , , , ]


[20080428] Recent development related to puffs, ReFUSE, rump, and more (Updated)
NetBSD's kernel is under very active development these days, and while many changes are related to improve SMP, it's not the only area. An area where very interesting and unique work is being done is the filesystem interfaces that Antti Kantee is working. Things started out as a past year's Google "userfs" SoC project to implement an interface for running filesystem code in userland. The project was imported into NetBSD some time ago. On top of that, a library that mimics the Linux interface for filesystems in userland. Following the Linux name FUSE, the re-implementation is called ReFUSE (pun intended :). See the webpage about puffs, refuse, FUSE on the NetBSD website for more information.

Another project that was started by Antti after his work to run filesystem code in userland is "rump". The project allows to use "ordinary" filesystems that usually run inside the kernel, and mimic an environment similar to what's available inside the kernel, and move the whole filesystem into userland - verbatime, with no code changes! This allows to develop filesystem code in userland, and later on move it inside the kernel with no further changes - a bit step forward for filesystem development!

This all sounds rather easy, but as filesystems need to move data between storage and memory, a big issue in filesystems is interfacing with the virtual memory subsystem, and adding interfaces like puffs and ReFUSE also needs to consider VM for efficient transfers and caching.

Work in this area is still ongoing, and I've asked Antti about his recent achievements in this area[1]. While the only user-visible change is caching and performance improvements in the Secure Shell filesystem's handler "mount_psshfs", most of the changes are on the inside. Antti wrote me: ``The interesting ones from a programmer's perspective are probably:

  • Splitting userspace transport out of puffs in the kernel (putter)
  • Using putter to implement support for userspace block/char device drivers (pud). pud does still not have a userspace library similar to libpuffs. libpuffs needs to become libputter and lib{puffs,pud}.

  • Removing special case handling for the puffs user/kernel protocol transport. This means that file system requests can now be read/written like any other protocol. This is covered in the AsiaBSDCon 2008 paper "Send and Receive of File System Protocols: Userspace Approach With puffs"

    With some minor work in libpuffs, it possible to e.g. do an ffs mount from a remote site with the help of rump.

Finally, while not really useful for anything except puffs development, I think the following is cool from the perspective of completeness:

  • Add support to rump to be able to run the puffs kernel module in userspace. This means that that *any* puffs file system (incl. rump ones) can be mounted so that requests pass once through the puffs kernel module running in the kernel and once through the puffs kernel module running in userspace before being delivered to the file system driver. Example:
      sys/rump/fs/bin/syspuffs> ./syspuffs mount_psshfs server.address /path 
''

With puffs and rump, there are two very interesting and active projects doing research in filesystems on NetBSD, which may lead to changes in the way filesystems are understood in the Unix world. While there, a third project that may be worth watching in this regards is this year's Google hurdt Summer of Code project by Marek Dopiera, which aims at implementing Hurd translators for NetBSD,

Update: Antti dropped me a note that another project related to filesystems is this year's "fs-utils" SoC project. The goal is to create a userland tool to manipulate filesystem images, and the idea is to reuse kernel code with the ukfs library. That way, no redundancy between kernel sources and userland sources are created, and both areas benefit from mutual testing and code maturity.

[Tags: , , , , , ]



[20080225] Mondo catch-up on source-changes (~Aug '07 'till Feb '08)
In the context of Mark Kirby stopping his NetBSD CVS Digest, I've felt an urge to catch up on source-changes, and put up some of the items here that I haven't found mentioned or announced elsewhere (or that I've plainly missed) after digging through some 7,000 mails. All those changes are available in NetBSD-current today and that will be in NetBSD 5.0:

  • Support C99 complex arithmetic was added by importing the "cephes" math library
  • POSIX Message queues were added
  • bozohttpd was added as httpd.
  • the x86 bootloader now reads /boot.cfg to configure banner text, console device, timeout etc. - see boot.cfg(5)
  • ifconfig(8) now has a "list scan" command to scan for access points
  • SMP (multiprocessor) support is now enabled in i386 and amd64 GENERIC kernels
  • Processor-sets, affinity and POSIX real-time extensions were added, along with the schedctl(8) program to control scheduling of processes and threads.
  • systrace was removed, due to security concerns
  • the refuse-based Internet Access Node file system was committed, which provides a filesystem interface to FTP and HTTP, similar to the old alex file system, see http://mail-index.netbsd.org/source-changes/2007/08/28/0081.html
  • LKMs don't care for options MULTIPROCESSOR and LOCKDEBUG, i.e. it's easier to reuse LKMs between debugging/SMP and non-debugging/SMP kernels now.
  • PCC, the Portable C Compiler that originates in the very beginnings of Unix, was added to NetBSD. The idea is that it is used as alternative to the GNU C Compiler in the long run.
  • In addition to the iSCSI target (server) code that is already in NetBSD 4.0, there'a also a refuse-based iSCSI initiator (client) now, see http://mail-index.netbsd.org/source-changes/2007/11/08/0038.html
Plus:
  • Many driver updates and new drivers, see your nearest GENERIC kernel config file
  • Many security updates, see list of security advisories
  • Many 3rd software packages that NetBSD ships with were updated: ipsec-tools (racoon), GCC 4.1, Automated Testing Framework 0.4, OpenSSH 4.7, wpa_supplicant and hostapd 0.6.2, OpenPAM Hydrangea
The above list is a mixed list of items. There are a number of areas where there is very active development going on in NetBSD. Andrew Doran is further working on SMP, fine-grained locking inside the kernel and interrupt priority handling. Antti Kantee has has done more work on his filesystems work (rump, puffs, refuse/fuse), and Jared McNeill and Jörg Sonnenberger have continued their work on NetBSD's power management framework. Those changes are large and far-reaching, and I've yet to look at them before I can report more here.

So much on this subject for now. If someone's willing to help out with continuing Mark Kirby's NetBSD CVS Digest either using his software-setup or by simply reading the list and writing a monthly/weekly digest of the "interesting" changes, I'd appreciate this very much. Put me on CC: for your postings! :)

[Tags: , , , , , , , , , , , , , ]



[20070313] Filesystems in User-Space: puffs, refuse vs. FUSE, and more (Updated)

[I'm taking the liberty to post this unquoted here, as I did the editing on the news item, based on input from Antti and Al]

NetBSD now offers full support for running file systems in userspace. The core component inside the kernel that exposes a file system interface towards the userland programs is "puffs", and communicates with vnodes, the kernel's own idea of files. The "libpuffs" library is the interface between userland file systems and the kernel component. To facilitate running the huge amount of file systems already available for the FUSE interface, but not dictate the capabilities of puffs by it, it was decided that FUSE support should be provided as a compatibility layer on top of the native puffs interface. This compatibility is offered by re-fuse, a BSD-licensed re-implementation of FUSE. It is designed to be source code compatible with FUSE, and more information can be found in the refuse(3) manpage.

While puffs is stable for users, the library programming interface and the binary interfaces are still likely to change, and therefore puffs and the dependant librefuse are not built by default. For people tracking -current, the build of all relevant puffs components (librefuse, psshfs, etc.) by "make build" can be enabled by adding "MKPUFFS=yes" to /etc/mk.conf. Using puffs also requires the kernel option "file-system PUFFS", or alternatively loading the puffs.o kernel module, which is included in a puffs-enabled build.

FUSE compatibility was added within pkgsrc, and besides the required infrastructure work a number of FUSE packages were added to pkgsrc in the new "filesystem" category. Example packages that are currently available include:

puffs was originally developed during the Google Summer of Code 2005 "userfs" project by Antti Kantee. It was further polished using funding from the Ulla Tuominen Foundation and committed into the NetBSD source tree in October 2006. The work was presented at AsiaBSDCon 2007 in a paper entitled "puffs - Pass-to-Userspace Framework File System". The "refuse" library was written by Alistair Crooks and committed to NetBSD in February 2007. All components will be available in NetBSD releases starting with NetBSD 5. Work for using FUSE-enabled packages via pkgsrc was done by Juan Romero Pardines following import of the "refuse" library.

For more information, see NetBSD's webpage on file systems in userspace at http://www.NetBSD.org/Documentation/misc/puffs.html.

Update: An in-depth technical description of puffs was presented at AsiaBSDCon 2007 in a paper entitled "puffs - Pass-to-Userspace Framework File System". The paper and slides are available.

[Tags: , , ]


Tags: , 2bsd, 3com, 501c3, 64bit, acl, acls, acm, acorn, acpi, acpitz, adobe, Advocacy, advocacy, advogato, aes, afs, aiglx, aio, airport, alereon, alex, alix, alpha, altq, am64t, amazon, amd64, anatomy, ansible, apache, apm, apple, arkeia, arla, arm, art, Article, Articles, ascii, asiabsdcon, aslr, asterisk, asus, atf, ath, atheros, atmel, audio, audiocodes, autoconf, avocent, avr32, aws, axigen, azure, backup, balloon, banners, basename, bash, bc, beaglebone, benchmark, bigip, bind, blackmouse, bldgblog, blog, blogs, blosxom, bluetooth, board, bonjour, books, boot, boot-z, bootprops, bozohttpd, bs2000, bsd, bsdca, bsdcan, bsdcertification, bsdcg, bsdforen, bsdfreak, bsdmac, bsdmagazine, bsdnexus, bsdnow, bsdstats, bsdtalk, bsdtracker, bug, build.sh, busybox, buttons, bzip, c-jump, c99, cafepress, calendar, callweaver, camera, can, candy, capabilities, card, carp, cars, cauldron, ccc, ccd, cd, cddl, cdrom, cdrtools, cebit, centrino, cephes, cert, certification, cfs, cgd, cgf, checkpointing, china, christos, cisco, cloud, clt, cobalt, coccinelle, codian, colossus, common-criteria, community, compat, compiz, compsci, concept04, config, console, contest, copyright, core, cortina, coverity, cpu, cradlepoint, cray, crosscompile, crunchgen, cryptography, csh, cu, cuneiform, curses, curtain, cuwin, cvs, cvs-digest, cvsup, cygwin, daemon, daemonforums, daimer, danger, darwin, data, date, dd, debian, debugging, dell, desktop, devd, devfs, devotionalia, df, dfd_keeper, dhcp, dhcpcd, dhcpd, dhs, diezeit, digest, digests, dilbert, dirhash, disklabel, distcc, dmesg, Docs, Documentation, donations, draco, dracopkg, dragonflybsd, dreamcast, dri, driver, drivers, drm, dsl, dst, dtrace, dvb, ec2, eclipse, eeepc, eeepca, ehci, ehsm, eifel, elf, em64t, Embedded, embedded, emips, emulate, encoding, envsys, eol, espresso, etcupdate, etherip, euca2ools, eucalyptus, eurobsdcon, eurosys, Events, exascale, ext3, f5, facebook, falken, fan, faq, fatbinary, features, fefe, ffs, filesystem, fileysstem, firefox, firewire, fireworks, flag, flash, flashsucks, flickr, flyer, fmslabs, force10, fortunes, fosdem, fpga, freebsd, freedarwin, freescale, freex, freshbsd, friendlyAam, friendlyarm, fritzbox, froscamp, fsck, fss, fstat, ftp, ftpd, fujitsu, fun, fundraising, funds, funny, fuse, fusion, g4u, g5, galaxy, games, gcc, gdb, gentoo, geode, getty, gimstix, git, gnome, google, google-soc, googlecomputeengine, gpio, gpl, gprs, gracetech, gre, groff, groupwise, growfs, grub, gumstix, guug, gzip, hackathon, hackbench, hal, hanoi, happabsd, hardware, Hardware, haze, hdaudio, heat, heimdal, hf6to4, hfblog, hfs, history, hosting, hotplug, hp, hp700, hpcarm, hpcsh, hpux, html, httpd, hubertf, hurd, i18n, i386, i386pkg, ia64, ian, ibm, ids, ieee, ifwatchd, igd, iij, image, images, imx233, imx7, information, init, initrd, install, intel, interix, internet2, interview, interviews, io, ioccc, iostat, ipbt, ipfilter, ipmi, ipplug, ipsec, ipv6, irbsd, irc, irix, iscsi, isdn, iso, isp, itojun, jail, jails, japanese, java, javascript, jetson, jibbed, jihbed, jobs, jokes, journaling, kame, kauth, kde, kerberos, kergis, kernel, keyboardcolemak, kirkwood, kitt, kmod, kolab, kvm, kylin, l10n, landisk, laptop, laptops, law, ld.so, ldap, lehmanns, lenovo, lfs, libc, license, licensing, linkedin, links, linksys, linux, linuxtag, live-cd, lkm, localtime, locate.updatedb, logfile, logging, logo, logos, lom, lte, lvm, m68k, macmini, macppc, macromedia, magicmouse, mahesha, mail, makefs, malo, mame, manpages, marvell, matlab, maus, max3232, mbr95, mbuf, mca, mdns, mediant, mediapack, meetbsd, mercedesbenz, mercurial, mesh, meshcube, mfs, mhonarc, microkernel, microsoft, midi, mini2440, miniroot, minix, mips, mirbsd, missile, mit, mixer, mobile-ip, modula3, modules, money, mouse, mp3, mpls, mprotect, mtftp, mult, multics, multilib, multimedia, music, mysql, named, nas, nasa, nat, ncode, ncq, ndis, nec, nemo, neo1973, netbook, netboot, netbsd, netbsd.se, nethack, nethence, netksb, netstat, netwalker, networking, neutrino, nforce, nfs, nis, npf, npwr, nroff, nslu2, nspluginwrapper, ntfs-3f, ntp, nullfs, numa, nvi, nvidia, nycbsdcon, office, ofppc, ohloh, olimex, olinuxino, olpc, onetbsd, openat, openbgpd, openblocks, openbsd, opencrypto, opendarwin, opengrok, openmoko, openoffice, openpam, openrisk, opensolaris, openssl, or1k, oracle, oreilly, oscon, osf1, osjb, paas, packages, pad, pae, pam, pan, panasonic, parallels, pascal, patch, patents, pax, paypal, pc532, pc98, pcc, pci, pdf, pegasos, penguin, performance, pexpect, pf, pfsync, pgx32, php, pie, pike, pinderkent, pkg_install, pkg_select, pkgin, pkglint, pkgmanager, pkgsrc, pkgsrc.se, pkgsrcCon, pkgsrccon, Platforms, plathome, pleiades, pocketsan, podcast, pofacs, politics, polls, polybsd, portability, posix, postinstall, power3, powernow, powerpc, powerpf, pppoe, precedence, preemption, prep, presentations, prezi, Products, products, proplib, protectdrive, proxy, ps, ps3, psp, psrset, pthread, ptp, ptyfs, Publications, puffs, puredarwin, pxe, qemu, qnx, qos, qt, quality-management, quine, quote, quotes, r-project, ra5370, radio, radiotap, raid, raidframe, rants, raptor, raq, raspberrypi, rc.d, readahead, realtime, record, refuse, reiserfs, Release, Releases, releases, releng, reports, resize, restore, ricoh, rijndael, rip, riscos, rng, roadmap, robopkg, robot, robots, roff, rootserver, rotfl, rox, rs323, rs6k, rss, ruby, rump, rzip, sa, safenet, san, sata, savin, sbsd, scampi, scheduler, scheduling, schmonz, sco, screen, script, sdf, sdtemp, secmodel, Security, security, sed, segvguard, seil, sendmail, serial, serveraptor, sfu, sge, sgi, sgimips, sh, sha2, shark, sharp, shisa, shutdown, sidekick, size, slackware, slashdot, slides, slit, smbus, smp, sockstat, soekris, softdep, softlayer, software, solaris, sony, sound, source, source-changes, spanish, sparc, sparc64, spider, spreadshirt, spz, squid, ssh, sshfs, ssp, statistics, stereostream, stickers, storage, stty, studybsd, subfile, sudbury, sudo, summit, sun, sun2, sun3, sunfire, sunpci, support, sus, suse, sushi, susv3, svn, swcrypto, symlinks, sysbench, sysctl, sysinst, sysjail, syslog, syspkg, systat, systrace, sysupdate, t-shirt, tabs, talks, tanenbaum, tape, tcp, tcp/ip, tcpdrop, tcpmux, tcsh, teamasa, tegra, teredo, termcap, terminfo, testdrive, testing, tetris, tex, TeXlive, thecus, theopengroup, thin-client, thinkgeek, thorpej, threads, time, time_t, timecounters, tip, tk1, tme, tmp, tmpfs, tnf, toaster, todo, toolchain, top, torvalds, toshiba, touchpanel, training, translation, tso, tty, ttyrec, tulip, tun, tuning, uboot, ucom, udf, ufs, ukfs, ums, unetbootin, unicos, unix, updating, upnp, uptime, usb, usenix, useradd, userconf, userfriendly, usermode, usl, utc, utf8, uucp, uvc, uvm, valgrind, vax, vcfe, vcr, veriexec, vesa, video, videos, virtex, virtualization, vm, vmware, vnd, vobb, voip, voltalinux, vpn, vpnc, vulab, w-zero3, wallpaper, wapbl, wargames, wasabi, webcam, webfwlog, wedges, wgt624v3, wiki, willcom, wimax, window, windows, winmodem, wireless, wizd, wlan, wordle, wpa, wscons, wstablet, X, x.org, x11, x2apic, xbox, xcast, xen, Xen, xfree, xfs, xgalaxy, xilinx, xkcd, xlockmore, xmms, xmp, xorg, xscale, youos, youtube, zaurus, zdump, zfs, zlib

'nuff. Grab the RSS-feed, index, or go back to my regular NetBSD page

Disclaimer: All opinion expressed here is purely my own. No responsibility is taken for anything.

Access count: 24157736
Copyright (c) Hubert Feyrer