Platform support for eMIPS is the first part of Microsoft's contribution. The second part includes the hardware accelerator framework and will be proposed on tech-kern soon.''

Update: The NetBSD Blog has an entry with more details, including a screenshot with boot output. Oh, and there's a NetBSD/emips port page, too!

Update #2: Antti was so kind to point me at the paper published by the people behind NetBSD/emips, titled An online scheduler for hardware accelerators on general- purpose operating systems: ``This paper presents an online scheduling algorithm for hardware accelerators and its implementation on the NetBSD operating system. The scheduler uses the current performance characteristics of the accelerators to select which accelerators to load and unload. [...] The NetBSD modifications consist primarily in loadable kernel modules, with minimal changes to the operating system itself.''. Secion 7 goes into details WRT NetBSD and the (LKM based) kernel modules..

[Tags: emips, lkm, microsoft, mips]

Today, I've found time to install NetBSD-current/i386, and configure things that I use here - /kern, /proc, and some NFS, in addition to a local disk. Now, looking at the list of loaded kernel modules reveals:

% modstat 
NAME            CLASS   SOURCE  REFS    SIZE    REQUIRES
compat          misc    builtin 0       -       -
coredump        misc    filesys 1       3067    -
exec_elf32      misc    filesys 0       7225    coredump
exec_script     misc    filesys 0       1187    -
ffs             vfs     boot    0       166292  -
kernfs          vfs     filesys 0       11131   -
nfs             vfs     filesys 0       145345  -
procfs          vfs     filesys 0       28068   -
ptyfs           vfs     filesys 0       8975    - 

Another interesting module is "coredump", which is loaded by the module to execure 32bit ELF programs, exec_elf32. This is an example of module dependencies, and again no manual intervention was needed.

So what modules are there? First, let's remember that kernel modules are object code that implements facilities for the running kernel, and which interfaces closely with the running kernel. As such, they need to match the kernel version, ideally. When one of the kernel's API or ABI interfaces changes, it's best to rebuild all modules. For NetBSD, the kernel's version is bumped e.g. from 5.99.15 to 5.99.16 for such an interface change, which helps tracking those changes.

Back to the question of what modules are there. Now that we know kernel modules are closely tied to the version of the kernel (which still is in the file /netbsd, btw), associated modules -- for the example of NetBSD/i386 5.99.15 -- can be found in /stand/i386/5.99.15/modules:

% cd /stand/i386/5.99.15/modules
% ls -F
accf_dataready/     drm/                lfs/                ptyfs/
accf_httpready/     efs/                mfs/                puffs/
adosfs/             exec_aout/          miniroot/           putter/
aio/                exec_elf32/         mqueue/             radeondrm/
azalia/             exec_script/        msdos/              smbfs/
cd9660/             ext2fs/             nfs/                sysvbfs/
coda/               fdesc/              nfsserver/          tmpfs/
coda5/              ffs/                nilfs/              tprof/
compat/             filecore/           ntfs/               tprof_pmi/
compat_freebsd/     fss/                null/               udf/
compat_ibcs2/       hfs/                overlay/            umap/
compat_linux/       i915drm/            portal/             union/
compat_ossaudio/    kernfs/             ppp_bsdcomp/        vnd/
compat_svr4/        ksem/               ppp_deflate/
coredump/           layerfs/            procfs/

% ls */*.kmod
accf_dataready/accf_dataready.kmod      layerfs/layerfs.kmod
accf_httpready/accf_httpready.kmod      lfs/lfs.kmod
adosfs/adosfs.kmod                      mfs/mfs.kmod
aio/aio.kmod                            miniroot/miniroot.kmod
azalia/azalia.kmod                      mqueue/mqueue.kmod
cd9660/cd9660.kmod                      msdos/msdos.kmod
coda/coda.kmod                          nfs/nfs.kmod
coda5/coda5.kmod                        nfsserver/nfsserver.kmod
compat/compat.kmod                      nilfs/nilfs.kmod
compat_freebsd/compat_freebsd.kmod      ntfs/ntfs.kmod
compat_ibcs2/compat_ibcs2.kmod          null/null.kmod
compat_linux/compat_linux.kmod          overlay/overlay.kmod
compat_ossaudio/compat_ossaudio.kmod    portal/portal.kmod
compat_svr4/compat_svr4.kmod            ppp_bsdcomp/ppp_bsdcomp.kmod
coredump/coredump.kmod                  ppp_deflate/ppp_deflate.kmod
drm/drm.kmod                            procfs/procfs.kmod
efs/efs.kmod                            ptyfs/ptyfs.kmod
exec_aout/exec_aout.kmod                puffs/puffs.kmod
exec_elf32/exec_elf32.kmod              putter/putter.kmod
exec_script/exec_script.kmod            radeondrm/radeondrm.kmod
ext2fs/ext2fs.kmod                      smbfs/smbfs.kmod
fdesc/fdesc.kmod                        sysvbfs/sysvbfs.kmod
ffs/ffs.kmod                            tmpfs/tmpfs.kmod
filecore/filecore.kmod                  tprof/tprof.kmod
fss/fss.kmod                            tprof_pmi/tprof_pmi.kmod
hfs/hfs.kmod                            udf/udf.kmod
i915drm/i915drm.kmod                    umap/umap.kmod
kernfs/kernfs.kmod                      union/union.kmod
ksem/ksem.kmod                          vnd/vnd.kmod

% find . -type f -print | wc -l
      58 

P.S.: I've seen one confusion WRT systems that use kernel modules to whatever extent, as they shrink the size of the actual kernel binary: Even with kernel modules, an operating system is still a monolithic kernel: The modules are tied in closely into the system once loaded, ending in a monolithic system. In contrast, a "microkernel" is something very different, and it doesn't have anything to do with kernel modules. :-)

[Tags: kernel, lkm, modules]

• In preparation of the NetBSD 5.0 release, a lot of documentation updates were made, esp. in the release notes. Also, many manual pages were added to the system, documenting existing userland tools, library, system and internal interfaces.

• Following some re-organization of binary packages on ftp.NetBSD.org some time ago, the official URLs are now:
  • ftp://ftp.NetBSD.org/pub/pkgsrc/packages/NetBSD/{MACHINE}/{VERSION}/All
  • ftp://ftp.NetBSD.org/pub/NetBSD/packages/current-packages/NetBSD/{MACHINE}/{VERSION}/All
  Both should have the same results, the latter is more safe on mirrors that don't carry /pub/pkgsrc. Adjust your PKG_PATHs!

• Syntax for /etc/rc.conf's ifconfig_xxN variables and /etc/ifconfig.xxN was changed to also allow line breaks via ';'s. This allows something like ifconfig_wi0="ssid 'my network'; dhcp"

• Martin Schuette's work on syslogNG from Google Summer of Code 2008 is now available in NetBSD's syslog

• X.org integration is advancing in big steps. It's on by default on a number of platforms (including alpha, i386, macppc, shark, sparc and sparc64), and instead of using the (now obsolete) MKXORG build variable it can be build with "build.sh -x".

• Old-style LKMs are dead, welcome to the new module framework! (XXX Documentation???) In the process, more and more kernel subsystems are being changed to be loadable as a module, e.g. POSIX AIO and semaphores, File System Snapshots, emulations, exec formats, coredump, NFS client and server, http and data accept filters, ppp compressors, and others.

  Hooks into UVM have been added to unload unused kernel modules if memory is scarce.

• MAKEVERBOSE now has two new levels, 3 and 4. The complete list is now:
  • 0 Minimal output ("quiet")
  • 1 Describe what is occurring
  • 2 Describe what is occurring and echo the actual command
  • 3 Ignore the effect of the "@" prefix in make commands
  • 4 Trace shell commands using the shell's -x flag
  The default remains MAKEVERBOSE=2, you can also set this via build.sh's -N switch.

• A POSIX conformant tabs(1) utility was added

• The haad-dm branch was merged to NetBSD-current. This adds Logical Volumen Management (LVM) functionality to the base NetBSD system. It uses Linux LVM2 tools and our BSD licensed device-mapper driver.

• The wrstuden-revivesa branch was merged into NetBSD-current, bringing Scheduler Activation based threading back to NetBSD, and giving NetBSD 5.0 and up both SA and 1:1 threads.

• Support for the ARM-based Cortina Systems SL3516 eval board was added to NetBSD/evbarm

• patch(1) got a major overhaul, based on DragonflyBSD and OpenBSD. There's better detection of double applied patches, rejected diffs remain in unified diff format, and and less limitation e.g. on line length.

• pxeboot now understands boot.cfg

• Boot CD ISO creation has been greatly overhauled, accomodating changes in boot.cfg, and moving away from a ramdisk-based system to using a file system on the cd-rom, which helps reduce RAM usage. Also, the GENERIC kernel can be used there.

• makefs(8)'s ISO-9660 (cdrom) support was enhanced to write RISC OS data. This allows to make bootable CDs for acorn{26,32} directly, without copying the bootloader to a native file system.

• The christos-time_t branch has been merged into NetBSD-current. This gives 64bit time_t and dev_t types (no more year 2038-problem!!!).

  Many related places like timeval and timespec were adjusted, kernel and userland APIs were touched, and shared library major versions (including libc) were bumped for this fairly exhaustive change.

  See src/UPDATING's entry on 20090110 for the full update path!

• New/updated drivers:
  • jme(4) for JMicron Technologies JME250 Gigabit Ethernet and JME260 Fast Ethernet PCI Express controllers
  • u3g(4) provides better support for 3G datacards than ugensa
  • dbcool(4) for dbCool(tm) family of Thermal Monitor and Fan Controller
  • ataraid(4) now supports Intel MatrixRAID and JMicron RAID
  • bwi(4) for Broadcom BCM4302 wlan controllers, otherwise known as Airport Extreme
  • alipm(4) for the Acer Labs M7101 Power Manage- ment controller
  • admtemp(4) for the Analog Devices ADM1021, Analog Devices ADM1023, Analog Devices ADM1032, Genesys Logic GL523SM, Global Mixed-mode Technology G781, Maxim 1617, and Xeon embedded temperature sensors
  • ipw(4),iwi(4),wpi(4),iwn(4): We ship the firmware now, but users have to accept the Intel license manually by setting sysctls like hw.ipw.accept_eula=1. The latter is also offered by sysinst.
  • nsp(4) adds support for the NSP2000 cryptographic processor which does crypto, hashing and arbitrary precision arithmetics in hardware, and which hooks into the opencrypto(9) interface.
  • pseye(4) makes the Sony Playstation Eye USB webcam usable with the new video(4) framework
  • ath(4) now uses the recently-released source-based version of the Atheros HAL, no more binary blob!

• Google Summer of Code is over for some time, but apparantly no final report has emerged so far (shame!). Still, a number of individual status reports came by on the official lists:
  • Zach Wegner's sysinst project
  • Adam Burkepile's subfile project (PDF)
  • Martin Schütte's syslogd project
  • Jeremy Morse's DVB project
  • Mark Dopiera's hurd translators project
  • Adam Hamsik's lvm project

  I know of at least one other project (uvc) that has completed but that I didn't see a report here - maybe I've missed it. Anyways, GSoC was another big success this year. Thanks, Google!

• Speaking of Adam Hamsik and Logical Volume Management (LVM), Adam has continued his work in that are, and he has written a device mapping library that interacts with his kernel driver. This allows to interact with his GSoC project without using any GPL code! See Adam's posting for more details.

• Force 10 Networks, producer of 10gbit switches that use an operating system based on NetBSD, have added a new feature as part of their FTOS operating system: VirtualView, which provides virtualization of Force 10 based equipment. From the xchange article: ``Force10 Networks Inc. this week introduced VirtualView software for benchmaking, troubleshooting and managing virtualized environments based on Force10 gear.''

  More information is available from the Force 10 Networks homepage, plus in articles by fibresystems.org, light reading, Zycko, and SmartBrief.

• Following the latest hype in portable computers, NetBSD has created a netbook page that intends to list models and the extent to which they are supported. Your contributions are most welcome here! (Contact me for sending updates and hardware :-)

• Zafer Aydogan has made RSS feeds available for CVS commits to single files - see his mail to netbsd-users for more details.

• New security advisory were released that I've missed in my last update:
  • NetBSD Security Advisory 2008-011: ICMPv6 MLD query
  • NetBSD Security Advisory 2008-012: Denial of service issues in racoon(8)

• A project that's been ongoing for quite some time is the move from "old-school" loadable kernel modules (LKMs) to new-style kernel modules. Important changes include the fact that modules can be either linked into the kernel at build time, or loaded into the kernel at runtime from the same file. Also, the bootloader was modified to load modules after the kernel, e.g. for a RAM-disk like the one that is used by the INSTALL kernel.

  In the same line, some parts are starting to be moved out of the GENERIC kernel, and installed as modules that can be loaded by the new framework then. The start is made by POSIX semaphores as a first step and proof-of concept, even if some details are still under hot debate, e.g. what the file system layout for modules is, and if the belong to the kernel and its build process, or to the userland.

• While talking about splitting the kernel into modules, Antti Kantee has continued his work to move parts of the kernel into userspace, in particular running file system code as userland in his RUMP, and puffs and (Re)FUSE works. The idea is to provide the interfaces that file systems need in the userland, and the result is that you can run code that used to run inside the kernel in userland now.

  Another subsystem running in the kernel that could be moved to userland by providing appropriate interfaces with the rest of the kernel is the network stack, and Antti has moved just that to the userland. See Antti's mail to tech-net@ for more information on this impressive work.

• NetBSD has shipped XFree in previous releases, and people who wanted to use X.org had to install it from pkgsrc. That's all fine, but to get a modern X, one had to compile things, as no precompiled binary packages are made available for many platforms. This is changing now, and NetBSD is getting X.org integrated via a reachover infrastructure which is also enabled for crosscompiling.

  The "user interface" for this is still in flux, but after some detour ("build.sh -V MKXORG=yes", without -x), "build.sh -x" now builds whatever X is considered the default for the platform. Some platforms already default to use X.org as X, and more will come, as changes that were made to NetBSD's copy of XFree are adopted to X.org.

  Platforms that use X.org by default now are macppc (see here and here), sparc sparc64, shark, amd64 and i386.

  As X.org is at Revision 7 now, it's installed in /usr/X11R7, which will lead to a lot of interesting effects. pkgsrc is already prepared for the new layout, but there are still many minor details that will need adjusting to the new directory. If you find one, post your patches to tech-x11.

• Besides the GNU C compiler, there's the BSD-licensed Portable C Compiler around for some time now. It doesn't offer the same support as its GNU cousin yet, but this may change now: The BSD Fund is currently doing a fund drive to get money to enhance PCC. The goal is to raise $12,000US to improve support for core compiler functionality as well as support for C99, gcc compatibility and the amd64 architecture. See the project page for further details.

• The NetBSD 5.0 release cycle has started! There's a netbsd-5 branch in CVS, daily binaries are available for testing, and some of the highlights of the upcoming release include file system journalling for FFS via WAPBL, and X.org.

  To help testing of NetBSD on Cobalt machines, Izumi Tsutsui has made a NetBSD 5.0_BETA based version of the Cobalt restore CD available. Enjoy!

• As the final point today, a word on NUMA support from Christoph Egger. Non-Uniform Memory Access is needed in massive parallel systems where some nodes have RAM more tightly associated than others, where the RAM is further away, resulting in different access times for different regions of memory. In order to support this, Christop Egger has made first steps.

  His example implementation uses information from ACPI, and shows some heavy dmesg-pr0n from a 16-core machine with four sockets. Yumm!

• Kernel Preemption: Andrew Doran has continued his work towards fine-grained locking, and he has proposed a patch to implement kernel preemption, i.e. that in a realtime environment, high-priority processes can interrupt system calls running inside the kernel.

  Handling the Floating Point Unit (FPU) was added later on -- the FPU needs special attention as saving and restoring is expensive, and doesn't need to be done in many cases. But if a program uses it, care must be taken to handle the case. The exact handling is explained by Christoph Egger.

  While there, Christoph also outlined the roadmap for getting realtime support in NetBSD - there are still a number of bits missing, but being able to preempt the kernel is a good first step!

• Fine-grained socket locking: In order to allow fine-grained locking (instead of blocking all other processes from entering the kernel, as is done in the "biglock" SMP approach), many kernel subsystems need to be changed. The socket system is the core part of interprocess communication, and Andrew Doran has changed it to use fine-grained locking now.

  In that context, the question of what code still runs with the biglock held, and Andrew gave an overview where more work is needed: some file systems (lfs, ext2fs, nfs), most of the drivers, protocols like TCP/IP, Veriexec, and some machine-dependent parts.

  Veriexec-Hacker Brett Lymn added details on the status of Veriexec with respect to its transition towards fine-grained locking.

• Kernel modules and ramdisk: A change in kernel modules was proposed some time ago, and Andrew Doran has used this scheme now to unify the way many ports handle the install media: There, the kernel loaded contains a ramdisk (miniroot) image inside the kernel, which is then used as root-filesystem for the kernel, containing the install tools.

  In order to split things and eventually use a stock GENERIC kernel for both running and installing, Andy has changed the x86 boot process to load the miniroot as a kernel module.

  When booting it may be useful to select one of several ramdisks: one for installing, and one for resuing the system, For this, the recently introduced boot.cfg file was extended to handle kernel modules in the boot menu.

  Izumi Tsutsui has made an ISO with all changes for testing available.

• Device File System (devfs): Another area of the kernel where a lot of work is currently being done by Matt Fleming is NetBSD's device driver infrastructure, esp. under aspects of dynamic attaching, detaching, and suspending (power management!). To talk to the various drivers, device nodes in the /dev directory are kept right now, but those are static and need to be updated when a new driver is added. Matt is working on a Device Filesystem (devfs) that dynamically created /dev from the list of devices inside the kernel. The fileysstem will also handle dynamic creation and deletion of nodes, and as an important case it will also keep permissions across reboots, if someone changes permissions manually.

  The work is at a very mature point right now and needs some testing - see Matt's mail to the tech-kern list for more information!

• Testing driver attachment: While talking about testing of device drivers, David Young has reminded driver developers to test individual drivers' detachment and re-attachment, suspension and resumption after changes. He has also posted a how-to for those tests, using drvctl(8). (The manpage needs some updating, sorry -- UTSL :-)

• Support C99 complex arithmetic was added by importing the "cephes" math library
• POSIX Message queues were added
• bozohttpd was added as httpd.
• the x86 bootloader now reads /boot.cfg to configure banner text, console device, timeout etc. - see boot.cfg(5)
• ifconfig(8) now has a "list scan" command to scan for access points
• SMP (multiprocessor) support is now enabled in i386 and amd64 GENERIC kernels
• Processor-sets, affinity and POSIX real-time extensions were added, along with the schedctl(8) program to control scheduling of processes and threads.
• systrace was removed, due to security concerns
• the refuse-based Internet Access Node file system was committed, which provides a filesystem interface to FTP and HTTP, similar to the old alex file system, see http://mail-index.netbsd.org/source-changes/2007/08/28/0081.html
• LKMs don't care for options MULTIPROCESSOR and LOCKDEBUG, i.e. it's easier to reuse LKMs between debugging/SMP and non-debugging/SMP kernels now.
• PCC, the Portable C Compiler that originates in the very beginnings of Unix, was added to NetBSD. The idea is that it is used as alternative to the GNU C Compiler in the long run.
• In addition to the iSCSI target (server) code that is already in NetBSD 4.0, there'a also a refuse-based iSCSI initiator (client) now, see http://mail-index.netbsd.org/source-changes/2007/11/08/0038.html

• Many driver updates and new drivers, see your nearest GENERIC kernel config file
• Many security updates, see list of security advisories
• Many 3rd software packages that NetBSD ships with were updated: ipsec-tools (racoon), GCC 4.1, Automated Testing Framework 0.4, OpenSSH 4.7, wpa_supplicant and hostapd 0.6.2, OpenPAM Hydrangea

So much on this subject for now. If someone's willing to help out with continuing Mark Kirby's NetBSD CVS Digest either using his software-setup or by simply reading the list and writing a monthly/weekly digest of the "interesting" changes, I'd appreciate this very much. Put me on CC: for your postings! :)

[Tags: alex, bozohttpd, c99, cephes, cvs, cvs-digest, digest, ian, iscsi, lkm, pcc, refuse, smp, systrace]

• an in-kernel loader/linker so there's no need to rely on running the ld(1), and rely on have a working userland, thus
• module dependencies so that one module can request to load other modules automatically
• support to load modules from the boot loader and provide them to the kernel
• use the same code for kernel modules and in-kernel use, so things that are currently used inside the kernel can be moved to a module easily, without changes in code.