Please join in and help test the code, and send your feedback to the lists. If no serious issues come up, the code will be merged within a week.

See Matt's posting to tech-kern for more details, inclusing diffs and links for amd64 and i386 GENERIC (+usbmp) kernels.

Further information on the state of the code - what is and what is not converted yet - can be found in the TODO.usbmp file.

[Tags: smp, usb]

``The NetBSD Foundation is pleased to announce completion of Multiprocessing Support for the port of its Open Source Operating System to the Xen hypervisor.

The NetBSD Fundation started the Xen MP project 8 month ago; the goal was to add SMP support to NetBSD/Xen domU kernels. This project has officially completed, and after a few bug fixes in the pmap(9) code it is now considered stable on both i386 and amd64. NetBSD 6.0 will ship with option MULTIPROCESSOR enabled by default for Xen domU kernels.

The availability of Xen MP support in NetBSD allows to run the NetBSD Open Source Operating Systems on a range of available infrastructure providers' systems. Amazon's Web Services with their Elastic Cloud Computing is a prominent examples here.

Xen is a virtualization software that enables several independent operating system instances ("domains") to run concurrently on the same computer hardware. The hardware is managed by the first domain (dom0), and further guest/user domains (domU) are spawned and managed by dom0. Operating systems available for running as dom0 and domU guests include Microsoft Windows, Solaris and Linux besides NetBSD.

NetBSD is a free, fast, secure, and highly portable Unix-like Open Source operating system. It is available for a wide range of platforms, from large-scale servers and powerful desktop systems to handheld and embedded devices. Its clean design and advanced features make it excellent for use in both production and research environments, and the source code is freely available under a business-friendly license. NetBSD is developed and supported by a large and vivid international community. Many applications are readily available through pkgsrc, the NetBSD Packages Collection.

NetBSD has been available for the Xen hypervisor since Xen 1 and NetBSD 2.0, released in 2004 , but until now only a single processor was supported in each NetBSD/xen domain.''

[Tags: amazon, ec2, smp, xen]

The article introduces real-time scheduling and the scheduling classes found in NetBSD 5.0, and gives an estimate on the response timeframe that can be expected for real-time applications. Setting scheduling policy and priority from a userland application is shown next, and programming examples for thread affinity, dynamic CPU sets and processor sets are shown. Besires C APIs, there are also a number or new commands in NetBSD 5.0 that can be used to control things from the command line, e.g. to define scheduling behaviour and manipulate processor sets. My favourite gem is the CPU used in the cpuctl(8) example, which is identified as "AMD Engineering Sample". :-)

[Tags: Articles, dmesg, posix, pthread, smp]

• Optimization for exec by using a cached copy of the file's exec header, and reducing locking-overhead by keeping locks instead of freeing them and the immediately re-locking them.

• The NetBSD kernel offers a few internal interfaces for allocating memory, depending on use of the memory, duration of use, size, etc. For some of the proper SMP-handling is important, and thus the number of CPUs has an impact on their performance. By adding caching, this can be mitigated to get linear behaviour, independent from the number of CPUs:

  

• The openat() and fstatat() system calls are not available in NetBSD yet, but other systems offer them, standards are about to pick them up, and ZFS assumes their presence. The system calls offer a way to way to specify a different directory to which relative paths are relative to, other than ".", by passing a file descriptor for that directory. Now there is a patch to add openat() and fstatat() to NetBSD.

• In theory, pipes are just a special case of (host-local) sockets, but it makes sense to keep a separate implementation for reasons of speed optimizations. NetBSD has the "PIPE_SOCKETPAIR" kernel option to force use of the socket code for pipes in order to reduce the memory footprint, but benchmarks reflect the performace hit. In order to improve the the situation, a number of improvements are under way, including better cache utilization and SMP-compliant memory allocation over homegrown memory management.

• Improved performance of exit(2) - this is important in environments with many short-running processes (think httpd, inetd).

• As a final step, freeing entries in the translation lookaside buffer (TLB) of the x86 (i386, amd64, xen) memory management unit AKA TLB shootdown were sped up to a point where TLB shootdown interrupts are 50% down during a system rebuild on an 8-cpu machine, and several million(!) calls page invalidation were optimized away, resulting in a 1% speed increase on the overall build.

• A partial(!) port of Sun's ZFS is also made available. It's not at the state where it can be used, but should be a good starting point for an experienced kernel hacker to continue working. See Andrew's mail to tech-kern for further directions.

SMP & audio: One area that hasn't been changed for moving towards fine-grained kernel locking was NetBSD's audio subsystem. As audio recording and playback is mostly done via interrupts, and as latency in those is critical, the audio subsystem was moved to the new interrupt handling system. The work can be found on the ad-audiomp branch, more information is available in Andrew's posting about the MP safe audio framework and drivers.

Benchmarking: Changing a system from inside out is a huge technical task. On the way, performance measurements and tuning are needed to make sure that the previous performance is still achieved while getting better performance in the desired development area. As a result, benchmarks results from Sun's libmicro benchmark suite were posted, which allow comparison not only against Linux and FreeBSD, but also between NetBSD-current and NetBSD 4.0, in order to identify if any bad effects were added. All performance tests were made on a machine with 8 CPUs, and the areas tested cover "small" (micro) areas like various system calls. Of course this doesn't lead to a 1:1 statement on how the systems will perform in a real-life scenario like e.g. in a database performance test, but it still help identifying problems and gives better hints where tuning can be done.

Another benchmark that was also made in that regard comes from Gregory McGarry, who has published performance measurements previously. This time, Gregory has run the lmbench 3.0 benchmark on recent NetBSD and FreeBSD systems as well as a number of previous NetBSD releases - useful for identifying performance degradation, too!

One other benchmark on dispatch latency run was made by Andrew Doran: on a machine that was (CPU-wise) loaded by some compile jobs, he started a two threads on a CPU that wasn't distracted by device interrupts, and measured how fast the scheduler reacted when one thread woke up the other one. The resulting graph shows that the scheduler handles the majority of requests in less than 10us - good enough for some realtime applications?

Kernel modules are another area that's under heavy change right now, and after recent changes to load modules from the bootloader and the kernel, the kernel build process was now changed so that pre-built kernel modules can be linked into a new kernel binary, resulting in a non-modular kernel. Eventually, this could mean that src/sys is built into separate modules, and that the (many) existing kernels that are present for each individual platform -- GENERIC, INSTALL is already gone, ALL, etc. etc. -- can be simply linked from pre-compiled modules, without recompiling things over again for each kernel. Of course the overal goal here is to speed up the system (and kernel!) build time, while maintaining maximum flexibility between modules and non-modular kernels.

With the progress in kernel modules, it is a question of time when the new kernel module handling supercedes the existing loadable kernel modules to such an extent that the latter will be completely removed from the system -- at least the latter was alredy proposed, but I'd prefer to see some documentation of the new system first. We'll see what comes first! (Documentation writers are always welcome! :-)

[Tags: benchmark, kernel, smp]

The story so far. Andrew Doran writes: `` The kernel synchronization model has been completely revamped since NetBSD 4.0, with the goal of making NetBSD a fully multithreaded, multiprocessor system with complete support for soft real-time applications.

Through NetBSD 4.0, the kernel used spinlocks and a per-CPU interrupt priority level (the spl(9) system) to provide mutual exclusion. These mechanisms did not lend themselves well to a multiprocessor environment supporting kernel preemption. Rules governing their use had been built up over many years of development, making them difficult to understand and use well. The use of thread based (lock) synchronization was limited and the available synchronization primitive (lockmgr) was inefficient and slow to execute.

In development branch that will becomple NetBSD 5.0, a new rich set of synchronization primitives and software tools have been developed to ease writing multithreaded kernel code that executes efficiently and safely on SMP systems. Some of these are:

• Thread-base adaptive mutexes. These are lightweight, exclusive locks that use threads as the focus of synchronization activity. Adaptive mutexes typically behave like spinlock, but under specific conditions an attempt to acquire an already held adaptive mutex may cause the acquring thread to sleep. Sleep activity occurs rarely. Busy-waiting is typically more efficient because mutex hold times are most often short. In contrast to pure spinlocks, a thread holding an adaptive mutex may be preempted in the kernel, which can allow for reduced latency where soft real-time application are in use on the system.

• Reader/writer locks. These are lightweight shared/exclusive locks that again use threads as the focus of synchronization activity. Their area of use is limited, most of it being in the file system code.

• CPU based spin mutexes, used mainly within the scheduler, device drivers and device driver support code. Pure spin mutexes are used when it is not safe, or impossible for, a thread to use a synchronization method that could block such as an adaptive mutex.

• Priority inheritance, implemented in support of soft real-time applications. Where a high priority thread is blocked waiting for a resource held by a lower priority thread, the kernel can temporarily "lend" a high priority level to the lower priority thread. This helps to ensure that the lower priority thread does no unduly obstruct the progress of the higher priority thread.

• Atomic operations. A full set of atomic operations implementing arithmetic and memory barrier operations has been provided. The atomic operations are available both in the kernel and to user applications, via the C library.

• Generic cross calls: a facility that allows one CPU or thread to easily make an arbitrary function call on any other CPUs in the system.

• The interrupt priority level interfaces (spl(9)) have long been used to block interrupts on a CPU-local basis. These interfaces have been simplified and streamlined to allow for code and algorithms that make use of low cost CPU-local synchronization. In addition, APIs are provided that allow detection of kernel preemption and allow the programmer to temporarily disable preemption across critical sections of code that cannot tolerate any interruption.

• "percpu" memory allocator: a simple memory allocator that provides arbitrary amounts of keyed storage. Allocations are replicated across all CPUs in the system and each CPU has its own private instance of any allocated object. Together, the cross call facility, atomic operations, spl(9) interfaces and percpu allocator make it easy to build lightweight, lock-free algorithms.

• Lockless memory allocators: the standard kernel memory allocators have been augmented with per-CPU caches which signficantly avoid costly synchronization overhead typically associated with allocation of memory on a multiprocessor system. ''

• New thread scheduler, named M2: it reduces the lock contention, and increases the thread affinity to avoid cache thrashing - this essentially improves the performance on SMP systems. M2 implements time-sharing class, and POSIX real-time extensions, used by soft real-time applications.

• Processor sets and affinity API provides possibility to bind the processes or threads to the specific CPU or group of CPUs. This allows applications to achieve better concurrency, CPU utilization, avoid cache thrashing and thus improve the performance on SMP systems.''

• A scheduler is responsible for distributing workdload on CPUs, and besides the 4BSD scheduler, a more modern "M2"-scheduler was recently added to NetBSD, see above. Parts of that scheduler were now suggested to be included in the general scheduling framework. That way, the 4BSD scheduler gets processor affinity (so threads / processes keep stuck to a single CPU and thus reduce cache misses when migrating between CPUs/cores).

  With other changes surrounding this, NetBSD-current beats FreeBSD 7.0 and all earlier NetBSD releases when running build.sh (i.e. compiling the whole system) on a 8-core machine. In the image, small values mean less time for the build, and are thus good. I find the results impressive. For more information, see Andrew's posting to tech-kern.

• Reader/writer locks are a locking primitive used to allow multiple readers, but to block them if one or more processes want to write to a ressource. Those locks are used in the NetBSD kernel, see the rwlock(9) manpage. In order to further optimize the performance of the rwlock primitives, a few optimizations were suggested by Andrew Doran which reduces the build time on an 8-cpu machine by 4%: ``There is less idle time during the build because the windows where a rwlock can be held but the owner is not running is reduced''.

• Another optimization was suggested which cuts down another 5% of the time for a complete system build via build.sh on an 8-core machine, this time by replacing a linear list of locks in the namei cache with a hash table for the locks. The namei cache helps to speed up translations from a path name to the corresponding vnodes, see namei(9).

• Following Wikipedia, Portability is ``the general characteristic of being readily transportable from one location to another'', and it's also a major goal of NetBSD. Things start to get interesting when looking into details, e.g. Wikipedia also states that ``Software is portable when the cost of porting it to a new platform is less than the cost of writing it from scratch. The lower the cost of porting software, relative to its implementation cost, the more portable it is said to be.'' So there's some room for interpretation when defining what is portable and what is not, and to what extent.

  Besides my essay on What makes an operating system portable, there was a posting to the netbsd-advocacy mailing-list that goes into a few details on NetBSD's current state of portability. The posting lists a number of reasons why the author considers NetBSD to be portable, including the low effort to start new projects, central maintenance in one source tree, and the efforts from machine-independent changes to all ports.

  After reading about people doing research on how to assess "security" of operating systems by counting number of exploits and how quick they are patched, I wonder if there are some metrics out there to also put "portability" into numbers.

• I've mentioned the mult project some time ago. In one of their latest recordings, there's also a interview with its creator, Kristaps Dzonsons, on it on BSDtalk, available in mp3 and ogg formats. Thanks to Mark Weinem for the hint!

• Following some discussion on NetBSD on the Freescale i.mx31 board, Matt Thomas has posted a dmesg output. Mentioned here for all the fans of dmesg pr0n. :-)

• To give new users hints on how to use NetBSD, Jeremy C. Reed has started a netbsd-tips fortune database. It's part of NetBSD-current and can be run from .login/.profile by running "fortune netbsd-tips". There's also a wiki page that allows easy submitting of new entries. Feel free to contribute your special NetBSD gems!

• NetBSD's handling of harddisks and file systems is pretty static right now - while one can add additional disks to a system, and even span them using RAIDframe and ccd(4), extending the filesystem on top of it is a problem. This is being mitigated by Juan Romero Pardines' port of growfs(8): ``I've just adapted growfs(8) from OpenBSD (they adapted the FreeBSD code), which is able to grow FFSv1 and FFSv2 filesystems.

  I tested growing a partition in FFSv1 and FFSv2 from 1GB to 4GB and the process was smooth (and fast); after this I ran 'fsck_ffs -yf /fs' and it found one error that was fixed correctly.'' For more information, including where to get the code and what to test, see Juan's posting.

  There were a few attempts to get logical volume management (LVM) onto NetBSD, which were not successful so far. This may change in the future, and when flexible handling of storage volumes, with growfs(8) will be useful to manage FFS/UFS file system sitting on top of them.

• Andrew Doran has continued his hacking to improve NetBSD on SMP machines, and he has posted about making the socket code and the Unix domain communication running fine-grained, and about speeding up device detection during booting by running device configuration in a number of concurrent kernel threads. If someone has actual numbers on boot time before/after that patch, please post them to the list!

• When needing sources for some Open Source package, I've used "make extract NO_DEPENDS=1" with pkgsrc in the past. It seems that was removed without further notice, and Obata Akio was kind enough to point out that this can be done now by using SKIP_DEPENDS=yes. Mmm, interface stability...

• Last but not least a note from the "products based on NetBSD" department: Saitoh Masanobu from IIJ, Japan, has notified us that the SEIL/X series that IIJ unveils at AsiaBSDCon 2008 is based on NetBSD. There's a brochure on SEIL/X that mentions a long list of features supported by the machine, including all state of chw art in routing, bridging, VPN, firewalling, quality of service and more. This is made possible by the "SEIL Engine", a software architecture that's based on NetBSD that allows porting the application stack to a number of hardware platforms easily, while offering flexibility to add support for custom hardware and software modules:

  

  For more information on the SEIL Engine, see IIJ/SEIL's homepage (Japanese). and PDF brochude (English).

  Also, for some impression of the SEIL/X machine on the geek level, there's dmesg output of the machine available.

• Support C99 complex arithmetic was added by importing the "cephes" math library
• POSIX Message queues were added
• bozohttpd was added as httpd.
• the x86 bootloader now reads /boot.cfg to configure banner text, console device, timeout etc. - see boot.cfg(5)
• ifconfig(8) now has a "list scan" command to scan for access points
• SMP (multiprocessor) support is now enabled in i386 and amd64 GENERIC kernels
• Processor-sets, affinity and POSIX real-time extensions were added, along with the schedctl(8) program to control scheduling of processes and threads.
• systrace was removed, due to security concerns
• the refuse-based Internet Access Node file system was committed, which provides a filesystem interface to FTP and HTTP, similar to the old alex file system, see http://mail-index.netbsd.org/source-changes/2007/08/28/0081.html
• LKMs don't care for options MULTIPROCESSOR and LOCKDEBUG, i.e. it's easier to reuse LKMs between debugging/SMP and non-debugging/SMP kernels now.
• PCC, the Portable C Compiler that originates in the very beginnings of Unix, was added to NetBSD. The idea is that it is used as alternative to the GNU C Compiler in the long run.
• In addition to the iSCSI target (server) code that is already in NetBSD 4.0, there'a also a refuse-based iSCSI initiator (client) now, see http://mail-index.netbsd.org/source-changes/2007/11/08/0038.html

• Many driver updates and new drivers, see your nearest GENERIC kernel config file
• Many security updates, see list of security advisories
• Many 3rd software packages that NetBSD ships with were updated: ipsec-tools (racoon), GCC 4.1, Automated Testing Framework 0.4, OpenSSH 4.7, wpa_supplicant and hostapd 0.6.2, OpenPAM Hydrangea

So much on this subject for now. If someone's willing to help out with continuing Mark Kirby's NetBSD CVS Digest either using his software-setup or by simply reading the list and writing a monthly/weekly digest of the "interesting" changes, I'd appreciate this very much. Put me on CC: for your postings! :)

[Tags: alex, bozohttpd, c99, cephes, cvs, cvs-digest, digest, ian, iscsi, lkm, pcc, refuse, smp, systrace]

• Make the virtual memory (VM) system and trap handling (e.g. for page faults) MP safe
• Make the file system framework MP safe

It's approaching stability and I'd like to invite anyone who is interested to test.''

Please see Andrew's mail for known problems, and instructions on how to test. People that are interested in the details, and that want to help out Portmasters working on the machine dependent (MD) parts can find more information in another mail.

[Tags: smp]

* NetBSD hires Andrew Doran for full-time SMP development

The NetBSD Foundation announces that it has hired Andrew Doran to work full-time on improving symmetrical multi-processing (SMP) in NetBSD. This work is made possible through a generous donation by Force10 Networks and internal funding by The NetBSD Foundation.

Andrew Doran is an independent, Dublin based Unix systems consultant with special interest in building scalable systems. He has been a NetBSD developer since 1999 and is currently working on the transition from a big-lock SMP implementation to a fine-grained model, which allows multiple CPUs to execute code in kernel context simultaneously. Hiring Andrew full-time will boost work in this area, with the final result of a SMP implementation that is ready for tomorrow's multi-core-CPUs.

Force10 Networks is a pioneer in building and securing reliable networks. The Force10 TeraScale E-Series family of switch/routers and the recently introduced C300 resilient switch rely on the NetBSD-based FTOS to deliver the reliability, network control and scalability required to build application ready networks.

The funding will be for two months initially, and The NetBSD Foundation would like to extend this period. As a non-profit organization with no fixed financial backing, this is not possible without donations from individuals and companies. To realize our plans, $10k would be needed short term, with a goal of raising $15k or more eventually.

If you would like to donate to the ongoing effort of keeping NetBSD the most portable Open Source operating system, please consider supporting us! Donations via Paypal can be sent to paypal@NetBSD.org, or visit our donations page at http://www.NetBSD.org/donations/ for more details. Donations are tax deductible in the United States.

More information about the NetBSD operating system is available at http://www.NetBSD.org/, information about The NetBSD Foundation is at http://www.NetBSD.org/foundation/. More information about Andrew Doran's SMP work is available on his webpage at http://www.NetBSD.org/~ad/.

Information about Force10 Networks can be found at http://www.force10networks.com/.

[Tags: donations, force10, smp]

• A new set of synchronization primitives in the kernel designed to make programming for multiprocessor systems easier and more efficient: mutexes, reader / writer locks, condition variables, sleep queues and MI memory barrier operations.
• A number of underlying kernel facilties have been made 'multiprocessor safe' including the scheduler, ktrace and the general purpose method of kernel synchronisation: sleep & wakeup
• Some application facilities have been made MP safe and can now run without the "big lock" on multiprocessor systems, including signalling, SysV messaging, and system calls that inspect process state, for example: wait().
• The number of system calls that will run without the big lock went from 1 up to 56, with more in the pipeline. For workloads that are fork intensive and make heavy use of signals this will show a small yet quantifiable benefit on multi-way systems.
• The branch introduces a new 1:1 threading model that allows multithreaded applications to take advantage of all available CPUs in a multi-way system. The scheduler activations implementation used from NetBSD 2.0 through NetBSD 4.0 provides execellent performance on single CPU systems, but restricts any instance of a threaded application to a single CPU in the system. Given that multicore and multi-CPU systems are increasingly commonplace and that single threaded CPUs are rapidly disappearing from the market, we made the decision to move to a new threading model, on the basis that providing increased concurrency is now the most important factor in ensuring good performance for threaded workloads.
• Those following source-changes already know what that new 1:1 threading model means for the scheduler-activations based m:n threading model: it's gone.

For NetBSD, there was (and is, to spoil the fun!) no clear roadmap on how to move from here to there, but there is an interesting discussion on tech-kern that circles around steps that can be taken to break up the big-lock approach. A central problem seems to me to move from the System Priority Levels, which are used in BSD systems to prevent low-level code paths from interrupting higher-priority code, to a solution that allows multiple CPUs to access the same code paths by setting proper locks instead, but without stepping on each others toes and without losing performance on the other side.

For all the details, see the discussion around the "splx() optimization" thread on tech-kern (started in may, and is still going now in june; too bad mail-index doesn't index threads :-/).

[Tags: netbsd, smp]