hubertf's NetBSD Blog
Send interesting links to hubert at feyrer dot de!
 
[20170421] g4u 2.6beta2 has been released - Happy 18th Birthday, g4u!
Just right in time for its 18th birthday, I have released the 2nd beta version for g4u 2.6 (2.6beta2). It took some time to get to this point, and I want to move to 2.6 soon - please take your time to test and get back to me soon, as I want to push out g4u version 2.6 in june 2017.

g4u ("ghosting for unix") is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common setup on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported.

For more information, see http://www.feyrer.de/g4u/.

Changes in 2.6beta2 include:

  • Make this build with NetBSD-current sources as of 2017-04-17, binaries were cross-compiled from Mac OS X 10.10
  • Go back to keeping the disk image inside the kernel as ramdisk, do not load it as separate module. Less error prone, and allows to boot the g4u (NetBSD) kernel from a single file e.g. via PXE (Testing and documentation updates welcome!)
  • Actually DO provide the g4u (NetBSD) kernel with the embedded g4u disk image from now on, as separate file, g4u-kernel.gz
  • Put all object files into one object directory. This may need more cleanup in the future. Feedback from people building g4u welcome!
  • Disable verbose device messages on Microchannel (MCA) machines
  • New drivers:
    • Toshiba Dynabook hotkeys
    • Intel S1200,C2000 (non-pch) SMBus storage controller
    • Non-Volatile Memory Express (NVMe) storage controllers and devices
    • Intel Wireless WiFi Link 7xxx PCI network
    • Intel 8259x 10 gigabit PCI network
    • Realtek 8188CE/8192CE 802.11b/g/n PCI network
    • VMware VMXNET3
    • Marvell Libertas PCMCIA network
    • ASIX AX88178a/AX88179 based USB network adapters
    • Realtek RTS5209/RTS5229 Card Reader
  • Happy 18th Birthday, g4u!
Download links:
  • The g4u 2.6beta2 floppy images (zipped/ uncompressed floppy one, floppy two, floppy three, and floppy four)
  • The g4u 2.6beta2 ISO CD image (zipped/uncompressed)
  • The g4u/NetBSD kernel for PXE boot (gzipped)
  • The g4u 2.6beta2 source code
  • Some md5 checksums:
    MD5 (g4u-2.6beta2-1.fs) = 8214c1ded55e78aed2f34d3e318dc75d
    MD5 (g4u-2.6beta2-2.fs) = 8b55f3fe6a4806c9a65a3d176463e1bc
    MD5 (g4u-2.6beta2-3.fs) = 0f9629353ddd470d17c5b4fd4ec7fd8e
    MD5 (g4u-2.6beta2-4.fs) = c25a955e5d876d64943fd03f957ed4c9
    MD5 (g4u-2.6beta2.iso) = acabb0fb73d4a2f9062fb070b4ddd825
    MD5 (g4u-2.6beta2.tgz) = f8387d5bf146f98dda040dfcc1c53e44
    MD5 (g4u-2.6beta2.fs.zip) = 8ef324c71ae7b000708e69019e55ef4e
    MD5 (g4u-2.6beta2.iso.zip) = 1f4bd37784405350ee3e8e6c75a6b6b6
    


[Tags: , ]


[20170408] Update on NetBSD and Google's Summer of Code 2017: student application period is over, ranking is in progress
Thanks to all students who have submitted project proposals! The student application period is now over, and we will have a look at the proposed projects in detail. From that, we will make a list with our top priorities (most-wanted / most-promising first), and give that list to Google. Depending on the overall amount of all proposals to all projects, Google will then assign slots of actually sponsored projects.

The announcement of this will happen on May 4th 2017 - stay tuned!

[Tags: ]



[20170408] Let's get meta: an interview with me (hubertf) about my NetBSD blog
BSD Magazine has approached me about my (this!) NetBSD blog, and in their section that introduces "Unix" blogs, they wanted an interview with me. And here we are!

The article starts out with printing a blog article of my choice, which is from the recent series on the NetBSD scheduler. This is followed by the actual interview with yours truly, asking about myself, how i started with blogging in general, why I do it and my best and worst posts. Things go to a wider focus then, musing about why Unix is the best since sliced bread, what role NetBSD plays today with its 7.1 release and in the future, and what my personal plans are in that regard. See pages 44-48 of March 2017's BSD Magazine (registration required for PDF download!) for the full text.

[Tags: , ]



[20170404] pkgsrc-2017Q1 released
NetBSD's package collection "pkgsrc" comes with quarterly stable releases, and the one for 2017Q1 has been Released. See the posting to netbsd-announce for all the defails. This includes:
``
The pkgsrc developers are proud to announce the 54th quarterly release
of pkgsrc, the cross-platform packaging system.  pkgsrc is available
with more than 17500 packages, running on 23 separate platforms; more
information on pkgsrc itself is available at https://www.pkgsrc.org/
A neutral overview can be found at https://www.openhub.net/p/pkgsrc

For the 2017Q1 release we welcome the following notable package
additions and changes to the pkgsrc collection:

  - python 3.6
  - Nextcloud 11
  - firefox 45.8.0 and 52.0.1
  - gradle 3.4
  - pkg_comp 2.0
  - qmail 1.03nb24 binary packages work, supporting common use cases
  - many additional Python, Perl and Ruby modules
  - many additional TeX packages

The default version of Apache has been changed to 2.4 (from 2.2); set
PKG_APACHE_DEFAULT=apache22 in mk.conf to stay with 2.2.

Package removals include gcc 4.5, 4.6, and 4.7 and Xen 3.1, 3.3 and 4.1;
these are old and notable only because of their stature.

The following infrastructure changes were introduced:
  - mk/curses.mk enabled packages to depend on curses without specifying
      a particular version

In total, 192 packages were added, 25 packages were removed, and 1,458
package updates were processed since the pkgsrc-2016Q4 release.

Instructions on using the binary package manager can be found at
http://pkgin.net, and pkgsrc itself can be retrieved from
https://github.com/jsonn/pkgsrc or via cvs or tar file -- see
https://www.netbsd.org/docs/pkgsrc/getting.html.  The branch name
for the 2017Q1 branch is "pkgsrc-2017Q1".''
Now off to play the update game on all the systems that I use pkgsrc on, including Mac OS X and Debian Linux.

[Tags: , ]


[20170325] NetBSD and Google's Summer of Code: Students can apply now
This year's Summer of Code has reached the phase where interested students can hand in project applications. Deadline for submissions is April 3rd 2017, so hurry up to get in line!

We have a list of project ideas available.

Please feel free to discuss project ideas beforehand on our mailing lists. When discussing project ideas, make sure you had a look at our guidelines, and answer as many questions as possible.

[Tags: ]



[20170228] NetBSD will be in Google's Summer of Code 2017
NetBSD will be in the 2017 Google Summer of Code as one of the mentoring organizations. Right now, NetBSD projects for this summer are under discussion internally, but interested students are again welcome to start looking and participating, discussing project ideas of existing or new projects on our mailing lists, The next still on the timeline will be March 20th 2017 when students will be able to apply for projects officially. Stay tuned!

[Tags: ]


[20170109] Documenting NetBSD's scheduler tweaks
NetBSD's scheduler was recently changed to better distribute load of long-running processes on multiple CPUs. So far, the associated sysctl tweaks were not documented, and this was changed now, documenting the kern.sched sysctls.

For reference, here is the text that was added to the sysctl(7) manpage:

     kern.sched (dynamic)
             Influence the scheduling of LWPs, their priorisation and how they
             are distributed on and moved between CPUs.

                   Third level name              Type       Changeable
                   kern.sched.cacheht_time       integer    yes
                   kern.sched.balance_period     integer    yes
                   kern.sched.average_weight     integer    yes
                   kern.sched.min_catch          integer    yes
                   kern.sched.timesoftints       integer    yes
                   kern.sched.kpreempt_pri       integer    yes
                   kern.sched.upreempt_pri       integer    yes
                   kern.sched.maxts              integer    yes
                   kern.sched.mints              integer    yes
                   kern.sched.name               string     no
                   kern.sched.rtts               integer    no
                   kern.sched.pri_min            integer    no
                   kern.sched.pri_max            integer    no

             The variables are as follows:

             kern.sched.cacheht_time (dynamic)
                     Cache hotness time in which a LWP is kept on one particu-
                     lar CPU and not moved to another CPU. This reduces the
                     overhead of flushing and reloading caches.  Defaults to
                     3ms.  Needs to be given in ``hz'' units, see mstohz(9).

             kern.sched.balance_period (dynamic)
                     Interval at which the CPU queues are checked for re-bal-
                     ancing.  Defaults to 300ms.  Needs to be given in ``hz''
                     units, see mstohz(9).

             kern.sched.average_weight (dynamic)
                     Can be used to influence how likely LWPs are to be
                     migrated from one CPU's queue of LWPs that are ready to
                     run to a different, idle CPU.  The value gives the per-
                     centage for weighting the average count of migratable
                     threads from the past against the current number of
                     migratable threads.  A small value gives more weight to
                     the past, a larger values more weight on the current sit-
                     uation.  Defaults to 50 and must be between 0 and 100.

             kern.sched.min_catch (dynamic)
                     Minimum count of migratable (runable) threads for catch-
                     ing (stealing) from another CPU.  Defaults to 1 but can
                     be increased to decrease chance of thread migration
                     between CPUs.

             kern.sched.timesoftints (dynamic)
                     Enable tracking of CPU time for soft interrupts as part
                     of a LWP's real execution time.  Set to a non-zero value
                     to enable, and see ps(1) for printing CPU times.

             kern.sched.kpreempt_pri (dynamic)
                     Minimum priority to trigger kernel preemption.

             kern.sched.upreempt_pri (dynamic)
                     Minimum priority to trigger user preemption.

             kern.sched.maxts (dynamic)
                     Scheduler specific maximal time quantum (in millisec-
                     onds).  Must be set to a value larger than ``mints'' and
                     between 10 and ``hz'' as given by the kern.clockrate
                     sysctl.  Provided by the M2 scheduler.

             kern.sched.mints (dynamic)
                     Scheduler specific minimal time quantum (in millisec-
                     onds).  Must be set to a value smaller than ``maxts'' and
                     between 1 and ``hz'' as given by the ``kern.clockrate''
                     sysctl.  Provided by the M2 scheduler.

             kern.sched.name (dynamic)
                     Scheduler name.  Provided both by the M2 and the 4BSD
                     scheduler.

             kern.sched.rtts (dynamic)
                     Fixed scheduler specific round-robin time quantum in mil-
                     liseconds.  Provided both by the M2 and the 4BSD sched-
                     uler.

             kern.sched.pri_min (dynamic)
                     Minimal POSIX real-time priority.  See sched(3).

             kern.sched.pri_max (dynamic)
                     Maximal POSIX real-time priority.  See sched(3).


[Tags: , ]


[20170109] NetBSD 7.1_RC1 available
Well, subject says it all. To quote from Soren Jacobsen's email: ``The first release candidate of NetBSD 7.1 is now available for download at:

http://cdn.NetBSD.org/pub/NetBSD/NetBSD-7.1_RC1/

Those of you who prefer to build from source can continue to follow the netbsd-7 branch or use the netbsd-7-1-RC1 tag.

There have been quite a lot of changes since 7.0. See src/doc/CHANGES-7.1 for the full list.

Please help us out by testing 7.1_RC1. We love any and all feedback. Report problems through the usual channels (submit a PR or write to the appropriate list). More general feedback is welcome at releng%NetBSD.org@localhost.''

[Tags: ]



[20170104] Hotplugging RAM - uvm_hotplug(9), the Xen balloon(4) driver and portmasters' FAQ
Adding and removing hardware components in operation is common in today's commoditized computing environments. This was not always the case - in the past century, one had to power down a machine in order to change network cards, harddisks or RAM. A major step towards changing a system's configuration at runtime for customers came with USB, but that's not where it ends - other systems like PCI support hotplugging as well.

Another area where changing of the system's configuration is the amount of Ramdom Access Memory (RAM) of a system. Usually fixed, this is determined at system start time, and then managed by the operating system's memory managent system. But esp. with today's virtualized hardware systems, even the amount of RAM assigned to a system can easily be changed. For example a VM can be assigned more RAM when needed, without even rebooting the system, leading to increased system performance without introducing swapping/paging overhead. Of course this required support from the operating system and its memory management subsystem.

For NetBSD, the UVM virtual memory system was now changed to support this via the uvm_hotplug(9) API, and a first user for this is the Xen balloon(4) driver. Quoting from the balloon(4) manpage, ``The balloon driver supports the memory ballooning operations offered in Xen environments. It allows shrinking or extending a domain's available memory by passing pages between different domains.''

The uvm_hotplug(9) manpage gives us more information on the UVM hotplug functionality: ``When the kernel is compiled with 'options UVM_HOTPLUG', memory segments are handled in a dynamic data structure (rbtree(3)) com- pared to a static array when not. This enables kernel code to add or remove information about memory segments at any point after boot - thus "hotplug".''

To answer more questions for portmasters who want to change their ports, Cherry G. Mathew has now posted a uvm_hotplug(9) port masters' FAQ. It covers questions on the background, affected files, and needed changes.

For more information on UVM, see Charles' Chuck' Cranor's PhD disertation on Design and Implementation of UVM (PDF) as well as his Usenix talk on the UVM Virtual Memory System (PS). There is also plenty of information available on Xen ballooning - check it out and share your experiences on NetBSD's port-xen mailing list!

[Tags: , , , , ]



[20161222] Bringing the scheduler saga to the finishing line
After my last blog postings on the NetBSD scheduler, some time went by. What has happened that the code to handle process migration was rewritten to give more knobs for tuning, and some testing was done. The initial problem state in PR kern/51615 is solved by the code. To reach a wider audience and get more testing, the code was committed to NetBSD-current today.

Now, two things remain to be seen:

  1. More testing. This best involved situations that compare the system's behaviour without and with the patch. Situations to test include
    • pure computation jobs that involve multiple parallel processes
    • a mix of CPU-crunching and input/output, again on a number of concurrent processes
    • full build.sh examples
    If you have time and an interesting set of numbers, please feel free to let us know on tech-kern@..

  2. Documentation. There is already a number of undocumented sysctls under "kern.sched", which was now extended by one more, "average_weight". While it's obvious to add the knob from the formula, testing it under various real-life conditions and see how things change is left to be determined by a PhD thesis or two - be sure to drop us your patches for src/share/man/man7/sysctl.7 if you can come up with a comprehensible description of all the scheduler sysctls!
So just now when you thought there is no more research to be done in scheduling algorithms, here is your chance to fame and glory! :-)

[Tags: , ]


Previous 10 entries
Disclaimer: All opinion expressed here is purely my own. No responsibility is taken for anything.

Access count: 23415243
Copyright (c) Hubert Feyrer