hubertf's NetBSD blog

BSD Professional (BSDP) Certification Exam Objectives published

Wed, 01 Sep 2010 21:28:00 GMT

In its mission to create certifications for BSD Un*x systems, the BSD Certification Group has now published the "BSD Professional" (BSDP) certification exam objectives:

`` The 78 page document outlines 11 knowledge domains, each containing a number of objectives, that form the basis of the certification. The document, a collaborative effort by members of the BSD community, is a followup effort to the BSDP Job Task Analysis report published earlier this year.

"The BSDP Certification Requirements Document provides a comprehensive set of objectives that a BSD Professional should master, ranging from general system administration to advanced security and system maintenance," said Babak Farrokhi, a seasoned BSD administrator and author of 'Network Administration with FreeBSD 7'.

Set to launch in 2011, the BSD Professional certification exam has been in development for over two years. Since the launch of the BSD Associate exam in 2008, there has been a continual effort to put together a certification for the next level of achievement for BSD system administrators.

"The BSD Professional certification is aimed at assessing practical skills of senior level BSD system administrators," said Jim Brown, BSDCG board member. "With the publication of these objectives, the BSD Certification Group is aiming to produce a high quality certification, giving the computing industry a new standard for excellence in system administration."

The BSDP Requirements document is available as a downloadable PDF in English''

The publication of the BSDP exam objectives is the next step after the BSD Associate (BSDA) certification exam objectives were published in 2005, which was then followed by quite a number of successful BSDA certifications. See the BSD Certification group's website for more details!

New NetBSD Core Team announced

Mon, 16 Aug 2010 22:30:00 GMT

NetBSD's core team is responsible for technical steering of the project. In the past the group was five people, and it was increased to seven people now. Read more in Alistair Crooks' announcement:

``We take great pleasure in announcing that the NetBSD core team, responsible for technical management within the NetBSD project, has increased its numbers to seven. This is to help in the running of a project with an ever-growing source base and developer community, and mirrors a similar change made to the board of directors, which has worked extremely well.

To help with the running of the project, we have asked Antti Kantee (pooka%NetBSD.org@localhost) and Chuck Silvers (chs%NetBSD.org@localhost), and they have very kindly agreed to join the core team. Antti is well known to many both inside and outside the project, and has contributed many new and exciting ideas, the most memorable and useful of these being the rump kernel architecture. Chuck is also well-known in NetBSD circles - his work on UBC, and his recent update of the Linux emulation code are just two examples of his contributions.

We therefore thank them both for their outstanding work to date, and to their joining the core team to lead and guide progress in the future.

For the current core team:

Alistair Crooks -- agc%NetBSD.org@localhost
Matt Green -- mrg%NetBSD.org@localhost
Antti Kantee -- pooka%NetBSD.org@localhost
Chuck Silvers -- chs%NetBSD.org@localhost
Yamamoto Takashi -- yamt%NetBSD.org@localhost
Matt Thomas -- matt%NetBSD.org@localhost
Christos Zoulas -- christos%NetBSD.org@localhost

Of course it runs ... 2.11BSD (or: PDP-11 in a FPGA)

Mon, 09 Aug 2010 22:55:00 GMT

It's about time NetBSD gets the PDP11 port done: the PDP-11/70 CPU core is now available as implementation on a FPGA-board, and there's need for a newer operating system than 2.11BSD! Citing from the homepage:

``The project contains a complete PDP-11 system: a 11/70 CPU with memory management unit, but without floating point unit, a basic set of UNIBUS peripherals (DL11, LP11, PC11, RK11/RK05), and last but not least a cache and memory controllers for SRAM and PSRAM. The design is FPGA proven, runs currently on Digilent S3BOARD and NEXYS2 boards and boots 5th Edition UNIX and 2.11BSD UNIX. ''

Janitorial code maintenance with coccinelle (Updated)

Sun, 08 Aug 2010 23:22:00 GMT

According to its homepage, coccinelle is ``a program matching and transformation engine which provides the language SmPL (Semantic Patch Language) for specifying desired matches and transformations in C code. Coccinelle was initially targeted towards performing collateral evolutions in Linux. Such evolutions comprise the changes that are needed in client code in response to evolutions in library APIs, and may include modifications such as renaming a function, adding a function argument whose value is somehow context-dependent, and reorganizing a data structure.''

Jean-Yves Migeon has applied coccinelle to NetBSD, and suggested performing two manual tasks, citing from his email to tech-kern:

replace all sizeof(fooarray)/sizeof(foo) calculations by its __arraycount() macro equivalent, provided in cdefs.h
detect places where aprint_* calls could be replaced by aprint_*_dev() to save manual printing of the device driver's name

Implementation of the coccinelle scripts, resulting patched for NetBSD and more information can be found in Jean-Yves Migeon's posting, his list of patches, and on his wiki page.

Update: Jean-Yves' follow-up includes further examples:

logical not with bitwise "&"

NULL-pointer dereference after first checking if it's NULL(!?), my favourite (example diff):

 	if (xfer == NULL) {
-		aprint_error_dev(xfer->fc->bdev, "xfer == NULL\n");
+		aprint_error("%s: xfer == NULL\n", __func__);
 		return;
 	}

NYCBSDCon 2010 Call for Papers

Wed, 14 Jul 2010 21:13:00 GMT

Citing from the call for papers: ``The New York City BSD Conference (NYCBSDCon) is the main technical conference on the US East Coast for the BSD community to get together to share and gain knowledge, to network with like-minded people, and to have fun. This event is organized by members of the New York City *BSD Users Group (NYC*BUG).
The NYCBSDCon program committee is accepting submissions for imaginative, embryonic and energizing presentations surrounding the BSD operating systems. We are looking to attract a wide range of speakers and attendees; therefore, topics of interest range from the esoteric to development to practical, everyday sysadmin life. Of course, original topics are preferred in most cases.
Each talk is expected to be 45-50 minutes, including a few minutes for questions and answers. All presentations will be recorded for audio and video. Presenters will have audio/visual and network connectivity.
Abstracts for presentations are due July 31, 2010.
Authors of accepted submissions should be able to provide the full presentation for publication on NYCBSDCon sponsored mediums. Further instructions will follow notification of acceptance. Submissions accompanied by a non-disclosure agreement or a product advertisement will be rejected.
Abstract submissions should be emailed to cfp@nycbsdcon.org in text, ps or pdf format.

Conference Location: Cooper Union, New York, NY Conference Dates: November 12-14, 2010''

Submission of NetBSD related entries is highly appreciated! See the call for papers for more information on important milestones, subsidizing of speakers and the mailing list to stay upto-date.

BSD Magazine archive available

Thu, 01 Jul 2010 00:02:00 GMT

Olga Kartseva writes: ``BSD Magazine archives available without subscribing to BSDMag newsletter for freebsd-announce subscribers!'' Here are direct PDF links:

Enjoy - and remember: more NetBSD content is good content, authors are always welcome!

NetBSD 5.1_RC3 binaries available for download

Sun, 20 Jun 2010 22:36:00 GMT

NetBSD release-engineer Soren Jacobsen announces: ``The third (and hopefully final) release candidate of NetBSD 5.1 is now available for download at:
http://ftp.NetBSD.org/pub/NetBSD/NetBSD-5.1_RC3/
Those of you who prefer to build from source can continue to follow the netbsd-5 branch, but the netbsd-5-1-RC3 tag is available as well.
See src/doc/CHANGES-5.1 for the list of changes from RC2 to RC3.

Please help us test this release candidate as much as possible. Remember, any feedback is good feedback. We'd love to hear from you, whether you've got a complaint or a compliment. That said, we hope your feedback is positive, as we would like this to be the final release candidate before 5.1. ''

EuroBSDCon 2010 - Call for Papers

Wed, 16 Jun 2010 23:56:00 GMT

From some mails I see: ``EuroBSDCon 2010 - Call for Papers

9th European BSD Conference
October 8 - 10, 2010
Karlsruhe, Germany
http://2010.eurobsdcon.org/

Introduction

The European BSD Community will meet again this year for the ninth conference in the EuroBSDCon series. This is a great opportunity to present new ideas to the community, inform your fellow BSD enthusiasts about the newest developments and work for the continued success of your favorite operating system. The two day conference program (October 9 - 10) will be preceeded by a tutorial day (Oct 8). Call for Papers

We are inviting contributions on all areas relating to the BSD family of operating systems, e.g. applications, architecture, implementation, administration and security of *BSD operating systems ranging from embedded systems to mainframes. Investigations on economic aspects regarding the operation of BSD systems are also welcome.

Prospective authors of contributions to the technical program are requested to submit an abstract via http://2010.eurobsdcon.org/. Presentations should last about 40 minutes including time for questions from the audience. Authors of accepted submissions should provide a full paper for publication in the conference proceedings and give permission to the organizers to publish the results in the printed proceedings and on the conference web site at www.eurobsdcon.org.

Call for Tutorial Proposals

Selected tutorials will be offered on the day before the conference. If you are interested in presenting a tutorial, please submit your suggestion on the conference website using the same mechanism as for submitting a paper. Please indicate if this would be a half- or full-day tutorial.

Sponsorship Opportunities

We are seeking companies or institutions to sponsor various elements of the conference in order to keep delegate fees as low as possible. Sponsorship opportunities include: paying for a speaker's travel or accommodation; providing bursaries for delegates who cannot pay the conference fee themselves; sponsoring the social event or the printing of proceedings. Please see the conference website for details.

Important Dates

Final abstract deadline: July 6th 2010
Final tutorial deadline: July 6th
Final papers due: September 1st
Tutorial day: October 8th
Conference: October 9 - 10

For more, see http://2010.eurobsdcon.org/''

Hiding other users' processes

Sat, 05 Jun 2010 18:42:00 GMT

Thus it was asked on #NetBSD:

<batence> I wanna set the top command work only for users process, not
                for all system
<batence> in freebsd command is sysctl security.bsd.see_other_uids=0/1
<batence> but I dunno for netbsd
<batence> eg I don't want users see other uids
<batence> only which they owned

Looking at the output of "sysctl -a" didn'r show anything obvious, but recalling the topic and with some digging, there actually is a sysctl switch for that in NetBSD: security.models.bsd44.curtain=1

Here's an example top(1) output with the default setting (0). My username is "feyrer", note that besides my processes, other users' processes are shown as well:

load averages:  0.02,  0.01,  0.00;               up 11+15:08:30                           18:38:56
24 processes: 23 sleeping, 1 on CPU
CPU states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Memory: 71M Act, 51M Inact, 552K Wired, 5560K Exec, 110M File, 27M Free
Swap: 512M Total, 335M Used, 178M Free

  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
    0 root     126    0     0K   16M pgdaemon   5:41  0.00%  0.00% [system]
  492 root      85    0  4792K  608K kqueue     0:06  0.00%  0.00% master
  113 root      85    0  2908K  860K select     0:05  0.00%  0.00% dhclient
  535 root      85    0  2900K  556K nanoslp    0:05  0.00%  0.00% cron
  155 root      85    0  2932K  548K kqueue     0:05  0.00%  0.00% syslogd
  496 postfix   85    0  4792K  888K kqueue     0:01  0.00%  0.00% qmgr
 4409 feyrer    43    0  2984K 1240K CPU        0:00  0.00%  0.00% top
 1197 root      85    0  8640K 3692K netio      0:00  0.00%  0.00% sshd
24830 root      85    0  8640K 3692K netio      0:00  0.00%  0.00% sshd
 6949 feyrer    85    0  8640K 2828K select     0:00  0.00%  0.00% sshd
28093 feyrer    85    0  8640K 2828K select     0:00  0.00%  0.00% sshd
12391 feyrer    85    0  2132K 1876K pause      0:00  0.00%  0.00% tcsh
25579 feyrer    85    0  2132K 1876K pause      0:00  0.00%  0.00% tcsh
 5773 postfix   85    0  4792K 1868K kqueue     0:00  0.00%  0.00% pickup
 1929 root      85    0  2128K 1828K ttyraw     0:00  0.00%  0.00% tcsh
29212 root      85    0  2972K 1164K kqueue     0:00  0.00%  0.00% inetd
25972 root      85    0  2824K 1076K pause      0:00  0.00%  0.00% ksh

Likewise, I see a number of processes in ps(1):

% ps -aux | wc -l
      26

Now let's change the sysctl:

# sysctl -d security.models.bsd44.curtain
security.models.bsd44.curtain: Curtain information about objects to users not owning them.
# sysctl -w security.models.bsd44.curtain=1
security.models.bsd44.curtain: 0 -> 1

After this, the top(1) output looks like this:

load averages:  0.02,  0.01,  0.00;               up 11+15:08:45                           18:39:11
5 processes: 4 sleeping, 1 on CPU
CPU states:  0.0% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.8% idle
Memory: 71M Act, 51M Inact, 552K Wired, 5416K Exec, 110M File, 28M Free
Swap: 512M Total, 335M Used, 178M Free

  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
 4409 feyrer    43    0  2984K 1240K CPU        0:00  0.00%  0.00% top
28093 feyrer    85    0  8640K 2828K select     0:00  0.00%  0.00% sshd
 6949 feyrer    85    0  8640K 2828K select     0:00  0.00%  0.00% sshd
12391 feyrer    85    0  2132K 1876K pause      0:00  0.00%  0.00% tcsh
25579 feyrer    85    0  2132K 1876K pause      0:00  0.00%  0.00% tcsh

This reduced set of processes is also shown in ps(1):

% ps -aux | wc -l
       7

In other words, only my processes are displayed. (If you wonder about the difference between the 7 processes shown in top and the seven ps(1)-lines: the latter includes a heading).

Note that this "filtering" does not apply to the root user, i.e. he can still see all processes.

Videos: Booting NetBSD [Update #3]

Mon, 31 May 2010 21:15:00 GMT

Jun Ebihara wrote me that there are a bunch of videos on YouTube, showing NetBSD boot on various machines:

Update: Links added for those not seeing the embedded videos (which I've seen happens via at least two RSS aggregators)

Update #2: Added booting NetBSD/hpcsh 5.1_RC2 from Windows CE on PERSONA, also submitted by Jun Ebihara. Thanks a lot!

Update #3: Added booting NetBSD/dreamcast 5.1_RC2 with IDE HDD and NE2000 NIC

NetBSD 5.1_RC2 binaries available for download

Fri, 28 May 2010 14:30:00 GMT

Soren Jacobsen writes on netbsd-announce: ``The second release candidate of NetBSD 5.1 is now available for download at:
http://ftp.NetBSD.org/pub/NetBSD/NetBSD-5.1_RC2/
Those of you who prefer to build from source can continue to follow the netbsd-5 branch, but the netbsd-5-1-RC2 tag is available as well.
See src/doc/CHANGES-5.1 for the list of changes from RC1 to RC2.

Please help us test this and any upcoming release candidates as much as possible. Remember, any feedback is good feedback. We'd love to hear from you, whether you've got a complaint or a compliment. ''

Source-changes ketchup Dec'09 - May'10 [Updated]

Fri, 28 May 2010 02:22:00 GMT

Here's what I have in my source-changes folder as interesting changes between Dec '09 and May '10. YMMV:

NetBSD/cats now uses X.org
NetBSD/sparc64 now runs a MULTIPROCESSOR kernel by default
NetBSD/evbsh3 now also supports the following boards: AlphaNet MS104-SH4, TAC T-SH7706LAN Ver.3, TAC T-SH7706LSR Ver.1
NetBSD/hpcarm now supports the Sharp W-ZERO3 series
wc(1) can print the longest line length now with -L
cdbr(3), cdbw(3) implement constand database reader/writer routines. this shrinks the services(5) database from 2.1MB to 307kB
/dev/{mem,kmem,zero,null} implementations are unified in machine independent code on the rmind-uvmplock branch
Many kernel systems were prepared to be built either into a monolithic kernel, or loaded as module at runtime. This includes verbose listing of PCI devices, Berkeley Packet Filters (bpf), loading modules only after the root filesystem is mounted, others.
New drivers:
- acpiwmibus: a pseudo-bus to which child Microsoft Windows Management Instrumentation (WMI, a subset of ACPI) devices attach
- u2g: split into parts: u3ginit attaches to those devices that only come as a umass device in the default configuration and forces them to reinitialize in 3D mode and detach. The u3g part attaches to individual interfaces for the 3G functionality, leaving the umass interface(s) for that driver. With this change I can use the MMC card in my Huawey stick (as well as the integrated windows driver CD, which of course is pretty useless) and the 3G modem at the same time.
- cas: Driver for Sun Cassini/Cassini+ (GigaSwift) Ethernet (also known as National Semiconductor Saturn)
- aibs: replaces aiboost for the ASUSTeK AI Booster hardware monitor
Software imports include NetPGP 3.99.2 (20100507 version), dhcpcd-5.2.2, BIND 9.7.0rc1, libelf from FreeBSD 8.0-RELEASE, tzcode2009k, pkg_install-20100220, mdocml-1.9.23, libarchive-2.8.2, ACPICA 20100121, bozohttpd-20100512

NetBSD ketchup - news from my mailbox

Fri, 28 May 2010 01:31:00 GMT

Here's another bunch of NetBSD-related news that has been lingering in my inbox for far too long:

Izumi Tsutsui's NetBSD/cobalt restore CD is available based on NetBSD versions 5.0.2 5.1_RC2. See the for information on what it is and how to use it.
A negative symbol lookup cache was added to NetBSD's loader for shared libraries and shared objects, ld.so_elf, by Roy Marples: ``I've been researching why Evolution from GNOME takes over 5 minutes to load on my quad core amd64 beast. It boils down to dlsym looking for a symbol that does not exist directly and as such examining every needed library. However, the current implementation does not remember what libraries it as already checked. Normally this isn't a problem, but with the way Evolution is built the search chain is massive. [...]
With this patch, Evolution (without the patches to and a glib I added to pkgsrc a few days ago) loads in under 2 seconds (5 seconds with initial disk thrashing). ''
The NetBSD Logo is available in many variants, but a new variant was submitted via www@ these days by "Tim" - which is actually plain HTML, no image:
⚑ NetBSD Powered!
SafeNet's ProtectDrive is ``a full disk encryption solution that encrypts the entire hard drive of laptops, workstations and servers, as well as USB flash drives, to protect data in the case of the theft or loss of a hardware device.''
How do you implement such preboot authentication and harddisk encryption software, esp. if you want to provide thinks like LDAP integration for the user/key handling and two-factor authentication? Little is known, but rumors say the 32bit version of the software is based on NetBSD, as is backed by this worker bio info: ``Duties: Working on pre-boot restricted environment with loads before operation system and implemented on NetBSD. Ported and optimized the KDrive X server to NetBSD. Developed and implemented user secure authentication interface with smart card support.
Environment and tools : NetBSD (3.0), C/C++, FLTK''
A german-language introduction of pkgsrc on OpenSolaris was given by Michael 'kvedulv' Moll at the Munich OpenSolaris User Group back in march. Slides and a video are available.
Running NetBSD on an ~~Oracle~~ Sun Fire X4140 Server? Check out this posting by Ignatios Souvatzis for the full dmesg pr0n of this machine with 12 CPU cores and 32GB RAM!
Are you still looking for a nice small ARM-based board to start hacking on NetBSD/arm? The http://www.friendlyarm.net/products/mini2440 may be a good start, esp. after Paul Fleischer is reaching completion of NetBSD support for the board. Citing from his mail to port-arm:
``I have now fairly good (i.e., it works for me) support for the MINI2440 on NetBSD with support for the following:
- S3C2440 UART
- DM9000 (MAC+PHY)
- S3C2440 SD Controller
- S3C2440 DMA Controller
- S3C2440 IIS Controller
- FriendlyArm 3,5" LCD Display
- S3C2440 USB Host Controller (OHCI)
- S3C2440 Touch Screen
- UDA1341TS audio codec

Currently, support for three things on the S3C2440 are missing:
- S3C2440 NAND Controller
- S3C2440 USB Device Controller
- S3C2440 RTC

I've also created a stage2 bootloader for use with u-boot, which ensures that the value of bootargs is passed to the NetBSD kernel. At this point I have only tested the code with the 64Mb version of the FriendlyArm MINI2440.
All the code is available in a Git repository[1] and is based on the netbsd-5 code base. Progress can be followed on my webpage[2]. ''
While talking about NetBSD on cool hardware: How about NetBSD/hpcarm on WILLCOM | W-ZERO3 (WS004SH) mobile devices? Here is a screenshot of Ebihara-san's WS011SH with CCW screen, and there is also a video "booting NetBSD/hpcarm on WILLCOM | W-ZERO3(WS004SH)" posted on YouTube:

For more details, see Izumi Tsutsui's posting on port-hpcarm.

Apple Magic Mouse driver

Sun, 16 May 2010 22:33:00 GMT

NetBSD's Bluetooth hacker #1, Iain Hibbert, wrote on tech-kern: ``I wrote a driver for the Apple Magic Mouse, as the protocol was mostly decoded by a Linux developer, and Somebody was kind enough to send me one. [...]
The mouse itself is a wireless Bluetooth mouse and operates with the USB HID protocol much like other mice, but it doesn't provide a proper descriptor and requires features to be activated and special interpretations of the touch surface reports, so doesn't fit exactly into our HID framework, which configures independent sub-devices to report id's from the descriptor.

The driver interprets the touch reports to allow emulation of a middle mouse button (for mulitple firm touches detected), and horizontal and vertical scroll actions (for touches moving over a certain distance). It works well on NetBSD-current and NetBSD-5 and the mouse is pretty slick. '' See Iain's posting for more details.

Silencing the boot process

Sat, 01 May 2010 10:52:00 GMT

NetBSD-current is able to boot the userland silently with "boot -z" for quite some time now, thanks to Alan Barrett. Those changes were never ported back to the netbsd-5 branch so far, but I'm in the process to change this now. Here's a preview:

Of course there are still many places left in the kernel that don't honor the boot flags (i.e. that use printf(9) instead of aprint_normal(9) and friends), but this will change over time - I hope :-).

NetBSD-current build status

Thu, 29 Apr 2010 00:54:00 GMT

Have you ever wondered how stable a -current build is at times? Here is an interesting page to give an overview on the success of NetBSD-current builds from the past few days (and much further back):

From the webpage: ``This web page visualizes the state of the NetBSD-current build by plotting the number of lines in the build log from build.sh as a function of time, and coloring the points red or green depending on whether the build succeeded or failed. Hopefully, this can help give some insight into the frequency and duration of build failures.

The plots were constructed using an automated procedure that builds NetBSD-current periodically (roughly twice a day) using make.sh -m i386 release, looks for any changes in exit status or any substantial changes in the size of the build log, and then pinpoints the times at which the changes occurred using binary search. The graph may not contain every build failure, because the algorithm used can miss cases where the build is broken and then fixed again between two consecutive periodic builds. The converse case, when the build is fixed and then broken again between two consecutive periodic builds, is usually detected because the size of the build log usually changes in this case.

Most of the builds in the graph are cross-builds using a 64-bit Linux host, while some of the older ones are native NetBSD builds.

If your browser supports SVG, you can use the SVG version. ''

Thanks to Andreas Gustafsson for providing this service!

Unixes of the world unite!

Tue, 13 Apr 2010 23:42:00 GMT

I've stumbled about this presentation by Unix (co)pioneer Robert Pike, which he gave in 2001. Some hilights:

``What is Unix^TM?
[...] Those operating systems derived from or inspired by the Research Unix systems of the 1970.
Includes [...] NetBSD [...] and others.''
``1. What is the best thing about Unix?
A: The community.
2. What is the worst thing about Unix? A: That there are so many communities.''

Read the whole presentation for an overview of the things that Unix got right and wrong. Definitely worth reading!

NetBSD and Google Summer of Code - Send in your proposals!

Tue, 06 Apr 2010 21:31:00 GMT

Just a friendly reminder to interested students who want to participate in this year's Google Summer of Code: You have only three days left to send in your proposal! Student application deadline is April 9th, 19:00 UTC.

See our list of suggested SoC projects if you need inspiration (other project suggestions are welcome!), and please have a look at our proposal HowTo for a bunch of questions we want to have answered so that we can properly judge your proposal. Get going!

NetBSD is part of Google's Summer of Code 2010

Thu, 18 Mar 2010 23:07:00 GMT

This year's Google Summer of Code mentoring organizations has been announced, and NetBSD is part of it, again! Possible projects are listed on the projects page, the GSoC wiki page, and esp. on the SoC-projects page.

Prospective students that are interested in working on a project are recommended to have a look at the Project Application / Proposal HowTo!

Google Summer of Code 2010 NetBSD swcryptX Project Suggestion (Updated #2)

Mon, 22 Feb 2010 22:57:00 GMT

Please see the update below before applying for this GSoC project!

I've been thinking of a neat-o project for this year's Google Summer of Code:

Abstract:

Overview of operation

The opencrypto(9) framework exists to coordinate hardware acceleration in NetBSD. Applications of the framework can be inside the kernel like the FAST_IPSEC IPsec implementation, or in userland like OpenSSL with the "cryptodev" engine. Crypto drivers can be realized in software or in hardware. Hardware drivers can be used to instruct e.g. the AMD Geode LX's AES block or a HIFN chip to perform cryptographic operations. Upon system startup, the crypto drivers at the opencrypto(9) framework, telling what operations they can perform. When an operation is required later, the framework will look which crypto device is currently not busy, and offload the operation to that device. Upon completion, the result is fed back to the application.

The following image illustrates the components and their interaction.

Limitations

Offloading the cryptographic requests involves some overhead. Data needs to be transferred to the hardware and back. On systems with a slow CPU, this overhead is relatively small compared to the operation speed of the CPU. On faster CPUs, the overhead becomes more of a burden, making the benefit of the crypto hardware negligible.

As examples, while a hifn(4) chip can provide worthwhile speedups on 500MHz and 1GHz CPUs, no performance win is experienced on a 2.4GHz CPU.

Proposal

The communication overhead involves data transfers over a PCI bus, which is of relatively low speed compared to today's modern CPUs. Preventing the data transfer is a worthwhile goal. In coordination with today's modern multi-core CPUs, using one or more CPUs solely for the purpose of crypto acceleration, a measurable improvement of crypto performance is expected. At the same time, no special hardware requirements beyond the CPU exist. This allows turning standard contemporary systems into fast crypto systems easily.

The following image illustrates the idea of interoperation between a CPU core that runs the kernel and application codes and three cores that are dedicated to crypto code.

Implementation Roadmap

This is where it gets fishy. ;) ~~The existing opencrypto(4) framework probably needs to be make MP-aware at the same time, employing proper use of NetBSD's locking framework.~~ (Already done) The existing swcrypto(4) needs to be adjusted for operation on multiple CPUs at the same time. A way to decide how many CPUs are dedicated to run swcrypto(4) instances. CPUs that run swcrypto(4) need to be taken out from the usual NetBSD CPU scheduling so that they are available exclusively for crypto.

Requirements

In no particular order:

Know how to build and install a kernel
Understanding of fine grained SMP and locking
How to use NetBSD's kernel threads, code-wise
How to interact with NetBSD's scheduler, code-wise
Tell the scheduler to pin a specific kernel thread to a specific CPU
Interaction between applications (IPsec, OpenSSL) with opencrypto(9), code-wise
Interaction of crypto providers with opencrypto(9), code-wise
Hardware! You won't be able to do this without at least two CPU cores in your machine. The more the better.
Benchmarking & a test setup for it

Project Applications

Please follow the NetBSD Project Application/Proposal HowTo if you're serious to work on this project.

If you have any questions let me know, public discussion should be led on the tech-crypto@ list.

Update: There was some discussion. In particular, my understanding of the interaction of the various layers as outlined above is not 100% accurate, and userland applications using opencrypto already seem to benefit from multiple kernel threads. In-kernel applications apparently do not, and before providing multiple crypto-servers in kernel (as suggested), work should probably done first to make sure such applications exist. Examples of this are IPsec (and the whole network stack), but also others like cgd (which AFAIU currently does not use opencrypto(9)).